SQL Clone
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


Single DW multiple sources


Single DW multiple sources

Author
Message
Daniel Fountain
Daniel Fountain
SSCommitted
SSCommitted (1.6K reputation)SSCommitted (1.6K reputation)SSCommitted (1.6K reputation)SSCommitted (1.6K reputation)SSCommitted (1.6K reputation)SSCommitted (1.6K reputation)SSCommitted (1.6K reputation)SSCommitted (1.6K reputation)

Group: General Forum Members
Points: 1641 Visits: 890
Hey all,

We currently have multiple systems that we report on. They all contain information on the same topic - but they are systems built externally by different companies.

However one major goal is to be able to report on these systems together.

Here is my general plan within SSIS

1) Create denormalised structure for each system.
2) Delete stuff for that system in unversal tables, and copy over the tables.

Now there are a couple of things that have sprung to mind.

Firstly i am doing step one to minimise the load on the central tables. So it should hopefully avoid much locking if multiple SSIS packages (one for each system) are running in parrallel. Is this the best way to handle this?

Secondly - keys..... some are ints - others are varchar. Hows the best way to split the primary keys? I am planning to have a source column to identify the system. So am i better off having a composite key for every table - or the other option i can think of is to append a letter to signify the system to each and every PK and FK.

I would appreciate any thoughts.

Dan
PaulB-TheOneAndOnly
PaulB-TheOneAndOnly
SSCoach
SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)

Group: General Forum Members
Points: 18625 Visits: 4639
danielfountain (5/17/2013)
We currently have multiple systems that we report on. They all contain information on the same topic - but they are systems built externally by different companies.

However one major goal is to be able to report on these systems together.


if all systems are on the same topic then it looks to me the sensible thing to do is to build a single datamart data warehouse.

I would focus on the design of the single datamart e.g. the FACT and DIM tables that would store the data then I would plan the ETL processes. Most probably I would create staging tables mimicking as close as possible what you have on the operational systems then define how to map such data into the data warehouse.

Hope this helps.

_____________________________________
Pablo (Paul) Berzukov

Author of Understanding Database Administration available at Amazon and other bookstores.

Disclaimer: Advice is provided to the best of my knowledge but no implicit or explicit warranties are provided. Since the advisor explicitly encourages testing any and all suggestions on a test non-production environment advisor should not held liable or responsible for any actions taken based on the given advice.
herladygeekedness
herladygeekedness
SSCommitted
SSCommitted (1.7K reputation)SSCommitted (1.7K reputation)SSCommitted (1.7K reputation)SSCommitted (1.7K reputation)SSCommitted (1.7K reputation)SSCommitted (1.7K reputation)SSCommitted (1.7K reputation)SSCommitted (1.7K reputation)

Group: General Forum Members
Points: 1748 Visits: 813
we have the concept of SourceSystemKey to identify where a record came from. that plus the business key of the table is all we need.

We have DimSourceSystem that we manually keep updated with system connection info and whether or not that system is enabled (every so often there will be a weather event that knocks out a line of business so we avoid that one until it is back online).
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search