Click here to monitor SSC
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


How to create staging table to handle incremental load


How to create staging table to handle incremental load

Author
Message
Greg Edwards-268690
Greg Edwards-268690
SSC Eights!
SSC Eights! (819 reputation)SSC Eights! (819 reputation)SSC Eights! (819 reputation)SSC Eights! (819 reputation)SSC Eights! (819 reputation)SSC Eights! (819 reputation)SSC Eights! (819 reputation)SSC Eights! (819 reputation)

Group: General Forum Members
Points: 819 Visits: 8267
Jim1234 (1/5/2014)
Thanks Tibor. Few questions are coming in my mind. The volume of data in source (for actual tables) is too high. So we are planning for a staging layer in which only those records which got Inserted/updated/deleted will be present. From the staging we need to load the target by using the join query.

The challange is how to design the staging table. Will it be exactly like the source tables or staging will have only one table like the target. How to handle insert/update/delete for the incremental records.

Feel free to let me know if you need more details. Thanks.

Jim


If you can capture keys for add / change / deletes you are interested in with triggers on the source system, then use this to drive the staging, you should be able to handle about any volume.
To mirror the source for the most part will make it easier to understand and maintain.

I have a hard time understanding why you would keep saying the target has to be a single table, not a view.
Your description is just a logical representation of the 2 tables, not any real transformation logic.
Give more thought to the SDC suggestions.
Architecture can have huge impacts to performance, scalability, and ease of maintaining and adding changes.

It will have much more payback when modeling more complex situations.
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search