Incremental Data Loading Using CDC

SQLServerCentral is supported by Red Gate Software Ltd.

Popular Topics

Home

Members

Calendar

Who's On

Home » Article Discussions » Article Discussions by Author » Discuss content posted by Mark Murphy » Incremental Data Loading Using CDC

Incremental Data Loading Using CDC

Rate Topic

Display Mode

Topic Options

Author

Message

markoos1

Posted Saturday, January 15, 2011 1:23 PM

Grasshopper

Group: General Forum Members
Last Login: Today @ 1:07 PM
Points: 18, Visits: 168

Comments posted to this topic are about the item Incremental Data Loading Using CDC

Post #1048369

sqlserver-1064800

Posted Monday, January 17, 2011 3:40 AM

Forum Newbie

Group: General Forum Members
Last Login: Tuesday, July 05, 2011 6:35 AM
Points: 2, Visits: 53

Hi Mark,

Thanks for writing this article it was interesting to see a different way of implementing CDC incremental loading. I have also implemented an incremental load using CDC and I found that the really important part is ensuring synchronisation between the data warehouse tables and the tracked tables. You mention a couple of methods for resynchronisation in your article and I have an additional method which works quite reliably for me which I thought you might be interested in.

In my solution I have a SQL replication step between the production OLTP and the data warehouse. This gives me control over the data flow into the monitored tables and allows me to pause replication apply changes to the data warehouse. Once finished I can then restart the data flow and CDC monitoring without losing any of the change data. This of course requires the replication subscriptions to be setup not to expire during the stoppage.

Cheers

Martin

Post #1048670

Koen Verbeeck

Posted Monday, January 17, 2011 5:49 AM

SSCrazy Eights

Group: General Forum Members
Last Login: Today @ 2:11 AM
Points: 9,378, Visits: 6,473

Nice article, thanks for sharing.

How to post forum questions.
Need an answer? No, you need a question.
What’s the deal with Excel & SSIS?

Member of LinkedIn. My blog at LessThanDot.

MCSA SQL Server 2012 - MCSE Business Intelligence

Post #1048710

Sree Arjun Div

Posted Monday, January 17, 2011 5:54 AM

SSCrazy

Group: General Forum Members
Last Login: Wednesday, May 08, 2013 3:44 AM
Points: 2,173, Visits: 1,350

got knowledge of incremental data loading thanks for the post

Post #1048712

casinc815

Posted Monday, January 17, 2011 7:29 AM

Grasshopper

Group: General Forum Members
Last Login: Friday, January 11, 2013 8:29 AM
Points: 22, Visits: 48

Mark:

Great article. Nice clarification and demonstration of CDC.
It has often been a problem explaining to programmers the
intent of CDC and to execute the project correctly.

One note, there are no footnotes to your previsous two article.
For completeness in discussing CDC, can you either post or send
them to me.

Thanks!

Post #1048753

markoos1

Posted Monday, January 17, 2011 10:56 AM

Grasshopper

Group: General Forum Members
Last Login: Today @ 1:07 PM
Points: 18, Visits: 168

casinc815,

The links are at the beginning. They are: Article 1 and Article 2.

-Mark

Post #1048869

casinc815

Posted Monday, January 17, 2011 11:52 AM

Grasshopper

Group: General Forum Members
Last Login: Friday, January 11, 2013 8:29 AM
Points: 22, Visits: 48

thanks, Mark!

Post #1048893

JOHNSON4INFO

Posted Monday, March 05, 2012 1:42 PM

Forum Newbie

Group: General Forum Members
Last Login: Yesterday @ 11:57 AM
Points: 8, Visits: 46

I read the article about Incremental Data Loading using CDC - but can you create an article and demo step by step on how to capture changes from a bulk load and incremental load from a source that does not have CDC (like SQL Server 2005). most of my tables don't have datetime column so I don't know if the record is new or updated by using datetime alone. Can you help me?

I need to do a bulk load initially from 3 servers merged into one server database and perform incremental updates realtime or each night.

Post #1261811

« Prev Topic | Next Topic »

Permissions