Level 3 of the Stairway to Integration Services - Adding Rows in Incremental Loads

SQLServerCentral is supported by Red Gate Software Ltd.

Popular Topics

Home

Members

Calendar

Who's On

Home » Article Discussions » Article Discussions by Author » Discuss Content Posted by Andy Leonard » Level 3 of the Stairway to Integration...

18 posts, Page 2 of 2

««1 2

Level 3 of the Stairway to Integration Services - Adding Rows in Incremental Loads

Rate Topic

Display Mode

Topic Options

Author

Message

JJ B

Posted Wednesday, September 21, 2011 10:41 AM

SSC Veteran

Group: General Forum Members
Last Login: Today @ 3:08 PM
Points: 255, Visits: 2,407

You have great skill in explaining things. Thanks for taking the time to share your information in a way that is actually helpful!

Post #1178844

Andy Leonard

Posted Wednesday, September 21, 2011 11:18 AM

Old Hand

Group: General Forum Members
Last Login: Monday, May 13, 2013 2:53 PM
Points: 375, Visits: 947

Hi JJ,

I have an excellent editor, Steve Jones.

:{>

Andy Leonard
CSO, Linchpin People
Follow me on Twitter: @AndyLeonard

Post #1178874

rannoune

Posted Monday, October 01, 2012 8:05 AM

Forum Newbie

Group: General Forum Members
Last Login: Friday, November 02, 2012 10:30 AM
Points: 2, Visits: 33

Hi Andy,

Thanks for this great article!!! I have a question; what if the Person.Contacts table is located on an another server, what would you do? We definitely don't want to get all rows as it impacts performance and may have timeout issues for huge data. How can we join Person.Contact and dbo.Contact when they are located on different servers, then only select the rows we need to insert from Person.Contact?

Thanks again!

Post #1366584

Rob de Vos

Posted Friday, December 14, 2012 5:39 AM

SSC Veteran

Group: General Forum Members
Last Login: Today @ 2:18 AM
Points: 287, Visits: 141

Andy I'am enjoing the series but I have a small problem with the 'Update'.

I'ts working, but it only updates about 700 records each run, I can't find any option to change this behavior.
The only thing I can imagine is that I use a very limited virtual server and that this is caused by the Full Cache option of the Lookup.
But I see the correct number of records at each Data Flow.

I do see 2 warnings, but I don't think they have any thing to do with the update problem:
[SSIS.Pipeline] Warning: Warning: Could not open global shared memory to communicate with performance DLL; data flow performance counters are not available. To resolve, run this package as an administrator, or on the system's console.

and
[SSIS.Pipeline] Warning: The output column "LkUp_BusinessEntityID" (131) on output "Lookup Match Output" (103) and component "Lookup" (101) is not subsequently used in the Data Flow task. Removing this unused output column can increase Data Flow task performance.

Rob

Post #1396598

Brandon Forest

Posted Friday, March 29, 2013 2:19 PM

Valued Member

Group: General Forum Members
Last Login: Friday, May 17, 2013 11:29 AM
Points: 60, Visits: 303

I agree that the second method is better than the first, but the duration of each method can only be measured correctly if you clear the procedure cache for the previous dataset. Wink

Post #1437092

shalinder.verma

Posted Sunday, March 31, 2013 10:17 PM

SSC Rookie

Group: General Forum Members
Last Login: Wednesday, May 15, 2013 9:19 AM
Points: 29, Visits: 123

This is a good article for the beginners. Great work Andy!

I would also stress on looking at Merge TSQL statement. Have used it quite a few times. It would be interesting to compare the performance and pros/cons with Lookup transformation.

Also, with Merge we can handle SCD1 and SCD2 quite easily. I reckon this is required in most scenarios for incremental dimension loading. This way we can do away with bad SCD transformation available with SSIS out of the box.

My 2 cents.

Post #1437354

JAMESC2003

Posted Wednesday, April 03, 2013 2:55 PM

Forum Newbie

Group: General Forum Members
Last Login: Sunday, May 05, 2013 4:19 PM
Points: 6, Visits: 67

Write a book on SSIS.
I suspect it will be 200,000 pages .....
The blow-by-blow (with background explanations) are
immensely helpful.

I will still buy it .... even if it costs $100

Post #1438585

Rob de Vos

Posted Thursday, April 04, 2013 12:22 AM

SSC Veteran

Group: General Forum Members
Last Login: Today @ 2:18 AM
Points: 287, Visits: 141

in fact there is a book about SSIS although the title suggests something else..
the title is Implementing a Data Warehouse with Microsoft SQL server 2012 (exam 70-463)
It covers a lot of SSIS but also Master data services, Data Cleansing and Data Warehouse, most of the tasks are done using SSIS.

I found 2 problems,
1 the ODBC connections only work in 32-bit mode
2 the Master Data web interface is in Dutch I have configured the server to be in my home country, suddenly the exercises are a real puzzle Smile

Post #1438654

« Prev Topic | Next Topic »

18 posts, Page 2 of 2

««1 2

Permissions