Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase 12»»

Building an Incremental Load Package – Level 3 Expand / Collapse
Author
Message
Posted Wednesday, August 7, 2013 12:02 AM


Old Hand

Old HandOld HandOld HandOld HandOld HandOld HandOld HandOld Hand

Group: General Forum Members
Last Login: Monday, December 15, 2014 2:46 PM
Points: 388, Visits: 1,047
Comments posted to this topic are about the item Building an Incremental Load Package – Level 3

Andy Leonard
CSO, Linchpin People
Follow me on Twitter: @AndyLeonard
Post #1481667
Posted Wednesday, August 7, 2013 4:11 AM
UDP Broadcaster

UDP BroadcasterUDP BroadcasterUDP BroadcasterUDP BroadcasterUDP BroadcasterUDP BroadcasterUDP BroadcasterUDP Broadcaster

Group: General Forum Members
Last Login: Wednesday, December 17, 2014 1:50 AM
Points: 1,473, Visits: 725
Andy, Superb article! Informative, nicely written and elaborated with appropriate sample data. Can't wait to read your next article on BIML!!

Thanks a ton.:)
Post #1481771
Posted Thursday, August 8, 2013 10:41 AM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Monday, May 12, 2014 5:44 AM
Points: 18, Visits: 82
Questions from a newbie that's only been scripting queries for 2+ years:

You have it set up for deltas that are in table 2 that are not in table 1, to which data from table 2 is added to table 1, which is a great idea. I see that potential on the ETL that we do. My question on this issue is what happens to the deltas that are in table 1 that are not in table 2? What we do is an update on a personnel system here and some people get deleted from the system. So I wonder if this system can remove the ones that are removed when the data is updated. I have not studied this series in super detail, so if the answer is in your series, please let me know and I apologize for the inconvenience.

Related to above, we get text files that are imported into tables within a staging database. Could this work without the use of creating additional tables for a pre_load? If not, then I guess we can do a pre_stage table and use this process to increment our actual staging tables.

Someone suggested I explore this process for a possible alternative to using bcp queryout and/or Data Flow tasks for a bunch of files created from sql scripts. Would this be part of your Stairway discussion down the road?

Thanks.

Post #1482469
Posted Thursday, August 8, 2013 11:34 AM


Old Hand

Old HandOld HandOld HandOld HandOld HandOld HandOld HandOld Hand

Group: General Forum Members
Last Login: Monday, December 15, 2014 2:46 PM
Points: 388, Visits: 1,047
Hi jbm6401,

You will want to check out the Stairway to Integration Services for answers to these types of questions.

Hope this helps,
Andy


Andy Leonard
CSO, Linchpin People
Follow me on Twitter: @AndyLeonard
Post #1482491
Posted Friday, August 9, 2013 7:56 AM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Monday, May 12, 2014 5:44 AM
Points: 18, Visits: 82
Thanks.
Post #1482782
Posted Friday, August 9, 2013 11:57 PM
Grasshopper

GrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopperGrasshopper

Group: General Forum Members
Last Login: Friday, September 19, 2014 3:17 PM
Points: 12, Visits: 38
Awesome post Andy. This will help tremendously in trying to explain some SSIS patterns to my team!
Post #1483008
Posted Sunday, August 11, 2013 9:38 PM


Old Hand

Old HandOld HandOld HandOld HandOld HandOld HandOld HandOld Hand

Group: General Forum Members
Last Login: Monday, December 15, 2014 2:46 PM
Points: 388, Visits: 1,047
Thank you, James!

:{>


Andy Leonard
CSO, Linchpin People
Follow me on Twitter: @AndyLeonard
Post #1483129
Posted Tuesday, August 20, 2013 3:22 PM
SSC Eights!

SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!SSC Eights!

Group: General Forum Members
Last Login: Saturday, December 13, 2014 9:58 AM
Points: 858, Visits: 1,353
Finally got around to working through the examples. Very thorough and well-documented. I look forward to that BIMLScript class before the PASS Conference now and am looking forward to the rest of the series.

Only issue I had that was odd was the test package from article #2 had a different encryption level. I rebuilt it so everything had the same encryption level, but the package wouldn't debug until I deleted that first test package.

I tried to anticipate the code you listed so I'd get a bit more experience seeing what was available. Sometimes successfully, sometimes not so much, but it was a good learning experience going through the example. Now on to #4.



Post #1486449
Posted Monday, September 9, 2013 8:07 AM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Thursday, October 30, 2014 3:37 AM
Points: 112, Visits: 89
Andy this is turning into a really interesting series and I can't wait to see what further BIML wizardry you've got in store for us.

Post #1492800
Posted Friday, November 14, 2014 4:00 PM
Mr or Mrs. 500

Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500

Group: General Forum Members
Last Login: Today @ 12:27 AM
Points: 542, Visits: 805
I just returned from PASS where I first heard about BIML, so this article was very timely. Thank you for this! It's very powerful stuff. I noticed that apparently biml compiler does a lot of things "behind the scenes". For example, we never specified column mapping between the output of the "Conditional Split" and stgUpdates destination. I guess since in this case the column names matched, they were simply mapped based on their names. But what if they were different, which, I suspect, is a more common scenario? We would probably need to write much more code.


Post #1634791
« Prev Topic | Next Topic »

Add to briefcase 12»»

Permissions Expand / Collapse