Using value from target table to filter values from source

SQLServerCentral is supported by Red Gate Software Ltd.

Popular Topics

Home

Members

Calendar

Who's On

Home » Data Warehousing » Integration Services » Using value from target table to filter...

Using value from target table to filter values from source

Rate Topic

Display Mode

Topic Options

Author

Message

Ronald Fuller

Posted Monday, February 11, 2013 4:35 PM

Forum Newbie

Group: General Forum Members
Last Login: Thursday, May 09, 2013 1:32 PM
Points: 1, Visits: 28

Hello, I want to import all the rows from a source table where the value of a column (ID) is greater than max(ID) in the target table (thereby getting only new rows).

Lookup transforms do joins, and both tables are > 200M rows, so that seems like a waste when all I need is a where filter.

This sounds simple, but I cant figure it out using SSIS. Linked servers would solve this, but that's not an option here.

Thanks in advance for any help.

Post #1418686

opc.three

Posted Monday, February 11, 2013 7:11 PM

SSCertifiable

Group: General Forum Members
Last Login: Today @ 12:39 PM
Points: 6,722, Visits: 11,765

I would do this in two steps:

1. Run an Execute SQL Task to retrieve the max ID from the destination table and store the result in a variable.
2. Run a Data Flow Task to copy the data from the source where the ID is greater than the ID you retrieved from step 1.

__________________________________________________________________________________________________
There are no special teachers of virtue, because virtue is taught by the whole community. --Plato

Believe you can and you're halfway there. --Theodore Roosevelt

Everything Should Be Made as Simple as Possible, But Not Simpler --Albert Einstein

The significant problems we face cannot be solved at the same level of thinking we were at when we created them. --Albert Einstein

1 apple is not exactly 1/8 of 8 apples. Because there are no absolutely identical apples. --Giordy

Post #1418714

Phil Parkin

Posted Tuesday, February 12, 2013 2:07 AM

SSCarpal Tunnel

Group: General Forum Members
Last Login: Today @ 9:20 AM
Points: 4,242, Visits: 9,494

opc.three (2/11/2013)

+1 exactly how I would do it.

____________________________________________________________________________________________

Help us to help you. For better, quicker and more focused answers to your questions, consider following the advice in this link:

http://www.sqlservercentral.com/articles/Best+Practices/61537/

If you are asking for help and your post does not contain a question, you should expect responses which do not contain any answers. Put a question mark in there somewhere - it's not rocket science.

Post #1418803

« Prev Topic | Next Topic »

Permissions