SQL Clone
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


Issue with temp table as destibation


Issue with temp table as destibation

Author
Message
Anjan Wahwar
Anjan Wahwar
SSC-Enthusiastic
SSC-Enthusiastic (106 reputation)SSC-Enthusiastic (106 reputation)SSC-Enthusiastic (106 reputation)SSC-Enthusiastic (106 reputation)SSC-Enthusiastic (106 reputation)SSC-Enthusiastic (106 reputation)SSC-Enthusiastic (106 reputation)SSC-Enthusiastic (106 reputation)

Group: General Forum Members
Points: 106 Visits: 249
Hi All,

I have the requirement to load source txt file and perform incremental update on my destination table. As I was not allowed to create any table to hold staging records (from file), I'm storing file data to a global temp table and then performing MERGE operation (temp & Dest table).

In the data low task i have a derived column that handls date column (if blank or null then pass db_null to dest db else pass the column as it it).

Now everything is working fine. The only problem is with performance. For transferring 1 million records through the derived column task my package is taking almost an hour.

Any help on where I'm loosing my performance??

Thanks,
Anjan

__________________________________________
---------------------------------------------------
Save our mother Earth. Go Green !!!
a4apple
a4apple
SSC Veteran
SSC Veteran (256 reputation)SSC Veteran (256 reputation)SSC Veteran (256 reputation)SSC Veteran (256 reputation)SSC Veteran (256 reputation)SSC Veteran (256 reputation)SSC Veteran (256 reputation)SSC Veteran (256 reputation)

Group: General Forum Members
Points: 256 Visits: 406
Can you not use a lookup transformation instead of Merge? It's always good over Merge with the type of activity you are doing.

Good Luck Smile .. Visit www.sqlsaga.com for more t-sql code snippets and BI related how to articles.
Phil Parkin
Phil Parkin
SSCoach
SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)

Group: General Forum Members
Points: 18279 Visits: 20410
Anjan Wahwar (4/15/2014)
Hi All,

I have the requirement to load source txt file and perform incremental update on my destination table. As I was not allowed to create any table to hold staging records (from file), I'm storing file data to a global temp table and then performing MERGE operation (temp & Dest table).

In the data low task i have a derived column that handls date column (if blank or null then pass db_null to dest db else pass the column as it it).

Now everything is working fine. The only problem is with performance. For transferring 1 million records through the derived column task my package is taking almost an hour.

Any help on where I'm loosing my performance??

Thanks,
Anjan


Are you able to work out the split between how long it takes to populate the temp table and how long the merge takes?

The derived column task is unlikely to be the problem. In my experience, they generally work fast and do not cause blocking.

Does the temp table have any indexes or primary keys? Dropping these before the import and then recreating them before the merge might speed things up.

If the source data does not contain duplicates, you could consider using a lookup with full cache on the target table and sending the 'not matched' rows directly into the target table. Then the MERGE at the end has much less to do - just the updates.

Do all of the rows selected contain updates or inserts, or can you potentially filter the source?


Help us to help you. For better, quicker and more-focused answers to your questions, consider following the advice in this link.

If the answer to your question can be found with a brief Google search, please perform the search yourself, rather than expecting one of the SSC members to do it for you.

Please surround any code or links you post with the appropriate IFCode formatting tags. It helps readability a lot.
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search