Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase

Single record processing Expand / Collapse
Author
Message
Posted Thursday, November 22, 2007 1:47 PM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Sunday, May 25, 2008 12:28 PM
Points: 6, Visits: 31
Hello everybody,
I've just begun to work on SSIS and I'm facing an unexpected issue.

My source is a flat file (csv) and I need to process records one by one.
At now records are processed in groups of hundreds and my process can't work properly. For example it seems Lookup components process records in group. So if I see a record with a new identifier I have to insert it in identifiers' table and the next row with same identifier has not to be inserted again but it has to update a statistic.
At now I find several duplicate identifiers in the identifiers table, all belonging to the same group.

May I have tip to solve this?
Regards.

Nicola
Post #425071
Posted Thursday, November 22, 2007 7:08 PM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: General Forum Members
Last Login: Yesterday @ 8:51 PM
Points: 35,606, Visits: 32,190
My source is a flat file (csv) and I need to process records one by one. At now records are processed in groups of hundreds and my process can't work properly


Then, you've written the process incorrectly. The only time you should write RBAR in any form, is when a 3rd party API requires it and you cannot compel the 3rd party to rewrite the API to accept result sets.

I'd suggest that you reevalute your code and rewrite it to operate in a set based fashion. If you're getting duplicates in a table, I'd also like to suggest that you look into a bit of a refresher course on proper data and table design.

Not trying to bust your chops here... from what you've posted, I see some pretty serious design flaws that will kill all chances of scalability and data integrity. I hate to see people go through that.


--Jeff Moden
"RBAR is pronounced "ree-bar" and is a "Modenism" for "Row-By-Agonizing-Row".

First step towards the paradigm shift of writing Set Based code:
Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column."

(play on words) "Just because you CAN do something in T-SQL, doesn't mean you SHOULDN'T." --22 Aug 2013

Helpful Links:
How to post code problems
How to post performance problems
Post #425101
« Prev Topic | Next Topic »

Add to briefcase

Permissions Expand / Collapse