SQL Clone
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


Single record processing


Single record processing

Author
Message
nba
nba
Grasshopper
Grasshopper (18 reputation)Grasshopper (18 reputation)Grasshopper (18 reputation)Grasshopper (18 reputation)Grasshopper (18 reputation)Grasshopper (18 reputation)Grasshopper (18 reputation)Grasshopper (18 reputation)

Group: General Forum Members
Points: 18 Visits: 31
Hello everybody,
I've just begun to work on SSIS and I'm facing an unexpected issue.

My source is a flat file (csv) and I need to process records one by one.
At now records are processed in groups of hundreds and my process can't work properly. For example it seems Lookup components process records in group. So if I see a record with a new identifier I have to insert it in identifiers' table and the next row with same identifier has not to be inserted again but it has to update a statistic.
At now I find several duplicate identifiers in the identifiers table, all belonging to the same group.

May I have tip to solve this?
Regards.

Nicola
Jeff Moden
Jeff Moden
SSC Guru
SSC Guru (116K reputation)SSC Guru (116K reputation)SSC Guru (116K reputation)SSC Guru (116K reputation)SSC Guru (116K reputation)SSC Guru (116K reputation)SSC Guru (116K reputation)SSC Guru (116K reputation)

Group: General Forum Members
Points: 116621 Visits: 41430
My source is a flat file (csv) and I need to process records one by one. At now records are processed in groups of hundreds and my process can't work properly


Then, you've written the process incorrectly. The only time you should write RBAR in any form, is when a 3rd party API requires it and you cannot compel the 3rd party to rewrite the API to accept result sets.

I'd suggest that you reevalute your code and rewrite it to operate in a set based fashion. If you're getting duplicates in a table, I'd also like to suggest that you look into a bit of a refresher course on proper data and table design.

Not trying to bust your chops here... from what you've posted, I see some pretty serious design flaws that will kill all chances of scalability and data integrity. I hate to see people go through that.

--Jeff Moden

RBAR is pronounced ree-bar and is a Modenism for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column.
If you think its expensive to hire a professional to do the job, wait until you hire an amateur. -- Red Adair

Helpful Links:
How to post code problems
How to post performance problems
Forum FAQs
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search