Post reply

Single record processing

  • November 22, 2007 at 1:47 pm

    #118977

    Hello everybody,

    I've just begun to work on SSIS and I'm facing an unexpected issue.

    My source is a flat file (csv) and I need to process records one by one.

    At now records are processed in groups of hundreds and my process can't work properly. For example it seems Lookup components process records in group. So if I see a record with a new identifier I have to insert it in identifiers' table and the next row with same identifier has not to be inserted again but it has to update a statistic.

    At now I find several duplicate identifiers in the identifiers table, all belonging to the same group.

    May I have tip to solve this?

    Regards.

    Nicola

  • November 22, 2007 at 7:08 pm

    #753491

    My source is a flat file (csv) and I need to process records one by one. At now records are processed in groups of hundreds and my process can't work properly

    Then, you've written the process incorrectly. The only time you should write RBAR in any form, is when a 3rd party API requires it and you cannot compel the 3rd party to rewrite the API to accept result sets.

    I'd suggest that you reevalute your code and rewrite it to operate in a set based fashion. If you're getting duplicates in a table, I'd also like to suggest that you look into a bit of a refresher course on proper data and table design.

    Not trying to bust your chops here... from what you've posted, I see some pretty serious design flaws that will kill all chances of scalability and data integrity. I hate to see people go through that.

    --Jeff Moden

    RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
    First step towards the paradigm shift of writing Set Based code:
    ________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

    Change is inevitable... Change for the better is not.

    Helpful Links:
    How to post code problems
    How to Post Performance Problems
    Create a Tally Function (fnTally)

Viewing 2 posts - 1 through 1 (of 1 total)

You must be logged in to reply to this topic. Login to reply