SSIS Large records batch processing

  • ptjj

    Mr or Mrs. 500

    Points: 500

    Hi,

    I have nearly 1 Crore records at my source side,i have to transfer these records to destination in batch wise let's suppose like 1 lakh records per batch.How can i implement this scenario.Pls help me.

    Thank you.

  • This was removed by the editor as SPAM

  • ptjj

    Mr or Mrs. 500

    Points: 500

    Please suggest me to do the same with FOR Loop or FOR EACH Loop.

    Thanks in Advance:-)

  • This was removed by the editor as SPAM

  • ptjj

    Mr or Mrs. 500

    Points: 500

    Thanks For Your Reply..

    My Source Data BAse is Oracle and the Target is SQL Server.

    Please help...:-)

  • ptjj

    Mr or Mrs. 500

    Points: 500

    How to split a Oracle Data source data into different Rawfiles with a fixed size of 10,000?On which we can split the source data of Oracle?

    Pls hlp..:-)

  • AmolNaik

    SSCarpal Tunnel

    Points: 4767

    But why do you need to use a For each loop? As mentioned by stewartc mentioned use batch size and commit size of 10,000 with TABLOCK option in the OLEDB Destination component, using TABLOCK option will speed up the data load operation. If you need more than 10K records to be processed increase the size to 100,000 (1 lakh).

    Thanks,

    Amol Naik

  • ptjj

    Mr or Mrs. 500

    Points: 500

    For each batch(10,000 records) i need to run a script.:(

  • AmolNaik

    SSCarpal Tunnel

    Points: 4767

    How about this then... Load all the records to a staging table with IDENTITY column. Using T-SQL Loop thru the records using IDENTITY range. This will be more efficient and easy too.

    Thanks,

    Amol Naik

Viewing 9 posts - 1 through 9 (of 9 total)

You must be logged in to reply to this topic. Login to reply