Home Forums Data Warehousing Integration Services SSIS 2008 – Read roughly 50 CSV files from a folder, create SQL table from them dynamically, and dump data. RE: SSIS 2008 – Read roughly 50 CSV files from a folder, create SQL table from them dynamically, and dump data.

  • Hello Lowell,

    Thank you very much for the response.

    I'm also having the same type of requirements for this client as you specified.

    I'm going to handle our requirement in more or less in a same manner as you described.

    I'm going to use Script task to read header columns of CSV files, preparing one SSIS variable which will be having SQL script to create the required table with if exists condition inside script task itself.

    I will be having "Execute SQL task" following the script task. And this will create the actual table for a CSV.

    Both these components will be inside a for each loop container and execute all 50 CSV files one by one.

    Some points to be clarified,

    1. In the bunch of these 50 CSV files there will be some exception for which we first need to purge the tables and then insert the data. Meaning for 2 files out of 50, we need to first clean the tables and then perform data insert, while for the rest 48 files, they should be appended on daily basis.

    Can you please advise what is the best way to achieve this requirement? Where should we configure such

    exceptional cases for the package?

    2. For some of the CSV files we would be having more than one file with the same name. Like out of 50 the 2nd file is divided into 10 different CSV files. so in total we're having 60 files wherein the 10 out of 60 have repeated file names. How can we manage this criteria within the same loop, do we need to do one more for each looping inside the parent one, what is the best way to achieve this requirement?

    3. There will be another package, which will be used to purge data for the SQL tables. Meaning unlike the above package, this package will not run on daily basis. At some point we would like these 50 tables to be purged with older than criteria, say remove data older than 1st Jan 2015. what is the best way to achieve this requirement?

    Please know, I'm very new in SSIS world and would like to develop these packages for client using best package development practices.

    Any help would be greatly appreciated.