October 3, 2013 at 11:27 am
Hi,
Created two Control flow Tasks for Dimensions and Fact Tables Load. Each Control Flow Tasks as some Data Flows (Flat File Source, Pivot, Unpivot, Multicast, Sorting, Aggregation, Dimension Table Lookup, Fact Table Lookup, Data Conversion, OLEDB Destination etc...)
Works fine with above Transformations. Need some Help to Handle Events for Data Flow Tasks with Error Handling. All these Data Flow Tasks and Error should be Tracked in SQL Server Table.
And How to Show a Pop-up for Input from User to get his emailID and to store the EmailID with Data Flow Tasks in SQL Table.
Attached Screen shot of my Package created so far....
October 3, 2013 at 2:06 pm
Are you sure it works fine? You have a whole lot of blocking transformations in your package, which can lead to serious performance issues for large data sets.
Anyway, on topic: every component in a data flow has an error path. (the red arrows) You need to configure the components to redirect rows when an error occurs.
You can group the error paths with union all components and write the error rows to a database table.
Regarding the user input: SSIS is not made for user input. It is an ETL tool and it is made to run unattended on a server.
The only way you can have user input is to create dialogs in a script component, but that would mean the user has the execute the package manually using visual studio. Not really ideal.
Need an answer? No, you need a question
My blog at https://sqlkover.com.
MCSE Business Intelligence - Microsoft Data Platform MVP
October 4, 2013 at 12:32 am
Hey Koen,
Thanks for ur reply. Error Log with Red Flow Direction is working, But unable to Union all Error Logs to handle it in SQL Server Table. Will try that it seems will works fine.
And Package Runs fine, since it as Apt Input file and if any Errors it will surely Blocks. Am new to SSIS and past 3 weeks working in this. Some Transformations will be Blocked in this Package. Please Suggest me to Enhance this Package Better.
I will post my Requirements and Required Output is detail.
1. Flat File (Products [7 to 8 Columns] With Month Names as Columns)
2. Unpivot this Month Column names in Rows and its Value as another column
3. Pivot (In Products there will be a Column Measures - Need to Pivot Measure Column Values to Column names).
4. Products Columns will be Inserted in Product Dimensions.
5. Month Columns in Month dimensions
6. Look-up Product and Month Dimensions in Fact Tables (Those Pivoted Columns will be in Fact Table)
7. And Fact Table Insert.
the above Steps is need to Perform for Each Flat file sources with some Validations.
Validations:
1. Any New Product Dimensions (Have to Pop-up those Values)
2. Any new Month Dimensions (Have to Pop-up)
3. Any Duplicates or Null Values (have to pop-up)
4. All those Transformations with Errors need to Capture it in SQL Server Table.
October 4, 2013 at 4:34 am
Basically you'll want to avoid the sort and aggregation transform in SSIS; as they are blocking transformations and load all rows into memory.
These types of transformations are better handled by SQL Server.
Need an answer? No, you need a question
My blog at https://sqlkover.com.
MCSE Business Intelligence - Microsoft Data Platform MVP
October 4, 2013 at 10:42 am
Its Fine. But am Sorting Transformation to get the Unique Values for some Columns.
And Aggregation Transformation to SUM the Values for Measure Columns.
Without Using these How Can i Use it thru Memory. Can you elaborate in detail. So that I can correct myself in my Package.
October 6, 2013 at 1:47 pm
kalaiselvan.rajendran (10/4/2013)
Its Fine. But am Sorting Transformation to get the Unique Values for some Columns.And Aggregation Transformation to SUM the Values for Measure Columns.
Without Using these How Can i Use it thru Memory. Can you elaborate in detail. So that I can correct myself in my Package.
Getting distinct values and sorting data is something the database engine excels at.
In SSIS, every row has to be read into memory in order to process these operations.
This can lead to performance issues with large datasets. If you have enough memory available, it's not a problem, however if you don't have enough memory SSIS will page to disk. When this happens a package can go from a few minutes to a few hours of execution.
If it is an option, store the data in a staging table and do the sorting and aggregation there.
Need an answer? No, you need a question
My blog at https://sqlkover.com.
MCSE Business Intelligence - Microsoft Data Platform MVP
Viewing 6 posts - 1 through 5 (of 5 total)
You must be logged in to reply to this topic. Login to reply