SQL Clone
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


SSIS Event Handling for Data Flow Tasks with Error Log in SQL Server Table


SSIS Event Handling for Data Flow Tasks with Error Log in SQL Server Table

Author
Message
kalaiselvan.rajendran
kalaiselvan.rajendran
SSC Rookie
SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)

Group: General Forum Members
Points: 31 Visits: 13
Hi,
Created two Control flow Tasks for Dimensions and Fact Tables Load. Each Control Flow Tasks as some Data Flows (Flat File Source, Pivot, Unpivot, Multicast, Sorting, Aggregation, Dimension Table Lookup, Fact Table Lookup, Data Conversion, OLEDB Destination etc...)
Works fine with above Transformations. Need some Help to Handle Events for Data Flow Tasks with Error Handling. All these Data Flow Tasks and Error should be Tracked in SQL Server Table.
And How to Show a Pop-up for Input from User to get his emailID and to store the EmailID with Data Flow Tasks in SQL Table.
Attached Screen shot of my Package created so far....
Attachments
Control flow.jpg (25 views, 9.00 KB)
DIMENSION DATA FLOW.jpg (28 views, 70.00 KB)
FACT DATA FLOW.jpg (20 views, 60.00 KB)
Koen Verbeeck
Koen Verbeeck
SSC Guru
SSC Guru (63K reputation)SSC Guru (63K reputation)SSC Guru (63K reputation)SSC Guru (63K reputation)SSC Guru (63K reputation)SSC Guru (63K reputation)SSC Guru (63K reputation)SSC Guru (63K reputation)

Group: General Forum Members
Points: 63392 Visits: 13298
Are you sure it works fine? You have a whole lot of blocking transformations in your package, which can lead to serious performance issues for large data sets.
Anyway, on topic: every component in a data flow has an error path. (the red arrows) You need to configure the components to redirect rows when an error occurs.
You can group the error paths with union all components and write the error rows to a database table.

Regarding the user input: SSIS is not made for user input. It is an ETL tool and it is made to run unattended on a server.
The only way you can have user input is to create dialogs in a script component, but that would mean the user has the execute the package manually using visual studio. Not really ideal.


How to post forum questions.
Need an answer? No, you need a question.
What’s the deal with Excel & SSIS?
My blog at SQLKover.

MCSE Business Intelligence - Microsoft Data Platform MVP
kalaiselvan.rajendran
kalaiselvan.rajendran
SSC Rookie
SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)

Group: General Forum Members
Points: 31 Visits: 13
Hey Koen,
Thanks for ur reply. Error Log with Red Flow Direction is working, But unable to Union all Error Logs to handle it in SQL Server Table. Will try that it seems will works fine.
And Package Runs fine, since it as Apt Input file and if any Errors it will surely Blocks. Am new to SSIS and past 3 weeks working in this. Some Transformations will be Blocked in this Package. Please Suggest me to Enhance this Package Better.
I will post my Requirements and Required Output is detail.

1. Flat File (Products [7 to 8 Columns] With Month Names as Columns)
2. Unpivot this Month Column names in Rows and its Value as another column
3. Pivot (In Products there will be a Column Measures - Need to Pivot Measure Column Values to Column names).
4. Products Columns will be Inserted in Product Dimensions.
5. Month Columns in Month dimensions
6. Look-up Product and Month Dimensions in Fact Tables (Those Pivoted Columns will be in Fact Table)
7. And Fact Table Insert.

the above Steps is need to Perform for Each Flat file sources with some Validations.
Validations:
1. Any New Product Dimensions (Have to Pop-up those Values)
2. Any new Month Dimensions (Have to Pop-up)
3. Any Duplicates or Null Values (have to pop-up)
4. All those Transformations with Errors need to Capture it in SQL Server Table.
Koen Verbeeck
Koen Verbeeck
SSC Guru
SSC Guru (63K reputation)SSC Guru (63K reputation)SSC Guru (63K reputation)SSC Guru (63K reputation)SSC Guru (63K reputation)SSC Guru (63K reputation)SSC Guru (63K reputation)SSC Guru (63K reputation)

Group: General Forum Members
Points: 63392 Visits: 13298
Basically you'll want to avoid the sort and aggregation transform in SSIS; as they are blocking transformations and load all rows into memory.

These types of transformations are better handled by SQL Server.


How to post forum questions.
Need an answer? No, you need a question.
What’s the deal with Excel & SSIS?
My blog at SQLKover.

MCSE Business Intelligence - Microsoft Data Platform MVP
kalaiselvan.rajendran
kalaiselvan.rajendran
SSC Rookie
SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)SSC Rookie (31 reputation)

Group: General Forum Members
Points: 31 Visits: 13
Its Fine. But am Sorting Transformation to get the Unique Values for some Columns.
And Aggregation Transformation to SUM the Values for Measure Columns.
Without Using these How Can i Use it thru Memory. Can you elaborate in detail. So that I can correct myself in my Package.
Koen Verbeeck
Koen Verbeeck
SSC Guru
SSC Guru (63K reputation)SSC Guru (63K reputation)SSC Guru (63K reputation)SSC Guru (63K reputation)SSC Guru (63K reputation)SSC Guru (63K reputation)SSC Guru (63K reputation)SSC Guru (63K reputation)

Group: General Forum Members
Points: 63392 Visits: 13298
kalaiselvan.rajendran (10/4/2013)
Its Fine. But am Sorting Transformation to get the Unique Values for some Columns.
And Aggregation Transformation to SUM the Values for Measure Columns.
Without Using these How Can i Use it thru Memory. Can you elaborate in detail. So that I can correct myself in my Package.


Getting distinct values and sorting data is something the database engine excels at.
In SSIS, every row has to be read into memory in order to process these operations.
This can lead to performance issues with large datasets. If you have enough memory available, it's not a problem, however if you don't have enough memory SSIS will page to disk. When this happens a package can go from a few minutes to a few hours of execution.

If it is an option, store the data in a staging table and do the sorting and aggregation there.


How to post forum questions.
Need an answer? No, you need a question.
What’s the deal with Excel & SSIS?
My blog at SQLKover.

MCSE Business Intelligence - Microsoft Data Platform MVP
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search