Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase

SSIS Event Handling for Data Flow Tasks with Error Log in SQL Server Table Expand / Collapse
Author
Message
Posted Thursday, October 03, 2013 11:27 AM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Thursday, November 07, 2013 10:28 PM
Points: 3, Visits: 12
Hi,
Created two Control flow Tasks for Dimensions and Fact Tables Load. Each Control Flow Tasks as some Data Flows (Flat File Source, Pivot, Unpivot, Multicast, Sorting, Aggregation, Dimension Table Lookup, Fact Table Lookup, Data Conversion, OLEDB Destination etc...)
Works fine with above Transformations. Need some Help to Handle Events for Data Flow Tasks with Error Handling. All these Data Flow Tasks and Error should be Tracked in SQL Server Table.
And How to Show a Pop-up for Input from User to get his emailID and to store the EmailID with Data Flow Tasks in SQL Table.
Attached Screen shot of my Package created so far....



  Post Attachments 
Control flow.jpg (7 views, 9.61 KB)
DIMENSION DATA FLOW.jpg (7 views, 70.57 KB)
FACT DATA FLOW.jpg (7 views, 60.87 KB)
Post #1501295
Posted Thursday, October 03, 2013 2:06 PM


SSChampion

SSChampionSSChampionSSChampionSSChampionSSChampionSSChampionSSChampionSSChampionSSChampionSSChampion

Group: General Forum Members
Last Login: Today @ 1:28 AM
Points: 12,208, Visits: 9,174
Are you sure it works fine? You have a whole lot of blocking transformations in your package, which can lead to serious performance issues for large data sets.
Anyway, on topic: every component in a data flow has an error path. (the red arrows) You need to configure the components to redirect rows when an error occurs.
You can group the error paths with union all components and write the error rows to a database table.

Regarding the user input: SSIS is not made for user input. It is an ETL tool and it is made to run unattended on a server.
The only way you can have user input is to create dialogs in a script component, but that would mean the user has the execute the package manually using visual studio. Not really ideal.




How to post forum questions.
Need an answer? No, you need a question.
What’s the deal with Excel & SSIS?

Member of LinkedIn. My blog at LessThanDot.

MCSA SQL Server 2012 - MCSE Business Intelligence
Post #1501347
Posted Friday, October 04, 2013 12:32 AM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Thursday, November 07, 2013 10:28 PM
Points: 3, Visits: 12
Hey Koen,
Thanks for ur reply. Error Log with Red Flow Direction is working, But unable to Union all Error Logs to handle it in SQL Server Table. Will try that it seems will works fine.
And Package Runs fine, since it as Apt Input file and if any Errors it will surely Blocks. Am new to SSIS and past 3 weeks working in this. Some Transformations will be Blocked in this Package. Please Suggest me to Enhance this Package Better.
I will post my Requirements and Required Output is detail.

1. Flat File (Products [7 to 8 Columns] With Month Names as Columns)
2. Unpivot this Month Column names in Rows and its Value as another column
3. Pivot (In Products there will be a Column Measures - Need to Pivot Measure Column Values to Column names).
4. Products Columns will be Inserted in Product Dimensions.
5. Month Columns in Month dimensions
6. Look-up Product and Month Dimensions in Fact Tables (Those Pivoted Columns will be in Fact Table)
7. And Fact Table Insert.

the above Steps is need to Perform for Each Flat file sources with some Validations.
Validations:
1. Any New Product Dimensions (Have to Pop-up those Values)
2. Any new Month Dimensions (Have to Pop-up)
3. Any Duplicates or Null Values (have to pop-up)
4. All those Transformations with Errors need to Capture it in SQL Server Table.
Post #1501431
Posted Friday, October 04, 2013 4:34 AM


SSChampion

SSChampionSSChampionSSChampionSSChampionSSChampionSSChampionSSChampionSSChampionSSChampionSSChampion

Group: General Forum Members
Last Login: Today @ 1:28 AM
Points: 12,208, Visits: 9,174
Basically you'll want to avoid the sort and aggregation transform in SSIS; as they are blocking transformations and load all rows into memory.

These types of transformations are better handled by SQL Server.




How to post forum questions.
Need an answer? No, you need a question.
What’s the deal with Excel & SSIS?

Member of LinkedIn. My blog at LessThanDot.

MCSA SQL Server 2012 - MCSE Business Intelligence
Post #1501518
Posted Friday, October 04, 2013 10:41 AM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Thursday, November 07, 2013 10:28 PM
Points: 3, Visits: 12
Its Fine. But am Sorting Transformation to get the Unique Values for some Columns.
And Aggregation Transformation to SUM the Values for Measure Columns.
Without Using these How Can i Use it thru Memory. Can you elaborate in detail. So that I can correct myself in my Package.
Post #1501680
Posted Sunday, October 06, 2013 1:47 PM


SSChampion

SSChampionSSChampionSSChampionSSChampionSSChampionSSChampionSSChampionSSChampionSSChampionSSChampion

Group: General Forum Members
Last Login: Today @ 1:28 AM
Points: 12,208, Visits: 9,174
kalaiselvan.rajendran (10/4/2013)
Its Fine. But am Sorting Transformation to get the Unique Values for some Columns.
And Aggregation Transformation to SUM the Values for Measure Columns.
Without Using these How Can i Use it thru Memory. Can you elaborate in detail. So that I can correct myself in my Package.


Getting distinct values and sorting data is something the database engine excels at.
In SSIS, every row has to be read into memory in order to process these operations.
This can lead to performance issues with large datasets. If you have enough memory available, it's not a problem, however if you don't have enough memory SSIS will page to disk. When this happens a package can go from a few minutes to a few hours of execution.

If it is an option, store the data in a staging table and do the sorting and aggregation there.




How to post forum questions.
Need an answer? No, you need a question.
What’s the deal with Excel & SSIS?

Member of LinkedIn. My blog at LessThanDot.

MCSA SQL Server 2012 - MCSE Business Intelligence
Post #1501952
« Prev Topic | Next Topic »

Add to briefcase

Permissions Expand / Collapse