Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 


 
     
Recent Posts
   
Popular Topics
Home   
Search
    Members    Calendar    Who's On

Home » Data Warehousing » Integration Services » Generic SSIS Package Creation
Add to briefcase

Generic SSIS Package Creation Expand / Collapse
Rate Topic
Display Mode
Topic Options
Author
Message
quillis131
Posted Tuesday, January 22, 2013 1:17 PM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Friday, May 10, 2013 10:46 AM
Points: 156, Visits: 512
Hello,
I have an Idea and want to see if this can be possible or any real time Developed solution already there

Scenario :
Package should be able to pull data from Sql Server and push it to Staging tables on OrcaleDB. There are no transformation used, it should be a straight pull.

My team figured out the number of SSIS packages to be created will be some Hundreds with Different Source and destination Tables. All this packages will be copy for one another. Which is a heck of work to do.

I think off building one SSIS package and pass in all the Data ( Source, Destination, SourceDB) dynamically from a table ( Meta Data table),As the all packages logic is same. It’s just the Source and Destination table names are different.
I have no idea how mappings would be handled b/w Source and destination and Change of Connection managers.
Post #1410244
Sam Vanga
Posted Tuesday, January 22, 2013 8:24 PM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Wednesday, May 01, 2013 11:47 AM
Points: 39, Visits: 264
You may want to look at BIML (Business Intelligence Markup Language). It comes with BIDS Helper.
It's metadata driven and allows you to automate SSIS package generation.

I've implemented this scenario, but moving tables from SQL Server source to staging tables in SQL Server (not Oracle). Source and staging tables should have the same table and column names. Then, columns will be automatically mapped for each table.

This script is an example to get started. It creates SSIS package to export each table in the database to a flatfile. You'll see it is dynamic and you don't have to manually create them.

I've a framework I about to publish that might also give you ideas. I'll let you know when it's published.

~Sam.

http://svangasql.wordpress.com
Post #1410316
quillis131
Posted Wednesday, January 23, 2013 8:30 AM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Friday, May 10, 2013 10:46 AM
Points: 156, Visits: 512
samvanga (1/22/2013)
You may want to look at BIML (Business Intelligence Markup Language). It comes with BIDS Helper.
It's metadata driven and allows you to automate SSIS package generation.

I've implemented this scenario, but moving tables from SQL Server source to staging tables in SQL Server (not Oracle). Source and staging tables should have the same table and column names. Then, columns will be automatically mapped for each table.

This script is an example to get started. It creates SSIS package to export each table in the database to a flatfile. You'll see it is dynamic and you don't have to manually create them.

I've a framework I about to publish that might also give you ideas. I'll let you know when it's published.



BIML can only be written for SQL to SQL loading and not SQL to Orcale, am I right?
Post #1410619
Sam Vanga
Posted Wednesday, January 23, 2013 9:16 AM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Wednesday, May 01, 2013 11:47 AM
Points: 39, Visits: 264
No. BIML can create packages to move data from SSIS to Oracle.

~Sam.

http://svangasql.wordpress.com
Post #1410667
quillis131
Posted Wednesday, January 23, 2013 9:34 AM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Friday, May 10, 2013 10:46 AM
Points: 156, Visits: 512
samvanga (1/22/2013)
You may want to look at BIML (Business Intelligence Markup Language). It comes with BIDS Helper.
It's metadata driven and allows you to automate SSIS package generation.

I've implemented this scenario, but moving tables from SQL Server source to staging tables in SQL Server (not Oracle). Source and staging tables should have the same table and column names. Then, columns will be automatically mapped for each table.

This script is an example to get started. It creates SSIS package to export each table in the database to a flatfile. You'll see it is dynamic and you don't have to manually create them.

I've a framework I about to publish that might also give you ideas. I'll let you know when it's published.


I have difficult in understanding this BIML concept….. I end up editing BIML for every package that I want to create (Source, source Query, Destination, data types). I could have done it in the same way by copying an SSIS package and editing it or standard template can be used to generate a SSIS package with CM and Task and DFs and edit source and Destinations, isn’t this same as BIML?
I am trying to understand what more flexibility BIML adds up when compared to customized SSIS Template.
Post #1410681
opc.three
Posted Saturday, January 26, 2013 12:05 PM


SSCertifiable

SSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiable

Group: General Forum Members
Last Login: Yesterday @ 8:27 PM
Points: 6,735, Visits: 11,788
If there are no transforms have you thought about setting up a Snapshot Replication Publication/Subscription to send data from SQL Server to Oracle?

If you must do this with SSIS then with hundreds of packages you may want to invest some effort building your own .NET application that can build and execute an SSIS package on the fly using the SSIS Object Model per metadata you store in a table, e.g. source SQL Server table and column names mapping to destination Oracle table and column name, but that sounds a lot like you'd be re-engineering what Replication can do for you.

__________________________________________________________________________________________________
There are no special teachers of virtue, because virtue is taught by the whole community. --Plato

Believe you can and you're halfway there. --Theodore Roosevelt

Everything Should Be Made as Simple as Possible, But Not Simpler --Albert Einstein

The significant problems we face cannot be solved at the same level of thinking we were at when we created them. --Albert Einstein

1 apple is not exactly 1/8 of 8 apples. Because there are no absolutely identical apples. --Giordy
Post #1412055
Phil Parkin
Posted Monday, January 28, 2013 4:02 AM


SSCarpal Tunnel

SSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal Tunnel

Group: General Forum Members
Last Login: Yesterday @ 9:39 AM
Points: 4,247, Visits: 9,500
quillis131 (1/23/2013)
samvanga (1/22/2013)
You may want to look at BIML (Business Intelligence Markup Language). It comes with BIDS Helper.
It's metadata driven and allows you to automate SSIS package generation.

I've implemented this scenario, but moving tables from SQL Server source to staging tables in SQL Server (not Oracle). Source and staging tables should have the same table and column names. Then, columns will be automatically mapped for each table.

This script is an example to get started. It creates SSIS package to export each table in the database to a flatfile. You'll see it is dynamic and you don't have to manually create them.

I've a framework I about to publish that might also give you ideas. I'll let you know when it's published.


I have difficult in understanding this BIML concept….. I end up editing BIML for every package that I want to create (Source, source Query, Destination, data types). I could have done it in the same way by copying an SSIS package and editing it or standard template can be used to generate a SSIS package with CM and Task and DFs and edit source and Destinations, isn’t this same as BIML?
I am trying to understand what more flexibility BIML adds up when compared to customized SSIS Template.


Good question. BIML attempts to be a simple text-based representation of a package, which can be 'built' (can't remember the correct term) into a package.

It's possible to create a routine - eg a stored proc - which will accept parameters & return an XML string in BIML format which can be built straight into a package.

So you can create packages which do not require any BIDS development. And recreate them at will.

This is particularly useful when you want to use the same simple package structure for numerous sources and destinations (eg, a data warehouse load).

____________________________________________________________________________________________

Help us to help you. For better, quicker and more focused answers to your questions, consider following the advice in this link:

http://www.sqlservercentral.com/articles/Best+Practices/61537/

If you are asking for help and your post does not contain a question, you should expect responses which do not contain any answers. Put a question mark in there somewhere - it's not rocket science.
Post #1412320
« Prev Topic | Next Topic »

Add to briefcase

Permissions Expand / Collapse

 
 
Copyright © 2002-2013 Simple Talk Publishing. All Rights Reserved. Privacy Policy. Terms of Use. Report Abuse.