Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase

Generic SSIS Package Creation Expand / Collapse
Author
Message
Posted Tuesday, January 22, 2013 1:17 PM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Wednesday, June 18, 2014 3:04 PM
Points: 162, Visits: 570
Hello,
I have an Idea and want to see if this can be possible or any real time Developed solution already there

Scenario :
Package should be able to pull data from Sql Server and push it to Staging tables on OrcaleDB. There are no transformation used, it should be a straight pull.

My team figured out the number of SSIS packages to be created will be some Hundreds with Different Source and destination Tables. All this packages will be copy for one another. Which is a heck of work to do.

I think off building one SSIS package and pass in all the Data ( Source, Destination, SourceDB) dynamically from a table ( Meta Data table),As the all packages logic is same. It’s just the Source and Destination table names are different.
I have no idea how mappings would be handled b/w Source and destination and Change of Connection managers.
Post #1410244
Posted Tuesday, January 22, 2013 8:24 PM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Yesterday @ 1:37 PM
Points: 43, Visits: 336
You may want to look at BIML (Business Intelligence Markup Language). It comes with BIDS Helper.
It's metadata driven and allows you to automate SSIS package generation.

I've implemented this scenario, but moving tables from SQL Server source to staging tables in SQL Server (not Oracle). Source and staging tables should have the same table and column names. Then, columns will be automatically mapped for each table.

This script is an example to get started. It creates SSIS package to export each table in the database to a flatfile. You'll see it is dynamic and you don't have to manually create them.

I've a framework I about to publish that might also give you ideas. I'll let you know when it's published.


~Sam.

http://svangasql.wordpress.com
Post #1410316
Posted Wednesday, January 23, 2013 8:30 AM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Wednesday, June 18, 2014 3:04 PM
Points: 162, Visits: 570
samvanga (1/22/2013)
You may want to look at BIML (Business Intelligence Markup Language). It comes with BIDS Helper.
It's metadata driven and allows you to automate SSIS package generation.

I've implemented this scenario, but moving tables from SQL Server source to staging tables in SQL Server (not Oracle). Source and staging tables should have the same table and column names. Then, columns will be automatically mapped for each table.

This script is an example to get started. It creates SSIS package to export each table in the database to a flatfile. You'll see it is dynamic and you don't have to manually create them.

I've a framework I about to publish that might also give you ideas. I'll let you know when it's published.



BIML can only be written for SQL to SQL loading and not SQL to Orcale, am I right?
Post #1410619
Posted Wednesday, January 23, 2013 9:16 AM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Yesterday @ 1:37 PM
Points: 43, Visits: 336
No. BIML can create packages to move data from SSIS to Oracle.

~Sam.

http://svangasql.wordpress.com
Post #1410667
Posted Wednesday, January 23, 2013 9:34 AM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Wednesday, June 18, 2014 3:04 PM
Points: 162, Visits: 570
samvanga (1/22/2013)
You may want to look at BIML (Business Intelligence Markup Language). It comes with BIDS Helper.
It's metadata driven and allows you to automate SSIS package generation.

I've implemented this scenario, but moving tables from SQL Server source to staging tables in SQL Server (not Oracle). Source and staging tables should have the same table and column names. Then, columns will be automatically mapped for each table.

This script is an example to get started. It creates SSIS package to export each table in the database to a flatfile. You'll see it is dynamic and you don't have to manually create them.

I've a framework I about to publish that might also give you ideas. I'll let you know when it's published.


I have difficult in understanding this BIML concept….. I end up editing BIML for every package that I want to create (Source, source Query, Destination, data types). I could have done it in the same way by copying an SSIS package and editing it or standard template can be used to generate a SSIS package with CM and Task and DFs and edit source and Destinations, isn’t this same as BIML?
I am trying to understand what more flexibility BIML adds up when compared to customized SSIS Template.
Post #1410681
Posted Saturday, January 26, 2013 12:05 PM


SSCertifiable

SSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiable

Group: General Forum Members
Last Login: Yesterday @ 8:51 PM
Points: 7,140, Visits: 12,763
If there are no transforms have you thought about setting up a Snapshot Replication Publication/Subscription to send data from SQL Server to Oracle?

If you must do this with SSIS then with hundreds of packages you may want to invest some effort building your own .NET application that can build and execute an SSIS package on the fly using the SSIS Object Model per metadata you store in a table, e.g. source SQL Server table and column names mapping to destination Oracle table and column name, but that sounds a lot like you'd be re-engineering what Replication can do for you.


__________________________________________________________________________________________________
There are no special teachers of virtue, because virtue is taught by the whole community. --Plato
Post #1412055
Posted Monday, January 28, 2013 4:02 AM


SSCertifiable

SSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiable

Group: General Forum Members
Last Login: Today @ 2:52 PM
Points: 5,317, Visits: 12,353
quillis131 (1/23/2013)
samvanga (1/22/2013)
You may want to look at BIML (Business Intelligence Markup Language). It comes with BIDS Helper.
It's metadata driven and allows you to automate SSIS package generation.

I've implemented this scenario, but moving tables from SQL Server source to staging tables in SQL Server (not Oracle). Source and staging tables should have the same table and column names. Then, columns will be automatically mapped for each table.

This script is an example to get started. It creates SSIS package to export each table in the database to a flatfile. You'll see it is dynamic and you don't have to manually create them.

I've a framework I about to publish that might also give you ideas. I'll let you know when it's published.


I have difficult in understanding this BIML concept….. I end up editing BIML for every package that I want to create (Source, source Query, Destination, data types). I could have done it in the same way by copying an SSIS package and editing it or standard template can be used to generate a SSIS package with CM and Task and DFs and edit source and Destinations, isn’t this same as BIML?
I am trying to understand what more flexibility BIML adds up when compared to customized SSIS Template.


Good question. BIML attempts to be a simple text-based representation of a package, which can be 'built' (can't remember the correct term) into a package.

It's possible to create a routine - eg a stored proc - which will accept parameters & return an XML string in BIML format which can be built straight into a package.

So you can create packages which do not require any BIDS development. And recreate them at will.

This is particularly useful when you want to use the same simple package structure for numerous sources and destinations (eg, a data warehouse load).




Help us to help you. For better, quicker and more-focused answers to your questions, consider following the advice in this link.

When you ask a question (and please do ask a question: "My T-SQL does not work" just doesn't cut it), please provide enough information for us to understand its context.

It is better to keep your mouth shut and appear stupid than to open it and remove all doubt. (Mark Twain)
Post #1412320
« Prev Topic | Next Topic »

Add to briefcase

Permissions Expand / Collapse