January 22, 2013 at 1:17 pm
Hello,
I have an Idea and want to see if this can be possible or any real time Developed solution already there
Scenario :
Package should be able to pull data from Sql Server and push it to Staging tables on OrcaleDB. There are no transformation used, it should be a straight pull.
My team figured out the number of SSIS packages to be created will be some Hundreds with Different Source and destination Tables. All this packages will be copy for one another. Which is a heck of work to do.
I think off building one SSIS package and pass in all the Data ( Source, Destination, SourceDB) dynamically from a table ( Meta Data table),As the all packages logic is same. It’s just the Source and Destination table names are different.
I have no idea how mappings would be handled b/w Source and destination and Change of Connection managers.
January 22, 2013 at 8:24 pm
You may want to look at BIML (Business Intelligence Markup Language). It comes with BIDS Helper.
It's metadata driven and allows you to automate SSIS package generation.
I've implemented this scenario, but moving tables from SQL Server source to staging tables in SQL Server (not Oracle). Source and staging tables should have the same table and column names. Then, columns will be automatically mapped for each table.
This script is an example to get started. It creates SSIS package to export each table in the database to a flatfile. You'll see it is dynamic and you don't have to manually create them.
I've a framework I about to publish that might also give you ideas. I'll let you know when it's published.
Sam Vanga
http://SamuelVanga.com
January 23, 2013 at 8:30 am
samvanga (1/22/2013)
You may want to look at BIML (Business Intelligence Markup Language). It comes with BIDS Helper.It's metadata driven and allows you to automate SSIS package generation.
I've implemented this scenario, but moving tables from SQL Server source to staging tables in SQL Server (not Oracle). Source and staging tables should have the same table and column names. Then, columns will be automatically mapped for each table.
This script is an example to get started. It creates SSIS package to export each table in the database to a flatfile. You'll see it is dynamic and you don't have to manually create them.
I've a framework I about to publish that might also give you ideas. I'll let you know when it's published.
BIML can only be written for SQL to SQL loading and not SQL to Orcale, am I right?
January 23, 2013 at 9:16 am
No. BIML can create packages to move data from SSIS to Oracle.
Sam Vanga
http://SamuelVanga.com
January 23, 2013 at 9:34 am
samvanga (1/22/2013)
You may want to look at BIML (Business Intelligence Markup Language). It comes with BIDS Helper.It's metadata driven and allows you to automate SSIS package generation.
I've implemented this scenario, but moving tables from SQL Server source to staging tables in SQL Server (not Oracle). Source and staging tables should have the same table and column names. Then, columns will be automatically mapped for each table.
This script is an example to get started. It creates SSIS package to export each table in the database to a flatfile. You'll see it is dynamic and you don't have to manually create them.
I've a framework I about to publish that might also give you ideas. I'll let you know when it's published.
I have difficult in understanding this BIML concept….. I end up editing BIML for every package that I want to create (Source, source Query, Destination, data types). I could have done it in the same way by copying an SSIS package and editing it or standard template can be used to generate a SSIS package with CM and Task and DFs and edit source and Destinations, isn’t this same as BIML?
I am trying to understand what more flexibility BIML adds up when compared to customized SSIS Template.
January 26, 2013 at 12:05 pm
If there are no transforms have you thought about setting up a Snapshot Replication Publication/Subscription to send data from SQL Server to Oracle?
If you must do this with SSIS then with hundreds of packages you may want to invest some effort building your own .NET application that can build and execute an SSIS package on the fly using the SSIS Object Model per metadata you store in a table, e.g. source SQL Server table and column names mapping to destination Oracle table and column name, but that sounds a lot like you'd be re-engineering what Replication can do for you.
There are no special teachers of virtue, because virtue is taught by the whole community.
--Plato
January 28, 2013 at 4:02 am
quillis131 (1/23/2013)
samvanga (1/22/2013)
You may want to look at BIML (Business Intelligence Markup Language). It comes with BIDS Helper.It's metadata driven and allows you to automate SSIS package generation.
I've implemented this scenario, but moving tables from SQL Server source to staging tables in SQL Server (not Oracle). Source and staging tables should have the same table and column names. Then, columns will be automatically mapped for each table.
This script is an example to get started. It creates SSIS package to export each table in the database to a flatfile. You'll see it is dynamic and you don't have to manually create them.
I've a framework I about to publish that might also give you ideas. I'll let you know when it's published.
I have difficult in understanding this BIML concept….. I end up editing BIML for every package that I want to create (Source, source Query, Destination, data types). I could have done it in the same way by copying an SSIS package and editing it or standard template can be used to generate a SSIS package with CM and Task and DFs and edit source and Destinations, isn’t this same as BIML?
I am trying to understand what more flexibility BIML adds up when compared to customized SSIS Template.
Good question. BIML attempts to be a simple text-based representation of a package, which can be 'built' (can't remember the correct term) into a package.
It's possible to create a routine - eg a stored proc - which will accept parameters & return an XML string in BIML format which can be built straight into a package.
So you can create packages which do not require any BIDS development. And recreate them at will.
This is particularly useful when you want to use the same simple package structure for numerous sources and destinations (eg, a data warehouse load).
If you haven't even tried to resolve your issue, please don't expect the hard-working volunteers here to waste their time providing links to answers which you could easily have found yourself.
Viewing 7 posts - 1 through 6 (of 6 total)
You must be logged in to reply to this topic. Login to reply