Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase ««12

T-SQL insert using SSIS Data Pump Expand / Collapse
Author
Message
Posted Wednesday, February 6, 2013 4:20 PM
SSC Journeyman

SSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC JourneymanSSC Journeyman

Group: General Forum Members
Last Login: Wednesday, December 10, 2014 1:21 PM
Points: 91, Visits: 418
Federico Iori (2/5/2013)
How could I launch from T-SQL a SSIS package ? The only way I know is using xp_cmdshell.. purpose of the article was a T-SQL stored procedure able to launch a SSIS package, and this requires xp_cmdshell


We use SQL Agent to launch and manage our SSIS packages, it's pretty trivial and easy to implement (and you could easily make it dynamic and adapt into your system).
Post #1416761
Posted Wednesday, February 6, 2013 9:26 PM


SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Thursday, December 11, 2014 10:00 PM
Points: 179, Visits: 592
SQL Agent is tricky because:

1. If you have only one SQL Agent job being your 'host' you can't run mutliple jobs in parallel
2. If you want to have a non power-user trigger a SQL Agent job from some kind of client you need quite a bit of custom plumbing to get it all to work

Here's another idea: use BULK INSERT instead of SSIS.


But anyway all of these various methods of loading data have already been covered ad nauseum elsewhere.
Post #1416797
Posted Thursday, February 7, 2013 2:07 AM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Wednesday, December 17, 2014 1:26 AM
Points: 33, Visits: 296
BULK INSERT on Windows OS is not pipelined : first you need to save all to a file, then Bulk Insert from the file.
SSIS use C# to achieve pipelining .
Post #1416894
Posted Thursday, February 7, 2013 4:24 AM


SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: Monday, December 15, 2014 7:14 PM
Points: 2,693, Visits: 1,223
nick.mcdermaid (2/6/2013)
SQL Agent is tricky because:

1. If you have only one SQL Agent job being your 'host' you can't run mutliple jobs in parallel
2. If you want to have a non power-user trigger a SQL Agent job from some kind of client you need quite a bit of custom plumbing to get it all to work

Here's another idea: use BULK INSERT instead of SSIS.


But anyway all of these various methods of loading data have already been covered ad nauseum elsewhere.

We load end-of-day transaction files from multiple banks using the same package. We have multiple SQL Agent jobs and use the /config parameter to provide a different config file for each bank. Package has no problems executing in parallel.

As for non-power users managing the jobs, that's done from a web-page that calls sp_start_job. The app pool identity of the site has the permissions required to manage the jobs.


Federico Iori (2/6/2013)
philcart (2/5/2013)

2) you call xp_cmdshell to execute your package. This won't be usable on systems that dis-allow usage of xp_cmdshell.

How could I launch from T-SQL a SSIS package ? The only way I know is using xp_cmdshell.. purpose of the article was a T-SQL stored procedure able to launch a SSIS package, and this requires xp_cmdshell


That is precisely the wrong approach to take. Forget the gaping security implications of having to keep xp_cmdshell enabled for the whole server. Why on earth would you contemplate launching an executable within the memory space of the Database Engine, consume it's resources and possibly adversely affect load performance?


Hope this helps
Phill Carter
--------------------
Colt 45 - the original point and click interface

Australian SQL Server User Groups - My profile
Phills Philosophies
Murrumbeena Cricket Club
Post #1416954
Posted Thursday, February 7, 2013 4:58 AM


SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Thursday, December 11, 2014 10:00 PM
Points: 179, Visits: 592
We load end-of-day transaction files from multiple banks using the same package. We have multiple SQL Agent jobs and use the /config parameter to provide a different config file for each bank. Package has no problems executing in parallel.


As I said if you only have one SQL Agent job there is a problem, as you have many there is not a problem

As for non-power users managing the jobs, that's done from a web-page that calls sp_start_job. The app pool identity of the site has the permissions required to manage the jobs.


I would call building a web app 'custom plumbing'

Anyway I don't think we disagree here. When we get down to specifics every solution has it's advantages and disadvantages.
Post #1416970
Posted Thursday, February 7, 2013 4:59 AM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Wednesday, December 17, 2014 1:26 AM
Points: 33, Visits: 296
philcart (2/7/2013)
nick.mcdermaid (2/6/2013)
SQL Agent is tricky because:

1. If you have only one SQL Agent job being your 'host' you can't run mutliple jobs in parallel
2. If you want to have a non power-user trigger a SQL Agent job from some kind of client you need quite a bit of custom plumbing to get it all to work

Here's another idea: use BULK INSERT instead of SSIS.


But anyway all of these various methods of loading data have already been covered ad nauseum elsewhere.

We load end-of-day transaction files from multiple banks using the same package. We have multiple SQL Agent jobs and use the /config parameter to provide a different config file for each bank. Package has no problems executing in parallel.

As for non-power users managing the jobs, that's done from a web-page that calls sp_start_job. The app pool identity of the site has the permissions required to manage the jobs.


Federico Iori (2/6/2013)
philcart (2/5/2013)

2) you call xp_cmdshell to execute your package. This won't be usable on systems that dis-allow usage of xp_cmdshell.

How could I launch from T-SQL a SSIS package ? The only way I know is using xp_cmdshell.. purpose of the article was a T-SQL stored procedure able to launch a SSIS package, and this requires xp_cmdshell


That is precisely the wrong approach to take. Forget the gaping security implications of having to keep xp_cmdshell enabled for the whole server. Why on earth would you contemplate launching an executable within the memory space of the Database Engine, consume it's resources and possibly adversely affect load performance?


PhilCart, your idea is :
T-SQL stored procedure should
1) Create with dynamic SQL and launch syncronously a job that launches BULK_TRANSFER_DATA.dtsx
2) Create with dynamic SQL and launch ( possibly syncronously) a second job that launches the SSIS package generated at point 1 and saved into a specific folder
3) Delete the 2 jobs and the created dtsx


It is possible of course, maybe safer than xp_cmdshell, but requires some coding ...
Would you try to provide it ? My article does not indicate the way, but offers a working solution..
Moreover, as long as I know, a SSIS package can create another SSIS package only to the file system ...
Post #1416972
Posted Thursday, February 7, 2013 5:53 AM


SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: Monday, December 15, 2014 7:14 PM
Points: 2,693, Visits: 1,223
Federico, No point in boring people unnecessarily. I've sent you a PM.

Hope this helps
Phill Carter
--------------------
Colt 45 - the original point and click interface

Australian SQL Server User Groups - My profile
Phills Philosophies
Murrumbeena Cricket Club
Post #1417014
Posted Thursday, February 7, 2013 2:54 PM
Mr or Mrs. 500

Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500

Group: General Forum Members
Last Login: Yesterday @ 12:27 AM
Points: 542, Visits: 805
Thank you for the article! Note that if you are using SQL 2012 there is now a way to invoke SSIS package from T-SQL without having to resort to xp_cmdshell: [url=http://sqlblog.com/blogs/davide_mauri/archive/2011/11/24/execute-a-ssis-package-in-sync-or-async-mode-from-sql-server-2012.aspx][/url]


Post #1417364
Posted Friday, February 8, 2013 2:01 AM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Wednesday, December 17, 2014 1:26 AM
Points: 33, Visits: 296
mishaluba (2/7/2013)
Thank you for the article! Note that if you are using SQL 2012 there is now a way to invoke SSIS package from T-SQL without having to resort to xp_cmdshell: [url=http://sqlblog.com/blogs/davide_mauri/archive/2011/11/24/execute-a-ssis-package-in-sync-or-async-mode-from-sql-server-2012.aspx][/url]


Nice to know Mishaluba, by the way I know personally Davide Mauri
I am not sure, anyway, if a SSIS package can create dynamically another SSIS package ( my case ) and save it elsewhere than the operating system ...
Post #1417520
« Prev Topic | Next Topic »

Add to briefcase ««12

Permissions Expand / Collapse