<a href="http://g.adspeed.net/ad.php?do=clk&zid=15220&wd=468&ht=60&pair=as" target="_blank"><img style="border:0px;" src="http://g.adspeed.net/ad.php?do=img&zid=15220&wd=468&ht=60&pair=as" alt="i" width="468" height="60"/></a>
Log in
::
Register
::
Not logged in
Home
Tags
Articles
Editorials
Stairways
Forums
Scripts
Videos
Blogs
QotD
Books
Ask SSC
SQL Jobs
Training
Authors
About us
Contact us
Newsletters
Write for us
<a href="http://g.adspeed.net/ad.php?do=clk&zid=15491&wd=120&ht=300&pair=as" target="_top"><img style="border:0px;" src="http://g.adspeed.net/ad.php?do=img&zid=15491&wd=120&ht=300&pair=as" alt="i" width="120" height="300"/></a>
Recent Posts
Recent Posts
Popular Topics
Popular Topics
Home
Search
Members
Calendar
Who's On
Home
»
SQL Server 2008
»
SQL Server 2008 - General
»
Process Architecture Optimization
Process Architecture Optimization
Rate Topic
Display Mode
Topic Options
Author
Message
sqlnaive
sqlnaive
Posted Wednesday, January 16, 2013 12:23 AM
SSCrazy
Group: General Forum Members
Last Login: Today @ 12:14 AM
Points: 2,797,
Visits: 1,746
I have this scenario where we have som huge set of data in Sybase. Now there is a procedure which contains some heavy processing on that set of data based on certain parameters and finally store that output to a temporary table. Later, in the same session, that whole processed data in the temporary table gets transferred to our SQL Server tables by java code in small chunks say 50k rows per page.
Now this whole thing takes lot of time. I have been assigned to optimize this whole architecture. I was just wondering if this can be done using SSIS ? Important thing here is that the data processing procedure runs parallely to max 10 connections at a time.
Post #1407632
opc.three
opc.three
Posted Wednesday, January 16, 2013 12:40 AM
SSCertifiable
Group: General Forum Members
Last Login: Yesterday @ 1:07 PM
Points: 6,826,
Visits: 11,951
Kickoff a proc in Sybase and then copy some data from a table to SQL Server when the proc is done? Sounds like an easy job for SSIS. The hardest part of it may actually be finding good Sybase drivers that work with SSIS and setting up your connections to retrieve the data.
__________________________________________________________________________________________________
There are no special teachers of virtue, because virtue is taught by the whole community.
--Plato
Believe you can and you're halfway there.
--Theodore Roosevelt
Everything Should Be Made as Simple as Possible, But Not Simpler
--Albert Einstein
The significant problems we face cannot be solved at the same level of thinking we were at when we created them.
--Albert Einstein
1 apple is not exactly 1/8 of 8 apples. Because there are no absolutely identical apples.
--Giordy
Post #1407636
sqlnaive
sqlnaive
Posted Wednesday, January 16, 2013 1:07 AM
SSCrazy
Group: General Forum Members
Last Login: Today @ 12:14 AM
Points: 2,797,
Visits: 1,746
Thanks. Apart from drivers, how it works parallely ? Say If i create a SSIS package. Will the same package work in parallel if invoked with different set of parameters at the same time ?
Post #1407650
Koen Verbeeck
Koen Verbeeck
Posted Wednesday, January 16, 2013 1:10 AM
SSCrazy Eights
Group: General Forum Members
Last Login: 2 days ago @ 6:30 AM
Points: 9,410,
Visits: 6,495
sqlnaive (1/16/2013)
Thanks. Apart from drivers, how it works parallely ? Say If i create a SSIS package. Will the same package work in parallel if invoked with different set of parameters at the same time ?
I don't think that should be an issue, unless you are writing to the same table.
How to post forum questions.
Need an answer? No, you need a question.
What’s the deal with Excel & SSIS?
Member of
LinkedIn
. My blog at
LessThanDot
.
MCSA SQL Server 2012 - MCSE Business Intelligence
Post #1407653
opc.three
opc.three
Posted Wednesday, January 16, 2013 1:14 AM
SSCertifiable
Group: General Forum Members
Last Login: Yesterday @ 1:07 PM
Points: 6,826,
Visits: 11,951
An SSIS package is just a set of instructions. If you kick off dtexec.exe three times in three different cmd shell windows and all call the same package then you'll have three instances of the package running at the same time.
SSIS supports parallel processing by default both internally in some of the Components as well as in separate components not connected by precedence constraints. In the example below Data Flows 1, 2 and 4 will immediately start executing in parallel when the package starts. Data Flow 3 will not start executing until Data Flow 2 completes successfully:
__________________________________________________________________________________________________
There are no special teachers of virtue, because virtue is taught by the whole community.
--Plato
Believe you can and you're halfway there.
--Theodore Roosevelt
Everything Should Be Made as Simple as Possible, But Not Simpler
--Albert Einstein
The significant problems we face cannot be solved at the same level of thinking we were at when we created them.
--Albert Einstein
1 apple is not exactly 1/8 of 8 apples. Because there are no absolutely identical apples.
--Giordy
Post Attachments
SSISDFT.jpg
(
35 views,
11.13 KB
)
Post #1407655
sqlnaive
sqlnaive
Posted Wednesday, January 16, 2013 1:16 AM
SSCrazy
Group: General Forum Members
Last Login: Today @ 12:14 AM
Points: 2,797,
Visits: 1,746
Fortunately or unfortunately, it is the same table. That was also in my mind. But I don't understand that how it is not giving any issue currently as it is writing to same table parallely.
Post #1407658
Koen Verbeeck
Koen Verbeeck
Posted Wednesday, January 16, 2013 1:22 AM
SSCrazy Eights
Group: General Forum Members
Last Login: 2 days ago @ 6:30 AM
Points: 9,410,
Visits: 6,495
sqlnaive (1/16/2013)
Fortunately or unfortunately, it is the same table. That was also in my mind. But I don't understand that how it is not giving any issue currently as it is writing to same table parallely.
Which option have you specified in the OLE DB Destination.
There's the option to use table locks, so if this is off, you might have less concurrency.
How to post forum questions.
Need an answer? No, you need a question.
What’s the deal with Excel & SSIS?
Member of
LinkedIn
. My blog at
LessThanDot
.
MCSA SQL Server 2012 - MCSE Business Intelligence
Post #1407663
opc.three
opc.three
Posted Wednesday, January 16, 2013 1:47 AM
SSCertifiable
Group: General Forum Members
Last Login: Yesterday @ 1:07 PM
Points: 6,826,
Visits: 11,951
sqlnaive (1/16/2013)
Fortunately or unfortunately, it is the same table. That was also in my mind. But I don't understand that how it is not giving any issue currently as it is writing to same table parallely.
It depends on how they are writing rows. Committing 50K at a time, if they are added to a heap or the tail end of the clustered index then having 10 processes doing that concurrently might cause some intermittent blocking but could increase overall throughput significantly.
__________________________________________________________________________________________________
There are no special teachers of virtue, because virtue is taught by the whole community.
--Plato
Believe you can and you're halfway there.
--Theodore Roosevelt
Everything Should Be Made as Simple as Possible, But Not Simpler
--Albert Einstein
The significant problems we face cannot be solved at the same level of thinking we were at when we created them.
--Albert Einstein
1 apple is not exactly 1/8 of 8 apples. Because there are no absolutely identical apples.
--Giordy
Post #1407676
« Prev Topic
|
Next Topic »
Permissions
You
cannot
post new topics.
You
cannot
post topic replies.
You
cannot
post new polls.
You
cannot
post replies to polls.
You
cannot
edit your own topics.
You
cannot
delete your own topics.
You
cannot
edit other topics.
You
cannot
delete other topics.
You
cannot
edit your own posts.
You
cannot
edit other posts.
You
cannot
delete your own posts.
You
cannot
delete other posts.
You
cannot
post events.
You
cannot
edit your own events.
You
cannot
edit other events.
You
cannot
delete your own events.
You
cannot
delete other events.
You
cannot
send private messages.
You
cannot
send emails.
You
may
read topics.
You
cannot
rate topics.
You
cannot
vote within polls.
You
cannot
upload attachments.
You
may
download attachments.
You
cannot
post HTML code.
You
cannot
edit HTML code.
You
cannot
post IFCode.
You
cannot
post JavaScript.
You
cannot
post EmotIcons.
You
cannot
post or upload images.
Copyright © 2002-2013 Simple Talk Publishing. All Rights Reserved.
Privacy Policy.
Terms of Use.
Report Abuse.
