March 31, 2016 at 4:21 pm
I've stumbled into an unexpected situation.
I'm observing significant performance differences depending on how a child Package is launched.
Scenario 1 : Single Master Package calling Specific Packages
Scenario 2 : ETL Framework approach - Specific package is a few layers in (nested child packages).
I would expect some initial overhead with scenario 2, but once the base package is running, I would expected performance to be similar.
WHat I've found is performance of the base packages is significantly better with scenario 1.
For Example : Same server, same data, etc.
Scenario 1 : 1 minutes.
Scenario 2 : 10 minutes.
There does not appear to be any "problem" step .... all the steps are slowed in Scenario 2.
I even looked at a step that didn't need any datebase - all it does is use variables to build a file path.
Scenario 1 : 1 sec
Scenario 2 : 7 sec.
Any thoughts ?
Is it possible that have multiple packages open in scenario 2 causes the system to reach a point where even variables are being written to disk?
Is there some threshold of "active packages" that I may have hit ?
April 1, 2016 at 5:24 am
larry.andrews (3/31/2016)
I've stumbled into an unexpected situation.I'm observing significant performance differences depending on how a child Package is launched.
Scenario 1 : Single Master Package calling Specific Packages
Scenario 2 : ETL Framework approach - Specific package is a few layers in (nested child packages).
I would expect some initial overhead with scenario 2, but once the base package is running, I would expected performance to be similar.
WHat I've found is performance of the base packages is significantly better with scenario 1.
For Example : Same server, same data, etc.
Scenario 1 : 1 minutes.
Scenario 2 : 10 minutes.
There does not appear to be any "problem" step .... all the steps are slowed in Scenario 2.
I even looked at a step that didn't need any datebase - all it does is use variables to build a file path.
Scenario 1 : 1 sec
Scenario 2 : 7 sec.
Any thoughts ?
Is it possible that have multiple packages open in scenario 2 causes the system to reach a point where even variables are being written to disk?
Is there some threshold of "active packages" that I may have hit ?
Are you seeing this behaviour after deploying the job to a server, or are these findings from running it in VS?
April 4, 2016 at 2:04 pm
Similar behavior - deployed to the server or running in VS (on the server).
April 4, 2016 at 2:31 pm
larry.andrews (4/4/2016)
Similar behavior - deployed to the server or running in VS (on the server).
OK. I was wondering whether the VS debug overhead might have been a factor, but it seems not.
I don't recall ever seeing anyone else do this testing, so I have nothing much useful to add. It's an interesting experiment though.
I would also say that I generally hate processes which rely on deep nesting of packages, because it usually means that it is difficult to unit test those packages. The framework we use here never goes beyond two levels:
(1) Framework master package calls
(2) Framework multi-threading package, which calls
(3) all the other packages,
based on meta data to control parallelism and execution order
The non-framework packages do not call each other.
Viewing 4 posts - 1 through 4 (of 4 total)
You must be logged in to reply to this topic. Login to reply