January 18, 2018 at 9:25 am
Hello,
I have a unique situation where i need to execute all the SQLs sitting in the directory on same server in parallel using SSIS. Can someone help how can this be achieved? Number of files in the folder can vary.
Please note that - execution of all SQL has to start at same time in parallel.
January 18, 2018 at 9:32 am
What do you mean exactly by "execution of all SQL"?
Do you have SQL script files (.sql) sitting in the folder?
January 18, 2018 at 9:39 am
nic2885 - Thursday, January 18, 2018 9:25 AMHello,I have a unique situation where i need to execute all the SQLs sitting in the directory on same server in parallel using SSIS. Can someone help how can this be achieved? Number of files in the folder can vary.
Please note that - execution of all SQL has to start at same time in parallel.
This is simple. Just create a package containing multiple execute SQL tasks. They will execute in parallel by default (do not connect them using precedence constraints).
If you haven't even tried to resolve your issue, please don't expect the hard-working volunteers here to waste their time providing links to answers which you could easily have found yourself.
January 18, 2018 at 9:59 am
How dynamic are these files going to be? How many total possible, are the file names dynamic?
January 18, 2018 at 10:00 am
Phil Parkin - Thursday, January 18, 2018 9:39 AMnic2885 - Thursday, January 18, 2018 9:25 AMHello,I have a unique situation where i need to execute all the SQLs sitting in the directory on same server in parallel using SSIS. Can someone help how can this be achieved? Number of files in the folder can vary.
Please note that - execution of all SQL has to start at same time in parallel.This is simple. Just create a package containing multiple execute SQL tasks. They will execute in parallel by default (do not connect them using precedence constraints).
No of files are not certain.
January 18, 2018 at 10:00 am
Paulo de Jesus - Thursday, January 18, 2018 9:32 AMWhat do you mean exactly by "execution of all SQL"?
Do you have SQL script files (.sql) sitting in the folder?
yes
January 18, 2018 at 10:01 am
ZZartin - Thursday, January 18, 2018 9:59 AMHow dynamic are these files going to be? How many total possible, are the file names dynamic?
yes they can range from 1 to 15 and name can be anything. But all files will be .sql
January 18, 2018 at 10:11 am
Have a look at this article:
http://www.sqlservercentral.com/articles/Integration+Services+(SSIS)/163434/
You’ll have to modify the package a bit to execute the SQL files instead of importing the data but the principle of running the files in parallel should be the same.
January 18, 2018 at 10:25 am
nic2885 - Thursday, January 18, 2018 10:01 AMZZartin - Thursday, January 18, 2018 9:59 AMHow dynamic are these files going to be? How many total possible, are the file names dynamic?yes they can range from 1 to 15 and name can be anything. But all files will be .sql
Create a script task which populates up to 15 package variables with the paths to the different files.
Create 15 file connections and use an expressions in each 1 to set the value of 'Connection String' to path1, path 2, ..., path15, using the variables populated above.
Create 15 exec sql tasks and put them all in a sequence container. The exec sql tasks will use file connections 1 to 15 as their source.
Connect the sequence task to the script task, to force the script task to be executed first.
You will need to refine this to cope with the situation where there are < 15 files, possibly by using expressions to control the value of the exec sql tasks' 'Disable' property at runtime.
If you haven't even tried to resolve your issue, please don't expect the hard-working volunteers here to waste their time providing links to answers which you could easily have found yourself.
January 18, 2018 at 12:31 pm
Paulo de Jesus - Thursday, January 18, 2018 10:11 AMHave a look at this article:http://www.sqlservercentral.com/articles/Integration+Services+(SSIS)/163434/You’ll have to modify the package a bit to execute the SQL files instead of importing the data but the principle of running the files in parallel should be the same.
Thank you this helps a lot.
January 18, 2018 at 2:24 pm
For reference, if anyone lands on this page, I have resolved this using C#. Script task -
#region Help: Introduction to the script task
/* The Script Task allows you to perform virtually any operation that can be accomplished in
* a .Net application within the context of an Integration Services control flow.
*
* Expand the other regions which have "Help" prefixes for examples of specific ways to use
* Integration Services features within this script task. */
#endregion
#region Namespaces
using System;
using System.IO;
using System.Linq;
using System.Threading;
using System.Collections;
using System.Collections.Generic;
using Microsoft.SqlServer.Dts.Runtime;
using System.Data.SqlClient;
using Microsoft.SqlServer.Management.Smo;
using Microsoft.SqlServer.Management.Common;
#endregion
namespace ST_someuniqueidentifier
{
/// <summary>
/// ScriptMain is the entry point class of the script. Do not change the name, attributes,
/// or parent of this class.
/// </summary>
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
#region Help: Using Integration Services variables and parameters in a script
/* To use a variable in this script, first ensure that the variable has been added to
* either the list contained in the ReadOnlyVariables property or the list contained in
* the ReadWriteVariables property of this script task, according to whether or not your
* code needs to write to the variable. To add the variable, save this script, close this instance of
* Visual Studio, and update the ReadOnlyVariables and
* ReadWriteVariables properties in the Script Transformation Editor window.
* To use a parameter in this script, follow the same steps. Parameters are always read-only.
*
* Example of reading from a variable:
* DateTime startTime = (DateTime) Dts.Variables["System::StartTime"].Value;
*
* Example of writing to a variable:
* Dts.Variables["User::myStringVariable"].Value = "new value";
*
* Example of reading from a package parameter:
* int batchId = (int) Dts.Variables["$Package::batchId"].Value;
*
* Example of reading from a project parameter:
* int batchId = (int) Dts.Variables["$Project::batchId"].Value;
*
* Example of reading from a sensitive project parameter:
* int batchId = (int) Dts.Variables["$Project::batchId"].GetSensitiveValue();
* */
#endregion
#region Help: Firing Integration Services events from a script
/* This script task can fire events for logging purposes.
*
* Example of firing an error event:
* Dts.Events.FireError(18, "Process Values", "Bad value", "", 0);
*
* Example of firing an information event:
* Dts.Events.FireInformation(3, "Process Values", "Processing has started", "", 0, ref fireAgain)
*
* Example of firing a warning event:
* Dts.Events.FireWarning(14, "Process Values", "No values received for input", "", 0);
* */
#endregion
#region Help: Using Integration Services connection managers in a script
/* Some types of connection managers can be used in this script task. See the topic
* "Working with Connection Managers Programatically" for details.
*
* Example of using an ADO.Net connection manager:
* object rawConnection = Dts.Connections["Sales DB"].AcquireConnection(Dts.Transaction);
* SqlConnection myADONETConnection = (SqlConnection)rawConnection;
* //Use the connection in some code here, then release the connection
* Dts.Connections["Sales DB"].ReleaseConnection(rawConnection);
*
* Example of using a File connection manager
* object rawConnection = Dts.Connections["Prices.zip"].AcquireConnection(Dts.Transaction);
* string filePath = (string)rawConnection;
* //Use the connection in some code here, then release the connection
* Dts.Connections["Prices.zip"].ReleaseConnection(rawConnection);
* */
#endregion
/// <summary>
/// This method is called when this script task executes in the control flow.
/// Before returning from this method, set the value of Dts.TaskResult to indicate success or failure.
/// To open Help, press F1.
/// </summary>
public void Main()
{
try
{
string sourceDirectory = Dts.Variables["User::ParallelDMLs"].Value.ToString();
string fileMask = "*.Sql"; //Dts.Variables["$Project::FileMask"].Value.ToString();
Int16 noOfPackageInstances = 4;
Int32 fileCount = 0;
Int32 fileGroup;
List<string>[] allFiles = new List<string>[noOfPackageInstances];
foreach (string file in Directory.GetFiles(sourceDirectory, fileMask))
{
fileCount += 1;
fileGroup = fileCount % noOfPackageInstances;
List<string> tempList = new List<string>();
if (allFiles[fileGroup] != null)
{
tempList = allFiles[fileGroup];
}
tempList.Add(file);
allFiles[fileGroup] = tempList;
}
var Threads = new List<Thread>();
for (Int16 i = 0; i < noOfPackageInstances; i++)
{
Int16 j = i; // due to closure and scoping of the variable in the loop
if (allFiles[j] != null)
{
Thread t = new Thread(() => ExecuteScripts(allFiles[j])) // need to revisit this piece
{
IsBackground = false
};
t.Start();
Threads.Add(t);
}
}
// TODO: Add your code here
while (Threads.Any(t => t.ThreadState == ThreadState.Running))
{
Thread.Sleep(500);
};
Dts.TaskResult = (int)ScriptResults.Success;
}
catch (Exception ex)
{
Dts.Log(ex.Message, 0, null);
Dts.TaskResult = (int)ScriptResults.Failure;
}
}
public void ExecuteScripts(List<string> fileList)
{
var sqlConnectionString = Dts.Variables["$Project::ODSConnectionStringForCSharp"].Value.ToString();
SqlConnection conn = new SqlConnection(sqlConnectionString);
Server server = new Server(new ServerConnection(conn));
foreach (var file in fileList)
{
FileInfo fileInfo = new FileInfo(file);
string script = fileInfo.OpenText().ReadToEnd();
server.ConnectionContext.ExecuteNonQuery(script);
// close fileInfo
fileInfo = null;
}
}
#region ScriptResults declaration
/// <summary>
/// This enum provides a convenient shorthand within the scope of this class for setting the
/// result of the script.
///
/// This code was generated automatically.
/// </summary>
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
}
}
January 18, 2018 at 2:38 pm
So, if there happens to be 50 scripts sitting in that folder, then all 50 will be executed starting at the same time? This sounds like it's intended to be a load testing tool. In any even, I wouldn't want to put this thing in the hands of end users, not if they can point it at the production server, or else when you least expect it they'll hammer your database so hard you'll think it's a denial of service attack.
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
January 19, 2018 at 8:50 am
Eric M Russell - Thursday, January 18, 2018 2:38 PMSo, if there happens to be 50 scripts sitting in that folder, then all 50 will be executed starting at the same time? This sounds like it's intended to be a load testing tool. In any even, I wouldn't want to put this thing in the hands of end users, not if they can point it at the production server, or else when you least expect it they'll hammer your database so hard you'll think it's a denial of service attack.
True.. Thats why in the code i have hardcoded threads to be maximum of 4. We can configure them depending on number of CPUs you have.
Therefore, if there are 12 scripts, there will be 3 groups in which scripts will be executed in parallel.
Viewing 13 posts - 1 through 12 (of 12 total)
You must be logged in to reply to this topic. Login to reply