Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 

A script task alternative to a massive SSIS multicast transformation

By Stan Kulp,

A Multicast Transformation can be used for many things, and one of those things is to write the same data file generated from a query in a Data Flow Task into multiple network locations. The only problem with this approach is that each destination connection and file must be configured manually, which can get tedious and messy as the number of destination files grows. Just look at the package in the image below.

An alternative to a Multicast Transformation is to create a Data Flow Task with a single destination file, then copy that file to all the other destination folders using a Script Task.

Run the following sript in SQL Server Management Studio to create and populate a table named Customer.

CREATE TABLE [dbo].[Customer](
    [CustomerId] [int] NULL,
    [LastName] [varchar](20) NULL,
    [FirstName] [varchar](20) NULL,
    [City] [varchar](20) NULL,
    [State] [varchar](2) NULL,
    [ZipCode] [varchar](10) NULL,
    [Phone] [varchar](12) NULL
)

GO

INSERT INTO TestDB.dbo.Customer
(CustomerId,LastName,FirstName,City,State,ZipCode,Phone)
VALUES
(7923,'Blow','Joe','Chicago','IL','12345-9876','555-555-5555'),
(7924,'Antoinette','Marie','Seattle','WA','84356-8456','777-777-7777'),
(7925,'Doe','Janet','Houston','TX','99354-9445','333-444-555'),
(7926,'Alverez','Desmond','Des Moines','IA','79684-8473','222-222-2222'),
(7927,'Contrary','Mary','Boston','MA','17545-4564','111-111-1111')

Next, create a subdirectory named C:\DestinationSubdirectories\, and create 10 subdirectories under it named Dest01, Dest02, Dest03,...Dest10.

To create a Data Flow Task, run the import/export wizard from an SSIS project in Business Intelligence Development Studio, right-click on the SSIS Packages node and select SSIS Import and Export Wizard.

Choose the default SQL Server Native Client data source and the server and database where you created the Customer file, then click the Next button

.

Choose Flat File Destination and pick the first folder from the Destination drop-down list, then enter the DataFile.txt as the file name and click Next.

Choose the Write a query radio button and click Next.

Enter the query SELECT * FROM Customer and click Next.

Click the Edit Mappings button of the Configure Flat File Destination window.

Note that the Create destination file is the only available radio button. Click OK.

After returning to the previous Configure Flat File Destination screen, click the Preview button.

Note that the data from the SELECT query is the same data entered by the create script. Click the OK button.

After returning to the previous Configure Flat File Destination window once again, click the Next button.

From the Complete the Wizard window click the Finish button.

Click the Close button of the final Import and Export Wizard window.

At this point we have successfully created a Data Flow task that runs a query against the Customer table and writes a file named DataFile.txt containing the results of that query into C:\DestinationsSubdirectory\ .

Execute the package and confirm that C:\DestinationSubdirectories\Dest01\DataFile.txt has been created by the package.

Open the C:\DestinationSubdirectories\Dest01\DataFile.txt file to confirm its contents.

Add a Script Task to the package by dragging-and-dropping it onto the main panel.

Add a precedence constraint from the Data Flow Task to the Script Task by holding the CTRL key while clicking first the Data Flow Task, then the Script Task, then right-click on the Data Flow Task and select Add Precedence Constraint.

Double click on the Script Task to bring up the Script Task Editor.

Select Microsoft Visual Basic 2008 for the Script Language, then click the Edit Script button.

The Script Task contains the text Add your code here.

Replace the highlighted text with the following VB.NET code.

Dim FileName As String = "DataFile.txt"
Dim SourceSubdirectory As String = "C:\DestinationSubdirectories\Dest01\"

Dim DestinationSubirectories As New ArrayList

DestinationSubirectories.Add("C:\DestinationSubdirectories\Dest02\")
DestinationSubirectories.Add("C:\DestinationSubdirectories\Dest03\")
DestinationSubirectories.Add("C:\DestinationSubdirectories\Dest04\")
DestinationSubirectories.Add("C:\DestinationSubdirectories\Dest05\")
DestinationSubirectories.Add("C:\DestinationSubdirectories\Dest06\")
DestinationSubirectories.Add("C:\DestinationSubdirectories\Dest07\")
DestinationSubirectories.Add("C:\DestinationSubdirectories\Dest08\")
DestinationSubirectories.Add("C:\DestinationSubdirectories\Dest09\")
DestinationSubirectories.Add("C:\DestinationSubdirectories\Dest10\")

Dim SeedFile As New System.IO.FileInfo(SourceSubdirectory & FileName)

If SeedFile.Exists = False Then

    Dim button0 As DialogResult = MessageBox.Show("Seed file '" & SourceSubdirectory & FileName & "' does not exist. " & Chr(13) & Chr(10) & Chr(13) & Chr(10) & " Package exiting.", "DATA FLOW TASK FAILED", MessageBoxButtons.OK)
    Dts.TaskResult = ScriptResults.Failure

Else

    Dim message1 As String = SourceSubdirectory & Chr(13) & Chr(10)
    Dim message2 As String = ""
    Dim SuccessCount As Integer = 1
    Dim FailureCount As Integer = 0
    Dim DestSub As String = ""

    For Each DestSub In DestinationSubirectories

Try
    Dim PreviousDestinationFile As New System.IO.FileInfo(DestSub & FileName)
    PreviousDestinationFile.Delete()
Catch ex As Exception

End Try

Try
    SeedFile.CopyTo(DestSub & FileName)
    message1 += DestSub & Chr(13) & Chr(10)
    SuccessCount += 1
Catch ex As Exception
    message2 += DestSub & Chr(13) & Chr(10)
    FailureCount += 1
End Try

    Next

    If FailureCount = 0 Then
        Dim button1 As DialogResult = MessageBox.Show("'" & FileName & "' was written to the following " & SuccessCount & " subdirectories: " & Chr(13) & Chr(10) & Chr(13) & Chr(10) & message1, "Success Confirmation", MessageBoxButtons.OK)
    End If

    If FailureCount <> 0 Then
        Dim button2 As DialogResult = MessageBox.Show("'" & FileName & "' was written to the following " & SuccessCount & " subdirectories: " & Chr(13) & Chr(10) & Chr(13) & Chr(10) & message1 & Chr(13) & Chr(10) & "'" & FileName & "' was NOT written to the following " & FailureCount & " subdirectories: " & Chr(13) & Chr(10) & Chr(13) & Chr(10) & message2, "FAILURE ALERT", MessageBoxButtons.OK)
    End If

    Dim Writer As System.IO.StreamWriter
    Writer = IO.File.CreateText("C:\DataFlowTaskLogFile" & "-" & Now.ToString.Replace("/", "").Replace(":", "").Replace(" ", "") & ".txt")
    Writer.WriteLine("'" & FileName & "' was written to the following " & SuccessCount & " subdirectories: " & Chr(13) & Chr(10) & Chr(13) & Chr(10) & message1)
    If message2 <> "" Then
        Writer.WriteLine("'" & FileName & "' was NOT written to the following " & FailureCount & " subdirectories: " & Chr(13) & Chr(10) & Chr(13) & Chr(10) & message2)
    End If
    Writer.Close()

    Dts.TaskResult = ScriptResults.Success

End If

The first section of code defines values for the FileName and SourceSubdirectory variables, which define the location of the seed file generated by the Data Flow Task.

     Dim FileName As String = "DataFile.txt"
     Dim SourceSubdirectory As String = "C:\DestinationSubdirectories\Dest01\"

The next section creates an array list named DestinationSubdirectories. This is all the subdirectories to which the seed file will be copied.

     Dim DestinationSubirectories As New ArrayList

     DestinationSubirectories.Add("C:\DestinationSubdirectories\Dest02\")
     DestinationSubirectories.Add("C:\DestinationSubdirectories\Dest03\")
     DestinationSubirectories.Add("C:\DestinationSubdirectories\Dest04\")
     DestinationSubirectories.Add("C:\DestinationSubdirectories\Dest05\")
     DestinationSubirectories.Add("C:\DestinationSubdirectories\Dest06\")
     DestinationSubirectories.Add("C:\DestinationSubdirectories\Dest07\")
     DestinationSubirectories.Add("C:\DestinationSubdirectories\Dest08\")
     DestinationSubirectories.Add("C:\DestinationSubdirectories\Dest09\")
     DestinationSubirectories.Add("C:\DestinationSubdirectories\Dest10\")

This code confirms that the seed file has been created.

     Dim SeedFile As New System.IO.FileInfo(SourceSubdirectory & FileName)

     If SeedFile.Exists = False Then

If the seed file does not exist, the package displays an error message and ends execution.


    Dim button0 As DialogResult = MessageBox.Show("Seed file '" & SourceSubdirectory & FileName & "' does not exist. " & Chr(13) & Chr(10) & Chr(13) & Chr(10) & " Package exiting.", "DATA FLOW TASK FAILED", MessageBoxButtons.OK)
    Dts.TaskResult = ScriptResults.Failure

If the seed file does exist, then the array list is looped through.

    Else
    For Each DestSub In DestinationSubirectories

The new seed file cannot be copied to any subdirectory where a previous version exists, so if a previous version of the file exists in the indexed subdirectory it is deleted.

    Try
        Dim PreviousDestinationFile As New System.IO.FileInfo(DestSub & FileName)
        PreviousDestinationFile.Delete()
    Catch ex As Exception
    End Try

The seed file is copied to the subdirectory. If it is successfully copied, the subdiretory name is added to the success message (message1) and the SuccessCount variable incremented by one. If it is not successfully copied, the subdirectory name is added to the failure message (message2) and the FailureCount varaible is incremented by one.

    Try
        SeedFile.CopyTo(DestSub & FileName)
        message1 += DestSub & Chr(13) & Chr(10)
        SuccessCount += 1
    Catch ex As Exception
        message2 += DestSub & Chr(13) & Chr(10)
        FailureCount += 1
    End Try

If, after the array list has looped through all the destination subdirectories, the FailureCount is still zero, a Success Confirmation popup is displayed.


    If FailureCount = 0 Then
        Dim button1 As DialogResult = MessageBox.Show("'" & FileName & "' was written to the following " & SuccessCount & " subdirectories: " & Chr(13) & Chr(10) & Chr(13) & Chr(10) & message1, "Success Confirmation", MessageBoxButtons.OK)
    End If

If, after the array list has looped through all the destination subdirectories, the FailureCount is not zero, a Failure Alert popup is displayed.


    If FailureCount <> 0 Then
        Dim button2 As DialogResult = MessageBox.Show("'" & FileName & "' was written to the following " & SuccessCount & " subdirectories: " & Chr(13) & Chr(10) & Chr(13) & Chr(10) & message1 & Chr(13) & Chr(10) & "'" & FileName & "' was NOT written to the following " & FailureCount & " subdirectories: " & Chr(13) & Chr(10) & Chr(13) & Chr(10) & message2, "FAILURE ALERT", MessageBoxButtons.OK)
    End If

The Success/Failure messages are also written to unique log files so that they are not overwritten.


    Dim Writer As System.IO.StreamWriter
    Writer = IO.File.CreateText("C:\DataFlowTaskLogFile" & "-" & Now.ToString.Replace("/", "").Replace(":", "").Replace(" ", "") & ".txt")
    Writer.WriteLine("'" & FileName & "' was written to the following " & SuccessCount & " subdirectories: " & Chr(13) & Chr(10) & Chr(13) & Chr(10) & message1)
    If message2 <> "" Then
        Writer.WriteLine("'" & FileName & "' was NOT written to the following " & FailureCount & " subdirectories: " & Chr(13) & Chr(10) & Chr(13) & Chr(10) & message2)
    End If
    Writer.Close()

This is what the package looks like upon successful completion.

You can confirm that the file has been copied to one or more of the destination subdirectories.

Conclusion


A Data Flow Task that writes a file to a single destination and copies that file to other destinations can be used in place of a Multicast Transformation.

Resources:

MultipleDestinationFileDataFlowTask.dtsx | CreateCustomers.sql
Total article views: 3987 | Views in the last 30 days: 10
 
Related Articles
FORUM

finding filepath, subdirectories, and filenames

in my local drive, i have a directory, d:\data\ with unlimited subdirectories liked the following: ...

FORUM

Send Excel SpreadSheet Filename to SSIS

Have SSIS take a variable filename in Excel.

FORUM

Use Database table to dynamically get the excel filename

Dynamic excel filename and path from database table

FORUM

SSIS - FTP TASK - DYNAMIC FILENAME

How to use the FTP Task with a dynamic localpath or filename

FORUM

How to load excel file with dynamic filename??

How to load excel file with dynamic filename??

 
Contribute

Join the most active online SQL Server Community

SQL knowledge, delivered daily, free:

Email address:  

You make SSC a better place

As a member of SQLServerCentral, you get free access to loads of fresh content: thousands of articles and SQL scripts, a library of free eBooks, a weekly database news roundup, a great Q & A platform… And it’s our huge, buzzing community of SQL Server Professionals that makes it such a success.

Join us!

Steve Jones
Editor, SQLServerCentral.com

Already a member? Jump in:

Email address:   Password:   Remember me: Forgotten your password?
Steve Jones