Can't upload all files from the S3 bucket

  • Hi all,

    I have a package that uploaded files from S3 bucket.

    Currently I need to add two another files. All previous files started with the phrase FA. The files that I need to add started from the phrase FARAN. And contain the name of the country in the format "Country_" or "_Counrty".

    The S3 bucket has folders with the number of each month and the number of each day. All files are stored separately. For example:

    2021/05/01 - Data of the 1st of May 2021

    2021/05/02 - Data of the 2nd of May 2021

     

    I changed C# code for this:

     

     


    public void Main()
    {
    string accessKey = Convert.ToString(Dts.Variables["User::AWS_Access_Key"].Value);
    string secretKey = Convert.ToString(Dts.Variables["User::AWS_Secret_Key"].Value);
    string startKey = Convert.ToString(Dts.Variables["User::LastFileName"].Value);
    string bucketName = Convert.ToString(Dts.Variables["User::S3_Bucket"].Value);
    string countryCode = Convert.ToString(Dts.Variables["User::Country"].Value);
    string sameFolder = startKey.Substring(0, startKey.LastIndexOf('/') + 1);
    var sameFolderFiles = Convert.ToString(Dts.Variables["User::SameFolderFiles"].Value).Split(',');

    try
    {
    AmazonS3Client s3Client = new AmazonS3Client(accessKey, secretKey, RegionEndpoint.EUWest1);
    System.Collections.ArrayList files = new System.Collections.ArrayList();
    ListObjectsRequest listRequest = new ListObjectsRequest
    {
    BucketName = bucketName,
    //Prefix = countryCode + "/"
    };

    ListObjectsResponse listResponse;
    do
    {
    // Get a list of objects
    listResponse = s3Client.ListObjects(listRequest);
    foreach (S3Object obj in listResponse.S3Objects)
    {
    if (
    (((startKey.CompareTo(obj.Key) < 0) && (!obj.Key.StartsWith(sameFolder))) || ((obj.Key.StartsWith(sameFolder)) && (!sameFolderFiles.Contains(obj.Key))))
    && (obj.Key.LastIndexOf('/') != obj.Key.Length - 1)
    && ((obj.Key.Contains("_" + countryCode)) || obj.Key.Contains(countryCode + "_"))
    && ((obj.Key.Contains("FA")) || obj.Key.Contains("FARAN"))

    )
    files.Add(obj.Key.ToString());
    }


    listRequest.Marker = listResponse.NextMarker;
    } while (listResponse.IsTruncated);

    Dts.Variables["User::Collection"].Value = files;
    Dts.TaskResult = (int)ScriptResults.Success;
    }
    catch (Exception ex)
    {
    MessageBox.Show(ex.ToString());
    Dts.TaskResult = (int)ScriptResults.Failure;
    }
    }
    }
    }?

    I have two variables: LastFileName, SameFolderFiles.

    This variables take data from the Audit using this code:

    DECLARE @FILEPROCESSKEYPREVIOUS INT = 0
    DECLARE @LASTFILENAME VARCHAR(500) = ''

    SELECT TOP 1
    @FILEPROCESSKEYPREVIOUS = ISNULL(File_Processing_Key,0),
    @LASTFILENAME = ISNULL([FileName], '')
    FROM Audit_File
    WHERE Data_Source = 'Bucket'
    AND Area = @Target_TableName
    AND COUNTRY = @COUNTRYCODE
    ORDER BY 1 DESC

    DECLARE @FILEPATH VARCHAR(500) = substring(@LASTFILENAME, 0, len(@LASTFILENAME) - charindex('/', reverse(@LASTFILENAME), 0) + 2)

    DECLARE @SAMEFOLDERFILES VARCHAR(1000) = ''

    SELECT @SAMEFOLDERFILES += ISNULL([FileName], '') + ','
    FROM Audit_File (NOLOCK)
    WHERE Data_Source = 'Bucket'
    AND Area = @Target_TableName
    AND COUNTRY = @COUNTRYCODE
    AND [FileName] LIKE @FILEPATH + '%'

    SELECT @FILEPROCESSKEYPREVIOUS, @LASTFILENAME, SUBSTRING(@SAMEFOLDERFILES,0,LEN(@SAMEFOLDERFILES))

    I still haven't uploaded these files.

    Could you help me with advice?

  • You've given us lengthy pieces of code and a description of what you are trying to do, which is good.

    But you have not described the problem in enough detail. What error messages are you seeing? Have you stepped through the code in debug mode? Which line of code is not working as intended?

    If you aren't happy single, you won't be happy in a relationship.

    Remember, happiness comes from guitars, not relationships.

  • Hi Phil,

    yes, I debugged the code and see that when the file with the name FARAN comes the package is down.

    Looks like this code doesn't work good.

  • christi1711 wrote:

    Hi Phil,

    yes, I debugged the code and see that when the file with the name FARAN comes the package is down. Looks like this code doesn't work good.

    Did you read my previous message? It contains two questions, of which you have answered precisely zero.

    If you aren't happy single, you won't be happy in a relationship.

    Remember, happiness comes from guitars, not relationships.

  • Hi Phil,

    1. I see only the message "Script was failed".

    2.Not for sure how correctly I can debug C# code. I put breakpoint in the main function in C# code, but after that Studio is closing..

  • Not sure whether I know how to debug C# 'correctly', but this is how I do it.

    1. Edit the code in the Script Task and put a breakpoint near the beginning of the code. In your case, I'd put the breakpoint on the line (string accessKey = Convert.ToString(Dts.Variables["User::AWS_Access_Key"].Value);). Position your cursor anywhere on the line and press F9 to enter the breakpoint.
    2. A red dot should appear in the left margin and the row of code should be permanently highlighted.
    3. Close the Script Task. After closing the Script Task, a red dot should be shown on the Script Task icon on the Control Flow.
    4. Run the package. When processing the Script Task, you should go into debug mode, with processing paused at the breakpoint created in (1). At this point, you have several options available in terms of how you step through the code. Of these, F10 is where I would start, this allows you to step through the code one line at a time. Press F10, and you should see execution advance to the next row.
    5. One of these presses of F10 will result in an error – now you know which line of code has caused the error and you can home in on that.
    6. Control should be passed to the 'Catch' section of the code. View the properties of the ex (exception) variable at that point to get more information about the error (particularly the 'innerException' property).

    Start with this and see how you get on.

    If you aren't happy single, you won't be happy in a relationship.

    Remember, happiness comes from guitars, not relationships.

Viewing 6 posts - 1 through 6 (of 6 total)

You must be logged in to reply to this topic. Login to reply