Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase

Cozyroc Parallel Loop Task and the Foreach Loop File Enumerator Expand / Collapse
Author
Message
Posted Friday, February 8, 2013 6:15 AM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Wednesday, November 12, 2014 12:07 PM
Points: 120, Visits: 489
Hi friends,

I'm trying to improve the the performance of a foreach loop task, that reads around 20.000 pdf files each day and registers them in a log table.
I found the Cozyroc Parallel Task but I'm not able to make it work.

I found a couple of Sample Packages, but one uses an Item Enumerator and the other a ADO Net Enumerator. Both declare a variable called "Iter" twice, one with a package scope and the other with a foreach loop scope. I soppose the name of the variable is not relevant.

This Iter variable is map as the foreach loop iterator. In my case, the file enumerator return to the enumeration variable the file name, in a particular format depending on the selected values (Fully qualify, name and extension, etc.).

It seems like the foreach and parallel loop need a "clean variable" to iterate, and I'm wondering if it´s possible to use these two components to improve the file iteration.

Any comment would be appreciated.


Paul Hernández
http://hernandezpaul.wordpress.com/
https://twitter.com/paul_eng
Post #1417656
Posted Friday, February 8, 2013 6:19 AM


SSCertifiable

SSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiable

Group: General Forum Members
Last Login: Today @ 4:18 AM
Points: 5,245, Visits: 12,161
Please explain what you mean by "registers them in a log table" - are you actually processing the files or just recording their names?


Help us to help you. For better, quicker and more-focused answers to your questions, consider following the advice in this link.

When you ask a question (and please do ask a question: "My T-SQL does not work" just doesn't cut it), please provide enough information for us to understand its context.
Post #1417660
Posted Friday, February 8, 2013 6:27 AM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Wednesday, November 12, 2014 12:07 PM
Points: 120, Visits: 489
Hi Phil,

Thanks for your quick response.

I need only to log the filename, path and modify date in a table.

Background: I receive digital card request in pdf format. The name of the file is a barcode containing a prefix, the card number and a checksum. I need to extract the card number, look up the contact_Id in a Microsoft Dynamics CRM. If the contact_Id is found I move the file to a specific directory, if move to another. Finally I update a custom CRM entity with the file name, path, modify date and contact_Id (only for the files that matched a contact).

I made a design that works, but I would be nice if I can improve the process.

Kind Regards,


Paul Hernández
http://hernandezpaul.wordpress.com/
https://twitter.com/paul_eng
Post #1417664
Posted Friday, February 8, 2013 6:43 AM


SSCertifiable

SSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiable

Group: General Forum Members
Last Login: Today @ 4:18 AM
Points: 5,245, Visits: 12,161
Have you considered a pure (ish!) SQL solution?

Haven't got much time right now, but have a look here & see whether it might be useful.



Help us to help you. For better, quicker and more-focused answers to your questions, consider following the advice in this link.

When you ask a question (and please do ask a question: "My T-SQL does not work" just doesn't cut it), please provide enough information for us to understand its context.
Post #1417670
Posted Wednesday, February 13, 2013 6:52 AM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Wednesday, November 12, 2014 12:07 PM
Points: 120, Visits: 489
Thanks so much again Phil,

I followed your suggestion and that makes the difference. I reduced the execution time from more than one hour using the foreach file enumerator, to just a few seconds, around 15 seconds. I just needed to adjust the code in your link to meet my requirements, because I need not only the file name but also the modified date, but at the end it works like a charm.

Kind Regards


Paul Hernández
http://hernandezpaul.wordpress.com/
https://twitter.com/paul_eng
Post #1419483
« Prev Topic | Next Topic »

Add to briefcase

Permissions Expand / Collapse