SQL Clone
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


Cozyroc Parallel Loop Task and the Foreach Loop File Enumerator


Cozyroc Parallel Loop Task and the Foreach Loop File Enumerator

Author
Message
Paul Hernández
Paul Hernández
Old Hand
Old Hand (390 reputation)Old Hand (390 reputation)Old Hand (390 reputation)Old Hand (390 reputation)Old Hand (390 reputation)Old Hand (390 reputation)Old Hand (390 reputation)Old Hand (390 reputation)

Group: General Forum Members
Points: 390 Visits: 661
Hi friends,

I'm trying to improve the the performance of a foreach loop task, that reads around 20.000 pdf files each day and registers them in a log table.
I found the Cozyroc Parallel Task but I'm not able to make it work.

I found a couple of Sample Packages, but one uses an Item Enumerator and the other a ADO Net Enumerator. Both declare a variable called "Iter" twice, one with a package scope and the other with a foreach loop scope. I soppose the name of the variable is not relevant.

This Iter variable is map as the foreach loop iterator. In my case, the file enumerator return to the enumeration variable the file name, in a particular format depending on the selected values (Fully qualify, name and extension, etc.).

It seems like the foreach and parallel loop need a "clean variable" to iterate, and I'm wondering if it´s possible to use these two components to improve the file iteration.

Any comment would be appreciated.

Paul Hernández
Phil Parkin
Phil Parkin
SSCoach
SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)

Group: General Forum Members
Points: 18784 Visits: 20458
Please explain what you mean by "registers them in a log table" - are you actually processing the files or just recording their names?


Help us to help you. For better, quicker and more-focused answers to your questions, consider following the advice in this link.

If the answer to your question can be found with a brief Google search, please perform the search yourself, rather than expecting one of the SSC members to do it for you.

Please surround any code or links you post with the appropriate IFCode formatting tags. It helps readability a lot.
Paul Hernández
Paul Hernández
Old Hand
Old Hand (390 reputation)Old Hand (390 reputation)Old Hand (390 reputation)Old Hand (390 reputation)Old Hand (390 reputation)Old Hand (390 reputation)Old Hand (390 reputation)Old Hand (390 reputation)

Group: General Forum Members
Points: 390 Visits: 661
Hi Phil,

Thanks for your quick response.

I need only to log the filename, path and modify date in a table.

Background: I receive digital card request in pdf format. The name of the file is a barcode containing a prefix, the card number and a checksum. I need to extract the card number, look up the contact_Id in a Microsoft Dynamics CRM. If the contact_Id is found I move the file to a specific directory, if move to another. Finally I update a custom CRM entity with the file name, path, modify date and contact_Id (only for the files that matched a contact).

I made a design that works, but I would be nice if I can improve the process.

Kind Regards,

Paul Hernández
Phil Parkin
Phil Parkin
SSCoach
SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)SSCoach (18K reputation)

Group: General Forum Members
Points: 18784 Visits: 20458
Have you considered a pure (ish!) SQL solution?

Haven't got much time right now, but have a look here & see whether it might be useful.


Help us to help you. For better, quicker and more-focused answers to your questions, consider following the advice in this link.

If the answer to your question can be found with a brief Google search, please perform the search yourself, rather than expecting one of the SSC members to do it for you.

Please surround any code or links you post with the appropriate IFCode formatting tags. It helps readability a lot.
Paul Hernández
Paul Hernández
Old Hand
Old Hand (390 reputation)Old Hand (390 reputation)Old Hand (390 reputation)Old Hand (390 reputation)Old Hand (390 reputation)Old Hand (390 reputation)Old Hand (390 reputation)Old Hand (390 reputation)

Group: General Forum Members
Points: 390 Visits: 661
Thanks so much again Phil,

I followed your suggestion and that makes the difference. I reduced the execution time from more than one hour using the foreach file enumerator, to just a few seconds, around 15 seconds. I just needed to adjust the code in your link to meet my requirements, because I need not only the file name but also the modified date, but at the end it works like a charm.

Kind Regards

Paul Hernández
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search