June 14, 2011 at 7:33 am
I want a requirement where i can get all the information about all the files in ftp server and get the only recently updated files based on recently modified date on that file and copy the entire ftp path to a text file.
Thanks in Advance.
Srikanth.
June 14, 2011 at 8:43 am
A third-party FTP tool component be best for this sort of thing. Here is a good one for .NET. The free version may do most of what you need. If not the paid version is still very affordable:
http://enterprisedt.com/products.html
On a design note...it would be cleaner if you split the responsibilities of getting the files and processing the files into two separate pieces. If you want to do the FTP work in an SSIS script task it's certainly possible, however a standard C# console app would be simpler to implement and debug. You can download Visual Studio C# Express Edition for free and create yourself a console app using the edtFTP component above.
There are no special teachers of virtue, because virtue is taught by the whole community.
--Plato
June 15, 2011 at 7:55 am
Ditto to opc.three's comment. But just to clarify, you can do these two pieces in the same SSIS package (I'm sure that's what he meant).
SSIS isn't meant to do "everything in one task". It's meant to put all tasks in one package to make things easier to manage.
June 15, 2011 at 8:59 am
Hi Brandie, I get what you're saying about keeping everything together for manageability and I agree, it's a good practice to keep related pieces of logic that make up what's viewed as a singular unit of work in the same SSIS package. It depends on how you like to manage things as to where you draw that line. In this case I did however mean to say that I would opt for either 2 separate SSIS packages or a C# console app and one SSIS package. I make a distinction between the retrieval of files and the processing of files that I don't make when it comes to retrieving data from a database. I try to split file retrieval and file processing responsibilities so they can be modified and tested independently.
My option 1 would have:
> C# console app that retrieved files from FTP site per business rules and parked files "on disk somewhere" for later use
> SSIS package that was looking at the location "on disk somewhere"
My option 2 would have:
> SSIS package 1 that retrieved files from FTP site per business rules via Script Task and parked files "on disk somewhere" for later use
> SSIS package 2 that was looking at the location "on disk somewhere"
That's just me. That distinction may not work out for everyone. I only do this because I have been bitten by the "we must implement SFTP with business partner X for compliance reasons" or the "our client is switching from a Windows FTP server to a Linux FTP server so we need to retest all our text file transfers" bugs before. In those cases I have had to work with processes where it would have been easier not to have to modify the piece that contained transformation logic in it because it opens up a whole Can of Worry regarding regression testing.
There are no special teachers of virtue, because virtue is taught by the whole community.
--Plato
June 15, 2011 at 9:26 am
Whereas in my office, we have so many feeds coming in from different places (or are grabbing feeds from different sources) that we just include the grab as part of the bigger business logic.
For instance, I do the grab in one task, then have 3 or 4 containers worth of processing steps. But usually, my grab is going for a specific file.
There is too the option of writing a separate application (or using a company FTP server) to grab files. One of my co-workers wrote up one that, based on data in a db table, checks the date, sees if there are any files that need to be grabbed on that date, then pulls back the file to our server. The process can grab from multiple sources and put each file in a different location (so the files don't have to be moved later).
So, I agree with the "it depends" logic of your statement. Matter of preference, I suppose, and which method creates less double work. I have found, however, that different sources require different connection methods, hence the sheer amount of options we have at my workplace.
Viewing 5 posts - 1 through 5 (of 5 total)
You must be logged in to reply to this topic. Login to reply