I have to import files into adatabase from the data provider through FTP. Each day the vendor provides 3,000 files which I need to import. The idea is the package will run every 30 minutes and I should grab files greater than last processed id during each run
20100217 is the folder which contains upto 3000 files
ex 1.txt, 2.txt, 3.txt, 4.txt
if in my previous run i have processed files 1,2,3 I should copy files only from 4.txt.
I see that the FTP Task has the option to define remote path as variable but how can i achieve the >lastprocessed option
may be i can save the ftp variable as
/20100217/*.txt but it will fetch all the files and each time i do not want to get 3000 files and then discard already processed ones. Is there an efficient way of doing this.
If FTP task has limitations then probably i will have to write a batch file that might do the same and I have to explore the possibilities
Your help in this regard is appreciated.