Thanks for your replies everybody. I finally got to the bottom of the problem and it can only be described as extremely annoying.
It turned out the package was technically running fine. It only appeared
to be running the same file over and over . As I was watching it it kept showing the same number of rows in each file but not the file name. There was no earthly reason to expect that the files would have precisely the same number of rows in each one so I thought that it was running the same one over and over. The actual fault occurred when the package hit a file with the column headers in a different order or that contained completely different data.
We've had to solve it by lining up volunteers in front of Excel and running a macro over the files to standardise them. I know we probably could have coded this but the high-ups decided that given the time pressure and relative quietness at the moment that was the quickest and easiest way!
On two occasions I have been asked, "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" ... I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.
—Charles Babbage, Passages from the Life of a Philosopher How to post a question to get the most help