11/13/2006

I have discovered the problem with my text file generation degradation.

It all comes back to the file system.  As files are being generated and more and more files are generated into one folder, the subsequent file generates get slower.  By moving to a new folder, the file generates pick right up.

In my final test scenario I did the following:

1. Using sp_OACreate, sp_OAMethod, and sp_OADestroy, I generated 75000 text files into test folder 1

2. Using sp_OACreate, sp_OAMethod, and sp_OADestroy, I generated 75000 text files into test folder 2, sub folder K, where K was incremented every 10000 files.

3. Using sp_OACreate, sp_OAMethod, and sp_OADestroy, I generated 75000 text files into test folder 3

Step 1 ran in 25 minutes with the file generation degrading from 16800/min down to 600/min.

Step 2 ran in 5 minutes with no noticeable degradation, averaging about 17000/min.

Step 3 ran in 24 minutes with the file generation degrading from 18000/min down to 350/min.

So I will include multiple subfolders in my file generation process.

11/15/2005

I modified my process to include subfolders and ran a partial test.  446,000 documents were generated in 68 minutes, giving an average rate of over 6500 documents generated per minute.