michael vessey (5/14/2012)
Lynn Pettis (5/14/2012)
Going with the "it depends" answer. How many files are being processed, how often are they processed, how big is each file that is being processed, how long does it take to process each file (average may be good here), what else is occurring on the system, what is the system configuration (processors, memory, disk (SAN or DASD)).As you can see there are a lot of variables. You may just need to play with the values until you find something that statisfies your SLA for processing files as well as the other requirements of the system.
ok - all good questions
number of files - thousands, each constains potentially 10k+rows
average time to process - 2 seconds - max time to process 40 seconds
the data from the files is pumped into tables that are used for reporting and aggregation, but it's really simple stuff, no joins
the hardware is Amazon 64 bit with 64 GB ram - unfortunatly we have no control over the san disk config as it's hidden from you on amazon - but it's very slooooow. CPU most likely 4 cores
i have no SLA and there are no other requirements (most of the work is being done by other systems, this is just for aggregation)
rather than anyone just telling me a figure, really i'm looking for some info/past expreience/best practice/guidlines so i can work it out myself without having to keep incrementing by 1 and looping through the test 😀 - once i get in the ballpark i can fine tune it later (plus also i would like to be able to explain why i chose that figure)
thanks
I missed a question, how many files (average would be good) are received over a set time period (again, you will have to define this as it could be every minute, every 10 minutes, etc). This will have a impact on how many readers you will need. If you receive 10 files a minute and each file is processed in less than a minute you only need 10 readers. Also, are there spikes and lulls in receiving the files and are they significant in time span.