thanks for the reply lynn

i have to assume that the files will be constant and a 24*7 basis, and that the number of files will be more than a single machine can handle (worst case scenario there is a potential that we would receive in excess of 150 million files in the space of 24 hours)

it is acceptable for us to have files pending and a large backlog - we are under no pressure to perform to any SLA.

the thing with this is that the data is not required immediatley - from receiving the first file we may have to wait 24 hours until the last file is actually sent to us (i can't discuss the actual product, but imagine it was a large retail chain and tranactional data and status data are sent to a head office - plus the head office has the ability to send out questions to the till systems for a status update - if it's a sunday and that store is closed then we wait until it comes online)

the solution we have will scale sideways rather than demanding more powerfull hardware (as we are limited by amazon IO) so the real task is optimising the solution so that we can run on as few machines as possible when we go live.

really i'm not looking for a solution, more of a best practice guide (ie a microsoft web page or a white paper) so that we can say - "this is the reason we chose a figure of 50"

my main line of thinking is how many spid threads can each CPU handle and how do i manage the locking of the inserts when more than 1 file goes into a single table