Without being able to see the query plan - can't say for certain, but I have in hte past seen similar types of degradation (or even worse) using derived tables where once the derived tables are linked together in one query it just causes the optimiser to barf and chose a plan that sucks.

Such circumstances are often a case for Cunning Use of Temp Tables - or a redesign as it usually only seems to occur when in the company of bad design. Temp table solution is quicker, less likely to cause greying/hair loss and easier to sell to management!

Try generating a temp table to hold the data - run one after the other into the temp table and see if you're back subsecond. If so - it's probably that scenario.