I have actually found if you split the tables into multiple files it will decrease the time it takes to have a table grow. I also have noticed a flaw when it comes to table growth, if you hit the wall on a table and it needs to become larger it will lock the file group into a single cpu which will slow everyone down not just people using this table when using only a single file group. If you use different file groups it seems not lock to a single cpu unless you are accessing tables inside that file group.

I would also suggest moving your temp tables into its own file group as they are the ones growing and shrinking a lot.