What you've done is a technique that I use all of the time. It's called "Divide'n'Conquer". To put it in simple terms, think of a plate juggler... which is more difficult to do and requires the most attention? Keeping 20 plates spinning on the end of sticks or just one. SQL Server works in proverbially the same way.

Keep the Temp Tables and relish the extreme gain in performance.