Of course, buying new hardware can get you some time. But how much time would you get if your app is full of crappy code? There's no piece of hardware that can handle massive nested-RBAR. Some time ago, I handled to improve a query that ran for over 30 hours and didn't finish to make it run in under 5 minutes. I assure you, more power wouldn't be a solution as the time to get it would have been more than the time it took to tune the code.

Yes, you might be going one step at a time, but you now that it will last forever. Tuning hardware is a fast and temporary solution that will demand attention in the near future. What will happen if management find out that the brand new server you got and solved the problem ended up with the same problem 6 months later? Is that cheaper?

It's not about pre-optimizing, is about doing things right from the beginning and correct anything that hasn't been done right. Changing configurations might help as well improving the performance on the whole system.