"But its not web scale". Well neither is your market, or your share of the market. Your market is mature rather than emerging so that isn't going to change any time soon. When it does change it is likely to be in decline.
Years ago I joked in response to someone extolling the power of the latest and greatest hardware. "I'm sure normal development practises will bleed off any excess performance". Many a true word said in jest.
In the data world you hear people talking about the "massive amount of data we have to process". Spinning up Spark clusters to process what they consider to be a vast amount of data. Or, what a modern DBA would call a sod allth of data. Don't misunderstand me, Apache Spark is an impressive piece of engineering and once you start working in the high TB range it is a very useful tool. When you are in the sub-TB range it is likely to be overkill.
There seems to be received wisdom that what an organisation needs in its IT infrastructure is micro-services, NOSQL Dbs, Elastic Search, Kubernetes, complex orchestration tools and God knows what else. All that comes at a cost and I think parts of the moderns stack exist to demand that other parts of the modern stack also exist.
Does it work? Well at a cost, yes, but is it necessary for what you actually have to do and at the scale you operate?
When I look at tools I ask
- What does this do that I couldn't do before?
- For the stuff I already do, does this make it easier?
- What is my return on investment for adopting this?
There are tools for which those questions get strong positive answers, but there is complexity and necessary complexity and frankly the former is in the majority and is our fault.