Great topic, Steve. I'm with you, I'd had to develop a monitoring system, because I now know the system being monitored would likely change. Who wants to constantly upgrading your monitoring system to accommodate changes in the system?
I think that knowing what's normal is good for people in other disciplines, like developers. It's good to know how a system should behave, especially before you go and change it.
But I also see a lot of not monitoring systems. One of my colleagues works in a small town several hundred miles south of the rest of us. Just a few minutes ago he texted me to inform me that the network he's on was down, again. This seems to happen to that location at least once a week! According to my colleague no one knows why it does this, but you can just about count on their being unavailable for a few hours at least, each week. it really frustrates my coworker, who loses so much productivity, when the network just goes down so often. And its appalling how the network people seem quite OK with this level of interrupted service. I guess this is some sort of "normal" to them, to which they've become. If "normal" means poor or unreliable service, then I think we need to come up with a new definition of normal.
Kindest Regards, Rod Connect with me on LinkedIn.