I was looking through the news this week and found a piece by Michael J. Swart that I really enjoyed. It talks about different types of metrics that he has used when monitoring and alerting on his database servers. Michaal talks about the difference between alarms and gauges, which are usually base on the time scale in which a DBA or administrator needs to react to a situation. Alarms are created when something grows in a short term and causes an issue. The gauges tend to be longer term items that eventually cause a problem.

I've had a similar view in the past. However I look at these types of metrics from the standpoint of how I tackle my job as a DBA. I also try to think about these issues as a developer, ensuring that telemetry helps track issues that occur over time and acute issues cause a graceful degredation (or exit) and not an unexpected one.

When I've needed to manage systems, I ensure that I have alarms set up. I need these for the situations where something goes wrong in a short period of time. However I never want alarms to be based on a lack of visibility into the system over time. I want to try and eliminate as many alarms as possible with better long term monitoring that looks for indicators that the system might have issues. I've found that many (not all) of the alarms that I receive could have been prevented if I'd been paying close attention.

In other words, I'm a proactive DBA, looking at gauges (using MIchael's terms), not a reactive one counting on alarms.

I can't predict everything, but when I find that looking back at data is helpful. If I find a trend that could have alerted me, I'll evolve my monitoring system to try and track those potential issues in the future. I've written my own custom metrics in the past, which track data, either PerfMon type counters or business data, and then let me know when the growth, decline, or change is unexpected. In this way, I tend to find I can often proactively prevent issues. If I can use my data to "predict" an issue my boss doesn't believe in, I've also found my budget for resources may increase a bit, allowing me to manage systems even better in the future.