Most teams have monitoring and still find out about outages from customers. The gap is not tooling — it is that the alerts fire on things that do not matter and stay silent on the things that do.
Alert on symptoms, not causes
CPU at 90% may be perfectly healthy. Checkout success rate dropping from 99% to 91% is never healthy. Alert on what users experience — error rate, latency, and the completion rate of your critical flows — and use resource metrics for diagnosis after the alert, not as the trigger.
Nobody has ever been paged usefully by a CPU graph. Page on what the customer feels.
Every alert needs an action
If the response to an alert is to look at it and do nothing, delete it. Alert fatigue is caused by exactly these, and it is what makes a team miss the real one. Each alert should have a runbook naming what is wrong, who owns it and the first three things to check.
Trace across service boundaries
In any system with more than one service, the question is which part of the request was slow. Propagate a trace ID from the entry point through every downstream call and into the logs. Without it, debugging a distributed latency problem is guesswork across several log files with unsynchronised clocks.
Log for the question, not for completeness
Logging everything produces an expensive haystack. Log structured events with the fields you will filter by — user, tenant, request, operation, outcome — and log decisions rather than narration. 'Selected provider B because provider A returned 503' is worth a hundred lines of 'entering function'.
Percentiles, never averages
Average latency hides the experience of the users having the worst time. Track p50, p95 and p99, and when they diverge, understand why: a slow tail usually indicates a cold cache, an unindexed query on a large tenant, or a dependency degrading under load.
Know what normal looks like
You cannot recognise an anomaly without a baseline. Spend an hour looking at your dashboards when nothing is wrong, so the shape of a healthy Tuesday is familiar. That hour is what makes an incident diagnosable in minutes rather than an argument about whether the graph is unusual.
