SRE
SLO burn-rate alerting
Rebuilt paging around error budgets so on-call wakes up for customer pain, not CPU graphs.
- Role
- SRE, on-call lead
- When
- 2023 · 3 months
- Industry
- Fintech
The problem
On-call took about 60 pages a week, mostly threshold alerts on CPU, memory and queue depth. People had started muting the pager, and two real outages were first reported by customers.
Why it matters
Alert fatigue is how real incidents go unanswered. Tying pages to SLOs means every page maps to users hurting right now, which is the only thing worth waking someone up for.
How it works
Each service declares SLOs in a short YAML spec next to its code. Sloth turns that spec into Prometheus recording rules and multi-window, multi-burn-rate alerts: a fast burn (2% of the monthly budget in an hour) pages, and a slow burn (5% in six hours) opens a ticket.
Grafana dashboards are generated from the same spec, so the graph an engineer opens during an incident shows the exact numbers the alert fired on.
Decisions worth explaining
- Availability and latency only, to start. Two SLOs per service got teams through the door. Freshness and correctness SLOs came later, where they mattered.
- Old alerts were archived, not deleted. They moved to a non-paging route for 30 days. Nothing in that route fired before a real incident, which settled the argument.
What I’d do next
Feed remaining error budget into the deploy pipeline so risky releases pause automatically when a service is close to its limit.