Start with a defined service-level indicator
An HTTP error ratio is an example of a service-level indicator (SLI), not automatically an availability objective. Define which user requests count, which HTTP statuses represent failures, and what period is relevant. Some 4xx responses reflect expected user behavior; not every failed background health check means the service failed for users.
A typical Prometheus counter named http_requests_total may carry labels such as service, route and status. The metric name and label schema must match the application instrumentation. A query copied without adapting its labels can return no data or a misleading aggregate.
Calculate a request-weighted error ratio
For an application that records server errors as status labels, a simplified five-minute request error ratio can be expressed as:
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))The numerator counts per-second server-error requests and the denominator all requests, both aggregated over the same population. This is a ratio: multiply by 100 to express a percentage. When the denominator is zero, there were no qualifying requests in the range; that is not proof of 100% availability. Build low-traffic handling into the dashboard and alerting rule.
Prefer rate() on counters before summing across instances. Separate routes or customer segments where service objectives genuinely differ. Label values with very high cardinality, such as raw URL IDs or request IDs, can cause excessive metrics storage and query cost.
Translate targets into an error budget
A 99.9% success target permits a 0.1% failure fraction under the defined indicator, traffic population and time window. For a time-based availability SLI, 99.9% over thirty days corresponds to about 43.2 minutes of downtime; a request-based success ratio instead budgets unsuccessful requests, not minutes.
Use the site's SLA / Error Budget Calculator for time-based arithmetic, but avoid confusing that result with a Prometheus request-success SLO.
A burn-rate alert measures how quickly the current error ratio consumes the allowed budget. Set evaluation windows and thresholds based on service criticality and real traffic volume. Short windows catch sharp failures; long windows help distinguish sustained regressions from noise.
Investigate before changing alert thresholds
Check instrumentation first: do counters reset correctly after restarts, do route labels represent the same logical service, and does the monitoring stack have scrape gaps? Compare error ratio changes with deployments, latency, saturation and downstream dependency health. Add relevant logs and traces when available.
When an alert fires, record a UTC timeline and the exact PromQL expression used. A dashboard screenshot without query and time bounds is insufficient evidence for a repeatable incident review.