Observability for production SaaS: the four signals that catch outages before users do
Dashboards green, synthetic check passing — and three customers email before you realise one region is failing. The four signals that surface outages first.
It is 10:46 AM on a Tuesday. Your Slack is quiet. The status page is green. Your synthetic uptime check fired successfully four minutes ago, as it does every five minutes, from a data-center in Virginia. Datadog — or whatever you're paying for this month — shows zero red boxes on the service map.
Then a Slack message arrives from a customer who is also friends with your founder on LinkedIn: "Hey, is the site down? Can't load anything." The founder forwards it with a single question mark.
You refresh your dashboards. Still green. You curl your own API from your laptop. 200, 180 ms, perfect. You check the CDN logs — except you can't, because you set up the CDN two weeks ago and you never actually wired its access logs into your pipeline. You find the CDN vendor's portal, log in, navigate to analytics, and discover that one of the edge regions — São Paulo, PoP SA-1 — has been returning 523 Connection Timed Out for the last forty-one minutes. Not 500. A 523. Your synthetic check runs from Virginia. Your error-rate metric counts HTTP 5xx responses that reach your origin. A timeout at the edge before the request ever hits your origin produces a status code your metric knows nothing about and a region your synthetic probe never visits.
While you are reading CDN documentation to understand what a 523 actually means (it means the edge node could not reach the origin — which, in this case, is because a cache rule update two weeks ago removed the exception for your WebSocket upgrade path, and the new rule was silently blocking the handshake), two more customers email support. A third posts on the community forum. The thread title is "Service broken?"
The fix takes eleven minutes once you find the rule. The incident ran for fifty-three minutes before you knew about it. Three customers experienced it. All three had already started evaluating alternatives before the banner went up on your status page.
This story is not exotic. Every engineer who has run a production SaaS long enough has a version of it. The dashboards were green because the dashboards were measuring the wrong thing. The alerts did not fire because the alerts were wired to signals that did not exist at the right point in the request path. The synthetic check was running, but it was checking the wrong place from the wrong geography using the wrong protocol.
Observability is the discipline of instrumenting a system so that you can ask arbitrary questions about its internal state from the outside — questions you did not know you would need to ask when you built the instrumentation. That is a meaningfully different goal from dashboards full of green lights. A dashboard full of green lights tells you that the conditions you thought to measure are currently fine. Observability tells you what is happening to real requests from real users, including in the conditions you forgot to think about.
This post is a practitioner guide. It assumes you have a running SaaS, you are past "tail the logs on the server," you probably have at least one existing observability tool, and you want to understand what you are actually building and why — not just which product to subscribe to.
What's actually happening
Before describing what to build, it is worth being precise about the four signals most observability frameworks talk about, because they are frequently conflated in a way that produces systems that look comprehensive but have large blind spots.
Logs
A log is a timestamped, structured or unstructured record of a discrete event. "User 4821 authenticated successfully at 2024-03-15T10:46:02Z" is a log entry. Logs are the oldest form of operational data and the most universally present.
Logs are excellent for answering "what happened in this specific context?" — especially after the fact, when you already know something went wrong and you need to reconstruct a sequence of events. They are poor at answering "what is the system doing right now, in aggregate?" because that requires scanning, parsing, and aggregating potentially enormous volumes of text. They are also poor at correlating events across service boundaries, because that correlation must be woven in explicitly via shared identifiers.
"We have logs" is not observability. It is the raw material from which observability can be constructed, but only if the logs are structured, correlated, retained at an appropriate horizon, and queryable at the cardinality you need. An unstructured log file on a server you SSH into is strictly better than nothing and strictly worse than instrumentation.
The trap is retention cost. Logs can be high-volume. A service handling 1,000 requests per second generating one log line per request produces 86 million lines per day. At typical log storage costs, retaining 30 days of uncompressed text is expensive enough that teams either stop logging important events to control cost or retain logs so briefly that they cannot investigate incidents that manifested slowly.
Metrics
A metric is a numeric measurement of system state over time. A metric has a name, a value, a timestamp, and a set of labels (also called dimensions or tags). "http_requests_total = 482931" is a metric.
Metrics are excellent at answering "how is the system behaving in aggregate over time?" They are efficient to store because they are numeric and highly compressible. They support alert conditions ("fire if this value crosses a threshold") naturally. They are the right tool for trending and capacity planning.
Metrics are poor at answering "why is this specific request slow?" The aggregation that makes metrics storage-efficient also destroys the per-request context. A p99 latency metric tells you that the 99th percentile of requests took longer than X milliseconds. It does not tell you which requests, from which users, traversing which code paths, waiting on which downstream calls.
This is the cardinality problem. A metric's labels are what allow you to slice it ("show me error rates by endpoint" or "show me latency by region"). But every unique combination of label values is a separate time series. If you add a "user_id" label to an HTTP metric on a service with 100,000 users, you have created 100,000 time series for that metric. Prometheus and similar systems store one time series per label combination in memory. Label cardinality that is too high crashes your metrics store. This means you structurally cannot answer "which users are experiencing high latency?" from a metrics system alone — you need a different signal for that.
Traces
A trace represents the full lifecycle of a single request as it flows through a distributed system. A trace is composed of spans. A span represents a unit of work: an HTTP handler, a database query, a downstream API call, a message queue publish. Each span records its start time, duration, service name, operation name, and an arbitrary set of attributes. Spans in the same trace share a trace ID. Parent-child relationships between spans are recorded, so you can reconstruct the call graph.
Traces are what close the gap between "the p99 is bad" and "here is the exact request that took 3.2 seconds, and it was slow because the PostgreSQL query on line 847 ran a sequential scan on a table with 14 million rows because the index was recently dropped during a migration."
The OpenTelemetry specification (https://opentelemetry.io/docs/specs/otel/) is the current standard API for producing traces (and metrics and logs) in a vendor-neutral way. The spec defines the data model — trace ID, span ID, parent span ID, attributes, events, status, kind — and the SDK contract for propagating context across process boundaries. Adopting OpenTelemetry means your instrumentation is not tied to a specific backend vendor. You can send the same telemetry to Jaeger, Tempo, Honeycomb, or Lightstep by changing a config.
Traces are the answer to unknown-unknowns. A metrics dashboard covers the dimensions you designed it to cover. A trace covers the actual execution path of an actual request, including the parts you did not know to measure. This distinction — known-unknowns (metrics tell you when a pre-specified condition is bad) vs unknown-unknowns (traces let you explore what actually happened) — is the central argument in Charity Majors and Liz Fong-Jones's work on observability. Their book, Observability Engineering (O'Reilly, 2022), is the most rigorous treatment of this distinction available.
Traces have a cost: storage and ingestion. A trace for a single request through a system with ten services and twenty spans per service is a meaningful amount of data at high request rates. This is why sampling exists. The tradeoff between sampling rate and coverage of the long tail is a real engineering decision, covered in the pattern section below.
Events
An event is a structured log entry that represents something significant that happened in the system at a specific point in time: a deployment, a feature flag change, a database failover, a cron job starting, a certificate expiring. Events are often overlaid on metric graphs ("this latency spike correlates with this deployment at 14:32") and are the signal you reach for to explain why a metric changed, as opposed to that it changed.
Events are underinvested in most observability stacks. Teams track deployments in CI/CD but do not push deployment events into the same time-series database as their metrics. When an incident is caused by a configuration change that was not a code deployment, the correlation takes manual detective work.
A minimal events integration pushes a Prometheus annotation (via the Alertmanager API or a webhook into Grafana) on every production deployment. A richer integration captures: deployment SHA and author, feature flag changes, database migration completions, scaling events (autoscaler added 3 nodes), and dependency version updates. The tooling overhead is low; the investigation value is high.
{
"time": "2024-03-15T14:32:07Z",
"type": "deployment",
"service": "order-service",
"version": "2.14.1",
"sha": "a3f8c91",
"author": "[email protected]",
"environment": "production",
"notes": "Added async order fulfillment queue"
}Push this JSON to your Grafana annotations API and it will appear as a vertical line on every dashboard panel, timestamped and labelled. When the p99 latency panel shows a spike at 14:33, the annotation at 14:32 is the first thing you look at.
How the four signals interact in practice
The signals are not redundant — they answer orthogonal questions and are most useful when all four are present and queryable in the same time range.
Consider a realistic incident scenario: your order service p99 latency starts climbing at 14:30. By 14:45 it reaches 2,800ms, triple the normal baseline.
What metrics tell you: the p99 latency burn-rate alert fires. You open the dashboard. The latency is elevated for POST /api/v1/orders specifically, not for other endpoints. The error rate on that endpoint is unchanged — requests are succeeding, just slowly. Database connection pool utilisation (a USE metric you instrumented) is at 98%.
What metrics do NOT tell you: whether all orders are slow, or just a subset. Whether the slowness is in your application code, in the database query, in a downstream payment gateway call, or in serialisation. Whether a specific customer segment is disproportionately affected.
What logs tell you: scanning the structured logs for endpoint="/api/v1/orders" with durationMs > 2000, you can see that the slow requests all have a common characteristic — they include an item from a specific product category that was added to the catalogue this morning. But correlating across the database logs to find the slow query requires the annotation you added to the SQL.
What traces tell you: opening any of the slow traces in Jaeger or Grafana Tempo, you see the waterfall immediately. The processOrder span takes 2.6 seconds. Within it, the validateInventory child span takes 2.4 seconds. Within that, the PostgreSQL query span takes 2.4 seconds and carries the query text: a SELECT with a JOIN to the new product category table that has no index on the foreign key column. The slow query is identified in under two minutes.
What events tell you: there is a Grafana annotation at 11:15 AM: "Database migration: add product_categories table, insert 45,000 rows." The missing index is the gap between adding the table and the migration that should have followed it.
The incident resolution time in this scenario is under 10 minutes from alert fire to root cause identified, because all four signals were present and queryable. Without traces, the investigation would have involved sequential log grep, SSH into the database server, and EXPLAIN ANALYZE on a query you first had to reconstruct from fragmented logs. That path typically takes 45–90 minutes.
RED and USE: two frameworks that tell different stories
Two frameworks from the SRE literature give you a principled set of metrics to instrument rather than asking "what should I measure?"
RED — Rate, Errors, Duration — applies to request-driven services. For each service and each endpoint:
- Rate: requests per second arriving at this endpoint
- Errors: proportion of requests returning an error (4xx that are client-caused, 5xx that are server-caused — decide which you count and be consistent)
- Duration: latency distribution (not average — use a histogram so you can compute p50, p90, p95, p99)
RED metrics answer the question "is this service serving its callers well?" They are the minimum viable instrumentation for any API endpoint. Tom Wilkie introduced the RED method while at Weave Works; the Grafana blog post (https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/) is the canonical reference.
USE — Utilisation, Saturation, Errors — applies to resources (CPU, memory, disk, network, connection pool slots, thread pool workers):
- Utilisation: what fraction of the resource's capacity is in use?
- Saturation: how much work is waiting because the resource is at capacity? (queue depth, wait time)
- Errors: how many operations on this resource are failing?
USE metrics answer the question "is any resource becoming a bottleneck?" Brendan Gregg introduced USE at Netflix; the methodology is documented at https://www.brendangregg.com/usemethod.html.
RED and USE are complementary. RED tells you the service is slow. USE tells you why — the CPU is saturated, the database connection pool is exhausted, the disk is at 100% utilisation. The Google SRE book (https://sre.google/sre-book/monitoring-distributed-systems/) makes the same decomposition under different labels: the four golden signals are latency, traffic, errors, and saturation — which maps closely to RED + the saturation half of USE.
Cardinality, sampling, and retention
These three dimensions determine whether your observability system is actually operable in production or just technically present.
Cardinality is the number of distinct time series in your metrics store, or the number of distinct field values in your log/trace store. High-cardinality fields — user ID, request ID, session ID, customer ID — are valuable for debugging but cannot be stored as metric labels. They belong in traces and in structured log fields, where you query them selectively rather than materialising every combination.
Sampling is the fraction of traces (or logs) you actually record and store. At 1,000 requests per second, recording every trace at 100% sample rate will overwhelm any self-hosted backend without dedicated infrastructure. The key insight is that not all requests are equally interesting. A fast 200 response is rarely worth storing. A slow response, an error response, or a response that involved an unusual code path always is. This asymmetry motivates tail-based sampling: make the sampling decision after the trace completes, so you can use the outcome (was it slow? did it error?) as a signal for whether to keep it. Head-based sampling makes the decision at the start of the request, before the outcome is known, which means you discard many of the most interesting traces.
Retention is how long you keep data. Metrics are cheap to retain long; 13 months of Prometheus metrics for a small service is a few gigabytes. Traces are expensive to retain; the same 13 months of full traces at even 5% sample rate can be terabytes. Design your retention policy before you deploy, not when you get the storage bill.
A reasonable starting point for most teams: 14 days of full-resolution metrics (scrape interval: 15s), 90 days of 5-minute aggregates, 13 months of 1-hour aggregates. Traces at 100% sample rate for 7 days if volume permits, or tail-sampled at 10–20% for 30 days. Logs compressed for 30 days hot, 90 days cold, then archived or deleted. These numbers are starting points — adjust based on how long your slowest incidents take to manifest and how far back you need to look for capacity planning.
The relationship between cardinality and cost is non-linear. Doubling the number of label dimensions can increase time-series count by an order of magnitude. Before adding any new label to an existing metric, estimate the cardinality: multiply the number of distinct values per label across all dimensions. An endpoint label with 50 values, a region label with 5 values, and a status-code label with 10 values yields 50 × 5 × 10 = 2,500 time series for that one metric. Add a service-instance label with 20 values and you are at 50,000 time series. That is still manageable. Add a user-tier label with 100 values and you are at 5 million — which is where self-hosted Prometheus starts straining under memory pressure.
The pattern
Observability systems do not have a single canonical architecture. The right shape depends on your scale, your budget, your team size, and what questions you are trying to answer. Recognising which tier you are in is the first step to not over-engineering or under-engineering.
Tier 0: Hobby project / side project
One service. One region. Fewer than ten requests per second. Probably running on a single VM or a small container.
The right observability posture here is: structured stdout logging, one health endpoint, and a single uptime check from an external service. Nothing more. Prometheus, Grafana, and Jaeger add operational overhead that exceeds the value they provide at this scale. The moment something goes wrong, you can read the logs in the terminal or in your cloud provider's log viewer. The important investment is making the logs structured JSON rather than printf strings, so that when you outgrow this tier, you do not need to rewrite your logging calls.
// Minimal structured logging — every log line is JSON
import pino from "pino";
const logger = pino({
level: process.env.LOG_LEVEL ?? "info",
base: { service: "api", version: process.env.APP_VERSION ?? "dev" },
});
// Good: structured, searchable
logger.info({ userId: req.user?.id, endpoint: req.path, durationMs: elapsed }, "request completed");
// Bad: unstructured, unsearchable
console.log(`User ${req.user?.id} hit ${req.path} in ${elapsed}ms`);Tier 1: Early-stage startup
Three to ten services. One region. Under 100 requests per second. A small engineering team.
At this tier, introduce Prometheus for metrics and a centralised log aggregator (Grafana Loki, or an inexpensive hosted service like Papertrail or Logtail). You do not necessarily need distributed tracing yet — if your call graphs are shallow and your services are few, structured logs with shared request IDs give you most of the correlation you need. Introduce a simple Grafana dashboard with RED metrics per service.
The critical investment is structured logging with a correlation ID that flows across all service calls. Every log line for a given user request should carry the same requestId. When something goes wrong, you can grep (or query) by that ID and reconstruct the full story across services.
The pattern for correlation ID propagation:
// Edge middleware: assign a requestId if none arrives
import { randomUUID } from "crypto";
app.use((req, res, next) => {
const requestId = (req.headers["x-request-id"] as string) ?? randomUUID();
req.requestId = requestId;
res.setHeader("x-request-id", requestId);
// Bind to async-local storage so downstream calls can read it
asyncLocalStorage.run({ requestId }, next);
});
// Downstream HTTP calls: propagate the ID
const { requestId } = asyncLocalStorage.getStore() ?? {};
const response = await fetch(upstreamUrl, {
headers: { "x-request-id": requestId ?? "" },
});// Every logger call inherits the requestId from context
function getLogger() {
const store = asyncLocalStorage.getStore();
return baseLogger.child({ requestId: store?.requestId });
}Metrics instrumentation at this tier is lightweight. Instrument your HTTP layer once with a middleware that records RED metrics for every endpoint:
import { Counter, Histogram, Registry } from "prom-client";
const registry = new Registry();
const httpRequestsTotal = new Counter({
name: "http_requests_total",
help: "Total number of HTTP requests",
labelNames: ["method", "route", "status_code"],
registers: [registry],
});
const httpRequestDurationSeconds = new Histogram({
name: "http_request_duration_seconds",
help: "HTTP request duration in seconds",
labelNames: ["method", "route", "status_code"],
buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10],
registers: [registry],
});
app.use((req, res, next) => {
const start = process.hrtime.bigint();
res.on("finish", () => {
const durationSeconds = Number(process.hrtime.bigint() - start) / 1e9;
const route = req.route?.path ?? req.path;
httpRequestsTotal.inc({ method: req.method, route, status_code: res.statusCode });
httpRequestDurationSeconds.observe(
{ method: req.method, route, status_code: res.statusCode },
durationSeconds
);
});
next();
});
// Prometheus scrape endpoint
app.get("/metrics", async (_req, res) => {
res.set("Content-Type", registry.contentType);
res.end(await registry.metrics());
});Tier 2: Growth-stage
Twenty or more services. Multi-region. More than 1,000 requests per second. A dedicated SRE or platform team, or at least an engineer whose job partly involves infrastructure.
This is where distributed tracing becomes essential. Without traces, correlating a slow request that traverses twelve services requires a log grep across twelve log streams with perfect correlation ID propagation. With traces, you open a single trace view and see the waterfall. OpenTelemetry is the right choice here — you instrument once with the OTel SDK, run an OTel Collector as a sidecar or DaemonSet, and route telemetry to whichever backend you choose.
This is also where you formalise SLOs, alert routing, and on-call processes. An alert that pages someone at 2 AM should represent a customer-impacting condition. Everything below that threshold should be a ticket in your backlog, not a wake-up call.
At growth-stage, you also need to standardise instrumentation across services. If every team chose its own logging library, its own metric naming convention, and its own span attribute names, your cross-service queries are exercises in normalisation rather than diagnosis. Define a platform standard: one logging library, one set of required log fields (requestId, service, version, environment, level), one metric naming convention ({service}_{subsystem}_{verb}_{unit}), and one set of required OTel span attributes (matching the OpenTelemetry semantic conventions for HTTP spans: http.method, http.route, http.status_code, http.url). The OpenTelemetry semantic conventions specification (https://opentelemetry.io/docs/specs/semconv/) is the reference here — using it means your tooling (Jaeger, Grafana Tempo, Honeycomb) can automatically understand your spans without custom parsing logic.
Deploy the OTel Collector as a DaemonSet on Kubernetes, one per node, receiving OTLP from all pods on that node and batching exports to your backend. This decouples your application's telemetry output from the backend — if you migrate from Jaeger to Grafana Tempo, you change the collector config, not every application.
# kubernetes/otel-collector-daemonset.yaml (excerpt)
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: otel-collector
namespace: monitoring
spec:
selector:
matchLabels:
app: otel-collector
template:
spec:
containers:
- name: otel-collector
image: otel/opentelemetry-collector-contrib:0.96.0
args: ["--config=/etc/otel/config.yaml"]
volumeMounts:
- name: config
mountPath: /etc/otel
ports:
- containerPort: 4317 # OTLP gRPC
- containerPort: 4318 # OTLP HTTP
resources:
requests:
memory: "200Mi"
cpu: "100m"
limits:
memory: "500Mi"
cpu: "500m"
volumes:
- name: config
configMap:
name: otel-collector-configThe resource limits on the collector matter. A collector without limits becomes a noisy neighbour on the node under trace bursts. Size the memory limit to accommodate your peak trace volume with a 2x safety margin, and set the CPU limit conservatively — the collector's batching logic is designed to absorb bursts.
The single best investment we made was running the OTel Collector as a DaemonSet and making it the only thing our apps talked to. We migrated backends twice in 18 months and neither time required a code change in the services.
Tier 3: The "we cannot afford Datadog" plateau
Many teams hit a moment where their Datadog bill exceeds their entire cloud infrastructure spend. This is a real constraint, not a complaint. At this point the options are:
- Negotiate a custom contract with the vendor. Often works — Datadog's list price is not what large customers pay.
- Move to open-source. Prometheus + Grafana + Tempo + Loki is the full OSS stack covering metrics, dashboards, traces, and logs respectively. The operational overhead is real but bounded.
- Move to a cost-optimised hosted alternative. Grafana Cloud's free tier and Observe.inc both have aggressive pricing for smaller teams.
The right answer depends on your team's operational capacity. Running Prometheus in HA mode with remote write to long-term storage (Thanos, Cortex, or Mimir) requires someone who understands stateful workloads on Kubernetes. If nobody on the team has that background, paying a modest bill for a managed service is rational.
### Tier 3 continued: Prometheus at scale
When Prometheus is handling hundreds of millions of active time series, a single Prometheus instance is insufficient. The typical path is Prometheus federation (a top-level Prometheus scrapes summaries from per-region or per-cluster Prometheus instances), or remote write to a horizontally scalable long-term storage backend.
The three OSS options are:
- Thanos (https://thanos.io/): stores blocks in object storage (S3, GCS, or Azure Blob). Query layer fans out across Thanos stores. Proven at large scale; operationally complex.
- Cortex (https://cortexmetrics.io/): multi-tenant, horizontally scalable. Each component (ingester, querier, compactor) scales independently. The architecture that Grafana Cloud is built on.
- Grafana Mimir (https://grafana.com/oss/mimir/): the productised successor to Cortex, with a simpler operational model and better documentation. Recommended for new deployments over Cortex as of 2024.
For teams not on Kubernetes or without the operational capacity to run distributed stateful workloads, the correct answer is a managed service. Grafana Cloud's free tier accommodates 10,000 series, 50 GB of logs, and 50 GB of traces per month — which covers Tier 1 and the lower end of Tier 2 at no cost.
The "log everything" anti-cost-pattern
Log volume is not a proxy for observability quality. Teams that emit a log line for every function call, every cache hit, every database query at the DEBUG level in production — and then fail to turn DEBUG off — routinely generate ten to one hundred times more log data than they can afford to retain. The logs they need are buried in noise and priced out of retention.
The discipline is: INFO for events that represent meaningful application state transitions. WARN for conditions that are unusual but handled. ERROR for conditions that caused a request to fail. DEBUG for developer use only, never enabled in production. Pair this with sampling: for high-volume repetitive events, log a statistical sample rather than every occurrence.
The fix / approach
This section is a rollout sequence. Each step builds on the previous one. You do not need to complete all steps before you get value — Step 1 alone is better than nothing, and the value compounds with each step.
Structured logging with correlation IDs everywhere
Start by making every log line a JSON object and ensuring every log line that is part of a user request carries a
requestIdthat was assigned at the edge and propagated through every service call and every database query.The implementation pattern is async-local storage (Node.js), context variables (Go), or MDC / mapped diagnostic context (Java). The ID must travel:
- From your load balancer or API gateway into the first service via an HTTP header (
x-request-idor the W3Ctraceparentheader if you are already adopting OpenTelemetry) - From the first service into every downstream HTTP call as the same header
- From the application into the database connection via a query comment or application name
The database propagation is frequently forgotten. A slow query log entry that carries
/* requestId=abc123 */as a comment can be correlated with the application trace. Without it, you know a query was slow but not which user request triggered it.// Add request context to database queries (example: node-postgres) async function queryWithContext<T>( pool: Pool, sql: string, params: unknown[] ): Promise<T[]> { const { requestId } = asyncLocalStorage.getStore() ?? {}; const annotatedSql = requestId ? `/* requestId=${requestId} */ ${sql}` : sql; const result = await pool.query(annotatedSql, params); return result.rows as T[]; }Validation: after implementing, pick a failed request from production, grab its
requestIdfrom the response header, and query your log store. You should see a single, coherent narrative of every log line generated by that request across every service. If you cannot reconstruct that narrative, the correlation ID propagation has a gap.- From your load balancer or API gateway into the first service via an HTTP header (
RED metrics per endpoint via Prometheus
Wire the HTTP middleware shown in the pattern section above to every service. Expose a
/metricsendpoint on each service. Configure Prometheus to scrape it. This gives you:- Rate:
rate(http_requests_total[5m])per endpoint - Errors:
rate(http_requests_total{status_code=~"5.."}[5m]) / rate(http_requests_total[5m]) - Duration:
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))
The histogram is critical. Do not use a gauge or a summary for latency. A histogram lets you compute any percentile after the fact using PromQL. A pre-computed summary is computed on the client and cannot be aggregated across instances. Prometheus best practices (https://prometheus.io/docs/practices/histograms/) are explicit on this point.
# p99 latency per endpoint, across all instances histogram_quantile( 0.99, sum by (le, route) ( rate(http_request_duration_seconds_bucket[5m]) ) )# Error rate per route over the last 5 minutes sum by (route) (rate(http_requests_total{status_code=~"5.."}[5m])) / sum by (route) (rate(http_requests_total[5m]))Build one Grafana dashboard per service. Each dashboard has one row: the RED panel trio for every instrumented endpoint. All three panels share a time range and can be filtered by a route variable. The pattern is described in the Grafana Prometheus documentation (https://grafana.com/docs/grafana/latest/datasources/prometheus/).
A concrete Grafana dashboard layout for the order service:
Row 1 — POST /api/v1/orders [Rate: req/s] [Error rate: %] [p50/p95/p99 latency: ms] Row 2 — GET /api/v1/orders/:id [Rate: req/s] [Error rate: %] [p50/p95/p99 latency: ms] Row 3 — Database (USE) [Connection pool utilisation: %] [Pool wait time: ms] [Query error rate: %] Row 4 — Runtime (USE) [CPU: %] [Memory: MB] [Event loop lag: ms]Each panel should use the same time range variable and a
$serviceor$instancevariable for filtering. The RED panels per endpoint live in rows 1–N; the USE panels for the backing resources live at the bottom. This layout means that when a RED alert fires, you open the dashboard, find the slow endpoint row at a glance, then scroll down to the USE rows to diagnose the resource bottleneck.For the latency panel, always show multiple percentiles simultaneously. A single p99 line hides the story. Show p50, p95, and p99 as three separate series on the same panel:
# p50 — median latency histogram_quantile( 0.50, sum by (le) (rate(http_request_duration_seconds_bucket{route="/api/v1/orders",method="POST"}[5m])) ) # p95 histogram_quantile( 0.95, sum by (le) (rate(http_request_duration_seconds_bucket{route="/api/v1/orders",method="POST"}[5m])) ) # p99 histogram_quantile( 0.99, sum by (le) (rate(http_request_duration_seconds_bucket{route="/api/v1/orders",method="POST"}[5m])) )When the p99 diverges from the p95 without the p50 moving, you have a long-tail problem affecting a small fraction of requests — often a specific customer, a specific region, or a specific data shape that triggers a pathological code path. This is exactly the class of problem that traces exist to debug.
- Rate:
Distributed tracing with OpenTelemetry
Install the OpenTelemetry Node.js SDK. Initialise it at process startup, before any other imports, so the automatic instrumentation patches land on the original module instances.
// tracing.ts — import this first in your entrypoint import { NodeSDK } from "@opentelemetry/sdk-node"; import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"; import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node"; import { Resource } from "@opentelemetry/resources"; import { SEMRESATTRS_SERVICE_NAME, SEMRESATTRS_SERVICE_VERSION } from "@opentelemetry/semantic-conventions"; const sdk = new NodeSDK({ resource: new Resource({ [SEMRESATTRS_SERVICE_NAME]: process.env.SERVICE_NAME ?? "api", [SEMRESATTRS_SERVICE_VERSION]: process.env.APP_VERSION ?? "dev", }), traceExporter: new OTLPTraceExporter({ url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? "http://localhost:4318/v1/traces", }), instrumentations: [ getNodeAutoInstrumentations({ "@opentelemetry/instrumentation-fs": { enabled: false }, // noisy, rarely useful }), ], }); sdk.start(); process.on("SIGTERM", () => sdk.shutdown());// entrypoint: import tracing FIRST import "./tracing"; import express from "express"; // ... rest of your appThe auto-instrumentation covers
http,express,pg,redis,mongodb, and many other common libraries automatically. For business-logic spans you care about:import { trace, SpanStatusCode } from "@opentelemetry/api"; const tracer = trace.getTracer("order-service"); async function processOrder(orderId: string): Promise<void> { return tracer.startActiveSpan("processOrder", async (span) => { span.setAttribute("order.id", orderId); try { await validateInventory(orderId); await chargePayment(orderId); await fulfil(orderId); span.setStatus({ code: SpanStatusCode.OK }); } catch (err) { span.recordException(err as Error); span.setStatus({ code: SpanStatusCode.ERROR, message: (err as Error).message }); throw err; } finally { span.end(); } }); }The W3C
traceparentheader (format:00-{traceId}-{spanId}-{flags}) is the propagation standard used by the OTel SDK when you use the HTTP auto-instrumentation. Your CDN and load balancer should be configured to pass this header through unchanged rather than stripping or rewriting it. If your CDN strips thetraceparentheader, spans generated at the edge will not be connected to spans generated in your application, and your trace waterfall will be broken into disconnected fragments.Validate end-to-end trace propagation by making a request from a curl command with a manually set
traceparentheader and verifying that the trace you find in your backend carries the same trace ID:# Set a known trace ID and check that it propagates end-to-end TRACE_ID="0af7651916cd43dd8448eb211c80319c" curl -H "traceparent: 00-${TRACE_ID}-b7ad6b7169203331-01" \ https://api.example.com/api/v1/orders # Then search for this trace ID in your backend # In Grafana Tempo: # {traceId="${TRACE_ID}"}Sampling strategy: at fewer than 200 requests per second, sample 100%. Above that, use tail-based sampling in the OTel Collector: always keep error spans and spans with duration above your p99 threshold; sample the rest at 10%. The OTel Collector's
tailsamplingprocessorsupports this logic:# otel-collector-config.yaml processors: tail_sampling: decision_wait: 10s policies: - name: errors-policy type: status_code status_code: { status_codes: [ERROR] } - name: slow-traces-policy type: latency latency: { threshold_ms: 1000 } - name: probabilistic-policy type: probabilistic probabilistic: { sampling_percentage: 10 }Alertmanager routing and SLO-based alerts
Before you can write SLO-based alerts, you need SLOs. Defining them is an engineering and product decision, not a monitoring decision. The right questions:
- What does "working" mean for each critical user journey? For a checkout endpoint, "working" might mean: the request succeeds (HTTP 2xx) AND completes within 800ms. These become your SLI (Service Level Indicator) definition.
- What fraction of requests must meet that bar? Start with your current measured performance. If your checkout endpoint today has a 99.5% success rate and p99 at 600ms, setting a 99.9% availability SLO is aspirational — you need to first understand whether the 0.5% failure rate is intentional (client errors you are correctly rejecting) or accidental. A reasonable first SLO is slightly below your current measured performance, because the goal is to alert on degradation, not to set a target you have not yet achieved.
- What is the measurement window? 30-day rolling windows are standard. They give you a month of error budget to work with and align with monthly operational reviews.
A 99.9% availability SLO over a 30-day window gives you 43.8 minutes of allowed downtime per month (0.1% × 30 × 24 × 60). That is your error budget. A burn-rate alert fires when you are consuming that budget at a rate that would exhaust it within the remaining window.
Wire Prometheus alerts through Alertmanager. The configuration has two goals: page the right person for the right reason, and never page anyone for a condition that does not impact customers.
The alerting hierarchy:
- Page now: SLO breach is imminent or occurring. A real customer is experiencing degraded service.
- Ticket: anomalous but not customer-impacting. High CPU, elevated error rate on a non-critical path, saturation warning.
- Dashboard annotation: deployments, config changes, batch job completions. No notification needed.
Multi-window, multi-burn-rate alerting is the standard approach for SLO-based paging. The Google SRE Workbook (https://sre.google/workbook/alerting-on-slos/) describes the algorithm in detail. The basic idea: define an error budget as
(1 - SLO) * window. Alert when you are consuming that budget at a rate that would exhaust it before the window ends.# prometheus/alerts/slo.yaml groups: - name: slo_api_availability rules: # Fast burn: consuming 14x budget — alert within ~1 hour - alert: ApiAvailabilitySLOBurnRateFast expr: | ( sum(rate(http_requests_total{status_code=~"5.."}[1h])) / sum(rate(http_requests_total[1h])) ) > (14 * 0.001) for: 2m labels: severity: page team: platform annotations: summary: "API error budget burning at 14x — SLO breach imminent" runbook: "https://runbooks.internal/api-availability" # Slow burn: consuming 5x budget — alert for trending issues - alert: ApiAvailabilitySLOBurnRateSlow expr: | ( sum(rate(http_requests_total{status_code=~"5.."}[6h])) / sum(rate(http_requests_total[6h])) ) > (5 * 0.001) for: 60m labels: severity: ticket team: platform annotations: summary: "API error budget trending toward breach over 6h window" runbook: "https://runbooks.internal/api-availability"# alertmanager/config.yaml route: group_by: ["alertname", "team"] group_wait: 30s group_interval: 5m repeat_interval: 4h receiver: default-tickets routes: - match: severity: page receiver: on-call-pager continue: false - match: severity: ticket receiver: ticketing-system receivers: - name: on-call-pager pagerduty_configs: - service_key: "${PAGERDUTY_SERVICE_KEY}" - name: ticketing-system webhook_configs: - url: "http://internal-ticketing/api/alert-webhook" - name: default-tickets webhook_configs: - url: "http://internal-ticketing/api/alert-webhook"On-call runbooks tied to every alert
An alert that fires without a runbook creates an on-call experience where the engineer pages through Slack history, Confluence, and memory trying to remember what this alert means and what to do. This produces slow incident response and alert fatigue in equal measure.
Every alert that can page someone must have a
runbookannotation pointing to a document that answers:- What exactly is this alert measuring? What PromQL expression defines it?
- What is the expected normal state, and what does the current firing state mean in plain terms?
- What are the first three things to check? (Link to specific dashboards, queries, log searches)
- What are the known causes and their remediation steps?
- Who owns this service, and who is the escalation path if you cannot resolve it in 30 minutes?
The runbook should be a living document. After every incident that touches a runbook, update it. If the runbook did not exist and someone had to invent the debugging steps, write them down immediately after resolution.
Anti-patterns
Observability stacks accumulate debt in specific ways. These are the most common failure modes, paired with the correction.
Alerting on CPU > 80%
Wrong
Right
CPU above 80% is a cause. Customers do not experience high CPU; they experience slow requests, errors, and timeouts. A CPU spike during a nightly batch job that runs in the background affects nobody. A CPU spike during peak traffic that is causing p99 latency to breach your SLO affects everyone. Alert on the symptom users feel — elevated error rate, latency above SLO — not on the resource metric that may or may not be causing it.
The Google SRE book's chapter on monitoring (https://sre.google/sre-book/monitoring-distributed-systems/) makes this point directly: alert on symptoms, use metrics and dashboards to diagnose causes.
Unbounded label cardinality
Wrong
Right
Adding user_id, customer_id, session_id, request_id, or any other field with high cardinality as a Prometheus label will crash your Prometheus instance or generate a bill that crashes your bank account in a managed service. Prometheus stores one time series per unique label combination. A service with 50,000 users generating one request per second has 50,000 active time series for a single metric if user_id is a label.
High-cardinality identifiers belong in trace attributes and in structured log fields. They are queryable there via point lookups, not by materialising every combination.
Prometheus documentation on label best practices (https://prometheus.io/docs/practices/naming/) is explicit: the combinatorial space of your labels should be bounded and manageable.
No service-level objectives
Without an SLO, there is no principled way to distinguish a good week from a bad week. You have metrics. You have dashboards. But "is the service performing well?" is answered by vibes — whoever looked at the dashboard most recently and their current tolerance for imperfection.
An SLO quantifies the agreement: "99.9% of requests to this endpoint complete successfully and within 500ms, measured over a 30-day rolling window." That number lets you compute an error budget, track burn rate, write principled alerts, and have a data-driven conversation with stakeholders about reliability vs velocity.
The SRE Workbook (https://sre.google/workbook/implementing-slos/) has a full treatment of SLO definition, including the difference between SLOs (internal engineering commitments to reliability targets) and SLAs (external contractual commitments with penalty provisions).
Retaining full-resolution histograms forever
A Prometheus histogram with 15 buckets, scraped every 15 seconds, for a service with 20 endpoints, over 13 months, produces a meaningful amount of data. Most teams set Prometheus retention to the default (15 days) and then discover they cannot answer "how did p99 latency compare to this time last month?" The temptation is to increase retention indefinitely.
The correct answer is downsampling: store full-resolution data for 2 weeks, 5-minute aggregates for 3 months, 1-hour aggregates for 13 months. Thanos (https://thanos.io/) and Grafana Mimir implement this via the compactor component. The cost reduction is substantial — hours-resolution data for capacity planning is almost always sufficient for data older than a few weeks.
Sampling 1% uniformly
Uniform 1% sampling means that on a service handling 2,000 requests per second, you collect 20 traces per second. That sounds reasonable until you consider that the requests you most want to see — errors, slow outliers, requests from specific customers who reported problems — appear at a rate far below 1% of total traffic. At 1% uniform sampling, a class of error that affects 0.1% of requests may have zero sampled traces on any given day.
Use tail-based sampling. Always keep errors. Always keep spans above your latency SLO threshold. Sample the healthy fast requests probabilistically. This is not more expensive than uniform sampling — you keep the same number of traces — but the traces you keep are dramatically more useful.
Shipping every JSON field as a separate metric
Teams connecting a metrics library to a rich event stream sometimes emit one metric per JSON field — event_order_status_pending, event_order_status_processing, event_order_status_fulfilled, etc. When the event schema has 40 fields, this creates 40 metrics. When the event schema changes, the metrics go stale or proliferate further.
The correct model is one metric with a label for the dimension being counted:
# One metric, multiple label values
order_events_total{status="pending"}
order_events_total{status="processing"}
order_events_total{status="fulfilled"}Or, for high-cardinality event tracking, a structured log event that is queryable in your log store rather than a metric at all.
Case studies / numbers
The following are composite examples drawn from observability migration patterns. Names are omitted and numbers are rounded to order of magnitude.
The 47-to-6 alert reduction
A B2B SaaS platform had instrumented its services thoroughly over three years of iterative additions. The result was 63 active alert rules, of which 47 fired at least once per week. Mean time to acknowledge was over 90 minutes. The on-call rotation had developed a habit of acknowledging alert storms with a bot command rather than investigating.
The audit revealed the following distribution of weekly alerts:
- 31 alerts were threshold-based resource alerts (CPU, memory, disk) with no direct mapping to customer experience. None had fired on a week where a customer-impacting incident did not also trigger a separate, more specific alert.
- 9 alerts were monitoring synthetic checks that were inconsistently configured — some checked endpoints the service had stopped serving, some had timeouts set below the service's normal cold-start time.
- 7 alerts were duplicates of each other, measuring the same condition from slightly different windows.
The team deleted or converted-to-ticket all resource threshold alerts, fixed the synthetic checks, and deduplicated the remainder. They added three SLO burn-rate alerts — one per critical service endpoint. The result was 6 actionable alerts per week on average. Mean time to acknowledge dropped to under 12 minutes because engineers had learned that if the alert fired, it was worth investigating immediately.
The total observability cost did not change. The signal-to-noise ratio changed dramatically.
The cardinality incident
A platform team at a growth-stage startup added user-cohort-level analytics to their metrics pipeline by appending a cohort_id label to 14 existing metrics. The cohort IDs were UUIDs assigned at account creation. There were approximately 8,000 active cohorts.
Within 48 hours, Prometheus memory consumption had grown from 12 GB to over 80 GB. The Prometheus instance OOM-killed. During the recovery window — approximately 3 hours of degraded observability — a deployment introduced a regression in the checkout flow that was not caught until customers reported it.
The fix was removing the cohort_id label and replacing the business analytics use case with a log-based query in Loki, where the UUID was a structured field rather than a time-series label. The analytics latency increased slightly (log queries are slower than instant metric queries) but the cardinality problem was resolved and the metrics store returned to stable operation.
The post-mortem added a Prometheus cardinality check to the CI pipeline: any change to metric label definitions that would increase total time-series count by more than 10% required an explicit review step.
The p99 that no one looked at
A platform had comprehensive error-rate monitoring and responded quickly to any spike above 1%. Latency metrics existed but no alerts were defined on them. For approximately six weeks, the p99 latency on the document search endpoint had been degrading — from 420ms to 1,100ms — as the search index grew without a corresponding index rebuild.
No customer filed a support ticket. The degradation was gradual enough that individual users attributed it to their connection or device. The engineering team discovered it only when a customer success manager mentioned in a meeting that "search feels a bit slow lately."
Introducing a p99 latency SLO (500ms target) and a corresponding burn-rate alert would have surfaced this within the first two weeks of degradation. The root cause — the missing index rebuild — would have been a 20-minute fix at week two rather than a discovery at week eight, by which point several enterprise customers had downgraded their internal rating of the product's performance.
The CDN blind spot (a common variant)
The scenario in the lede of this post — a CDN misconfiguration causing silent failures in a specific region — represents a class of incident that observability inside your application layer cannot catch. The requests never reach your application. Your error rate is zero because the error rate is computed over requests you received.
The fix has two components. First, scrape CDN-level metrics and errors into the same observability system as application metrics. Most CDN vendors expose metrics via API; an exporter process can poll this API and publish the results as Prometheus metrics. Second, run synthetic checks from multiple geographic regions, including regions where your real traffic is concentrated, using the actual protocols those users use.
When both of these are in place, the alert for the CDN incident described in the lede would fire within 5 minutes: the CDN error rate metric for SA-1 would spike, and the São Paulo synthetic check would return a connection timeout. The on-call engineer receives an alert titled "CDN SA-1 connection timeout — likely origin connectivity or rule misconfiguration" with a runbook link, before the customer Slacks the founder.
The database connection pool silent saturation
A service handling 300 req/s was configured with a database connection pool of 20 connections. Under normal load, pool utilisation averaged 40%. A scheduled report generation job was added that, when triggered, consumed 18 connections for approximately 90 seconds.
During report generation windows, the connection pool reached 100% utilisation. Requests that arrived during this window waited for a connection, adding up to 800ms of latency. The p99 latency spiked during these windows but the average latency barely moved. No error rate increase occurred because the requests eventually succeeded.
Without latency SLO alerts, this was invisible. With a p99 latency burn-rate alert, the first report generation triggered the alert within the first window. The fix — separating the reporting database connection pool from the application pool — was implemented before any customer-facing SLO was breached.
The lesson is that queue saturation (the S in USE) is often invisible in error rates and averages. Histogram-based p99 alerts catch it; gauge-based average alerts do not.
Deployment event correlation
A platform team introduced Grafana annotations triggered by their CI/CD pipeline on every production deployment. Over the first month, they reviewed 12 latency incidents and found that 9 of them had a deployment event within 15 minutes of the incident start.
This correlation prompted a change to the deployment process: a canary phase that routes 5% of traffic to the new version for 10 minutes before full rollout. The canary window is monitored by the same RED metrics and latency SLO. If the canary's error rate or latency exceeds the SLO threshold, the deployment is automatically reverted.
Over the 8 months following this change, deployment-caused incidents dropped from an average of 1.8 per week to 0.3 per week. The remaining incidents were from configuration changes and dependency updates that bypassed the canary process.
The deployment annotation alone did not prevent any incidents. It provided the correlation evidence that motivated the process change. This is the most common pattern of observability value — it does not prevent problems directly, but it accelerates the identification of patterns that humans then act on.
Limits + caveats
Observability, done well, is genuinely valuable. It is also limited in ways that are worth stating explicitly to avoid substituting instrumentation investment for other necessary investments.
Observability does not surface bugs you cannot reproduce. A race condition that manifests once every ten million requests will show up in your error rate as a rounding error and may appear in a trace if you are lucky with sampling. The investment in debugging ephemeral, low-rate bugs is testing infrastructure and reproducible environments, not instrumentation.
Observability does not measure customer perception. Your p99 latency is 320ms. Is that good? That depends on what your customers are comparing you to, what their network conditions are, and what mental model they have built over time. Customer satisfaction surveys, NPS scores, and support ticket analysis measure customer perception. Latency metrics are a proxy for the experience, not the experience itself.
Observability does not catch supply-chain incidents in time. When a widely-used open-source library has a vulnerability that is disclosed publicly, or when a dependency publishes a broken release, your observability stack will tell you that things are broken — after the breakage reaches production. Supply-chain risk requires a different class of tooling: dependency scanning, lockfile integrity checks, and SBOM generation. Observability is a post-event signal; supply-chain security requires pre-event controls.
Observability does not replace business metrics. Revenue dashboards, conversion funnels, activation rates, and churn signals tell you whether the business is healthy. Your service availability SLO could be 99.99% on a week where your signup conversion dropped by 30% because a marketing campaign drove unqualified traffic. Keep your engineering observability and your business metrics as separate dashboards consumed by different people for different decisions.
Observability does not cover security events. Anomalous authentication patterns, credential stuffing, data exfiltration, and privilege escalation events require a SIEM (Security Information and Event Management) system with rules specifically calibrated for security patterns. The logs and traces you generate for operational observability are necessary input to a SIEM but not sufficient as a security detection layer on their own.
The observability stack itself has a cost that needs observing. A misconfigured OTel Collector that duplicates trace exports, or a Loki ingest pipeline without rate limits, can generate costs that materially affect your infrastructure budget. Monitor your observability infrastructure with the same rigour as your application infrastructure. Track Prometheus time-series count per week. Track log ingest volume per service. Set cost alerts.
Sources & further reading
The sources below are primary references for the concepts in this post. All links were verified at the time of writing.
Google SRE Book — Monitoring Distributed Systems https://sre.google/sre-book/monitoring-distributed-systems/ The foundational treatment of the four golden signals (latency, traffic, errors, saturation), the principle of symptom-based alerting, and the relationship between monitoring and on-call burden. Chapter 6 of the SRE Book. Available free online.
Google SRE Workbook — Alerting on SLOs https://sre.google/workbook/alerting-on-slos/ The multi-window, multi-burn-rate alerting algorithm. Describes how to set burn-rate thresholds for different alert windows to minimise false positives while catching real SLO breaches quickly. Chapter 5 of the SRE Workbook. Available free online.
Google SRE Workbook — Implementing SLOs https://sre.google/workbook/implementing-slos/ Practical guidance on selecting SLI metrics, setting SLO targets, computing error budgets, and building error budget policies. Available free online.
OpenTelemetry Specification https://opentelemetry.io/docs/specs/otel/ The normative specification for the OTel data model, API, and SDK. The trace data model (trace ID, span ID, parent span ID, attributes, events, status), the context propagation protocol (W3C traceparent), and the OTLP wire protocol are all defined here.
Prometheus Best Practices — Histograms and Summaries https://prometheus.io/docs/practices/histograms/ Why histograms are preferred over summaries for latency metrics. Explains the aggregation properties of histograms that allow cross-instance quantile computation, and why pre-computed summaries cannot be aggregated.
Prometheus Best Practices — Metric and Label Naming https://prometheus.io/docs/practices/naming/ The label cardinality problem explained with guidance on which dimensions belong as labels vs which should be stored elsewhere. Essential reading before instrumenting any service.
Charity Majors, Liz Fong-Jones, George Miranda — Observability Engineering O'Reilly Media, 2022. ISBN 978-1-4920-7644-0 https://www.oreilly.com/library/view/observability-engineering/9781492076438/ The most rigorous treatment of the distinction between monitoring (known-unknowns, pre-specified dashboards) and observability (unknown-unknowns, arbitrary queries over high-cardinality events). Chapter 2 on the difference between metrics and events is particularly valuable.
Brendan Gregg — The USE Method https://www.brendangregg.com/usemethod.html The original description of Utilisation, Saturation, Errors as a framework for resource metrics. Includes checklists for Linux system resources. The complementary framework to RED for request-driven services.
Tom Wilkie — The RED Method: How to Instrument Your Services https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/ The canonical description of Rate, Errors, Duration as the three metrics every request-driven service should expose. Originally presented at GrafanaCon 2018.
Cindy Sridharan — Distributed Systems Observability O'Reilly Media, 2018 (free report) https://www.oreilly.com/library/view/distributed-systems-observability/9781492033431/ A concise (80-page) treatment of the three pillars — logs, metrics, traces — with specific attention to the failure modes of each and the conditions under which you need all three rather than any single one.
Grafana Loki Documentation https://grafana.com/docs/loki/latest/ The operational guide for Grafana Loki, the log aggregation system designed to work alongside Prometheus. Relevant to teams building an all-OSS observability stack.
Alertmanager Documentation
https://prometheus.io/docs/alerting/latest/alertmanager/
The configuration reference for Alertmanager routing, inhibition, and silencing. The routing tree configuration is described in the routing section.
- Tech
Server-Sent Events vs WebSockets in 2026: when SSE wins, backpressure, reconnect semantics
A streaming LLM endpoint that worked on Chrome failed silently on Safari at 30 KB/s. The SSE vs WebSocket decision and the backpressure, reconnect, and heartbeat patterns that survive production.
- Tech
Webhook signature verification done right: HMAC, timing-safe compare, and replay defense
A webhook endpoint validated the signature with `===` and got owned by a side-channel attack three months later. The HMAC, timing-safe, and replay-window patterns that actually work.
- Tech
Postgres connection pooling at scale + serverless cold starts: pgbouncer, Supavisor, RDS Proxy
A serverless API that opened 4,200 idle Postgres connections during a Black-Friday burst. The pool-mode, prepared-statement, and transaction-scope choices that survive serverless.