0009: Sample logs per request, by trace ID¶
Status: accepted · Reviewed: 2026-10-31
Context¶
Successful requests write most of the log volume, and fastapiobserver_log_bytes_total shows which routes. Dropping a share of lines at random would leave requests with half their lines, and lose the lines a failure needs. Traces are already sampled by trace ID, so logs sampled any other way would point at traces that were never kept.
Decision¶
LOG_SAMPLE_RATEdecides per request, not per line. A request is sampled when the low 64 bits of its trace ID are below the ratio's bound. That's the rule OpenTelemetry'sTraceIdRatioBasedsampler uses. Without a trace ID, a 64-bit hash of the request ID is used instead. Services forward both, so every service that sees a request makes the same decision.- Only
INFOand lower lines are sampled. For a request that isn't sampled, they're held in the per-request state thatLOG_LEVEL_ON_ERRORalready uses (0008). They're written if the request fails or takesLOG_SLOW_REQUEST_MSor longer.WARNINGand higher are always written. - Metrics aren't sampled.
Evidence¶
test_sampling_follows_the_trace_id_as_opentelemetry_doessends two traces at a ratio of 0.5. The one whose lines are written is the one OpenTelemetry's sampler keeps.test_unsampled_requests_write_only_warnings_unless_they_fail_or_are_slowchecks the failed, slow andWARNINGrules.- CPU per request, log line included, from
examples/benchmarks/request_cpu.pyon an Apple M1, median of 6 runs:
| CPU per request | |
|---|---|
| 1.10.0 | 51.1 µs |
| Sampling off | 51.2 µs |
LOG_SAMPLE_RATE=0.1 |
38.0 µs |
What it cost¶
- Logs and traces line up only at matching ratios. If each service samples traces with OpenTelemetry's ratio sampler at the same ratio, a request's lines are kept exactly when its trace is. A lower log ratio keeps a subset of the traced requests. A higher one also keeps requests that have no trace. Other trace samplers, or an upstream service that decides differently, can disagree.
- Unsampled requests are held in memory until they end. At most 200 lines are held, the newest kept, so a slow request with more loses its oldest lines.
- 4xx lines are always written, because
request.client_erroris aWARNING. That includes scanners' 404s.