Structured Logging Guide¶
fastapi-observer emits structured request and WebSocket diagnostics and provides an optional
non-blocking JSON logging pipeline. By default it configures logging; established applications
can keep their own pipeline with configure_logging=False.
Application-owned logging¶
Configure your application logger first, then install diagnostics:
from fastapiobserver import install_observability
# Your existing logging configuration, including your redaction, is in place.
install_observability(app, configure_logging=False)
This option preserves root, application and Uvicorn handlers, filters, formatters, levels and propagation settings. It adds no output handler, queue, log processor or log exporter. Each diagnostic follows the application's existing routing and level rules. Uvicorn access logs stay enabled when your application has enabled them.
The fastapiobserver.middleware logger captures request_id, user_context and
available trace_id / span_id before existing logger filters and handlers run.
This also works with application-owned queues: formatting on another thread
does not lose the request context. Request fields remain on LogRecord.event;
your formatter must serialize event and the correlation attributes to include
them in output.
The application owns output sanitization, including free text, captured context
and exceptions. Body-capture controls and captured-body masking still apply,
but SecurityPolicy does not sanitize the application's output handlers.
OBS_LOG_* pipeline controls (levels, sampling, held logs, files, queues and
sinks), registered log filters/enrichers and otel_logs_settings do not configure
the application pipeline. Runtime-control requests to change log levels return 409.
Request metrics, database/HTTP diagnostics, the event-loop watchdog and optional
tracing and metric export remain available; see
OpenTelemetry ownership.
For correlation on your application's own records, you can attach RequestIdFilter
and TraceContextFilter from fastapiobserver.logging to your producer-side handlers.
See examples/existing_logging_app.py
for a runnable stdlib JSON integration. The rest of this page describes the default
library-owned pipeline.
What Gets Logged on Requests and WebSockets¶
Request middleware emits one primary event per HTTP request or WebSocket session:
| Message | Level | Trigger |
|---|---|---|
request.completed |
INFO |
Successful HTTP response (< 400) |
request.client_error |
WARNING |
Client-side HTTP errors (4xx) |
request.server_error |
ERROR |
Server-side HTTP errors (5xx without exception) |
request.failed |
ERROR + traceback |
Unhandled exception in HTTP request path |
request.cancelled |
WARNING |
Cancelled before completion (status_code is 499 if no response started) |
request.client_disconnected |
WARNING |
Application received http.disconnect before response completion and ran on (after_disconnect_ms and after_disconnect_calls report wasted work). See servers |
websocket.closed |
INFO |
Accepted WebSocket closed by either side, including a client disconnect the handler does not catch (close_code, messages_sent, messages_received, bytes_sent, bytes_received) |
websocket.rejected |
WARNING |
WebSocket closed before websocket.accept |
websocket.failed |
ERROR + traceback |
Unhandled exception during WebSocket session, other than WebSocketDisconnect |
Standard request event fields include:
- method, path, status_code, client_ip, error_type, and OTel aliases (http.request.method, url.path, http.response.status_code)
- http.route: the matched route template, omitted when no route matched
- duration_ms and app_duration_ms
- response_start_ms, first_body_ms, stream_duration_ms, and response_bytes (total response body bytes sent). See route and timing
- response_completed, cancelled, client_disconnected, after_disconnect_ms and after_disconnect_calls. See client disconnects
- diagnostics and, on 422 responses, validation_errors. See per-request diagnostics
- observation and log_sampling: see observation coverage and sampling policy
- db and http: count and total duration of completed database queries and outbound HTTP calls. See database tracing and HTTP diagnostics.
What the durations mean¶
duration_ms and the HTTP duration histogram start at middleware entry and stop
after the final response body has been passed to ASGI send (or after final
trailers). They include handler work, serialization and streaming, and exclude
work performed after the response.
app_duration_ms stops when the ASGI application returns or raises, including
background tasks and dependency cleanup. The request line is written then, while
HTTP metrics are recorded once when the response finishes.
response_completed=false means no final send was observed (duration_ms
measures time until interruption). Database and HTTP totals cover work completed
before the application ends, including background tasks, so their summed durations
can exceed duration_ms when calls overlap or run after the response.
Debug logs for failed requests¶
Set OBS_LOG_LEVEL_ON_ERROR below OBS_LOG_LEVEL to buffer a request's lower-level lines in memory while it runs:
OBS_LOG_LEVEL=INFO
OBS_LOG_LEVEL_ON_ERROR=DEBUG
- If the request ends in a
5xxor an unhandled exception, up to the last 200 held lines are flushed just beforerequest.failedorrequest.server_errorwith the samerequest_idandtrace_id. - Otherwise they are dropped without formatting or queue overhead.
- A message longer than 32 KiB, held or not, is cut to that length and ends with
... [N characters cut]. - A level set through the runtime control plane replaces
OBS_LOG_LEVEL, and holding stays active.
Sampling request logs¶
To sample successful INFO requests while retaining all warnings, errors, and slow requests:
OBS_LOG_SAMPLE_RATE=0.1 # Keep 10% of fast 2xx/3xx request logs
OBS_LOG_SLOW_REQUEST_MS=1000 # Always keep requests >= 1000 ms
- A request is kept or dropped as a whole, including its terminal
request.completedline. UnsampledINFOrequests skip terminal event dictionary construction and sanitization entirely. - Any request that fails (
5xxor unhandled exception), emitsWARNING+, or takesOBS_LOG_SLOW_REQUEST_MSor longer is written in full. - Prometheus metrics still count every request.
- WebSocket sessions are not sampled. A session can stay open for hours, so its lines are written as they happen.
- Decisions use OpenTelemetry's
TraceIdRatioBasedrule when atrace_idis present, or a hash ofrequest_idotherwise, keeping whole requests aligned across services.
Calls that block the event loop¶
A synchronous call inside async def (requests.get, time.sleep, sync DB driver) stalls every concurrent request on that worker. Enable the watchdog via the production profile (250 ms) or explicitly:
OBS_LOOP_BLOCK_THRESHOLD_MS=200
When the loop stalls past the threshold, the watchdog thread logs one warning per block with the offending call stack and tags in-flight requests with "event_loop.blocked" in their diagnostics array:
{"level": "WARNING", "logger": "fastapiobserver.event_loop", "message": "event_loop.blocked", "event": {"blocked_ms": 212, "stack": "... File \"/app/orders.py\", line 41, in get_order\n rates = requests.get(RATES_URL).json()\n"}}
Garbage-collection pauses¶
A garbage collection stops every thread until it ends. An old-generation collection
on a large heap can take hundreds of milliseconds, and Python reports no duration
for it. From the app's lifespan on, the library times each collection. An HTTP request
that collections stopped for 100 ms or more in total gets "gc.pause" in its
diagnostics and the time in gc_pause_ms, and fastapiobserver_gc_pause_total
counts it (skipped on WebSocket sessions):
{"message": "request.completed", "event": {"http.route": "/orders", "duration_ms": 912.4, "diagnostics": ["gc.pause"], "gc_pause_ms": 847}}
Collections that often pause long usually come from many long-lived objects;
gc.freeze() after startup keeps the objects created at import out of later
collections.
Full thread and database pools¶
Two pools make requests wait without using CPU:
- AnyIO's thread pool runs sync (def) endpoints and dependencies (40 threads by default).
- SQLAlchemy's QueuePool hands out at most pool_size + max_overflow connections.
fastapi-observer checks both at request entry, tags affected requests in event.diagnostics, and emits a rate-limited warning (at most once per minute per pool):
{"level": "WARNING", "logger": "fastapiobserver.pools", "message": "threadpool.saturated", "event": {"busy": 40, "limit": 40, "waiting": 12}}
{"level": "WARNING", "logger": "fastapiobserver.pools", "message": "db.pool_exhausted", "event": {"engine": "postgresql://app@db/orders", "checked_out": 15, "limit": 15}}
The pool gauges and counters on /metrics are listed under metrics.
JSON Schema Contract¶
| Field | Purpose |
|---|---|
timestamp, level, logger, message |
UTC time, severity, logger name, and event name |
app_name, service, environment, version |
Service identity |
log_schema_version, library, library_version |
Schema and producer identity |
request_id, trace_id, span_id, user_context |
Correlation and plugin context |
event |
Structured request/diagnostic payload |
error |
Structured exception payload (type, message, stacktrace, fingerprint) |
Exception payloads include error.fingerprint (a 128-bit digest of the exception type and normalized call frames, excluding dynamic message text and line numbers) and automatically strip credentials from URLs in error.message and error.stacktrace (postgresql://***:***@db/app).
Output, Queue, and Shutdown Behavior¶
- Local JSON logs write to
stdoutby default; setOBS_LOG_DIRfor rotating file logs, orlogs_mode(local_json,otlp,both) when OTel logs are enabled. - Uvicorn logs route through the same queue, and duplicate
uvicorn.accesslines are suppressed automatically. - The bounded in-memory queue supports
drop_oldest(default),drop_newest, andblock, with per-sink circuit breakers. See bounded queue and sink circuit breaker. flush_logging(timeout_seconds=5.0)andshutdown_logging()drain queued records cleanly when the FastAPI lifespan ends or at process exit (atexit).