Skip to content

Production Deployment Guide

Reference architecture

flowchart LR
  A["FastAPI services (fastapi-observer)"] --> C["OTel Collector"]
  C --> D["Tempo (traces)"]
  C --> E["Loki (logs)"]
  A --> F["Prometheus (/metrics scrape)"]
  F --> G["Grafana"]
  D --> G
  E --> G

The bundled manifests in examples/k8s deploy this stack with one command. See the Kubernetes guide.

Minimal collector config

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    limit_mib: 512
    spike_limit_mib: 128
    check_interval: 5s
  batch:
    send_batch_size: 512
    timeout: 5s

exporters:
  otlphttp/tempo:
    endpoint: http://tempo:4318
  otlphttp/loki:
    endpoint: http://loki:3100/otlp

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/tempo]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/loki]

Failure modes

Failure mode Expected behavior Immediate action
OTel Collector down App still serves traffic; local logs still available if OBS_OTEL_LOGS_MODE=both Fail over Collector or temporarily switch to local-json mode
Tempo down Traces unavailable; logs/metrics continue Restore Tempo, keep incident correlation via logs
Loki down Logs unavailable in Grafana; metrics/traces continue Restore Loki, use app stdout logs temporarily
Prometheus down No metrics/alerts; app traffic unaffected Restore Prometheus and alertmanager path
High cardinality on paths Prometheus pressure increases Use route templates and exclude noisy paths
Spoofed forwarded headers Incorrect client IP/request ID trust Tighten OBS_TRUSTED_CIDRS and proxy chain config

OTLP data sent during a Collector outage is lost; put durable retry queues in the Collector. A full log queue drops records too: alert on fastapiobserver_log_queue_dropped_total.

Redaction presets cover the library's log output. Span attributes and events follow the OpenTelemetry provider's policy, so check URLs, query strings and exception messages in traces before exporting them from a sensitive app.

Alert queries

Ready-to-import Prometheus alert rules covering all 10 diagnostic counters, 5xx error rates, log queue drops, and sink circuit breakers live in examples/full_stack/config/alerts.yml.

# 5xx fraction per service (5 minutes); add a minimum-traffic guard for alerts.
sum(rate(http_requests_total{status_code=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)

# p95 latency per service
histogram_quantile(
  0.95,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)

# Traffic drop detection
sum(rate(http_requests_total[5m])) by (service)

During an incident, the runtime control plane raises the log level or trace sampling ratio without a restart.