0014: Diagnosis works without tracing, and is on where it's cheap¶
Status: accepted · Reviewed: 2026-11-01 · Narrows 0006 for SQLAlchemy query counts
Context¶
The request line's db totals and the db.n_plus_one warning came from OpenTelemetry's client spans. They worked only with tracing on, only for sampled requests, and only where SQLAlchemy's instrumentation runs, which is not on SQLAlchemy 2.1 (opentelemetry-instrumentation-sqlalchemy 0.66b2 supports sqlalchemy < 2.1.0).
The event-loop watchdog and held DEBUG logs were opt-in everywhere, so the library reported a blocked loop only to someone who already suspected one.
Decision¶
- Queries on the SQLAlchemy engines passed as
db_engine, or set up withinstrument_sqlalchemy(), are counted from SQLAlchemy'sbefore_cursor_executeandafter_cursor_executeevents (db_queries.py). The span processor skips spans from SQLAlchemy's instrumentation, so a query counts once. Other clients, such as Redis or HTTP, are still counted from their spans. - The
productionprofile turns the watchdog on at 250 ms. - Held
DEBUGlogs (OBS_LOG_LEVEL_ON_ERROR) stay opt-in, in every profile.
Evidence¶
- Tests: in
test_request_insights.py,test_sqlalchemy_queries_are_counted_and_n_plus_one_reported_without_tracing,test_async_sqlalchemy_queries_are_counted_without_tracingandtest_sqlalchemy_queries_count_once_with_tracing_on; the profile intest_production_profile_enforces_strictintest_profiles.py. - The watchdog costs nothing measurable per request.
request_cpu.pywatchdogagainstdefault, three runs each on an Apple M1: 51.6 to 52.4 µs of CPU per request against 51.8 to 52.2. Idle, it uses about 2.6 ms of CPU per second, 0.25% of one core per worker, from waking every 125 ms. - Each held
DEBUGrecord costs about 4 µs.request_cpu.py held absoluteagainstdefault absolute: 75.6 to 82.7 µs with oneDEBUGline per request, and 81.0 to 158.3 µs with twenty (DEBUG_LINES=20), about whathttpcorewrites for two outbound calls. The cost follows how much every library in the app logs atDEBUG, which the library can't bound.
What it cost¶
- An engine traced with OpenTelemetry's
SQLAlchemyInstrumentordirectly, and not passed asdb_engineor throughinstrument_sqlalchemy(), is no longer counted, since its spans are skipped. - The library depends on SQLAlchemy's event names, which have been stable since 0.9.
- Production apps run one more thread per worker, and log
event_loop.blockedfor any stall over 250 ms, including a CPU-heavy endpoint written asasync def. SettingOBS_LOOP_BLOCK_THRESHOLD_MSchanges or, with a large value, effectively silences it.