Adding OpenTelemetry to FastAPI is not mainly a package-installation task. It is the design of a bounded telemetry path: the application creates standards-based HTTP spans, important business operations add a few manual spans, OTLP carries batches to a Collector, and the Collector filters, samples, and exports them. A production rollout is complete only after context propagation, errors, background work, cardinality, privacy, backpressure, and shutdown have all been tested.

What problem should FastAPI tracing solve?

An API request can cross a reverse proxy, FastAPI middleware, MongoDB, Redis, an HTTP dependency, and a model endpoint. A log line saying “request took 800 ms” cannot identify which segment consumed the time, whether a retry occurred, or whether the error came from the local process. Distributed tracing models the entire request as a trace and each observable operation as a span. Propagation fields such as traceparent preserve the relationship as work crosses process boundaries.

The OpenTelemetry HTTP semantic conventions define the expected names and attributes for HTTP client and server spans. Reusing those conventions makes queries portable across services and backends. Avoid inventing nearly equivalent fields such as http_method_name or putting concrete URLs in span names. If your application still mixes request work with unbounded in-process jobs, first separate those lifecycles using the FastAPI background-task architecture guide; otherwise, tracing will document an ambiguous execution model rather than fix it.

How should automatic and manual instrumentation divide the work?

Automatic instrumentation is best for framework facts: route template, method, status, duration, propagation, and standard exceptions. Manual spans are best for business operations such as selecting retrieval candidates, invoking a model, or committing a publication snapshot. A useful service normally uses both. Framework-only traces contain many HTTP spans but hide the operation users care about. Instrumenting every function manually creates noise, inconsistent attributes, and extra maintenance.

The OpenTelemetry FastAPI instrumentation reference documents FastAPIInstrumentor.instrument_app(app), excluded URLs, request and response hooks, and header sanitization. A small explicit setup looks like this:

from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

resource = Resource.create({"service.name": "catalog-api"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(endpoint="http://otel-collector:4318/v1/traces")
    )
)
trace.set_tracer_provider(provider)

app = FastAPI()
FastAPIInstrumentor.instrument_app(
    app,
    excluded_urls="/healthz,/metrics",
    http_capture_headers_sanitize_fields=["authorization", "cookie", "set-cookie"],
)
tracer = trace.get_tracer("catalog-api")

BatchSpanProcessor keeps request handlers from synchronously waiting on every export. service.name gives backends a stable aggregation dimension. The OpenTelemetry Python exporter guide recommends sending OTLP to a Collector in production and documents HTTP/protobuf and gRPC exporters. The endpoint above is an architectural example, not a claim about a measured deployment.

Pin the SDK, exporter, and instrumentation versions together. The core API and SDK can have a different stability level from contrib instrumentation and semantic conventions. Review release notes before an upgrade, especially when an instrumentation is migrating between semantic-convention versions. A lockfile prevents accidental upgrades; a trace-contract test prevents a deliberate upgrade from silently changing dashboards.

Which business operations deserve spans?

A span should represent a unit that can fail or become slow independently. It should not mirror every function call. Use a low-cardinality operation name such as catalog.search, then attach bounded attributes that help explain the result.

@app.get("/search")
async def search(q: str):
    with tracer.start_as_current_span("catalog.search") as span:
        span.set_attribute("search.mode", "hybrid")
        span.set_attribute("search.query_length", len(q))
        try:
            rows = await repository.search(q)
        except TimeoutError as exc:
            span.record_exception(exc)
            span.set_status(trace.Status(trace.StatusCode.ERROR))
            raise
        span.set_attribute("search.result_count", len(rows))
        return {"items": rows}

Do not attach the raw query, authorization headers, cookies, complete SQL statements, model prompts, or retrieved documents merely because the API allows attributes. Span data is commonly indexed, copied, and retained outside the application database. The application team remains responsible for classifying that data and limiting access.

Long-lived responses need another boundary decision. A single span that stays open for an entire streaming session may be useful for connection duration, but appending an event for every message can create an unbounded payload. The FastAPI SSE production guide describes connection lifecycle, disconnect detection, and buffering constraints that should inform whether you create one session span, bounded child spans, or metrics instead.

Why put a Collector between the application and the backend?

Direct-to-vendor export is convenient for a demo, but it puts retry policy, credentials, sampling, and backend coupling into every application process. A Collector centralizes receivers, processors, exporters, and pipelines. The Collector configuration guide also makes an important operational point: defining a component does not enable it; the component must appear in a service.pipelines entry.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128
  batch: {}

exporters:
  otlp:
    endpoint: tracing-backend:4317
    tls:
      insecure: false

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]

This is a shape, not a production-ready file. Official examples sometimes bind to 0.0.0.0 for convenience, while the configuration guide warns that localhost is preferable when all clients are local. Production also needs authentication, TLS validation, network policy, resource limits, queue monitoring, and a deliberate failure policy.

Keep the telemetry failure domain separate from request-serving state. For example, do not let an overloaded Collector exhaust the same Redis instance used by the online application. The Redis 8.6 upgrade guide explains eviction and operational gates that are relevant when reviewing shared infrastructure, but telemetry buffering should still have its own capacity plan.

How should sampling be designed?

Recording everything is useful during a short test, not automatically a sustainable production policy. Head sampling decides near the beginning of a trace. It is efficient and easy to reason about, but it cannot know that a later span will fail or that total latency will cross a threshold. Tail sampling waits for most or all spans and can keep errors or slow traces, but it requires state, memory, and a centralized decision point.

The OpenTelemetry sampling guide describes these tradeoffs and warns that poor sampling can remove the evidence you wanted to preserve. A practical sequence is:

  1. Temporarily record all test traffic and validate the trace structure.
  2. Start production with consistent probability head sampling and measure volume and cost.
  3. Add Collector-based tail sampling only when requirements justify preserving errors, high latency, or specific release cohorts.
  4. Run a fixed request set after every rule change to detect bias by route, tenant, or status.

Do not apply independent random sampling decisions at several layers. That can leave fragments rather than complete traces. A ten-percent sampling rate also does not guarantee that ten percent of every rare failure will be captured; a low-frequency error can disappear for a long interval.

How do you control sensitive data and cardinality?

Cardinality explodes when values such as resource IDs, user IDs, task IDs, raw exception messages, or full URLs become span names or indexed attributes. Use a route template such as /users/{user_id}, not /users/84721. If a business identifier is genuinely needed for correlation, consider a stable nonreversible hash or keep it in a more tightly controlled log store.

The OpenTelemetry sensitive-data guidance describes attributes, filter, redaction, and transform processors. Prefer an allowlist: enumerate the headers and business fields that may leave the process, then drop the rest. A denylist tends to miss the next authentication field introduced by another team.

Review at least these sources:

  • authorization, cookie, set-cookie, and API-key headers;
  • URL query strings, database parameters, and personal data in exception text;
  • model inputs, retrieved passages, and uploaded filenames;
  • identifiers such as user.id and tenant.id that create unbounded combinations;
  • Collector debug logs that might print data rejected from the export path.

Sanitizing captured headers is only one layer. A request hook can still add a secret as a custom attribute, a database instrumentation can capture a statement, and an exception can contain an input value. Test the final exported payload, not just the application configuration.

Why do background tasks and retries produce broken traces?

The server span can finish when the response is sent, while a BackgroundTasks callback continues. Treating that callback as ordinary request code can produce a child whose parent already ended, an unexpectedly long request span, or an orphan. For short in-process work, capture the current context and create a dedicated span when the task executes. For a queue, inject propagation context into message metadata and extract it in the worker. For an unrelated scheduled job, start a new trace and use a link if a historical request is relevant.

Retries also need explicit semantics. One parent operation can contain several client-attempt spans, or the retry count can be a bounded attribute. Hiding all attempts inside a single duration makes a slow dependency look like a slow local function. Creating a new trace for every attempt loses the reason those attempts belong together.

Status handling should follow the service contract. An unhandled exception normally records an error, but a business validation response such as 409 may be expected. Conversely, an application that converts a downstream failure into HTTP 200 must still record that the operation failed. Define the mapping once and enforce it with tests.

What should a production verification suite assert?

Prepare repeatable requests for a normal response, validation failure, unhandled exception, downstream timeout, concurrent calls, background work, and client cancellation. Assert the trace data rather than checking that “something appeared” in a UI.

Gate Passing evidence
Propagation Server, database, cache, and HTTP-client spans for one call share a trace ID
Parenting Retries and background work have the intended parent or link
Errors Exceptions and status follow policy without leaking request bodies
Naming Span names and indexed attributes remain low-cardinality
Privacy Tokens, cookies, personal data, prompts, and retrieved text are absent
Backpressure A Collector outage does not cause unbounded request latency or memory growth
Sampling A fixed corpus demonstrates the intended error and slow-trace retention
Shutdown The process performs a bounded flush and can still terminate

Test propagation through every client library that matters. An incoming trace can be correct while an HTTP or queue client fails to inject context. Check cancellation because abandoned streaming requests often take a path that ordinary response tests miss. Check concurrency because context stored in a global variable rather than a context-local mechanism may cross-contaminate requests.

Finally, monitor the telemetry path itself: export failures, dropped spans, queue depth, Collector memory, refused data, and backend throttling. A trace pipeline is another distributed system. If it has no capacity, privacy, recovery, and versioning gates, it can fail precisely when the application needs evidence most.

Common questions

Does automatic instrumentation change application behavior?

It normally observes framework calls through middleware or monkey patching, but it is still runtime code. Pin versions and regression-test middleware order, exception behavior, latency, background tasks, and shutdown before enabling it broadly.

Must traces, metrics, and logs launch together?

No. Start with the signal that answers a concrete diagnostic question. Traces are often the first choice for cross-component latency and causality. Add metrics and logs when you have defined how they correlate and who will operate them.

Should a trace ID be returned to the caller?

A public, opaque request ID can be useful for support and may map internally to a trace. Whether the trace ID itself is safe to expose depends on the threat model and backend access controls. Never return internal tracing URLs, Collector credentials, or telemetry payloads.