Distributed Tracing: How It Works and When You Need It.

When a page takes four seconds to load and the request touched a gateway, three services, a cache and two databases, logs from each component tell you a little and none of them tell you where the four seconds went. Distributed tracing records one request's whole journey, so the slow or failing step stands out.

Key takeaways

• A trace is one request's journey; it is made of spans, one per operation.
• Context (trace ID and parent span ID) travels between services in request headers.
• Sampling keeps cost under control - keep all errors and slow requests, sample the rest.
• OpenTelemetry provides vendor-neutral instrumentation for most languages.
• Tracing explains slowness inside your system; uptime checks confirm users can reach it at all.

Traces and spans

A span records one unit of work: an HTTP call, a database query, a cache lookup. Each has a start time, a duration, a name, attributes (such as http.status_code or db.statement) and a status. Spans nest: the span for "handle checkout" contains child spans for "reserve stock", "charge card" and "send receipt".

GET /checkout                          1,840 ms
├── auth.verify_token                     12 ms
├── inventory.reserve                    210 ms
│   └── SELECT ... FROM stock             190 ms
├── payments.charge                    1,520 ms   <- the slow part
│   └── POST gateway /charge            1,505 ms
└── email.queue_receipt                    9 ms

Reading a trace like this, the conversation moves from "the site is slow" to "the payment gateway call takes 1.5 seconds" in one look.

Context propagation

For spans from different services to join into one trace, each outgoing request carries the trace context. The W3C standard header looks like this:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

It contains the trace ID, the parent span ID and a sampling flag. Instrumentation libraries add and read it automatically for common HTTP clients, servers and message queues. A component that drops the header breaks the trace into pieces - a frequent cause of "missing" spans.

Sampling

Tracing every request in a busy system is expensive. Common strategies:

• Head sampling - decide at the start (for example keep 10%). Simple, but may drop the one interesting request.
• Tail sampling - decide after the trace completes, keeping all errors and slow traces plus a small share of normal ones. More useful, needs a collector that buffers traces.

Instrumenting with OpenTelemetry

OpenTelemetry (OTel) offers SDKs and automatic instrumentation for most languages and frameworks. A typical setup:

1. Add the OTel SDK and auto-instrumentation for your framework.
2. Set the service name and environment as resource attributes.
3. Export to an OpenTelemetry Collector, which forwards to the tracing backend you choose.
4. Add manual spans around important business operations.

export OTEL_SERVICE_NAME=checkout
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317

Making traces useful

• Log the trace ID in every log line so you can jump from a trace to its logs.
• Name spans by operation, not by value: GET /orders/{id}, not GET /orders/58114.
• Record errors on spans with the exception type and message.
• Don't put secrets in attributes - query strings and headers often contain tokens.

Where tracing stops

Traces start when a request reaches your instrumented code. They can't show a DNS failure, an expired certificate or a network problem that stops requests from arriving. Keep external uptime and API monitoring in place for that outside view, and use the response times it records from several cities to see whether slowness is inside your system or on the way to it.

Frequently asked questions

• Do I need tracing for a monolith?

  +

  Less urgently. A single application can often be diagnosed with good logs and a profiler. Tracing pays off once requests cross processes - queues, separate services, external APIs.

• Does tracing slow the application down?

  +

  The overhead of modern instrumentation is usually small, and sampling keeps export costs low. Measure it in your own environment before rolling out widely.

• What is the difference between a trace ID and a request ID?

  +

  They play the same role - linking everything one request did. A trace ID follows a standard format and is understood by tracing tools; many teams simply use it as their request ID.