Distributed Tracing

A single request in a microservices system rarely stays within one service — it might hit an API gateway, call three backend services, each of which queries its own database or calls another service in turn. When that request is slow, or fails, "which of these dozen hops was the problem?" is not a question logs on any one service can answer alone, because no single service's logs show the whole journey. Distributed tracing solves exactly this: it follows one request across every service it touches and reconstructs the full picture.

Trace IDs and spans

The mechanism is conceptually simple: when a request first enters the system, it's assigned a unique trace ID. That ID is passed along in headers on every subsequent call the request triggers — every service it reaches propagates the same trace ID to whatever it calls next. Each individual unit of work (one service handling its part of the request) is recorded as a span, with a start time, an end time, and a reference to its parent span, so the full set of spans sharing a trace ID can be reassembled into a single timeline.

System Design Lab

Reassembled this way, a trace turns into exactly the kind of waterfall diagram that immediately shows where a slow request's time actually went — in the example above, the Orders Service span (45ms) is dominated almost entirely by its own database query (38ms), which is precisely the detail a single service's isolated logs would show clearly, but that a caller of Orders would have no way to see on its own.

Why this is a genuinely new problem in microservices

A monolith doesn't need this: one process, one stack trace, one set of logs already shows a request's entire path through the code. Microservices Architecture trades that away deliberately in exchange for independent deployability and scaling — but one direct cost of that trade is that a request's path is now scattered across many separate services' separate logs, with nothing connecting them by default. Distributed tracing is the tool that reintroduces the visibility a monolith gave away for free, by threading one identifier through every hop.

Where trace context actually gets attached

Since a trace ID has to propagate through every hop a request makes, it naturally rides along wherever APIs already carry metadata — as HTTP headers on service-to-service calls, and as message attributes when a request's processing continues asynchronously through a message queue. An API Gateway, sitting at the entry point of nearly every request, is a natural place to originate a trace ID for requests that don't already carry one, for the same reason it's a natural home for rate limiting and circuit breaking — it's already on the path of every request, so it's a convenient single place to inject cross-cutting behavior instead of every service reimplementing it.

Tracing vs. logging vs. metrics

It's worth being precise about how tracing complements, rather than replaces, the other two pillars of observability: metrics answer "is the system healthy right now, in aggregate" (error rates, latency percentiles); logs answer "what happened in this one service, in detail"; traces answer "what was this one specific request's full path, and where did its time go." A slow-response alert typically comes from metrics, but diagnosing why that specific class of request is slow is exactly what a trace is built to show, by making the request's cross-service path visible as one connected timeline instead of scattered, unlinked log lines.

Why this matters in an interview

In any design with more than a couple of services calling each other, proactively naming distributed tracing as the answer to "how would you debug a slow or failing request in production" is a strong, concrete signal — it's a specific, well-known answer to a question that a design without it tends to leave unaddressed. Naming trace IDs and spans specifically, and where they get propagated (headers, message metadata), shows the mechanism is understood, not just the term.