Understanding spans in distributed tracing

A distributed tracing span is the atomic unit of work in a trace: one named, timed operation with a start timestamp and a duration. When a request touches your API gateway, an auth service, two databases, and a message queue, each of those operations becomes a span, and the collection of spans becomes the trace.

Engineers use the trace to see that a request was slow; they use the spans to identify the slow operation, the service, the parameters, and the execution order.

For platform engineers debugging microservices, this granularity is the difference between “checkout is degraded” and, for example, “the inventory service’s PostgreSQL query added 400ms because a connection pool was exhausted.”

What is a span in distributed tracing

A span records one operation, such as an HTTP handler executing or a database query running. The Trace API specifies that a span encapsulates a name, an immutable SpanContext, a parent reference, a SpanKind, start and end timestamps, attributes, links, events, and a status. That structure lets a backend reassemble thousands of operations, emitted by services that never talk to each other directly, into one coherent picture of a request.

In a monolith, a stack trace shows you the call path. In microservices, the call path crosses process and network boundaries, so no single stack trace exists. Spans reconstruct it: each service records its own operations, stamps them with a shared trace ID, and the backend stitches the pieces together.

Span vs. trace

A trace is the full end-to-end journey of a request; a span is one operation within it. Every span in a trace carries the same trace ID, and each span (except the root) carries a parent span ID pointing at the operation that invoked it. Because each span has zero or one parent, the parent/child relationships alone form a rooted tree.

Batch and async work need more than that tree, so OpenTelemetry defines a trace as a DAG of spans. Span links supply the extra causal edges: a batch consumer span can link to the SpanContexts of every producer message it processed, including spans in entirely different traces. The parent field stays a tree; links make the full causal graph a DAG.

Anatomy of a span

The OpenTelemetry API separates what you must provide at creation from the fields a span contains once it exists. At span creation, the Trace API requires only the span name. Parent context, SpanKind, attributes, links, and start timestamp are optional; the end timestamp is set via the End operation. Once created, the span has a SpanContext containing its trace ID and span ID.

Timestamps must support nanosecond precision at maximum and millisecond at minimum. In the OTLP wire format, trace_id, span_id, and name are the required fields; parent_span_id is empty for root spans.

When to use spans

Spans earn their keep whenever one request crosses a service boundary. A p99 regression in a monolith is a profiler problem; a p99 regression across twelve services is a correlation problem, and spans are the correlation mechanism.

A log line lives inside one service and carries no built-in identity connecting it to the request that produced it; an engineer investigating a slow checkout greps five services’ logs and joins them by timestamp and guesswork.

Spans carry the trace ID with them, so the query “show me everything this request touched” returns a complete, ordered answer.

Instrumenting spans with OpenTelemetry

OpenTelemetry defines the de facto standard for span structure and semantics. Its Trace API specifies the fields above, and its semantic conventions (current release v1.43.0) standardize attribute names like http.request.method and db.system.name so spans stay queryable across vendors and backends.

You get spans into existence three ways:

• Auto-instrumentation: OTel agents and SDKs wrap known frameworks (HTTP servers, database drivers) and emit spans with zero application code changes, though launch configuration changes and restarts are usually required, such as the Java -javaagent flag.
• Manual instrumentation: You call the SDK directly to create custom spans and attach business-logic attributes that no automatic layer can derive. OBI limits apply here: eBPF capture alone cannot produce custom spans or application-specific attributes.
• eBPF capture: Kernel-level instrumentation observes protocol traffic (HTTP, gRPC, database wire protocols) and emits spans without touching application code at all, covering languages that resist SDK instrumentation.

groundcover’s Flora eBPF sensor takes the third path: it captures spans at the Linux kernel level with zero code changes, covering every service on the cluster including Go and Rust workloads. Flora covers C++ services too. Services that already use OTel SDKs keep their spans too. groundcover ingests OTel spans as a first-class data source alongside the eBPF-captured signals, so SDK spans with custom business attributes and Flora spans coexist in the same trace view.

Visualizing spans

The waterfall diagram, a Gantt-style timeline, is the dominant span visualization. Each span renders as a horizontal bar positioned by its start timestamp and sized by its duration, with children indented under parents.

Overlapping bars mean parallel execution; a staircase of non-overlapping bars means sequential calls, which often indicates suboptimal execution that could be parallelized. Gaps between a parent and its children point at uninstrumented code or waiting time.

Jaeger enables critical path highlighting by default in its timeline view, and Grafana renders the critical path as a darker segment in its trace view. The darker path marks the chain of spans where a differential increase in segment time would increase end-to-end latency by the same amount. Tracing UIs also offer a flame graph view of the same hierarchy, with width representing duration.

Common issues and pitfalls

Most span problems come from three failure modes:

• Losing cardinality control
• Misclassifying status
• Sampling away the evidence you need during an incident

High-cardinality span naming

The span name should identify a class of operations, not an instance. The Trace API is explicit: “‘get_user’ is a reasonable name, while ‘get_user/314159’, where ‘314159’ is a user ID, is not a good name due to its high cardinality.” Dynamic values belong in attributes, including user IDs and invoice numbers. Raw request paths belong there too, where observability backends can support and query high-cardinality values.

The HTTP semantic conventions enforce this with route templates: server span names should use http.route values like /users/{id}, and instrumentation “MUST NOT default to using URI path” as the target.

Grafana Tempo documents that high-cardinality span names in generated metrics increase active series counts and storage costs, because every unique name becomes its own time series. Datadog enforces a hard limit of 1,000 unique resources per environment, service, and operation name.

Misusing span status

Span status is a three-value field, Unset, Ok, and Error, and it is deliberately separate from HTTP status codes. Unset is the default and usually the correct final state: the spec says instrumentation libraries “SHOULD leave the status code as Unset unless there is an error,” and Ok is reserved for cases where an application developer or operator has explicitly validated success.

The HTTP semantic conventions explain why the separation exists. For a 4xx response, HTTP semantic conventions require server instrumentation to leave status unset, because a bad client request is not a server failure, while client instrumentation should set Error, because the client’s call did fail. Instrumentation that mechanically maps every non-2xx code to Error pollutes error-rate dashboards with intentional behavior like validation rejections and 404s for optional resources.

Sampling and lost signal

Teams use sampling primarily to control cost and volume, and the two standard strategies trade different failure modes:

• Head-based sampling: The keep-or-drop decision happens in the SDK, at span creation, before the outcome of the request is known. It is cheap and simple, but it cannot guarantee capture of errors or latency outliers because neither exists yet at decision time. Datadog’s default is head-based at 100 traces per second per instance.
• Tail-based sampling: The decision happens on the collection path after spans complete, so policies like “keep every trace with an error status” become possible. The price is operational weight: the OTel Collector’s tail sampling processor buffers traces in memory (30-second decision_wait by default) and requires all spans of a trace to reach the same collector instance, which forces trace-ID-aware load balancing across a two-tier collector deployment.

Either way, a sampled-out trace is unrecoverable at 2 AM. groundcover takes a different path: Flora captures spans without sampling, so engineers do not have to decide in advance which failures are worth retaining. The data plane runs in your environment: traces, logs, and Kubernetes events stay in ClickHouse inside your VPC, while metrics stay in VictoriaMetrics. If you want to validate that on your own cluster, deploy Flora on one cluster and compare the trace view against your sampled pipeline.

The span hierarchy

The root span is the entry point of the trace: the one span with no parent span ID, typically the SERVER span where the request first entered your system. Its duration bounds the whole trace, and every other span nests somewhere beneath it.

Each child span records the ID of the span that caused it. When service A calls service B, A’s CLIENT span becomes the parent of B’s SERVER span, and B’s internal work produces further children under that.

The backend needs nothing more than these parent IDs to reconstruct the full call tree. In the checkout example, the gateway’s CLIENT span parents the inventory service’s SERVER span, and the PostgreSQL query span sits one level below that.

Types of span metadata and relationships

Span metadata explains what happened during an operation. Span relationships explain why operations belong together.

Span attributes

Attributes are key-value pairs describing the whole operation: HTTP method, response status code, database system, table name. Keys must be non-empty strings; values can be strings, booleans, doubles, 64-bit integers, or homogeneous arrays, and the SDK default caps a span at 128 attributes.

Because the spec standardizes http.request.method (required on both client and server spans) and db.system.name (required, stable for PostgreSQL, MySQL, MariaDB, and Microsoft SQL Server), a query for slow PostgreSQL operations works identically whether the spans came from a Java SDK, a Python agent, or an eBPF sensor. Attribute names should be constants rather than dynamically constructed strings, and values should stay bounded, with strings over 1 KB belonging in log bodies instead.

Span events

An event is a timestamped annotation inside a span, for occurrences that deserve their own timestamp but not their own span. Exceptions are the canonical case; the spec defines RecordException as a specialized event with the required name "exception". Retries and cache misses fit the same pattern. State transitions do too.

The decision rule from the OTel semantic conventions:

• Span for an operation with duration and a meaningful boundary
• Event for a distinct occurrence within it
• Attribute for a property of the whole operation that needs no timestamp

Span links

A link associates a span with one or more other SpanContexts, in the same trace or a different one, implying causality without claiming parenthood. Links are the correct model when strict parent-child breaks down.

The messaging semantic conventions make links the default producer-consumer correlation mechanism for a structural reason: a span can only have a single parent, so a batch consumer processing, say, 50 messages cannot be a child of 50 producers, but it can link to all 50. The scatter-gather pattern works the same way, with the aggregating span linking to every forked operation. One operational constraint: record links at span creation, because samplers inspect links only when they are present at span creation.

Span kind

SpanKind declares the span’s communication role so backends can assemble the call graph correctly. The five values encode direction and interaction style: CLIENT (outgoing, awaits response), SERVER (incoming, client awaits), PRODUCER (outgoing, deferred), CONSUMER (incoming, deferred), and INTERNAL, the default, for operations with no remote parent or child.

Backends depend on this field. Grafana Tempo’s service graph processor forms service-to-service edges from CLIENT/SERVER pairs and messaging edges from PRODUCER/CONSUMER pairs, and it excludes INTERNAL spans from edge formation entirely. Jaeger’s monitoring queries default to server spans specifically to avoid double-counting the same RPC from both sides. Mislabel your kinds and your service map lies to you.

Span context and context propagation

SpanContext is the portable, immutable identity of a span: the trace ID, the span ID, trace flags (currently Sampled and Random), and trace state. This small structure crosses process boundaries, and downstream services must be able to trust it, so it cannot be modified.

When a service makes an outbound call, the tracer injects the current SpanContext into the request headers; the receiving service extracts it, marks it remote (the spec requires IsRemote to return true for extracted contexts), and creates its SERVER span as a child of the caller’s CLIENT span.

Break this chain anywhere, a service mesh dropping unknown headers, a Kafka message published without context, an uninstrumented middle service, and the trace fractures into disconnected fragments with useless hierarchy. Flora observes propagation headers at the kernel level for the protocols it covers, so those services need no injection code.

W3C Trace Context and baggage

The W3C Trace Context Recommendation standardizes the header format so vendors and services interoperate. The traceparent header packs four fields into one lowercase hex string:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

That is version (2 hex chars), trace-id (32 chars), parent-id (16 chars), and trace-flags (2 chars, where bit 0 is the sampled flag). All-zero trace or parent IDs are invalid. The companion tracestate header carries vendor-specific key-value pairs, at most 32 list members, and it must not be parsed if traceparent fails validation.

Baggage rides alongside as a separate W3C specification for propagating business metadata: user IDs and tenant identifiers. Origin IPs are another example of the context available at the start of a request that downstream services would otherwise lack.

Baggage uses a separate key-value store, unassociated with span attributes unless you explicitly copy entries onto spans. And it travels in plaintext HTTP headers to every downstream service, so the OTel docs warn against putting credentials or API keys in it. PII does not belong there either.

Related concepts

Concepts that sit next to spans:

• Distributed tracing: Correlating spans across services into request-level views.
• OpenTelemetry: The standard defining the span data model, SDKs, and semantic conventions.
• Service maps: Topology graphs generated from span parent-child and kind relationships.
• Golden signals: RED (rate/error/duration) metrics, computed from span streams.
• APM: The product category built on span collection and analysis.
• eBPF instrumentation: Kernel-level span capture without application code changes, the mechanism behind Flora.

FAQ

How does a span differ from a trace? A trace is the complete journey of one request across all services; a span is a single operation inside it. All spans in a trace share one trace ID, and parent span IDs arrange them into a tree rooted at the entry-point operation.

What fields make up a span? A name (the only required creation parameter), trace ID, span ID, optional parent span ID, start and end timestamps, a SpanKind, attributes, events, links, and a status. Trace IDs are 16 bytes and span IDs 8 bytes, both rendered as lowercase hex.

How does the span hierarchy work? The root span has no parent ID and anchors the trace at the request’s entry point. Every other span carries the ID of the span that invoked it, which is all a backend needs to rebuild the call tree.

How does context propagation carry span identity across services? The outbound service injects its SpanContext into request headers (the W3C traceparent format), and the receiving service extracts it and parents its new span to the caller’s. Any hop that fails to propagate the header breaks the trace at that boundary.

What is the difference between span metadata and links? Attributes describe the whole operation with key-value pairs. Events mark timestamped moments within it, like an exception. Links connect the span causally to spans in other traces or branches, which parent-child relationships cannot express.

How does span kind affect visualization? Backends pair CLIENT with SERVER and PRODUCER with CONSUMER to draw service-map edges, and INTERNAL spans contribute none. A mislabeled kind shows up as a missing or duplicated edge.

How does OpenTelemetry standardize spans? The OTel Trace API defines the span data model and SDKs. OTLP defines the wire format, and the semantic conventions standardize attribute names like http.request.method so spans from any SDK, agent, or eBPF sensor stay queryable in any compliant backend.