APM tracing follows individual requests through an application to show where they slow down or fail. It captures each observed operation as a timed span and links related spans into a trace. When checkout latency rises, a trace can reveal whether the delay came from the payment service or an external API rather than simply showing that response times increased.
This guide explains how APM traces work, how to collect them with OpenTelemetry or eBPF, and how to investigate slow requests using waterfall views. It also covers sampling and retention, along with how groundcover connects traces with Kubernetes telemetry and AI-assisted debugging.
What Is an APM Trace?
An APM trace records the operations involved in an individual application request, while application performance monitoring (APM) tracks latency, error rates, and throughput across many requests. For example, metrics may show that p95 latency on /checkout doubled, but a trace can reveal that one checkout spent 1,100 ms in the payment service, including 1,080 ms across two calls to an external payment API.
Traces work at two scopes. A trace can capture operations inside a single service: a request handler, the database query it runs, and an internal function you chose to time. Distributed tracing extends the same trace across service boundaries, connecting operations in the API gateway, checkout service, payment service, and their dependencies. This is particularly useful in microservice applications, where a slow or failed request may involve several services.
How APM Traces Are Structured
A span represents one operation performed by one component. Each span carries identifiers, timing information, and metadata that allow a tracing backend to reconstruct the recorded request path:
A tracing backend groups spans by trace ID and connects them using their span IDs and parent relationships. Each span has at most one parent, while the root span has none. Start and end timestamps establish when operations occurred and how long they lasted, allowing the backend to display them on a timeline.
Span status is Unset by default. Instrumentation generally leaves it unchanged for successful operations and sets it to Error when an operation fails. The Ok status explicitly indicates successful completion, although instrumentation libraries generally avoid setting it automatically. For HTTP spans, a 4xx response normally leaves a SERVER span Unset but marks a CLIENT span as Error. A 5xx response generally marks both as Error, unless additional request context warrants a different classification.
Example: A Checkout Request Across Microservices
The following trace follows a checkout request through an API gateway, checkout service, inventory and pricing services, PostgreSQL, a payment service, and an external payment API. The timestamps and durations are illustrative, and the letters A through L identify spans in the example rather than representing their actual span IDs.
Every span shares trace ID 4bf92f3577b34da6a3ce929d0e0e4736. Span A is the root, while the parent references show how the remaining operations connect. In a waterfall view, these spans appear on a timeline, making overlapping calls, sequential operations, and long-running dependencies easier to identify. For example, inventory span D runs from 12 to 97 ms, while pricing span F runs from 14 to 74 ms. The two operations therefore consume 145 ms of combined span time but only 85 ms of elapsed time because they overlap. This also explains why the durations of checkout-service’s direct child spans total 1,271 ms, exceeding the parent span’s 1,224 ms.
The payment call accounts for most of the request’s latency. Checkout-service’s CLIENT span I lasts 1,104 ms, while payment-service’s SERVER span J lasts 1,100 ms. Within J, the first external API attempt (K) times out after 500 ms, and the retry (L) starts 5 ms later, taking another 580 ms before returning HTTP 503. Although K and L execute sequentially, they are siblings under J rather than parent and child. Together, their durations account for 1,080 ms of J’s 1,100 ms, leaving 20 ms outside those child spans that cannot automatically be classified as CPU processing time. The span hierarchy establishes relationships, while timestamps and application dependencies help identify which operations delayed the request.
How APM Tracing Works
When a request reaches an instrumented service, tracing starts a span for the incoming operation. Calls to supported databases, APIs, and downstream services create additional spans. Context propagation carries the trace ID and parent information between services, connecting those operations into a distributed trace. As spans finish, the instrumentation processes and exports them, typically using the OpenTelemetry Protocol (OTLP).
The spans can go directly to a tracing backend or through an OpenTelemetry Collector, which receives, processes, and forwards telemetry. In Kubernetes, a Collector running as a DaemonSet can receive data near each workload, while a gateway Collector handles shared processing, routing, or tail sampling. You can use both when local collection and centralized processing are needed.
Automatic and Manual Instrumentation With OpenTelemetry
OpenTelemetry provides APIs, SDKs, automatic instrumentation, exporters, and the Collector for generating and transporting telemetry. It is not a tracing backend: it does not store traces or provide waterfall views. You can send its spans to a backend such as Jaeger, where they become searchable and available for investigation. Automatic instrumentation captures operations from supported libraries and frameworks, while manual SDK instrumentation adds details specific to your application.
Automatic instrumentation creates spans for supported frameworks and libraries without changing application code. Each language has its own mechanism:
- Java: A Java agent JAR attaches to Java 8+ applications and injects bytecode into supported libraries.
- Python: The
opentelemetry-instrumentcommand wraps application startup and configures instrumentation through command-line flags or environment variables. - .NET: OpenTelemetry .NET Automatic Instrumentation collects traces and metrics without source-code changes, with supported instrumentations enabled by default.
- Node.js: Loading
@opentelemetry/auto-instrumentations-node/registerat startup, directly or throughNODE_OPTIONS, enables supported instrumentation libraries. - Kubernetes: The OpenTelemetry Operator injects automatic instrumentation into selected pods using an
Instrumentationresource and annotations such asinstrumentation.opentelemetry.io/inject-java: "true". Injection happens when an eligible pod is created, so existing pods must be restarted to receive it.
Manual instrumentation fills the gaps that automatic instrumentation cannot cover. Language agents can capture HTTP handlers, database drivers, and supported client libraries, but they do not know when your application calculates a discount, checks a customer’s subscription tier, or applies a business rule. You can create spans around those operations and attach attributes that explain what happened inside the application.
The following Python example creates a root span with two child spans and sends them to a local Jaeger backend over OTLP/HTTP. You’ll need Python 3.10 or newer and Docker for the verification step. Install the OpenTelemetry SDK and HTTP exporter:
Save the following code as app.py:
The nested start_as_current_span blocks create two child spans under POST /orders. The pass statements stand in for actual inventory and payment operations, so this example demonstrates the span hierarchy rather than reproducing the complete checkout trace from the previous section. BatchSpanProcessor queues completed spans for export, and provider.shutdown() flushes the queue before this short-lived script exits.
The exporter sends spans to http://localhost:4318/v1/traces, the standard OTLP/HTTP traces endpoint. OTLP/gRPC normally uses port 4317 and does not append an HTTP signal path such as /v1/traces.
To verify the exported spans, start a local Jaeger backend. Its all-in-one configuration accepts OTLP on ports 4317 and 4318, serves the interface on port 16686, and uses temporary in-memory storage:
Leave the container running, open another terminal, and execute the script:
Open http://localhost:16686 and search for checkout-service. You should find a trace containing POST /orders with the two child spans for inventory and payment. If the service does not appear, confirm that Jaeger is running, port 4318 is available, and the script completed without an export error. The in-memory storage is suitable for local verification but not for retaining production traces.
Trace Context Propagation Across Services
Context propagation connects spans created by different services into one distributed trace. When service A calls service B, A’s instrumentation injects the current trace ID and span ID into the outgoing request. B extracts that information and creates its server span with A’s span as the parent. OpenTelemetry uses tracecontext,baggage as its default propagation configuration, allowing compatible instrumentations to share request context.
For HTTP requests, the context travels through the W3C traceparent header. A version 00 header contains four fields separated by hyphens:
Each field has a specific purpose:
- Version (
00): Identifies the trace-context format using two hexadecimal characters. The valueffis invalid. - Trace ID (
0af7651916cd43dd8448eb211c80319c): A 32-character hexadecimal identifier shared by the operations in the trace. An all-zero value is invalid. - Parent ID (
b7ad6b7169203331): The 16-character hexadecimal span ID supplied by the caller. The receiving service uses it to establish the parent-child relationship. - Trace flags (
01): A bit field that communicates sampling information. Its least significant bit is the sampled flag, indicating that the caller may have recorded trace data.
The companion tracestate header carries additional vendor-specific tracing information, including metadata used by some sampling implementations. On public APIs, blindly honoring an incoming sampled flag can let callers increase tracing overhead or ingestion costs. Apply appropriate sampling controls and treat trace context from untrusted requests carefully.
Trace context also crosses non-HTTP boundaries. gRPC transports it through request metadata, while messaging systems such as Kafka carry it in message headers. Producers inject context and consumers extract it to associate downstream work with the originating operation. For batch processing, span links connect a consumer operation to multiple message creation contexts, since an individual span cannot have multiple parents.
When propagation fails, downstream operations may start a separate trace instead of joining the original one. The upstream trace might show a CLIENT span without a matching downstream SERVER span. Check that the caller injects context, the receiver extracts it, and proxies or other intermediaries preserve the required headers. For asynchronous consumers, also check that the message’s trace context reaches the handler.
eBPF Tracing Without Application Code Changes
eBPF tracing captures supported application and network operations without requiring SDK instrumentation in every service. Tracing agents load kernel probes and, where needed, user-space probes that observe network activity and functions in supported runtimes or libraries. In Kubernetes, a DaemonSet can run a sensor on each node to monitor multiple workloads without modifying their application code. This reduces instrumentation work, although the sensor still consumes CPU and memory, and visibility depends on supported protocols and environments.
OpenTelemetry eBPF Instrumentation (OBI) provides this collection approach across supported Linux applications. Its coverage, system requirements, and distributed tracing capabilities vary by protocol and runtime:
- Coverage: Supported protocols include HTTP/S, HTTP/2, gRPC, Kafka, NATS, MQTT, and SQL databases such as PostgreSQL and MySQL. Instrumentation supports applications written in Java, .NET, Go, Python, Ruby, Node.js, C, C++, and Rust, with feature-specific limitations.
- Requirements: Linux 5.8+ or supported RHEL-family 4.18+ kernels with the necessary backports, BTF support, and root privileges or the Linux capabilities required for the enabled features.
- Propagation limits: OBI reads supported incoming trace context, but outgoing network-level propagation must be enabled explicitly. Plaintext HTTP propagation can use standard headers, while network-level HTTPS propagation relies on TCP/IP metadata that other OBI-instrumented services can read. L7 proxies can break that connection. Go library-level instrumentation supports additional propagation paths, including injecting context before TLS encryption. Some incoming network-level parsing features also require Linux 5.17+ or an equivalent kernel backport.
- Business context: eBPF cannot automatically recover every application-specific operation or business attribute. Language agents and manual instrumentation are still useful for custom spans, business events, and operations that are not visible from the supported observation points.
Encrypted traffic requires a different observation point because kernel network probes normally see TLS ciphertext. For supported TLS libraries, eBPF tracers can attach user-space probes (uprobes) to functions such as OpenSSL’s SSL_read and SSL_write to observe plaintext before encryption or after decryption. This requires compatibility with the application’s TLS library and runtime. Trace context propagation is a separate concern because the ability to observe an encrypted request does not necessarily mean the tracer can inject context into it.
eBPF helps you identify which supported service call was slow or failed without changing the application. It cannot automatically tell you that applyLoyaltyDiscount() took 300 ms inside a request handler if that operation is not visible to its probes. groundcover uses an eBPF sensor deployed as a Kubernetes DaemonSet to collect supported service interactions and accepts OpenTelemetry spans alongside its own telemetry, allowing you to investigate network-level operations together with application-specific spans.
When Should You Use APM Tracing?
Use APM tracing when you know an application is slow or failing but need to identify which operation is responsible. Metrics can alert you to rising latency or error rates, while traces show the recorded operations behind individual requests. This is especially useful in microservice applications, where a request may pass through several services, databases, and external APIs before returning a response.
Tracing is also valuable when a problem affects only certain requests or is difficult to reproduce. Comparing successful and failed traces can expose differences in execution paths, repeated database queries, retries, or dependencies that become slow under particular conditions. These patterns help narrow the investigation before you change application code or infrastructure.
Diagnosing Microservice Latency and Failed Requests
Several common production problems become easier to investigate when you can follow the operations involved in each request:
- Slow downstream services: A request spends significantly longer than expected waiting for an HTTP, gRPC, or database operation. Its
CLIENTspan can help locate the dependency associated with the delay, while comparison with other traces shows whether the slowdown is isolated or widespread. - N+1 database queries: An application fetches a collection and then executes another query for each item. Instead of one query for the required related data, the trace shows many similar database spans under the same request. This pattern can explain why latency increases as the number of records grows.
- Unnecessary sequential calls: Independent operations execute one after another rather than concurrently. Their spans appear in consecutive intervals, allowing you to identify calls that could potentially run in parallel and reduce elapsed request time.
- Timeouts and repeated retries: Several attempts to contact the same dependency increase the request’s duration. When the instrumentation captures individual attempts, you can inspect their timing, error information, and attributes such as
http.request.resend_countto understand how retries contributed to the slowdown. - Cascading failures: A dependency becomes slow or overloaded, causing upstream requests to wait, time out, or retry. Those retries can increase pressure on an already struggling service. Comparing traces across affected services helps you follow the failures and determine where the disruption may have started.
Correlating Traces With Metrics and Logs
APM tracing becomes more useful when you can connect individual requests to the broader condition of the application. Suppose a latency alert shows that checkout-service p95 increased from 400 ms to 1.3 seconds while its error rate also rose. Those metrics establish the scope and timing of the problem. Traces collected during the affected period can then show whether the delay is concentrated in payment calls, database operations, or another dependency.
Logs provide details that span timing may not capture, such as exception messages, retry decisions, and application-specific errors. To correlate them reliably, include trace context in the log records. OpenTelemetry supports TraceId and SpanId fields, commonly represented as trace_id and span_id in structured application logs. Logging integrations can populate these values from the active trace context, allowing you to retrieve records associated with a particular request. Without those identifiers, you may need to rely on timestamps and service metadata, which are less reliable when many requests execute concurrently.
Using Profiles and Infrastructure Metrics to Explain Slow Spans
A long-running span does not necessarily mean the application spent that time executing code. A payment operation might take one second because it is waiting for an external response, while another operation could be delayed by CPU contention, lock waits, or memory pressure. CPU profiles show where processor time is spent, while off-CPU profiling can help reveal blocking and waiting behavior. Where profile samples contain trace and span identifiers, you can correlate code-level activity with individual traced operations.
Infrastructure metrics provide additional context. In Kubernetes, you can examine CPU throttling, memory pressure, pod restarts, and node-level resource usage to determine whether the slowdown is associated with the application or its runtime environment. Comparing these measurements with the affected traces helps distinguish resource constraints from slow downstream dependencies.
Metrics identify the degradation, traces narrow it to the affected request path, logs explain recorded application events, and profiles or infrastructure data help establish why an operation took longer than expected. With that context, you can examine individual span timelines and attributes to investigate the failure.
How to Read and Troubleshoot an APM Trace
Reading an APM trace starts with identifying where a slow or failed request spent its time. The span timeline shows which operations ran, their durations, and how they relate to one another. You can then examine errors, compare other requests, and correlate logs or infrastructure metrics to determine why the delay or failure occurred.
Understanding Waterfall Views, Trace Explorers, and Service Maps
Tracing backends provide different views for investigating request performance. The right one depends on whether you’re examining a specific request, searching for affected requests, or investigating dependencies across your application.
A waterfall view arranges spans along a time axis, with their parent-child relationships shown as nested operations. Each bar represents the elapsed duration of an operation. Overlapping sibling bars show operations whose lifetimes overlapped, while consecutive bars can reveal repeated or sequential calls that increase request latency. Error markers help identify failed operations, although their appearance depends on the tracing backend.
When investigating latency, focus on operations that delay the request’s completion rather than simply choosing the longest span. In the checkout example, inventory and pricing run during overlapping intervals, finishing at 97 ms and 74 ms respectively. If checkout must wait for both responses, reducing pricing latency alone would not shorten that wait because inventory finishes later. The payment call is more significant: it takes 1,104 ms of the root request’s 1,240 ms. Identifying the exact critical path also requires understanding which operations depend on one another, since parent-child relationships and timestamps do not capture every execution dependency.
A trace explorer helps you find representative requests before opening their waterfalls. Filter by service, operation, duration, error information, or span attributes to compare slow and successful requests over the same period. Service maps provide a broader view by summarizing communication between services across many recorded requests. Where performance metrics are available, they can also help identify dependencies associated with increased latency or errors.
Trace waterfalls should not be confused with CPU flame graphs. A span’s duration includes elapsed time spent waiting for network responses, locks, or other operations, whereas a CPU flame graph shows which call stacks consumed sampled CPU time. The checkout trace establishes that payment processing occupied most of the request’s elapsed time, but you need additional evidence, such as downstream spans, profiling, and infrastructure metrics, to determine whether the application was computing or waiting.
Finding the Slow or Failing Operation
You can investigate the checkout failure by following the request from the API gateway to the external payment dependency:
- Find affected requests. Filter the trace explorer to service
api-gateway, operationPOST /checkout, and requests with an error status or a duration above 1 second during the affected period. Open several traces rather than relying on one request, and include successful requests for comparison. - Follow the latency-critical operations. In trace
4bf92f3577b34da6a3ce929d0e0e4736, the root span lasts 1,240 ms. Inventory and pricing overlap and both finish within the first 97 ms. Checkout then records an order insert before calling the payment service. The checkout-service paymentCLIENTspan I lasts 1,104 ms, while payment-service’sSERVERspan J lasts 1,100 ms. Those spans account for most of the request’s elapsed time. - Account for time outside child spans. Payment-service’s span J lasts 1,100 ms. Its two external API attempts, K (500 ms) and L (580 ms), execute sequentially and account for 1,080 ms. The remaining 20 ms falls outside those child spans, including the interval between attempts. This is elapsed time not covered by the child spans, not necessarily CPU processing time. Use additional instrumentation or profiling to investigate what happened during those intervals.
- Examine span attributes and errors. Span K carries
error.type=timeout, while span L carrieshttp.request.resend_count=1and an HTTP 503 response. The first external payment request timed out after 500 ms, and the retry began 5 ms later before failing with a 503. These attributes show that the payment-service request was delayed by a timeout followed by an unsuccessful retry. - Validate the hypothesis. Search the payment-service logs using the trace ID to inspect timeout exceptions and retry decisions. Compare failing traces with successful requests from the same period and with historical latency for the external payment API. Check whether failures occur across multiple pods and nodes or are concentrated on particular workloads. If the slowdown affects every caller, investigate the external dependency; if it is limited to certain pods or nodes, examine local networking, DNS, deployment changes, and resource pressure before deciding on a fix.
The trace identifies the operation responsible for most of the latency, while comparisons across requests and correlated telemetry help establish the underlying cause. Once you’ve confirmed whether the problem originates in the application, its infrastructure, or an external dependency, you can choose the appropriate corrective action.
APM Traces vs. Logs vs. Metrics
APM traces, logs, and metrics describe different aspects of application behavior. Metrics measure performance and resource usage, logs record individual events, and traces connect the recorded operations involved in a request. The main difference is the level of detail each signal provides and the questions it helps you answer.
Consider a checkout endpoint returning HTTP 502 errors. Metrics can reveal when its error rate increased and whether latency rose at the same time. Logs may contain the payment client’s timeout exception and retry messages. An APM trace connects the gateway, checkout service, payment service, and external API calls, showing how the timeout and retry contributed to the failed request.
These signals can also share information. Logs can carry trace and span IDs, traces can contain exception events, and metrics can be generated from span data. However, each signal has limitations: metrics generally lack individual request paths, logs do not inherently establish cross-service relationships, and traces only show operations captured by instrumentation and retained by the tracing pipeline. Because trace volume increases with the number of recorded requests and operations, sampling and retention policies become important when collecting traces at production scale.
Trace Sampling, Retention, and Cost
Trace sampling determines which requests have their spans retained for investigation, while retention policies control how long those traces remain available. Both help manage costs as request volume grows. Your total tracing costs depend on request traffic, spans generated per request, processing overhead, storage requirements, and how your observability platform charges for telemetry.
Head sampling reduces the volume of spans recorded and exported by applications. Tail sampling reduces downstream export and storage volume but requires the Collector to receive and buffer spans before making a decision. Retention policies control how long stored traces remain available.
Head-Based vs. Tail-Based Sampling
Head-based sampling decides whether to record and export a trace near the beginning of a request, before its outcome is known. OpenTelemetry’s ParentBased sampler allows downstream services to follow the caller’s sampling decision, reducing the risk of incomplete traces. By default, the SDK uses parentbased_always_on, which samples new root traces unless an incoming parent context directs otherwise. You can configure a 10% sampling probability for new root traces in SDKs that support these environment variables:
This reduces the expected number of exported traces, but it cannot prioritize a request that fails later. With tail sampling, you can retain traces based on their recorded errors, latency, and attributes. For example, you might keep failed checkouts and requests exceeding two seconds while sampling only a fraction of successful requests. The OpenTelemetry Collector’s tail_sampling processor supports these policies.
Tail sampling requires additional resources because the Collector must buffer spans before making a decision. With the default trace-complete strategy, it waits 30 seconds and can hold up to 50,000 traces by default. In a scaled deployment, you also need trace-aware routing so spans sharing a trace ID reach the same Collector. Buffer exhaustion, late-arriving spans, and changes to Collector endpoints can result in missing data, making memory capacity, decision caches, and exporter retries important.
You can combine both methods, but tail sampling cannot recover traces discarded upstream. If head sampling retains only 10% of requests, the downstream tail sampler can examine only those retained traces. Keep that limitation in mind when deciding how much traffic to sample before it reaches the Collector.
Estimating Trace Volume and Retention Costs
Trace volume increases with request rate and the number of spans generated per request. Consider an application handling 500 requests per second, averaging 12 spans per trace. Over 24 hours, it produces 43.2 million traces and 518.4 million spans. Assuming an illustrative 500 bytes per span, uniform sampling would produce approximately these data volumes:
Actual storage requirements depend on span attributes, compression, indexing, replicas, and backend implementation. Doubling the average span count from 12 to 24 would also double the estimated data volume, even if request traffic stayed unchanged. These figures estimate telemetry volume rather than vendor charges.
Retention determines how much historical data remains in storage. For example, collecting 10 GB of traces daily and retaining them for seven days requires approximately 70 GB before compression and storage overhead. Longer retention helps you compare incidents across wider periods, while shorter retention reduces storage consumption. You can also use different retention periods for production and development workloads or move older traces to lower-cost storage where supported.
When Should You Use Full-Fidelity Tracing?
Full-fidelity tracing retains every trace produced by your configured instrumentation instead of discarding a percentage through sampling. It is useful during load testing, investigations involving rare failures, and workloads where missing a particular request could prevent you from diagnosing an incident. However, full-fidelity tracing does not capture operations that were never instrumented, and retaining every trace increases processing and storage requirements.
You can limit those costs without discarding traces by adjusting retention periods, removing unnecessary attributes, and using lower-cost storage for older data where supported. For example, you might retain all production traces for seven days while keeping selected diagnostic data for longer investigations. The appropriate policy depends on request volume, troubleshooting requirements, and available infrastructure capacity.
Common APM Tracing Problems
Missing spans and disconnected traces can make troubleshooting difficult. The problem may be caused by incomplete instrumentation or broken context propagation. Spans can also disappear during collection or export. Before changing your configuration, find out where the expected trace data is missing.
Isolating Collector and Exporter Failures
If traces are missing, start with the Collector logs. In Kubernetes, replace the placeholders with your namespace and Collector Deployment name:
Look for connection failures or authentication errors. Also check whether the Collector is refusing spans or timing out during export. If the cause is still unclear, temporarily add a debug exporter alongside your normal exporter. Generate a test request and compare the output with the traces stored in your backend.
- No spans in the debug output: Check whether the application generated and exported them. Then verify the receiver configuration and look for sampling or filtering earlier in the pipeline.
- Spans appear in debug output but not in the backend: Inspect the normal exporter’s endpoint and authentication settings. Check network connectivity and backend ingestion errors.
- Spans reach the backend but the trace is incomplete: Compare trace IDs across services. Check for missing instrumentation and review any sampling rules that could discard spans.
Disable the debug exporter after testing. Detailed output may expose sensitive span attributes.
If tracing affects application performance, measure its overhead under realistic traffic. Compare CPU and memory usage with tracing enabled and disabled. Include p95 latency and garbage collection where applicable. Change one instrumentation setting at a time. Repeat the test after each change so you can confirm the improvement without losing the spans needed for troubleshooting.
APM Tracing and Incident Investigation With groundcover
groundcover combines eBPF tracing with OpenTelemetry ingestion to help you investigate application performance across Kubernetes. You can follow requests through services and examine the infrastructure behind failures. AI-assisted investigations and BYOC deployment extend those capabilities to production environments.
Automatic eBPF Tracing and OpenTelemetry Integration
groundcover’s Flora sensor runs as a Kubernetes DaemonSet and observes supported service interactions without application code changes. It captures request latency and errors across HTTP, gRPC, databases, and other supported protocols. Existing OpenTelemetry instrumentation can also send spans through OTLP to capture internal operations that eBPF cannot observe.
Both sources can generate APM measurements, so select the appropriate source to avoid double-counting. The eBPF sensor measures supported requests while smart sampling determines which detailed traces are stored. You can force sampling when an investigation requires more complete trace retention.
Correlating Traces With Kubernetes Telemetry
Traces become more useful when you can examine the workloads behind them. groundcover enriches trace data with Kubernetes context so you can investigate related logs and infrastructure metrics alongside individual requests.
For the checkout incident, you could start with a p95 latency alert and inspect failing payment traces. The service map identifies affected dependencies, while the waterfall reveals the timeout and retry. Workload metrics and Kubernetes events help determine whether resource pressure or a recent deployment contributed to the failure. Use gcQL to query traces and logs, with trace IDs providing exact correlation where available.
AI-Assisted Debugging With Agent Mode
Agent Mode lets you investigate incidents using natural-language questions. Ask “Why did checkout latency increase after 14:00?” and it can examine request traces alongside application metrics and Kubernetes events. Follow-up questions can narrow the investigation to specific pods or operations.
You can also generate monitor and dashboard drafts from the findings. The underlying queries and results remain available for verification, so you can confirm a suspected root cause before acting. Connected external tools can provide additional context within your configured permissions.
Bringing Traces Into AI Coding Workflows With MCP
The groundcover MCP Server gives compatible coding assistants access to production telemetry. Tools in Cursor or Claude Code can query traces and logs, inspect metrics, and run AI-assisted root cause analysis.
For example, your assistant could identify repeated payment requests following a timeout. You can then review the retry logic in your code, deploy a correction, and query fresh traces to validate the result. MCP tools also generate monitor and dashboard drafts, but they do not automatically deploy changes to your applications.
Tracing LLM Calls and Agent Workflows
AI Observability extends tracing to applications using language models. For supported providers, eBPF captures model calls without code changes and provides visibility into latency, token usage, and estimated cost. Prompt and response capture can be restricted through privacy controls.
OpenTelemetry GenAI instrumentation adds agent workflows and tool execution spans. You can trace multi-step operations and examine how individual model calls contribute to overall execution time. groundcover also normalizes supported GenAI attributes from different SDK generations.
BYOC Deployment and Predictable Costs
With groundcover’s BYOC architecture, observability data stays in your cloud environment. The managed backend stores traces and logs in ClickHouse, while VictoriaMetrics handles metrics. You retain control over data retention and storage policies.
Pricing depends on monitored nodes rather than telemetry ingestion volume or user count. Cloud infrastructure and AI inference remain separate costs. BigBasket reported cutting observability spending by 50% while expanding monitoring coverage.
FAQs
A span records one operation, such as an HTTP request or database query. It contains timing information and attributes describing the operation. A trace connects spans belonging to the same request through a shared trace ID and parent-child relationships. In the checkout example, POST /charges is one span within a larger trace containing twelve spans.
Start by instrumenting your application with an OpenTelemetry SDK or automatic instrumentation agent. Set OTEL_SERVICE_NAME to identify the service and configure an OTLP exporter to send spans to a tracing backend. OTLP/gRPC commonly uses port 4317, while OTLP/HTTP uses 4318. Generate a test request and confirm that its spans appear in a trace viewer such as Jaeger. For Kubernetes workloads, you can also use eBPF to capture supported service interactions without changing application code.
Yes. Generating spans and exporting telemetry consumes application resources. The impact depends on your instrumentation and the number of spans produced. For example, a PHP-FPM benchmark measured 23.08 ms per request with the OpenTelemetry PHP Distro, compared with 17.36 ms without instrumentation. Your results will depend on the workload, so compare CPU usage and p95 latency with tracing enabled and disabled under similar traffic. If overhead is excessive, reduce unnecessary instrumentation or adjust sampling.
OpenTelemetry SDKs and automatic instrumentation agents generate spans, while the OpenTelemetry Collector processes and forwards them to a tracing backend. OpenTelemetry does not provide trace storage or visualization itself. Jaeger and Grafana Tempo are common tracing backends. groundcover combines eBPF collection with OpenTelemetry ingestion, allowing you to investigate traces alongside Kubernetes telemetry.
Yes. eBPF captures supported service interactions without application code changes, while OpenTelemetry instrumentation records internal operations and business-specific attributes. Combining them can provide visibility into network activity and application logic within the same observability platform. However, connecting spans into a single distributed trace still requires compatible context propagation and appropriate support from the tracing tools.
Application performance monitoring (APM) measures application health through latency, error rates, and throughput. It also covers alerting and performance investigation. Distributed tracing focuses on individual requests by connecting recorded operations across services. It helps you identify which dependency or operation contributed to a slowdown, making it one technique within the broader practice of APM.
groundcover uses eBPF to capture supported application traffic across Kubernetes without modifying your code or installing SDKs. It measures request latency and errors while using smart sampling to retain traces that help investigate failures and performance bottlenecks. You can examine service dependencies and correlate traces with Kubernetes logs and infrastructure metrics to identify the cause of slow requests. groundcover also ingests OpenTelemetry spans for deeper visibility into application logic. With BYOC deployment, your tracing data remains in your cloud environment.





