Better Together: Why You Should be Layering eBPF with OpenTelemetry

There's a recurring pattern in observability adoption that probably sounds familiar. Teams log everything, tell themselves they'll add tracing later, and then later never comes. By the time distributed traces actually matter, the codebase has grown and retrofitting instrumentation feels like a project in itself. Understanding why this keeps happening, and what actually fixes it,  requires looking honestly at how teams build observability in practice, not in theory.

In this post we'll walk through why the three pillars of observability rarely look equal in practice, what eBPF brings to the table that traditional instrumentation approaches can't, where it still falls short, and why combining eBPF with OpenTelemetry SDKs gives you something neither can deliver on its own.

The Three Pillars of Observability - Theory vs. Practice

Everyone in observability knows the three pillars: logs, traces, and metrics. The way they're usually presented, as equals sitting side by side, implies you should be using all three. But in reality, what most teams actually have looks very different, shaped more by what was urgent last sprint than by any deliberate observability strategy. Each pillar gets adopted at a different pace and for different reasons, leaving significant gaps in coverage.

Logs come first and accumulate fastest. Logging is the first thing developers learn and it never really leaves the muscle memory. It's elastic enough to capture almost anything, a debug message, a full HTTP request, a counter, which is exactly why it becomes the default answer to every observability question. The problem is that trying to do everything with one pillar creates noise and puts real pressure on your backends.

Metrics tend to emerge from that pain point naturally. Logs are a blunt instrument for measuring things at scale, and at some point the cost of querying them for counts, trends, and rates becomes unsustainable. Metrics adoption often happens without much deliberate planning, accumulating as a byproduct of the cloud providers, databases, and third-party integrations teams are already using.

Then there are traces. Where logs tell you what happened and metrics tell you how much or how often, traces tell you why, following a request across services, queues, and databases to show the complete path and where time was spent. That end-to-end visibility is something neither logs nor metrics can replicate. But traces require broad participation across every service to be meaningful, and that coordination challenge is exactly what leads to the procrastination described at the outset. Teams defer tracing in favor of more pressing work, and by the time an incident exposes the gap, the codebase has grown too complex to instrument quickly.

Where Existing Instrumentation Falls Short & the Unique Benefits of eBPF

Manual instrumentation, writing custom code to push traces into whatever backend you're using, is the most direct approach, but it's always the last thing that gets done. There's always another feature to ship, another bug to fix, and maybe if there's time you can add some spans somewhere. It's difficult to maintain, difficult to keep consistent across teams, and realistically it gets perpetually deprioritized.

Auto-instrumentation SDKs improved this significantly. For example, OpenTelemetry's Java agent, the Python zero-code instrumentation package, and the Node.js SDK wrapper can automatically capture traces for every HTTP call, every SQL query, and every outbound request without a single line of instrumentation code. This approach has proven genuinely useful, although there are real hurdles. 

First, it doesn't work for every framework, and it doesn't work well for some modern languages, Go being a notable example. Second, it adds meaningful overhead to applications that are already running in production optimized for performance. That overhead is often overlooked or difficult to measure precisely, but it's real and worth considering depending on your use case.

Enter eBPF

We've covered eBPF extensively elsewhere, but it's worth grounding it in this specific context of observability and its relationship with OpenTelemetry. Because every network call an application makes ultimately goes through the kernel, you can inspect those packets and determine exactly what happened. An HTTP call, a SQL query, a gRPC request. If you know how to dissect the protocol, you can reconstruct it from the kernel level.

And because the operating system is also responsible for running processes, containers or otherwise, when you track a call, you know which process made it. That lets you associate an HTTP call with the specific container that initiated it. You can see what's happening inside every application from the outside, as a sideband, without anyone touching the code and without any instrumentation decisions being made in advance.

This distinction matters. Logs, metrics, and even manual traces represent what a developer decided was important and worth capturing at the time they wrote the code. eBPF gives you something fundamentally different. It gives you the actual behavior of your programs whether you anticipated it or not, and without requiring instrumentation. It starts running within minutes, covers everything in your stack including third-party services you have no ability to instrument, and does all of this with very minimal overhead. But that outside-in view is also its natural boundary. eBPF sees what crosses the wire, but what happens inside the application logic, the decisions, the branches, the business context, remains out of reach.

Where eBPF Hands Off to OpenTelemetry - FTW!

This boundary is where OpenTelemetry SDKs complete the picture. eBPF gives you comprehensive coverage of applicative behavior across your entire stack, but the internal logic of your applications, the decisions, the conditions evaluated, the business rules triggered, remains outside its line of sight by design. Capturing the successful completion of a payment flow, a feature flag routing a user down a specific path, or a downstream action triggered by a particular business rule requires instrumentation that understands the application from the inside.

Wire-level visibility and application-level understanding are two different things, and conflating them leaves critical business logic effectively unobserved. AB testing, feature flag decisions, complex branching logic, anywhere in your applications where the meaning of what happened lives inside the application rather than in the network traffic, these require a different layer of instrumentation entirely.

The practical approach is to use eBPF for network-level visibility and OpenTelemetry SDKs for application logic, letting each do what it was designed for. Enter eBPF for OpenTelemetry.

On the eBPF side, that means deploying a dedicated eBPF agent that feeds directly into the OpenTelemetry ecosystem, handling kernel-level observation and distributed tracing context propagation across your services in a way that integrates seamlessly with your existing OpenTelemetry pipelines. This is an approach groundcover has been refining for years, with an agent battle-tested across thousands of production environments that outputs standard OpenTelemetry-compatible telemetry out of the box.

What makes this integration particularly powerful is what groundcover's eBPF sensor brings to OpenTelemetry traces that SDK instrumentation alone cannot. OpenTelemetry on its own is limited to the application-level spans and metadata that are manually instrumented by engineers, and cannot capture critical details such as full payloads, HTTP headers, or low-level network and kernel interactions. 

groundcover's eBPF sensor closes that gap by automatically enriching OpenTelemetry traces with full HTTP request and response payloads, headers, query parameters, cross-AZ indicators, and PII identification, without any additional instrumentation work. The result is a trace that carries both the distributed context that OpenTelemetry SDKs provide and the granular kernel-level detail that only eBPF can deliver.

The OpenTelemetry community has since embraced the same philosophy with OBI (OpenTelemetry eBPF Instrumentation), an open source project bringing eBPF-based tracing into the OpenTelemetry ecosystem, with production readiness as a stated focus for this year. Both reflect the same conviction: that eBPF and OpenTelemetry are stronger together, and that the kernel is the right place to start building observability coverage.

Layered Observability Made Easy

The observability gap that opens up when teams defer tracing is real, but it's no longer inevitable. OpenTelemetry provides the foundation on both sides of this approach, with a mature SDK ecosystem that is straightforward to adopt incrementally, starting with your most critical flows and expanding from there. On the eBPF side, groundcover's agent brings that foundation to life, deploying into a Kubernetes cluster in minutes and immediately enriching your OpenTelemetry traces with kernel-level detail that no amount of manual instrumentation could replicate. For teams looking for a purely open source path, OBI offers a community-driven alternative working toward production readiness.

The goal is not to choose between eBPF and OpenTelemetry SDKs. It's to let eBPF handle the coverage problem while SDKs handle the depth problem. Start with groundcover's eBPF agent as your baseline across everything, automatically capturing network-level behavior and enriching your OpenTelemetry traces from day one, then layer in SDK instrumentation deliberately for the flows where business context and application-level understanding actually matter. That combination gives you something neither approach can deliver on its own, full visibility from the wire to the application logic, without the instrumentation debt that has historically made distributed tracing so difficult to sustain and manage over time.