eBPF auto instrumentation gives every workload on a Kubernetes node a telemetry baseline: traces, RED metrics, and service maps. You get that coverage without adding an SDK or redeploying services. The alternative is an instrumentation sprint: coordinating library upgrades across dozens of services, in multiple languages, owned by teams with their own roadmaps. Kernel-level capture gives platform teams immediate cluster-wide telemetry, but it depends on three constraints: kernel support, privileged deployment, and workload models that eBPF can correlate reliably.
What is eBPF auto-instrumentation
eBPF auto-instrumentation captures telemetry by running sandboxed programs inside the Linux kernel, attached to the syscalls and library functions your applications already execute. Network syscalls expose socket activity at a fixed host-level boundary, and TLS-library probes expose plaintext before encryption or after decryption. Probes at those kernel and library boundaries observe processes across the node at once.
The contrast with SDK-based capture is where the code runs and who has to change it. An OpenTelemetry SDK executes in user space, inside your process: you import a library, wrap handlers, and redeploy. An eBPF program executes in kernel space, attached from outside the process. It requires zero application code changes. Kernel-level capture does not require a restart, and coverage includes services nobody remembered to instrument. Grafana Beyla's documentation states: "All data capture occurs without any modifications to application code or configuration."
How eBPF auto-instrumentation works
The pipeline runs from a kernel probe firing on a network call to an OpenTelemetry span landing in your backend. Each stage has specific mechanics worth understanding before you deploy one of these agents.
Kernel-space capture with uprobes and kprobes
Two probe types do the capture work. Kprobes attach to kernel functions and syscalls: hook send() and recv() and you see socket payloads at the syscall layer, while TLS payloads appear as ciphertext there. Agents need uprobes on TLS libraries to capture plaintext. Uprobes attach to user-space functions in application binaries and shared libraries; the kernel implements them by replacing the target instruction with an INT3 (0xCC) breakpoint, applied to the file inode so the probe persists across new invocations of that binary (bpftrace user probes).
eBPF maps are the bridge between kernel and user space. Probe handlers write events and connection state into maps; a userspace agent reads them, reassembles protocol payloads into requests, and correlates entry and exit events into latency measurements. Pixie, for example, tracks Go TLS function arguments through an active_tls_conn_op_map keyed by process ID and goroutine ID.
Go deserves a footnote here because uretprobes, the return-side counterpart to uprobes, are unsafe on Go binaries. Go stacks grow and move at runtime, which corrupts the return-address rewrite uretprobes depend on (Go uretprobe issue). Beyla and the OTel Go instrumentation work around this by analyzing the binary and placing a uprobe at every return statement instead.
From kernel probe to OpenTelemetry backend
Once the userspace agent has assembled spans and metrics, export is standard OTLP. OBI (OpenTelemetry eBPF Instrumentation) emits OTLP-native output over gRPC or HTTP, and since v0.5.0 it can run as a receiver inside the OpenTelemetry Collector, so eBPF-captured spans flow through the same processors and exporters as SDK spans (OBI receiver docs). The 2026 OBI roadmap includes a dedicated Collector distribution with OBI built in.
Pixie chose a different path. Pixie stores data in proprietary in-memory tables on the nodes themselves, queryable through its PxL language; getting that data into an OTel backend requires a Collector deployment, a PxL export script, and Plugin System configuration (Pixie OTel export). If OTLP is your standard, native export matters more than it looks on a feature list.
System requirements: kernel version, BTF, privileges
Kernel support and privileges gate every eBPF deployment; BTF determines whether CO-RE works without kernel headers. Minimum kernel versions per official documentation:
BTF (BPF Type Format) is kernel metadata that encodes type and struct-layout information, and it enables CO-RE: compile an eBPF program once, relocate it at load time to match whatever kernel it lands on, no kernel headers needed. Most distributions enable BTF by default from kernel 5.14, making that the practical zero-configuration floor; Ubuntu 20.04 notably shipped with it disabled. Check any node with ls /sys/kernel/btf/vmlinux. Without BTF, fallbacks exist (BTFHub's pre-built per-kernel files, tailored BTF embedded in the agent binary), but each adds operational friction.
Privileges are the third gate. DaemonSet deployment requires hostPID: true, and Beyla's security documentation lists the Linux capabilities it needs, including CAP_BPF, CAP_PERFMON, and CAP_SYS_ADMIN for library-level uprobes; on AKS and EKS, default perf_event_paranoid settings force CAP_SYS_ADMIN regardless. Odigos illustrates why kernel version and privileges interact: kernels 5.4–5.7 require CAP_SYS_ADMIN, while 5.8+ allows the narrower CAP_BPF plus CAP_SYS_PTRACE.
Key features and capabilities
What you get on day one, before writing any configuration, breaks down into four buckets.
RED metrics out of the box
Beyla and OBI deliver RED metrics for supported HTTP/S and gRPC services on the node without application changes. Other eBPF agents vary by protocol, runtime, and TLS-library coverage. Beyla captures "trace spans related to web transactions and Rate Errors Duration (RED) metrics for Linux HTTP/S and gRPC services" with no application changes. Because the agent measures latency at the kernel's network layer, it includes queue wait time that an in-process SDK never sees. The agent builds service maps from the kernel's connection data: it sees every connection between workloads, so it can construct dependency graphs at deployment.
Distributed tracing and context propagation
Trace context propagation is where eBPF's correlation model matters most and where it breaks. For Go HTTP and gRPC, Beyla and OBI use bpf_probe_write_user to inject the traceparent header directly into library memory. This requires CAP_SYS_ADMIN or privileged mode, and if the kernel runs in integrity lockdown mode (common with Secure Boot), bpf_probe_write_user is unavailable and distributed tracing is disabled. A network-level path using TC eBPF programs exists as well; parsing incoming headers there requires kernel 5.17+.
The breakage points are async execution models, because eBPF correlates request context by OS thread ID or goroutine ID:
- Goroutine nesting: OBI supports up to 6 nested goroutine levels for context tracking.
- Thread pools and reactive frameworks: work that moves through unobserved framework queues (Spring WebFlux, Netty/Reactor) loses its trace context; Beyla's docs say that reactive Java "is not a good fit for eBPF trace correlation."
- Java virtual threads: a virtual thread can unmount from one carrier OS thread and remount on another, breaking thread-ID correlation. OBI tracks mount and unmount operations to follow the virtual thread, but this requires JDK 21+.
- Python asyncio: thread-based correlation works for WSGI frameworks like Flask and Django but not asyncio; OBI v0.7.0 added support for asyncio workloads running on uvloop.
- Reused HTTP/2 connections: OBI "can write context for HTTP/2 and gRPC only on new, non-HTTPS connections" (OBI context propagation).
Encrypted traffic visibility without decryption
eBPF sees HTTPS payloads in plaintext without holding keys or performing decryption, because uprobes attach to the TLS library's API rather than the socket. A kprobe on send() sees only ciphertext; a uprobe on OpenSSL's SSL_write captures the buffer before encryption, and one on SSL_read captures it after decryption (Pixie eBPF TLS). Pixie uses four probes per library: entry and return for both functions.
Coverage varies by library. Go's crypto/tls is hooked at (*Conn).Read and (*Conn).Write using binary symbol offsets. BoringSSL is typically statically linked, so probes land on the application binary instead of a shared library. Java is the hard case: JSSE is not a native shared library and exports no SSL_write-style symbols for eBPF to attach to, so Beyla and OBI fall back to a Java agent for TLS capture (Java TLS instrumentation). For propagating context through gRPC/HTTP2, OBI v0.10.0 added sk_msg-based HPACK header injection (OBI v0.10.0 release).
Supported languages, protocols, and databases
OBI v0.10.0 instruments Java (JDK 8+), .NET, Go, Python, Ruby, Node.js, C, C++, and Rust. Protocol coverage spans HTTP/HTTPS/HTTP2, gRPC, gRPC-Web, JSON-RPC, MQTT, NATS, AMQP 1.0, and Memcached, plus databases: PostgreSQL, MySQL, MSSQL, MongoDB, Redis, and Couchbase (OpenTelemetry eBPF overview). Beyla adds custom detectors for GraphQL, Elasticsearch, and AWS S3 and SQS via HTTP payload extraction.
Known gaps are concrete: tools still do not support fully stripped static binaries because they defeat symbol resolution, Rust's rustls has heavily mangled symbols and never calls read or write itself, and applications using custom OpenSSL BIO patterns that decouple encryption from I/O remain roadmap items (Pixie TLS roadmap).
Deploying eBPF agents in Kubernetes
The DaemonSet is the dominant pattern: one agent pod per node, hostPID: true so it can see every process on the host, and coverage for each new workload the moment the scheduler places it. Beyla recommends this for production. The sidecar alternative uses shareProcessNamespace: true to give a per-pod agent visibility into its neighbor's processes, which limits blast radius but reintroduces per-workload deployment work, the exact toil eBPF was supposed to remove.
The security cost is explicit under Kubernetes Pod Security Standards. A container with hostPID: true and privileged: trueviolates the Baseline profile; both fields are forbidden outside the Privileged profile (Kubernetes Pod Security). In practice that means deploying the agent into a dedicated namespace with a pod-security.kubernetes.io/enforce: privileged label, or an exempted namespace like kube-system, while the rest of the cluster stays on Baseline or Restricted.
Treat that exemption as a real attack surface. hostPID allows reading /proc/[pid]/environ on host processes, exposing secrets passed as environment variables. A privileged container can escape to the host with nsenter -t 1 -m -u -i -n -p -- bash, and breakout can expose kubelet credentials. Sysdig documented this attack pattern in CVE-2026-39987 (2026-05-29): an attacker created a privileged container with host PID, network, and IPC namespaces, entered the host via nsenter, and replayed a stolen service-account token to dump the cluster's secret store (Sysdig incident analysis). Compensating controls belong in your rollout plan: admission policies scoped to the privileged namespace, restricted RBAC on who can create workloads there, and runtime detection.
How eBPF auto-instrumentation fits in
eBPF sets a baseline of fleet-wide visibility. OpenTelemetry SDKs provide code-level depth. The practical model is breadth by default, depth by exception.
eBPF vs OpenTelemetry SDKs
The trade-off across the dimensions that matter for a platform team:
Beyla's own documentation draws the line: "Beyla only provides generic metrics and transaction level trace span information. Language agents and manual instrumentation are still recommended so that you can specify the granularity of each part of the code to be instrumented." Use eBPF to guarantee a floor of visibility across the whole cluster on day one. Use SDKs where you need business-logic attributes, custom spans, or reliable async correlation, which in most fleets is a minority of services.
Combining eBPF and SDKs (hybrid model)
Running both raises an obvious question: do you get every span twice? Beyla and OBI suppress duplicate spans by detecting OTLP exports. Both ship exclude_otel_instrumented_services defaulting to true; the agent watches for a service performing a successful OTLP export and then stops instrumenting it (OBI service discovery). Detection is behavioral, and it has had bugs: Python gRPC exporters were missed until OBI PR #2712 fixed the port comparison.
For derived metrics, both tools automatically tag their traces with span.metrics.skip=true when span metrics or service graph generation is enabled, so a downstream Collector can avoid double-generating span metrics from spans the agent already counted. That attribute is an eBPF-side convention outside the OTel semantic conventions. Odigos takes a different approach entirely: instrumentation is opt-in per workload via a Source CRD, and spec.disableInstrumentation: true excludes a workload outright.
Tool comparison: OBI, Grafana Beyla, Odigos, Pixie
Four projects dominate the open-source field, and they differ more in architecture than in probe technology:
The Beyla-to-OBI relationship is worth internalizing: Grafana Labs donated Beyla's code to OpenTelemetry in May 2025, all Beyla maintainers now work full time on the upstream OBI repository, and maintainers merge new features into OBI first. OBI has the broader protocol list; Beyla is the stable, vendor-supported distribution. Odigos belongs in the comparison as a hybrid: its OSS default uses eBPF only for Go, with precompiled memory-offset tables for supported library versions, and SDK injection everywhere else. groundcover's Flora eBPF sensor sits in the commercial category with a different scope: a single DaemonSet capturing metrics, traces, logs, and Kubernetes events at the kernel level, exporting into a data plane (ClickHouse and VictoriaMetrics) that runs in your own cloud account, with OTel ingestion as a first-class source alongside the eBPF signals.
What's next for eBPF auto-instrumentation
Three fronts are moving fast:
- GenAI observability: OBI v0.10.0 already instruments OpenAI, Anthropic Claude, Google AI Studio (Gemini), Amazon Bedrock, Qwen, MCP over JSON-RPC, and vector retrieval systems including Pinecone, Qdrant, and Weaviate (OpenTelemetry eBPF overview). groundcover shipped eBPF-based LLM API observability in August 2025, capturing every LLM request and response with zero code changes, and extended it to Amazon Bedrock, OpenAI, and Anthropic in December 2025; the design pairs eBPF capture with OTel ingestion for AI workloads, and groundcover's own April 2026 write-up describes the eBPF path as still maturing.
- GPU tracing: Polar Signals shipped parca-agent v0.43.0 in October 2025, describing it as the first open-source, low-overhead NVIDIA CUDA profiler suitable for always-on production use, built on CUPTI plus USDT probes and eBPF (Polar Signals GPU profiling). At larger scale, Alibaba's SysOM-AI integrates GPU kernel tracing and NCCL instrumentation via eBPF at under 0.4% overhead across more than 80,000 GPUs (SysOM-AI paper). The constraint is NVIDIA's proprietary driver, which exposes exactly one tracepoint (
nvidia:nvidia_dev_xid); everything deeper relies on kprobes against undocumented internals. - Trace-log correlation: OBI already enriches logs with trace context, though OBI skips enrichment for requests on Java virtual threads. Two open OTel proposals (OTEPs #4719 and #4947) would have SDKs publish process- and thread-level attributes through memory mappings that external eBPF readers can consume, replacing today's fragile trick of inferring SDK presence by watching OTLP exports.
Limitations and production considerations
Async context propagation is the most practically significant gap. The pattern is consistent across every tool: eBPF correlates by OS thread or goroutine identity, and any framework that migrates logical work across those boundaries can produce orphaned spans. If your fleet leans on Spring WebFlux or heavy thread pooling, including deep goroutine fan-out, validate trace continuity in staging before you trust the spans.
Performance overhead numbers do not generalize, and you should distrust any single figure:
- Delft University: an independent study measured Beyla at 1.5 m CPU and 110 MiB memory on a 5G core workload, adding at most 0.33 ms to P99 latency.
- Independent k6 benchmark: a high-concurrency workload measured a ~24% throughput drop and ~28% p95 latency increase with Beyla enabled, against Grafana's vendor figure of roughly 0.1% CPU on a low-rate demo app.
- ACM 2026 storage I/O paper: eBPF tracing overhead reached 41% under small, highly concurrent storage I/O (ACM storage paper).
- groundcover vendor benchmark: groundcover's own April 2023 test measured Flora at +9% CPU and +0% memory against +59%/+27% for the OTel SDK and +249%/+227% for Datadog on a Go HTTP workload at 3,000 req/s, a vendor-published result without a 2025–2026 independent replication.
Benchmark on your own traffic profile; nobody else's numbers are yours.
Two more constraints shape adoption. eBPF requires kernel access, so managed PaaS platforms and non-Linux hosts are out of reach; kernel lockdown mode disables write-based context propagation even on Linux. And manual instrumentation stays necessary wherever you need custom spans, business attributes, or granular control that kernel-level capture cannot derive, which is why the hybrid model above is the realistic end state rather than a transition phase.
.svg)
