Understanding eBPF monitoring performance overhead

An eBPF sensor executes inside the same kernel that serves your production traffic, so every nanosecond it spends per event comes out of the same CPU budget as your workloads. Published overhead measurements span two orders of magnitude depending on workload and event rate, which means “eBPF is lightweight” tells you nothing about your cluster specifically. Before you deploy any eBPF tool on shared Kubernetes nodes, measure the overhead and identify the variables that drive it.

What is eBPF monitoring overhead

eBPF monitoring overhead is the CPU time and memory an eBPF-based tool consumes per node: the in-kernel cost of running probe programs on every event, the cost of moving event data to user space, the memory pinned in BPF maps, and the footprint of the sensor process itself. It matters on shared Kubernetes clusters because all of it comes out of node allocatable resources that would otherwise serve pods.

The published range is wide. DoorDash’s BPFAgent reports under 0.3% of a typical node’s CPU cores and memory, Netflix’s flow exporter reports under 1% of CPU and memory on any instance, and an IEEE ISPASS 2024 study of latency-sensitive datacenter workloads found median overhead below 0.5%. At the other end, an ACM CHEOPS 2026 workshop paper on storage I/O measured eBPF tracing overhead reaching 41% under small I/O requests with high concurrency.

The variable that explains the spread is event rate. Brendan Gregg’s operational model states it as overhead equals per-event instrumentation cost times event frequency, with rough thresholds: below 10,000 events per second overhead is probably acceptable, above 100,000 it becomes measurable. Two variables determine the result: the cost of each probe invocation and the number of times it fires.

How eBPF keeps overhead low

The low-overhead claim rests on specific kernel mechanisms, each of which you can verify independently.

Kernel space vs. user space execution

An eBPF program runs at its hookpoint inside the kernel, in the execution context of the event itself. There is no context switch per event and no per-event copy of raw data into a user-space agent; programs aggregate in BPF maps and hand summaries to user space in batches. A user-space agent pays the kernel-to-user boundary cost on every event it observes; an eBPF program pays it only when it exports.

The context-switch savings show up in measurement, though the published comparisons are thin. One independent service-mesh benchmark on ARM64 nodes counted 188 context switches per request for Cilium’s eBPF datapath against 482 for an Istio sidecar. Treat that as a single data point from one author’s test rig rather than a settled figure — the structural argument is stronger than any one number backing it.

The verifier and safety sandbox

Before any eBPF program loads, the kernel verifier statically checks it, rejecting unsafe memory access and unbounded control flow. LWN’s comparison draws the contrast with SystemTap plainly: the BPF verifier keeps programs from crashing the system, whereas the kernel-module approach has to implement equivalent safety checks at runtime.

This is the prerequisite for accepting any overhead trade-off at all. A kernel-module agent that panics a node costs you far more than a few percent of CPU, and Brendan Gregg has documented SystemTap’s history of panics and freezes.

Verification is mostly a load-time cost, but not entirely. The verifier and JIT also insert runtime work: pointer masking and other Spectre mitigations, patched bounds checks, and constant blinding when bpf_jit_harden is on. These are small, but they are not free, and they land on the per-invocation path.

JIT compilation

The kernel JIT-compiles eBPF bytecode to native machine code, removing the interpreter from the per-invocation path.

The kernel’s own measurements are older than eBPF itself and worth reading carefully. The x86-64 table in commit bd4cf0ed331a benchmarks a libpcap-generated packet filter: 90 ns per call cache-hit for the classic BPF interpreter, 31 ns for the new internal (eBPF) interpreter, and 12 ns for the classic JIT. The eBPF JIT row was left as TBD. Eric Dumazet’s original x86-64 JIT benchmark, on an 18-instruction network-address filter, reported roughly 50 ns saved per invocation. Both are classic-BPF packet-filter numbers, not tracing programs, so read them as evidence that JIT compilation matters rather than as a cost model for your sensor.

You almost certainly already have the JIT. CONFIG_BPF_JIT_ALWAYS_ON landed in Linux 4.15, motivated by Spectre v2, and commit 81c22041d9f1 turned the JIT on by default for x86-64 and arm64 in Linux 5.6 without that config. The result is a fixed, predictable native-code cost per invocation instead of a data-dependent interpreter cost.

Hook types and their overhead

Per-invocation cost varies by roughly two orders of magnitude across attachment mechanisms. Cloudflare’s ebpf_exporter benchmark measured an empty program at 15 ns on a tracepoint, 24 ns on fentry, and 137 ns on a kprobe, against a 117 ns getpid() baseline on Linux 6.5-rc1. A separate run on Linux 6.7-rc3 put an empty uprobe near 1,670 ns.

Read those absolute numbers with the test rig in mind: Cloudflare ran the benchmark on a MacBook Air (M1, 2020) under QEMU, and the README says as much. The ordering is the durable finding; the nanosecond values will move on your hardware.

perf_event reads pay a syscall boundary of their own: PAPI counter reads cost roughly 885–1,634 cycles through read() versus 142–199 with rdpmc, and a CERN PMU study measured heavy PMU multiplexing as high as 25% overhead, which is why perf_events fit sampling and profiling rather than per-event tracing.

The relative ranking is consistent across every independent comparison in the literature: fentry and the BPF trampoline are cheapest, then tracepoints, then kprobes, with kretprobes more expensive than kprobes — the kernel’s kprobes documentation puts the gap around 50–75%, though its benchmark table dates from mid-2000s hardware — and uprobes an order of magnitude above everything else, because they trap from user space into the kernel. On Cloudflare’s numbers, a tracepoint costs about a ninth of a kprobe per event and fentry about a sixth. The practical rule for platform engineers: prefer tracepoints and fentry on hot paths, and give uprobe-heavy tools extra scrutiny.

Maps and memory footprint

BPF maps are where eBPF monitoring memory lives, and they behave differently by type:

• BPF_MAP_TYPE_HASH: preallocates the full element backing store at creation by default; BPF_F_NO_PREALLOC disables this when preallocation costs too much memory. Budget for max_entries regardless of fill level.
• BPF_MAP_TYPE_PERCPU_HASH: removes lock contention by giving each CPU its own value copy, at the cost of a per-CPU formula. On a 64-core node, that multiplier is the sizing decision.
• BPF_MAP_TYPE_RINGBUF: max_entries is the buffer size in bytes, power-of-2 and page-aligned. The kernel maps the data area twice in virtual memory so wrapping records appear contiguous, but only one physical copy exists. Falco’s default 8 MB buffer shows as a 16 MB virtual area for this reason.

Sizing guidance from major projects gives you starting points: Cilium sizes large maps at 0.25% of total system RAM by default, and Falco defaults to one 8 MB buffer per two CPUs. Since kernel 5.11 under cgroup v2, BPF memory accounting charges BPF memory to the allocating cgroup, so an undersized memory.max on your monitoring DaemonSet triggers the OOM killer. bpftool map show reports a memlock figure rather than actual allocation, and the discrepancy can reach roughly 10x between it and /proc/meminfo for non-preallocated maps.

How to measure eBPF overhead in your cluster

Vendors ran their benchmarks on someone else’s workload. The kernel gives you the tools to measure any tool’s probes on yours, and the workflow takes minutes:

1. Enable statistics with sysctl -w kernel.bpf_stats_enabled=1 (kernel 5.1+). The instrumentation itself costs about 20 ns per program invocation; the implementing commit notes that fast BPF programs slow from roughly 10 ns to roughly 30 ns. On kernel 5.8+, the BPF_ENABLE_STATS bpf() command gives you an fd-scoped alternative that disables itself when the fd closes, so multiple tools don’t fight over the sysctl.
2. Read the counters with bpftool prog show. Output looks like this:

10: xdp  name some_prog  tag 005a3d2123620c8b  gpl
        run_time_ns 81632  run_cnt 10
        loaded_at 2017-09-29T20:11:00+0000  uid 0
        xlated 528B  jited 370B  memlock 4096B  map_ids 10

1. Divide run_time_ns by run_cnt for the average cost per invocation, then multiply by the event rate on your nodes. A program averaging 500 ns that fires 200,000 times per second consumes 10% of one core, before transport costs.
2. Go deeper with hardware counters: bpftool prog profile id 337 duration 10 cycles instructions reports per-program cycles plus instruction behavior. You can add cache-miss metrics when needed. You need bpftool prerequisites: BTF in the kernel build and a bpftool binary with skeleton support.
3. Confirm at the workload level with a fixed-time benchmark: run a representative load on one node with the sensor deployed and one without, and compare throughput and tail latency over the same interval.

Per-program stats capture in-kernel cost only. The user-space side of the sensor shows up in ordinary pod metrics, so measure both.

eBPF vs. sidecar agents and traditional tools

The structural difference is the deployment unit. An eBPF sensor is a DaemonSet, one pod per node; a sidecar is one proxy per pod; a user-space APM agent is a process plus per-language SDKs. Measured costs from current sources:

The multiplication is what hurts with sidecars, but the arithmetic depends on load. Istio’s 0.20 vCPU figure is per proxy at 1,000 requests per second, not a fixed per-pod tax. A node running 50 sidecars that each sustain 1,000 req/s would carry roughly 10 vCPU and 3 GB of proxy overhead where an eBPF DaemonSet runs a single pod. Halve the request rate and you roughly halve the CPU side, though the memory floor per proxy stays. Run the numbers against your own per-pod traffic before quoting a total.

Per-language agents carry a different tax: one process plus per-language SDKs, with resource use that scales outside the DaemonSet model. SystemTap’s per-event cost is competitive with eBPF, but ACM ICPE 2025 measurements found that its data-transfer path degrades disproportionately under load, and its compile-to-kernel-module pipeline carries the startup and stability costs above.

The hidden cost: overhead on untraced processes

Installing a hookpoint taxes every process that crosses it, whether or not you are tracing that process. An ACM SIGCOMM eBPF ‘24 paper on Linux 6.8 measured pre-eBPF filtering slowing an untraced process’s read syscall by 54 ns, a 15% slowdown, and sendmsg by 112 ns (6%). Untraced memcached throughput dropped between 1.5% and 10.1% depending on whether filtering ran before, inside, or after the eBPF program.

The effect gets larger with network hooks. ACM CoNEXT 2025 measured a co-located, untraced echo server’s median response latency increasing 45% when an eBPF offload ran on the same host. Cilium’s own documentation acknowledges Hubble overhead somewhere in the 1–15% range, depending on traffic patterns and aggregation settings.

The failure mode compounds at scale: a 2014 LKML report of enabling roughly 17,000 kprobe events describes a system slowed almost to a hang.

For multi-tenant planning, eBPF monitoring overhead lands on node allocatable capacity, so per-namespace quotas never see it. USENIX Security 2023 states the scope plainly: eBPF tracing can observe every process on the kernel, including the host’s and other containers’. Budget it against node allocatable capacity and hold every candidate tool to a hookpoint inventory before it touches a shared cluster.

Techniques to reduce overhead in production

Once a tool is deployed, most of its overhead comes from a handful of controllable decisions:

• Filter and aggregate as early as you can, in the kernel first. Data transport is expensive: the CHEOPS 2026 storage paper attributed up to 24% of total tracing overhead to userspace event retrieval versus 11% to probe triggering. Datadog’s eBPF file-integrity monitoring shows the layered version of this. In-kernel approvers and discarders pre-filter up to 94% of events before they cross the ring buffer, which is what keeps the agent from dropping events. A second, richer evaluation in user space then cuts what actually ships to the backend, taking a fleet-wide stream of over 10 billion events per minute down to roughly one million. Both stages earn their place: the kernel stage protects the node, the user-space stage protects the network and the bill.
• Use per-CPU maps on hot paths. Per-CPU hash maps remove lock contention between CPUs updating shared counters; pay the memory multiplier only where write contention exists.
• Keep the per-event path short. Every bpf_map_lookup_elem or bpf_probe_read adds to it. Cloudflare’s benchmark shows how fast key complexity alone adds up: an empty tracepoint costs 15 ns, a simple PID-keyed map increment 35 ns, and a composite key of PID plus a random value plus a 32-character command name 96 ns. Choose the narrowest key that answers your question.
• Filter selectively. Event volume dominates: the FOSDEM 2021 comparison found tracing only scheduling events cost about 3% while tracing all events cost 107–174%. Attach only the hookpoints you need; a bpftrace project discussion puts the practical limit around 500k events per second before you must narrow the syscall set.
• Prefer ring buffers for transport, and tune notifications. In the kernel’s own ringbuf benchmarks, a shared ring buffer hit 12.054 million records per second against 0.931 million for a perf buffer waking the consumer on every sample. Give perfbuf sampled notification and it reaches 9.889 million against ringbuf’s 11.563 million, so most of that headline gap is notification cost rather than the buffer itself. Ring buffers still win on ordering and on memory scaling as CPU count grows.
• Confirm the JIT is active. On kernels older than 5.6 without CONFIG_BPF_JIT_ALWAYS_ON, check net.core.bpf_jit_enable; the interpreter multiplies per-invocation cost several-fold.

Sampling barely helps with uprobe-based tools. The TU Delft Beyla measurements found 100% and 10% sampling nearly identical in cost (1.5 m vs. 1.6 m CPU) because kernel uprobes fire on every call regardless of the sampling decision. Sampling shrinks stored data volume while the probes keep firing at full rate.

How overhead scales as your cluster grows

Node count is the benign axis. A DaemonSet adds one sensor per new node, and each sensor’s cost depends on that node’s local event rate, not on cluster size. The capacity-planning model per node is the linear one from earlier: sum each hookpoint’s average invocation cost times its firing rate, add map memory and transport CPU.

Event rate is the axis that bites, and it can exhaust a single core before it exhausts the node. Falco’s 2021 capacity guidance put 0.5 cores and 512 MiB at a ceiling of 150K syscalls/sec, and its single-threaded hot path cannot spread past one CPU. Those 2021 figures predate the modern eBPF probe; Falco 0.40.1 cut CPU by about 38% and memory by about 30% against 0.39.2, so the capacity table works only as an order-of-magnitude anchor. Watch for drop counters before you trust any dashboard fed by a saturated sensor.

Two nonlinear effects break the simple model. Map cardinality degrades throughput: CoNEXT 2025 measurements show hash-map throughput 30.5% lower at 100K flows than at 100. Published scaling curves for probe count are scarce; the SIGCOMM eBPF ‘24 work found in-eBPF overhead scaling with both programs per hookpoint and hookpoint count, and one NetDev study measured a 46% throughput drop for full-functions tracking.

Kernel version sets your feature floor. The x86-64 eBPF JIT dates to 3.16, BTF vmlinux support for CO-RE portability arrived with CONFIG_DEBUG_INFO_BTF in 5.2, kernel.bpf_stats_enabled in 5.1, and BPF_MAP_TYPE_RINGBUF in 5.8. The low-overhead transport and measurement features all favor 5.8 or later.

How groundcover approaches eBPF overhead

groundcover’s Flora eBPF sensor deploys as a single DaemonSet, one pod per node, and captures observability telemetry and Kubernetes events from Linux workloads with no SDKs, no per-service agents, and no application restarts. Flora is purpose-built for Kubernetes and Linux workloads and needs kernel-level access; full-cluster visibility is available within hours of deployment.

The architecture applies the highest-return technique from the previous section: Flora uses kernel-level collection and in-flight processing so raw event volume shrinks before it ever crosses into user space.

In groundcover’s own head-to-head benchmark, Pixie added 32% CPU and 9% memory over application baseline at 3,000 requests per second. These are vendor-run numbers on our own test rig, and you should treat them the way we suggest treating everyone else’s: the measurement workflow above lets you reproduce them on your own nodes.

The collection unit is per node, so groundcover defaults to 100% signal fidelity across observability telemetry rather than using sampling as the main CPU-control lever. The sampling data from Beyla shows why this is the right trade for probe-driven collection: sampling saves storage while the CPU cost of the probes stays put, so discarding signal buys little. All telemetry lands in ClickHouse and VictoriaMetrics inside your own VPC under the default BYOC (Bring Your Own Cloud) architecture, so transport overhead stays inside your network too.

The free tier includes the full BYOC architecture, so you can deploy Flora on a single cluster, enable kernel.bpf_stats_enabled, and read run_time_ns off the programs yourself.

FAQ

These are the questions platform teams ask most often when sizing an eBPF sensor.

How much CPU and memory does eBPF monitoring use compared to sidecars?

Production eBPF deployments at DoorDash and Netflix each report under 1% of node CPU and memory combined, and the ISPASS 2024 academic study found sub-0.5% median overhead on datacenter workloads. Istio’s official 1.24 benchmark puts each sidecar at roughly 0.20 vCPU and 60 MB per pod at 1,000 req/s, so sidecar cost multiplies with pod density and per-pod traffic while a DaemonSet sensor stays at one pod per node. Under adversarial conditions (small I/O, high concurrency), the CHEOPS 2026 storage paper measured eBPF tracing overhead as high as 41%, so validate against your own event rates.

How do I profile eBPF overhead with bpftool?

Set kernel.bpf_stats_enabled=1, then run bpftool prog show and divide run_time_ns by run_cnt for average nanoseconds per invocation. bpftool prog profile adds hardware counters such as cycles and instructions if your kernel has BTF and bpftool was built with a skeleton. The stats instrumentation itself adds roughly 20 ns per invocation, so disable it after measuring.

Which hook types have the lowest overhead?

Tracepoints and fentry, at 15 ns and 24 ns per empty-program invocation in Cloudflare’s benchmark, against 137 ns for a kprobe and roughly 1,670 ns for a uprobe. Those absolute values come from a virtualized ARM test rig and will differ on your hardware, but the ordering holds everywhere it has been measured: fentry and tracepoints first, then kprobes, then kretprobes, with uprobes roughly two orders of magnitude above a tracepoint. Ask any vendor which attachment mechanisms their sensor uses on your hot paths.

Do eBPF map types change the memory footprint?

Substantially. Standard hash maps preallocate for max_entries by default, per-CPU hash maps multiply value storage by the number of possible CPUs in exchange for lock-free updates, and ring buffers allocate one physical copy of their pages while appearing twice in virtual memory. Since kernel 5.11 under cgroup v2, the kernel charges BPF memory to the sensor’s cgroup, so an undersized memory limit triggers the OOM killer.

Does eBPF monitoring slow down processes it isn’t tracing?

Yes. Measurements on Linux 6.8 show a hookpoint adding 54 ns (15%) to an untraced process’s read syscall, and a co-located untraced server’s median latency rising 45% under an eBPF network offload on the same host. Treat hookpoint installation as a node-wide cost in multi-tenant capacity plans.

What kernel version do I need?

Flora needs Linux-based Kubernetes nodes with kernel-level access. For the full low-overhead toolkit, target 5.8 or later: that gets you BPF_MAP_TYPE_RINGBUF, fd-scoped BPF_ENABLE_STATS, BTF-based CO-RE portability (5.2), runtime stats (5.1), and a JIT that is on by default on x86-64 and arm64 (5.6).