Understanding APM agent CPU overhead

APM agents consume the same CPU budget as your production workloads. Datadog measured their Agent version 7.39.0 using two full cores at 130,000 spans/sec on an AWS c5.2xlarge, while the NodeSource benchmark measured a Datadog tracer cutting throughput by 69.2% on its Node.js 22.17.1 dashboard workload. Those costs come directly out of your capacity budget.

Telemetry data is growing at roughly 29% a year according to IDC figures quoted by Cribl, which doubles volumes about every 2.7 years. As those volumes grow, in-process and node-local agents run on the same nodes as your production workloads. First, this article defines APM agent CPU overhead and explains where the CPU goes. Second, it gives a diagnostic workflow and the configuration and sizing changes that reduce overhead. Third, it compares agent-based collection with eBPF and looks at where collection is heading. Finally, it closes with an FAQ.

What is APM agent CPU overhead

APM agent CPU overhead is the sum of two costs: the in-process instrumentation cost paid on every traced request, and the data-pipeline cost of serializing, compressing, batching, and exporting telemetry to a backend. Both are separate from your application's own CPU consumption, but they run on the same cores, inside the same containers, against the same cgroup limits.

Cores per spans-per-second is a useful practical unit for measuring overhead when the hardware and agent version are specified, because agent CPU tracks span throughput. Datadog's documentation states this directly: "The Agent is CPU-bound and its CPU usage is correlated with the number of spans received per second." Lower span throughput generally produces lower agent CPU demand; higher throughput can require whole cores on specified hardware.

That framing matters for capacity planning. A "3% overhead" marketing figure is meaningless without the span rate. Instrumentation depth and sampling configuration further shape the result. New Relic says as much: the company states it is "unable to provide exact information about how much CPU and memory resource consumption is expected" for its .NET agent because overhead varies with application characteristics and the combination of instrumentation points with throughput. Having defined the cost, the next question is where the cycles go.

How APM agent CPU overhead works

Span volume and instrumentation depth generate most agent CPU, while language-runtime specifics determine how much each operation costs. Insufficient CPU can then trigger a throttling-to-OOM cascade.

Span volume and throughput as the primary driver

Agent CPU scales close to linearly with traced request volume. Elastic's Node.js documentation states that "the amount of work the APM agent needs to do, generally scales linearly with the number of traced requests," and an ICPE 2026 microbenchmark quantified the per-invocation cost: 315.28 ns per invocation level for the OpenTelemetry Java agent 1.56.0 and 384.66 ns for Elastic APM 9.2.1, with R² above 0.999 in both cases (ICPE 2026 preprint).

Export is often the largest single contributor to agent overhead. A 2025 academic study of distributed tracing overhead across Python, Java, Go, and Node.js microservices measured median throughput reductions of 38.6% for OpenTelemetry v1.25.1 and 49.4% for Elastic APM v6.22.0, and identified trace-data export as the biggest overhead source (ICPE 2025 study). Coroot's eBPF-based measurement of an OpenTelemetry Go service at 10,000 RPS found CPU rising from 2 cores to about 2.7, with roughly 10% of total CPU spent inside BatchSpanProcessor (Coroot).

Code instrumentation and stacktrace collection

Stacktrace capture is a disproportionate cost you can disable independently of tracing. Elastic's agents expose span_stack_trace_min_duration (Java, default 5ms) and spanStackTraceMinDuration (Node.js, default -1s, meaning disabled); the Node.js agent ships with span stack traces off precisely "because of the possibility of this CPU overhead."

Stack frames also inflate payload size, which feeds back into pipeline CPU. Elastic's APM Server sizing guidance puts a span with 10 frames at about 4 KB and a span with 50 frames at about 20 KB, a 5x difference in bytes to serialize, compress, batch, and ship. Instrumentation depth carries a second, less visible cost: bytecode injection can inhibit JVM method inlining, adding overhead beyond the instrumentation code itself (Elastic apm-agent-java issue #971).

The CPU throttling cascade into memory and OOM

CPU starvation in an APM agent becomes a memory problem, then a crash. Datadog documents the chain explicitly: "The Agent buffers unprocessed payloads in memory, so throttling the Agent process because of an insufficient CPU limit can lead to an out-of-memory issue." The terminal state shows up in agent logs as a watchdog kill: Killing process. Memory threshold exceeded: 8238.08M / 715.26M.

The same pattern repeats across vendors:

• Elastic .NET agent: at CPU utilization above 60%, thread starvation stopped transaction and span delivery; the default 1,000-event queue filled and the agent dropped events (issue #1571).
• Elastic Java agent: under ~200 req/s load, the disruptor ring buffer exhausted with "Ring buffer has no available slots" until the operator raised max_queue_size (issue #975).
• OpenTelemetry Collector: high memory triggers frequent GC, GC burns CPU, and the CPU shortage "limits capability to offload existing queued data," ending in an OOMKill (issue #12450).

Kubernetes CFS quotas make the cascade worse. When a process exhausts its CPU quota for a period, the kernel throttles it until the next period; Indeed Engineering measured worst-case response latency dropping from over two seconds to 30 ms after removing CPU limits, and the diagnostic counters live in /sys/fs/cgroup/cpu.stat as nr_throttled and nr_periods (Indeed Engineering).

Language-specific overhead: Java and Node.js

Java agents pay through bytecode transformation and JIT interaction. The JVM invokes the agent's premain at startup and routes every loaded class through registered ClassFileTransformer instances (Java SE 21 API); the OpenTelemetry Java agent's nightly macrobenchmark shows startup time rising from 9,857 ms to 12,271 ms with the agent attached (OTel benchmark-overhead). Individual instrumentation modules can dominate: a field report on OTel Java agent 2.1.0 documented CPU utilization rising from ~35% to ~60% after OpenTelemetry enabled Reactor instrumentation by default (issue #13261).

Node.js pays through async context tracking, and the cost varies by an order of magnitude with workload shape. A promise-dense microbenchmark showed a 96.8% penalty with AsyncLocalStorage enabled, reduced to 49.42% after the V8 PromiseHook fix landed in Node v16.2.0, while a timer-dominated workload paid only 1.75% (Node.js issue #34493). At the agent level, NodeSource's benchmark dashboard on Node.js 22.17.1 measured 71,883.8 requests/sec for vanilla Node against 22,136.6 with Datadog (−69.2%) and 11,491.8 with New Relic (−84.0%) (NodeSource benchmark). Having established where the CPU goes, the next step is confirming whether the agent, and not the application, is the process burning it.

Diagnosing high CPU usage from an APM agent

Diagnosis starts with OS-level attribution and continues with runtime thread analysis. Profiling resolves ambiguous cases, while the agent's own metrics provide final confirmation. Work top down:

1. Attribute CPU to a process. Run Brendan Gregg's 60-second checklist: vmstat 1 (a run-queue r greater than CPU count means saturation), pidstat 1, then top to separate the agent process from the application.
2. Attribute CPU to a thread. top -H -p <PID> shows per-thread CPU. For JVMs, convert the hot thread's decimal Linux TID to hex with printf '0x%x\n' <tid> and match it against nid=0x... in jstack output, focusing on RUNNABLE threads (Baeldung workflow). Agent reporter and serialization threads show up by name here.
3. Profile if the thread dump is ambiguous. perf record -F 99 -p PID -g -- sleep 10 plus a flame graph, or asprof -d 30 -f flamegraph.html <PID> with async-profiler, which falls back to ctimer when container seccomp policies block perf_events (async-profiler docs).
4. Check the agent's own metrics. Datadog exposes datadog.trace_agent.cpu_percent as a gauge in percent of one core (50 means half a core, 200 means two cores); compare it against the configured DD_APM_MAX_CPU_PERCENT, which defaults to 50 in non-Kubernetes environments. The metric carries no deprecation notice as of Agent 7.82.0. On Elastic agents, set recording: false to confirm the agent is the cause without a restart; if CPU stays high, set enabled: false and restart.

One container-specific trap inflates JVM-based agent CPU: the JVM can size its thread pools from the host's CPU count rather than the container quota. Before full container support in JDK 10 (backported to 8u191), a JVM seeing 32 host cores would spawn roughly 20 GC threads for a workload entitled to 2 cores, and those GC threads then trigger CFS throttling (JDK-8146115). The fix is -XX:ActiveProcessorCount=N, which overrides the processor count used for GC and ForkJoinPool sizing; note that the JVM rounds container limits up (500m becomes 1 CPU, 1500m becomes 2), and a practitioner guide recommends setting the flag to roughly 2x the container CPU limit (Pretius). Once you have confirmed the agent as the source, the next question is how to cut its cost.

Key features to reduce APM agent CPU overhead

Overhead reduction comes from configuration and sizing against published throughput tables. A controlled A/B test then measures your actual delta.

Configuration knobs that cut overhead

Sampling rate is the highest-leverage knob because agent work scales with traced requests. Elastic agents default transaction_sample_rate (Java, Python) and transactionSampleRate (Node.js) to 1.0, tracing everything. Lowering the rate primarily reduces span recording and downstream processing, including export work; the sampler and other synchronous instrumentation paths still run. On Datadog, DD_TRACE_SAMPLE_RATE is deprecated: use DD_TRACE_SAMPLING_RULES instead, for example [{"sample_rate":0.1}].

Beyond sampling, four knobs carry documented impact:

• Span stack traces: set Elastic's span_stack_trace_min_duration to -1ms to disable collection entirely, or raise the threshold so only slow spans pay the cost. Node.js already defaults to disabled.
• Queue and buffer sizes: Elastic's maxQueueSize (Node.js default 1,024 events) trades memory against drops; lower values mean less heap and CPU overhead, higher values mean fewer lost events. On Datadog, the trace channel is unbuffered by default and setting DD_APM_TRACE_BUFFER=16 fixed payload drops in a documented case, at the cost of more memory (issue #22469).
• Unused instrumentation modules: a single module can be the whole problem. New Relic's Java agent 8.10.0 caused elevated CPU and memory through its HttpUrlConnection instrumentation, fixed in 8.11.0, with a documented flag to disable the module as a workaround.
• Check collection intervals: for Datadog integration checks driving agent CPU, raise min_collection_interval to 60 seconds or more in the integration's conf.yaml.

Sizing guidelines: spans per second vs CPU and memory

Datadog publishes a practitioner-usable cores-per-span-rate table. Datadog measured it with Agent 7.39.0 on an AWS c5.2xlarge:

Size Kubernetes CPU limits against this table with headroom, because an undersized limit triggers the throttling-to-OOM cascade described earlier. Elastic publishes APM Server throughput by memory tier instead: on Elastic Cloud, a 1 GB server handles roughly 19,000 events/sec on AWS while a 32 GB server handles 127,000, with no CPU-core column published. New Relic's Infinite Tracing on-premise reference sizes 3 collector instances at 4 vCPUs and 8 GB RAM each for about 16,000 spans/sec.

Backend saturation flows back into the agent. When the APM server slows or drops connections, Elastic agents buffer events in a bounded queue (512 events default in Java). When the queue fills, Elastic agents reject events, so you lose transactions and spans. A CFS-throttled Elastic APM Server compounds this; one documented case measured 349,657 ms of throttled CPU time without automaxprocs versus 57,422 ms with it, and warned that connected agents "will suffer from throttling and/or request timeout when shipping APM events" (apm-server issue #7967).

Measuring true overhead with A/B load testing

The only trustworthy overhead number for your workload comes from two identical load tests, one with the agent attached and one without. Hold these variables constant:

• JVM version, flags, heap size, and GC policy
• Application build, host, CPU governor, and backend endpoint
• Agent and sampling settings, plus the workload script
• Traffic schedule and test timing, including duration and ramp-up

Start a fresh JVM per run, because measurements within one JVM invocation are not statistically independent (OOPSLA 2007 study).

Warm-up discipline decides whether the numbers mean anything. The OpenTelemetry benchmark suite uses a 30-second warm-up; Datadog's PetClinic macrobenchmark uses 5 minutes; MooBench discards the first 1,000,000 of 2,000,000 executions (MooBench paper). Record JVM user CPU and total machine CPU separately, since Elastic agents serialize and compress in background threads whose cost never appears in request latency, and collect p95 at minimum rather than averages.

If you run the test with JMeter, follow its documented practices: CLI mode (-n -t <jmx> -l <jtl>) with listeners disabled, CSV output rather than XML, an explicit warm-up phase via Startup Delay and Duration on Thread Groups, a Graphite or InfluxDB Backend Listener to compare both arms, and never run JMeter on the application server itself (JMeter best practices). A well-run A/B test tells you what the agent costs; the architectural question is whether that cost has to live inside the application at all.

How APM agent overhead fits the observability landscape

In-process agents and eBPF sensors put the collection cost in different places. A language agent runs inside your application's process and pays per request in that process's CPU budget; an eBPF sensor reads telemetry at the Linux kernel from a separate process, so the collection cost moves to an out-of-band pod you can size and limit independently. In one independent 10,000 req/s Go benchmark, InfoQ reported that eBPF metrics collection used under 0.3 cores while full OpenTelemetry tracing used about 2.7 (InfoQ).

groundcover's Flora eBPF sensor is built on that model. Flora deploys as a Kubernetes DaemonSet, one pod per node, running out-of-band in its own namespace with no per-service SDKs or code changes. It requires no application restarts; a documented high-throughput sizing example sets CPU requests at 160m and limits at 800m per sensor pod (groundcover docs). In groundcover's own benchmark, a Go HTTP service under a constant 3,000 requests/second showed +9% application CPU overhead and +0% memory with Flora, versus +249% CPU and +227% memory with the Datadog agent; the same test produced the figures that Flora consumed 73% less total CPU than the Datadog agent and 96% less total memory than the Pixie agent (groundcover benchmark). These are vendor-run, directional figures: groundcover did not disclose the node hardware or agent versions, and groundcover describes the figures as lacking independent replication.

eBPF shifts overhead rather than erasing it, and it has real constraints. groundcover states that eBPF alone does not provide full waterfall traces; those still require instrumentation (groundcover blog), which is why groundcover accepts OpenTelemetry as a first-class data source alongside Flora-captured signals. Flora also requires privileged containers to load eBPF programs, needs Kubernetes 1.21 or later and Linux kernel 4.16 or later, and does not support AWS Fargate or Docker Desktop.

For a platform team, the operational trade is one DaemonSet to size and upgrade instead of a per-language agent matrix, and an overhead budget that no longer scales with instrumentation depth inside each service. If you want to validate that trade on your own cluster, the free plan includes BYOC (Bring Your Own Cloud, where your data plane runs in your own cloud account) and requires no credit card: deploy Flora on one cluster and evaluate full-cluster visibility within hours. The same pressure that makes per-service agents expensive is also reshaping where the rest of the industry collects data.

What's next for APM agent overhead

OpenTelemetry-native pipelines are consolidating the export path. Datadog shipped its DDOT Collector distribution in May 2025. The pipeline itself carries CPU cost worth watching: tail sampling added roughly 0.5 to 1.5 cores in one practitioner test (Medium analysis), and Elastic's disk-backed tail-sampling benchmark cut RSS by 51.7% at the price of 90.7% more CPU.