Benchmark Kubernetes observability agent overhead

A DaemonSet agent requesting 200m CPU per pod reserves 10 cores of scheduling capacity across a 50-node cluster, even if the agents use far less. kubectl top node shows actual CPU and memory usage, while kubectl describe node lists those requests under Allocated resources. The requests reduce scheduling headroom without changing the node's reported allocatable value, so a cluster can look underused even when capacity is already committed to monitoring. That's why observability overhead needs to be measured rather than inferred from utilization alone.

In this guide, you'll learn how to benchmark Kubernetes agent CPU and memory usage at idle and under load, use the results to set resource requests and limits, and compare agents under consistent conditions.

What benchmarking Kubernetes agent overhead means

Benchmarking Kubernetes agent overhead means measuring the CPU and memory an agent consumes at idle and under load, along with any additional resource usage it causes outside its pod. The agent's cgroup metrics show its direct CPU and memory consumption in millicores and MiB. To capture the broader impact, compare node-level usage before and after deploying the agent under the same workload. Published comparisons and resource configurations provide useful reference points, but they can't replace measurements from your own cluster.

This is different from the CIS Kubernetes Benchmark, which defines security configuration recommendations for Kubernetes. CIS compliance scanning checks configurations against those recommendations, while agent overhead benchmarking measures resource consumption and performance impact.

Resource requests matter because Kubernetes uses them to decide whether a pod can fit on a node, regardless of how much CPU or memory it actually consumes. Node allocatable is the capacity available for pods after system reservations and applicable eviction thresholds. An agent's requests consume part of that scheduling headroom without changing the node's reported allocatable value. If you set a resource limit without a request, Kubernetes uses the limit as the request unless an admission-time default provides one.

For a DaemonSet, those requests accumulate across nodes. Each eligible node running an agent pod carries its own reservation, so the cluster-wide scheduling cost grows with node count even when actual resource usage remains low.

How agent overhead measurement works

Agent overhead shows up in two places: the CPU and memory the agent consumes directly, and the additional work it causes elsewhere on the node. Cgroup metrics such as container_cpu_usage_seconds_total and container_memory_working_set_bytes capture the agent's direct consumption. Comparing node usage with and without the agent under the same workload reveals its broader impact. Measurements at idle, under sustained load, and during bursts show how that overhead changes with telemetry volume.

Baseline overhead before any agent

Kubernetes consumes CPU and memory before you install an observability agent. Kubelet, containerd, and container shim processes all contribute to the node's existing resource usage. This baseline changes with pod density, traffic, and cluster configuration, so it needs to come from your own environment.

An agent's cgroup metrics already isolate its direct consumption. You therefore don't subtract kubelet or containerd usage from those measurements. Instead, compare total node usage before and after installing the agent under an equivalent workload. This captures both the agent's direct consumption and any additional work it causes elsewhere on the node.

For node-level measurements, the kubelet exposes node_cpu_usage_seconds_total and node_memory_working_set_bytes through /metrics/resource. If Prometheus scrapes that endpoint, calculate node CPU usage in millicores with:

rate(node_cpu_usage_seconds_total{instance="NODE_KUBELET_TARGET"}[5m]) * 1000

Measure the node's memory working set in MiB with:

node_memory_working_set_bytes{instance="NODE_KUBELET_TARGET"} / 1024 / 1024

Replace NODE_KUBELET_TARGET with your kubelet scrape target, or select the node using the labels available in your monitoring setup. Use equivalent workloads and matching measurement windows for both runs.

For example, suppose you collect these measurements before and after installing an agent:

The node's CPU usage increased by 150m, while the agent consumed 100m directly. That leaves an estimated 50m of additional CPU usage outside the agent's cgroup, assuming other node activity remained consistent.

Memory requires more care. The node's working set increased by 180 MiB, while the agent reported 140 MiB. The remaining 40 MiB isn't an exact measurement of indirect memory overhead because node and container working-set measurements aren't necessarily additive.

This distinction matters when comparing agents. The same agent can have different total overhead on nodes with different workloads, pod densities, or runtime configurations, even when its direct CPU and memory measurements are similar.

Measuring agent CPU and memory footprint

Cgroup metrics isolate the CPU and memory consumed by an agent's containers. You can get a quick view with kubectl top pod --containers, which displays current usage through Kubernetes Metrics Server. Its memory figure represents the working set, which can include some cached memory, rather than total memory usage.

However, Metrics Server retains only recent values and is intended primarily for autoscaling. To compare consumption across longer periods, use Prometheus with cAdvisor metrics.

The following queries measure CPU, memory, and throttling for an agent. Replace AGENT_NAMESPACE and AGENT_POD_REGEX with the namespace and pod-name pattern for your deployment.

CPU usage

container_cpu_usage_seconds_total tracks cumulative CPU time. Apply rate() to calculate CPU usage in cores, then multiply by 1,000 to convert it to millicores:

sum by (namespace, pod) (
  rate(container_cpu_usage_seconds_total{
    namespace="AGENT_NAMESPACE",
    pod=~"AGENT_POD_REGEX",
    container!="",
    container!="POD"
  }[5m])
) * 1000

The five-minute window smooths short spikes, making it useful for measuring sustained consumption. To find the highest one-minute CPU rate over a 30-minute period, use:

max_over_time((
  sum by (namespace, pod) (
    rate(container_cpu_usage_seconds_total{
      namespace="AGENT_NAMESPACE",
      pod=~"AGENT_POD_REGEX",
      container!="",
      container!="POD"
    }[1m])
  ) * 1000
)[30m:1m])

Adjust the rate window and test period to match your scrape interval and workload duration.

Memory usage

container_memory_working_set_bytes tracks the memory working set. Divide by 1,024 twice to convert bytes to MiB:

sum by (namespace, pod) (
  container_memory_working_set_bytes{
    namespace="AGENT_NAMESPACE",
    pod=~"AGENT_POD_REGEX",
    container!="",
    container!="POD"
  }
) / 1024 / 1024

You can also use container_memory_rss to inspect resident memory. RSS and working set represent different measurements, so use the same metric across agents and test runs. Capture both average and peak memory over the measurement period.

CPU throttling

CPU throttling reveals whether configured CPU limits restrict the agent's processing. Calculate the ratio of throttled enforcement periods to total enforcement periods with:

sum by (namespace, pod) (
  rate(container_cpu_cfs_throttled_periods_total{
    namespace="AGENT_NAMESPACE",
    pod=~"AGENT_POD_REGEX",
    container!="",
    container!="POD"
  }[5m])
) /
sum by (namespace, pod) (
  rate(container_cpu_cfs_periods_total{
    namespace="AGENT_NAMESPACE",
    pod=~"AGENT_POD_REGEX",
    container!="",
    container!="POD"
  }[5m])
)

The result shows how frequently throttling occurred, not the percentage of CPU time lost. It's most useful when the agent has an enforced CPU quota.

These queries combine container measurements into one result per pod. For a DaemonSet running one agent pod per node, each result represents that pod's footprint on its node. To measure total deployment consumption, aggregate the matching pods with sum() instead of sum by (namespace, pod). For throttling, aggregate the underlying counters before calculating the ratio rather than adding per-pod percentages.

Metric availability depends on the container runtime, cgroup version, and monitoring configuration. Confirm that the expected series exist before interpreting the results.

Idle vs active state consumption

An agent's CPU and memory usage changes with the telemetry it processes. Measuring only its idle footprint can underestimate the resources it needs during normal traffic or sudden increases in activity.

Three operating conditions reveal different aspects of its consumption:

• Idle: Shows the background CPU and memory used when no additional traffic is generated. For a DaemonSet, this footprint is repeated across its nodes.
• Sustained load: Shows resource consumption while processing telemetry at a steady rate representative of normal operation. These measurements help determine appropriate resource requests.
• Bursts: Reveal short CPU spikes, peak memory usage, and whether buffers grow when incoming telemetry exceeds the agent's processing capacity.

CPU and memory limits also behave differently under load. When an agent reaches its CPU limit, throttling delays processing and can cause telemetry queues to grow. If it exceeds its memory limit, the container can be OOMKilled, interrupting collection and potentially losing buffered data.

Sustained CPU usage helps size CPU requests, while peak memory consumption helps establish a memory limit with enough headroom for bursts. CPU peaks and throttling measurements also show whether a configured CPU limit is restricting collection.

This is why two agents with similar idle consumption can behave differently under load. The comparison needs to account for how their resource usage changes as telemetry volume increases.

Secondary overhead sources

CPU and memory measurements alone don't explain every difference between agents. Telemetry volume, networking activity, and collection configuration can affect both direct consumption and the wider node impact.

• Metric volume: For agents that maintain active time series, higher cardinality can increase memory consumption, while higher sample rates increase CPU and network activity. Changing a scrape interval from 15 to 60 seconds reduces samples collected by 75% when the targets and active series remain unchanged. Keep scrape intervals, target sets, and collection settings consistent when comparing agents. Learn more about metric cardinality.
• Network and kernel activity: CNI components, kube-proxy, and kernel-level instrumentation can affect total node usage outside the agent's containers. Keep the network configuration and traffic conditions consistent, and account for additional processing caused by the agent when interpreting node-level measurements.
• Buffering and duplicate collection: Slow exports can cause telemetry queues to grow, increasing memory consumption. Duplicate scraping can also increase processing and network traffic. Check for growing buffers, OOMKilled events, and targets being collected more than once before treating unusually high resource usage as normal agent overhead.

These factors help explain why the same agent can produce different results across environments. Separating direct consumption from the wider node impact makes those differences easier to interpret.

Benchmark setup and methodology

Benchmark Kubernetes agent overhead by measuring node CPU and memory under a repeatable workload without the agent, then repeating the test with it installed. Track container_cpu_usage_seconds_total, container_memory_working_set_bytes, and CPU throttling during idle, sustained-load, burst, and pod-churn scenarios. Compare the before-and-after node measurements for total overhead and the agent's cgroup metrics for direct consumption. Repeat each scenario under the same conditions to check consistency.

Prepare a consistent test environment

A meaningful comparison requires the same cluster environment, workload, and measurement conditions for every agent. Before testing, establish the following:

• Cluster and hardware: Keep the worker-node count, pod density, instance type, CPU architecture, vCPU count, and memory capacity consistent. Record whether the agent runs as a DaemonSet, sidecar, or central collector, since the deployment model determines how its resource usage scales.
• Kubernetes and runtime: Use the same Kubernetes distribution and version, container runtime, and cgroup version. Check whether PodAndContainerStatsFromCRI is enabled and whether the kubelet obtains container statistics from the CRI or cAdvisor. The source can affect reported measurements, so use consistent metrics throughout the comparison.
• Agent configuration: Keep telemetry signals, sampling rates, collection scope, and processing settings equivalent wherever possible. Note each agent's version, resource requests, and limits. Agents receiving the same telemetry volume may still perform different amounts of processing.
• Workload: Use the same application, pod count, traffic rate, event sizes, and test duration. Run idle, sustained-load, and burst scenarios separately so changes in traffic don't distort the results.
• Metrics collection: Use consistent scrape intervals and query windows. Confirm that CPU, memory, and throttling metrics are available before starting. Include network throughput and disk I/O where relevant.
• Warm-up and repetition: Allow each agent to initialize before measuring steady-state consumption. Use the same warm-up and measurement periods, and repeat tests to identify variations between runs.

Identical CPU and memory limits alone don't make two agents comparable. Their configurations and telemetry-processing workloads must also be considered.

Run the benchmark

With the environment prepared, follow these steps:

1. Establish the baseline. Run the application workload without the observability agent and measure node CPU and memory. Keep the monitoring infrastructure and workload generator consistent between runs so their resource consumption doesn't distort the comparison. Establish a matching agent-free baseline for each workload scenario.
2. Deploy the agent and allow it to warm up. Install the agent with the configuration you're testing. Confirm that it's collecting telemetry and allow its resource usage to stabilize. Exclude startup activity from steady-state measurements.
3. Measure idle consumption. With no additional generated traffic, measure the agent's CPU and memory usage over a fixed period. This captures the background footprint that remains when application traffic is low.
4. Apply sustained load and bursts. Generate telemetry at a fixed rate, then introduce short increases above that rate. For OpenTelemetry traces, telemetrygen supports controlled trace generation. Use an appropriate generator for logs, metrics, or application traffic, depending on what the agent collects. Keep event rates, payload sizes, and test durations consistent, and verify that the expected telemetry is processed rather than dropped.
5. Test pod churn separately. Use kube-burner or ClusterLoader2 to create and delete pods at a controlled rate. Repeat the same workload with and without the agent. Compare API-server request rates using apiserver_request_total and request latency using apiserver_request_duration_seconds. Where available, check etcd_request_duration_seconds and kubelet_pod_start_duration_seconds for changes in etcd latency and pod startup time.
6. Repeat and compare. Run each scenario several times under matching conditions. When comparing different agents, test them separately where possible and restore the same starting conditions between runs. Use the queries from the previous section to compare each agent's direct consumption with the corresponding node-level baseline.

For every scenario, measure average and peak CPU in millicores, average and peak memory working set in MiB, and CPU throttling where limits are enforced. CPU usage is calculated over a rate window, while memory is measured at individual points in time. Keep the measurement windows consistent and compare the variation between repeated runs.

You can also check secondary costs:

• Network throughput: Use the rates of container_network_receive_bytes_total and container_network_transmit_bytes_total to measure incoming and outgoing traffic.
• Disk I/O: Use the rates of container_fs_reads_total and container_fs_writes_total to measure read and write operations per second.
• Control-plane activity: Compare API-server request rates and latency against the agent-free baseline during the same pod-churn workload.
• Application latency: Compare p95 and p99 request latency under the same sustained traffic with and without the agent. Higher latency can reveal application overhead that isn't visible in the agent's container metrics.
• API responsiveness: Compare request latency using apiserver_request_duration_seconds, particularly during pod churn. This helps identify whether monitoring activity is affecting Kubernetes API responsiveness.
• Container startup time: Use kubelet_pod_start_duration_seconds to measure the time from when kubelet first sees a pod until it starts running. Where available, kubelet_pod_start_total_duration_seconds measures startup from pod creation, including image pulls and init containers. Compare equivalent pod-creation workloads to identify startup delays.

Metric availability depends on the Kubernetes environment and monitoring setup. API-server and etcd metrics also reflect activity from other cluster components, so a change between runs isn't automatically attributable to the agent.

When interpreting the results, keep direct and total overhead separate. The agent's cgroup metrics capture its own CPU and memory consumption, while the change in node usage captures its broader impact. As explained earlier, node and container memory working sets aren't necessarily additive, so their difference shouldn't be treated as an exact measure of indirect memory consumption.

A useful benchmark should establish how much CPU and memory the agent consumes and whether it handles the expected telemetry without excessive throttling, memory growth, or dropped data.

Using benchmark results to size and compare Kubernetes agents

Benchmark results help you determine how much CPU and memory an observability agent needs, how much scheduling capacity it reserves, and whether another agent could perform the same work with fewer resources. The important distinction is between configured resource requests and actual consumption. Requests affect where Kubernetes can schedule pods, while measured usage shows what the agent consumes under load.

For a DaemonSet, each agent pod reserves resources on its node. You can express the reservation and measured consumption as percentages of the node's allocatable capacity. Kubernetes exposes these values through kube_node_status_allocatable and kube_node_status_capacity when kube-state-metrics is installed.

Use these formulas:

• Reserved capacity (%) = Agent resource request ÷ Node allocatable × 100
• Consumed capacity (%) = Measured agent usage ÷ Node allocatable × 100

For example, consider a node with 16 CPU cores, of which 14.5 are allocatable. An agent requesting 200m CPU reserves approximately 1.38% of allocatable CPU. If it consumes 50m under sustained load, its actual consumption is only 0.34%. You can apply the same calculation to memory using the memory request and measured working set.

To estimate cluster-wide consumption, add the measurements across all agent instances. A DaemonSet typically runs one pod per eligible node, while sidecar instances scale with the pods they accompany. Account for differences in node capacity and workload instead of multiplying a single measurement across dissimilar nodes.

Resource requests and limits

Use measurements from representative workloads to set resource requests and limits. Requests should accommodate normal consumption with headroom for variation, while limits control what happens when the agent needs more resources.

CPU and memory require different decisions:

• CPU requests: Base these on sustained usage with enough headroom for normal fluctuations. Oversized requests reduce scheduling capacity, even when the agent consumes much less CPU.
• CPU limits: Reaching a CPU limit causes throttling, which can delay telemetry collection and increase queue sizes. Leaving the limit unset avoids quota-based throttling, although the agent can then use additional CPU when capacity is available.
• Memory requests: Account for the agent's typical working set and expected variation. A higher request reserves more scheduling capacity, while a lower request can increase eviction risk under node memory pressure.
• Memory limits: Set these above measured peaks with enough headroom for bursts. Exceeding a memory limit can cause an OOMKill, interrupting collection and potentially losing buffered telemetry.

Setting a memory request equal to its limit reserves that amount on every node, including capacity the agent rarely uses. It also doesn't automatically give the pod Guaranteed QoS, which requires matching CPU and memory requests and limits under the applicable Kubernetes rules.

As a configuration example, groundcover's resource-tuning documentation provides the following settings for its eBPF sensor in a large cluster:

These values illustrate how requests and limits can differ. They aren't measurements of actual sensor consumption or universal recommendations. The sensor runs as a per-node DaemonSet, while other groundcover components, including the Kubernetes watcher, have separate resource settings. Use your own measurements to determine suitable values.

Comparing agents head-to-head

Comparing agents requires more than measuring CPU and memory. The agents must process equivalent telemetry under consistent conditions, and you need to check whether they successfully handle that workload. Lower consumption means little if an agent drops events or collects fewer signals.

A March 2026 Kubernetes log-collector benchmark measured CPU and memory consumption at approximately 10,000 logs per second. The collectors ran on a single-node kind cluster with 1-core CPU and 1 GiB memory limits, using their Helm chart configurations without additional performance tuning. The results come from one benchmark workload, so validate them against your own workloads.

The results were:

CPU consumption ranged from 62m to 655m, while memory consumption ranged from 27.91 to 153.50 MiB. The collectors ran simultaneously on the same node. Actual figures vary with your traffic, telemetry configuration and cluster conditions.

These figures show how resource consumption differs under a common log workload. They don't establish default requirements for the collectors or predict their performance when processing metrics, traces, or different log volumes.

The same principle applies when comparing conventional collectors with eBPF-based sensors such as groundcover's eBPF sensor. A log collector's resource footprint isn't directly comparable to that of a sensor capturing application and network activity through kernel hooks. For a meaningful comparison, measure the sensor's own CPU and memory consumption, any additional overhead on the application and node, and the telemetry each agent captures.

Ultimately, the better choice is the agent that provides the required visibility with acceptable CPU, memory, and scheduling costs under your workload.

How deployment and collection methods affect agent overhead

The resources an observability agent consumes depend on where it runs and how it collects telemetry. Per-node agents, sidecars, and API-based collectors place different demands on cluster resources. Some overhead appears in the agent's own containers, while other costs show up in application CPU, kernel activity, or Kubernetes control-plane operations.

Agent-based vs agentless collection

Agentless monitoring collects available Kubernetes and cloud metrics through APIs without installing a monitoring component on every node. This avoids the resource consumption of per-node collectors, but API requests and backend processing still have a cost. It also provides less visibility into application execution and network activity than node-level instrumentation.

Agent-based monitoring introduces a measurable resource footprint in exchange for deeper runtime visibility. groundcover uses a per-node eBPF sensor to collect supported application and network telemetry without requiring SDKs or separate instrumentation agents inside individual services. This provides application visibility while keeping collection separate from application containers.

Per-node agents vs sidecars

A DaemonSet runs an agent on each eligible node, while a sidecar runs alongside individual application pods. In a three-node cluster with 20 services and five pods per service, the sidecar approach creates 100 collector instances compared with three per-node agents.

This difference affects how overhead scales. Sidecar resource consumption grows as more application pods are deployed, while the number of DaemonSet agents grows with the number of eligible nodes. However, each per-node agent must process telemetry from the workloads running on its node, so its CPU and memory usage can increase as container count and traffic grow.

A central gateway introduces another processing layer. It can handle shared tasks such as filtering, batching, and exporting telemetry, but its resource consumption must also be included when calculating the total cost of collection.

Telemetry volume and control-plane overhead

Agent consumption also depends on how much telemetry the cluster generates. More containers can produce more logs, metrics, and traces, increasing CPU processing, memory buffering, and network traffic.

Metric cardinality is another factor. Each additional combination of metric labels creates a distinct time series. For collectors that maintain series state, higher cardinality can increase memory usage, while more frequent sampling increases the number of data points processed and transmitted. That's why two clusters with the same node count can have different monitoring overhead.

API-based collection has a different cost. Tools that query or watch Kubernetes objects can increase requests to the kube-apiserver and contribute to etcd activity. As the number of monitored objects and updates grows, this load can change even when the collector's own CPU and memory consumption remain low.

The impact is best measured by comparing API-server request rates and latency, along with etcd latency, under the same pod-churn workload with and without the agent. The benchmark methodology above covers the relevant metrics and test procedure.

eBPF collection and application overhead

eBPF changes how application telemetry is collected. Instead of deploying SDKs, language agents, or separate instrumentation components inside every application, a per-node eBPF sensor can observe supported application and network activity through kernel-level instrumentation.

This reduces the number of instrumentation components needed across services while keeping collection separate from application containers. The sensor still consumes CPU and memory, and its kernel programs perform work when monitored events occur. Measuring both the sensor and the application therefore gives a more complete view of overhead than measuring the sensor pod alone.

groundcover's eBPF sensor uses this approach. It runs as a per-node DaemonSet and captures supported application and network telemetry without requiring application code changes. Kubernetes metadata is handled by separate components such as k8s-watcher.

In a benchmark of a Go HTTP application handling 3,000 requests per second, groundcover's eBPF sensor added just 9% CPU overhead to the application, with no measured increase in application memory. Under the same workload, the Datadog agent increased application CPU usage by 249% and memory usage by 227%.

When the application's resource consumption and monitoring overhead were measured together, groundcover used 73% less CPU than Datadog and 96% less memory than Pixie. These results show how node-level eBPF collection can reduce resource consumption while providing application visibility without per-service instrumentation.

What's next for low-overhead agents

As Kubernetes clusters generate more telemetry, the challenge is collecting logs, metrics, traces, and profiling data without increasing resource consumption unnecessarily. Node-level eBPF collection is helping reduce dependence on application-specific instrumentation, while shared telemetry pipelines make it easier to process multiple signals without deploying separate agents for every service.

eBPF profiling and shared telemetry pipelines

OpenTelemetry's adoption of eBPF profiling reflects the growing interest in collecting performance data without modifying application code. With OpenTelemetry Profiles entering public alpha, its eBPF profiler can operate as a Collector receiver and use existing pipelines to process profiling data alongside other telemetry signals.

This development extends low-overhead collection to continuous profiling, where capturing CPU activity across applications can help identify performance bottlenecks. It also shows how instrumentation and telemetry processing are becoming more closely integrated, reducing the need for separate collection components.

Keeping observability overhead and costs predictable

groundcover already uses node-level eBPF collection to capture supported application and network telemetry without requiring instrumentation inside individual services. Its sensor runs as a DaemonSet, allowing one component to observe workloads across a node rather than deploying an agent inside every application container.

This approach becomes particularly valuable as clusters grow. Adding services doesn't require a corresponding increase in application-level instrumentation agents, while the sensor's CPU and memory can be monitored and tuned independently of application workloads.

Collection efficiency also affects observability costs. groundcover's per-node pricing is based on monitored Kubernetes hosts rather than ingestion volume. Increasing log, metric, or trace volume therefore doesn't directly increase the platform's per-host charge, although additional telemetry can still consume more compute, storage, and network resources in the customer's environment.

As collection technologies improve, benchmarks will need to account for more than an agent's idle CPU and memory usage. Measuring consumption under sustained traffic, increasing container counts, and higher telemetry volumes will show whether those improvements translate into lower resource costs in production.