Your CPU limits aren't stopping noisy neighbors
This post explains what a noisy neighbor is, why CPU requests (not CPU limits) determine how CPU time is divided under contention, and why memory contention ends in OOM kills and evictions instead of added latency. It then explains why sizing each workload at a percentile of its own usage still overloads nodes, and how groundcover's Autonomous Infrastructure addresses that.

Your dashboard shows the node at 40% average CPU utilization, yet p99 latency on the checkout service has doubled, and nothing about the checkout service changed. A batch job on the same node started a CPU-heavy run, and for a few seconds at a time it takes CPU time that checkout needs. You set CPU limits on every container because the standard advice is that limits keep one workload from starving its neighbors, and those limits are not preventing this.
This post explains what a noisy neighbor is, why CPU requests (not CPU limits) determine how CPU time is divided under contention, and why memory contention ends in OOM kills and evictions instead of added latency. It then explains why sizing each workload at a percentile of its own usage still overloads nodes, and how groundcover's Autonomous Infrastructure addresses that. Noisy neighbors are where cost and stability conflict: packing Pods onto fewer nodes lowers cost and raises the risk that co-located Pods compete for the same CPU and memory. The goal is to reduce cost while keeping workloads stable, and that starts with how a node's resources are shared.
What a noisy neighbor is
A noisy neighbor is a workload that takes a disproportionate share of a node's CPU, memory or I/O (for example a single workload that hogs most of the node's resource) and degrades the performance or availability of other workloads on the same node. Containers on a node share one Linux kernel, the node's CPU cores and its physical memory. Linux namespaces isolate what each container can see, and control groups (cgroups) limit and account for what each container can consume. How a noisy neighbor affects other workloads therefore depends on how cgroups divide each resource type.
Kubernetes resources fall into two categories, and contention behaves differently in each. CPU is a compressible resource: when demand exceeds supply, processes wait for CPU time and run slower. Memory, ephemeral storage and PIDs are incompressible: when demand exceeds supply, the kernel or the kubelet must terminate a process or evict a Pod to free capacity. CPU contention shows up as higher latency, and memory contention shows up as OOM kills, evictions and restarts. Both depend on how the scheduler places Pods on nodes, which is the next topic.
A request is a reservation, and the scheduler ignores actual usage
A resource request instructs the scheduler to reserve that amount of a node's capacity for the container. The scheduler sums the requests of all Pods on a node and compares that sum to the node's allocatable capacity. When the sum reaches allocatable capacity, the scheduler places no additional Pods on that node. The scheduler does not consider actual CPU or memory usage at any point in this calculation.
Because placement is based on requests rather than usage, cost and stability conflict in every cluster. If requests exceed usage, nodes reach full allocation while most of their CPU and memory stays unused, and you pay for idle capacity. If usage exceeds requests, the scheduler co-locates Pods whose combined peak usage can exceed the node's capacity. Usage above requests is not a problem in itself, because spare capacity on a node is there to be used. Noisy neighbors become possible when several co-located Pods exceed their requests at the same time, so the next question is how the kernel divides CPU and memory when combined usage exceeds capacity.
CPU: requests set the share under contention, limits do not
A CPU request continues to have an effect after scheduling, because it also sets the container's CPU weight. Kubernetes converts each requested core to 1,024 CPU shares on cgroups v1, or to the equivalent cpu.weight value on cgroups v2, which is the default on current Linux distributions. When CPU demand on a node exceeds supply, the Completely Fair Scheduler (CFS) allocates CPU time in proportion to those weights. If two CPU-bound containers compete, the container that requested 2 cores receives twice the CPU time of the container that requested 1 core. This proportional weight is what protects a workload from a noisy neighbor, and it applies only while the node's CPU is fully used.
CPU limits use a different mechanism. Kubernetes converts a CPU limit into a CFS quota, which is a fixed amount of CPU time per 100ms scheduling period. A container with a 0.5-core limit receives 50ms of CPU time per 100ms period. If that container runs four threads in parallel, it consumes the 50ms quota in about 12.5ms of wall-clock time and is throttled for the remaining 87.5ms. Throttling occurs even when other cores on the node are idle, so the limit adds latency to the container without reducing contention for other workloads.
Figure 1 (draft visual for design to replace): one 100ms CFS period for a container with a 0.5-core CPU limit, which gives it 50ms of quota per period.
With one thread, the container runs for 50ms and waits 50ms. With four threads, it uses the same 50ms of quota in 12.5ms and then waits 87.5ms, while other cores on the node sit idle.
Two other settings are often assumed to affect CPU allocation, and neither does. QoS class (Guaranteed, Burstable or BestEffort) does not change how much CPU time a container receives. Its main effect is setting the OOM score adjustment that determines which processes the kernel kills first under memory pressure. Pod priority determines preemption and eviction order, and the kubelet does not evict Pods for CPU pressure. Under CPU contention, the request is the setting that determines each container's share, which is why many teams remove CPU limits from production services and focus on accurate CPU requests. Memory has no equivalent proportional-share mechanism.
Memory: contention ends in OOM kills and evictions
Memory cannot be divided by weight, because a process cannot wait for memory the way it waits for CPU time. A memory request reserves capacity for scheduling, and a memory limit sets a hard ceiling that the cgroup enforces. A container that exceeds its memory limit is OOM-killed immediately. A container that uses more than its request but less than its limit keeps running until the node runs short of memory. At that point the kubelet evicts Pods, or the kernel OOM killer terminates processes first, and the order depends on QoS class and on how far each Pod's usage exceeds its request. Under memory contention, a noisy neighbor causes terminations, not added latency.
Consider a service that uses 1Gi of memory for most of the day and 4Gi at its 9am traffic peak. A 4Gi request prevents OOM kills at 9am, but 3Gi stays reserved and unused for about 23 hours a day, and the scheduler cannot place other Pods in that capacity. A 1Gi request allows denser packing, but at 9am the service needs 3Gi more than it requested. If two replicas of this service run on one node, they need an additional 6Gi at the same time. Whether that memory is available depends on the usage of every other Pod on the node, which per-workload sizing does not measure.
The isolation illusion
Most sizing tools, including VPA and common open source recommenders, size each workload independently. They analyze one workload's usage history, select a percentile such as P95, and recommend a request based on it. Each recommendation can be accurate for its workload while the node still overloads, because a P95 request does not account for when the remaining 5% of usage occurs. If five workloads on one node all reach their top 5% of usage during the same morning traffic peak, the node must serve all five peaks simultaneously. The Wand team, now part of groundcover, calls this the isolation illusion: a node can run out of CPU or memory even when every workload on it has P95-based requests.
Denser bin-packing increases this risk. Karpenter and the Cluster Autoscaler consolidate Pods onto fewer nodes and remove nodes with low allocation, which reduces cost. Over-provisioned clusters had unused capacity that absorbed overlapping peaks, and consolidation removes that capacity. If Pods are consolidated without data on which workloads peak at the same time, the cost reduction is offset by OOM kills, evictions and CPU contention.
Many environments show both problems at the same time. In one environment we analyzed, across ten clusters, about 80% of requested CPU was unused at P95. Its production nodes were 74 to 86% allocated by requests, but only about 11% of their CPU was used. In the same environment, non-production workloads used more memory than they requested and recorded more than 770 OOM kills per week. A second team, already running a commercial rightsizing tool with Karpenter, recorded about 21,600 OOM kills per week. Both teams had invested in sizing, and both results point to a node-level problem that per-workload tuning does not address.
What manual tuning achieves, and where it stops scaling
Experienced platform teams can reduce both waste and contention manually. They set CPU requests from load-test results and remove CPU limits. They set memory requests equal to memory limits for critical services and configure JVM and Go runtime flags so heap sizes match container memory. They use priority classes and dedicated node pools to separate critical services from batch workloads. On a small cluster with stable traffic, these practices address most of the stability problem, but at a high cost. Dedicated node pools split the cluster's capacity into smaller pools, which restricts bin-packing and leaves capacity stranded in each one. And no engineer tuning requests by hand can know in real time which workloads will be co-located on a node, so requests get padded for neighbors nobody can predict.
Manual tuning also depends on coordination across teams. The developers who write a service set its runtime flags, such as JVM heap settings, when they build it. The platform team sets requests and limits later, often across hundreds of services it did not write, and no single team owns the relationship between the two. The workload labels that sizing tools rely on are applied by different engineers following different conventions, and they drift when a Helm chart changes or an engineer moves to another team. Keeping practices current also requires every team to track changes in Kubernetes itself, such as the removal of cgroup v1 support in Kubernetes 1.35. These are organizational limits, not limits of skill, and they add to the technical limits described next.
Manual tuning stops scaling because sizing is a continuous control loop, not a one-time task. Each deployment, traffic change and newly co-located workload changes the correct values. Engineers add headroom to requests intentionally because they cannot monitor every node, and that headroom is the stability margin that appears as waste on the cloud bill. Engineers also cannot choose which Pods are co-located, or respond to node-level contention within seconds across hundreds of services. TCP congestion control is automated for the same reason, not because network engineers lack skill. A control loop at this speed belongs in software, with engineers defining the policy it enforces.
How groundcover reduces cost while assuring stability
groundcover's Autonomous Infrastructure reduces cost while assuring stability by sizing workloads based on node-level usage, not on each workload's history alone. It analyzes the actual CPU and memory usage of the workloads on each node to determine which peak at the same time, which peak at different times, and how much headroom the node needs. Workloads with non-overlapping peaks can share headroom instead of each reserving capacity for its own peak, which produces the cost savings. Workloads with overlapping peaks receive headroom sized for their combined peak, which preserves stability. Per-workload tools cannot calculate this, because they analyze one workload at a time.
Autonomous Infrastructure operates within limits you define. You select which workloads it manages. It lowers CPU and memory requests for over-reserved workloads, raises them for workloads that need more capacity, all within the boundaries you declared. It applies changes to running Pods, in place on Kubernetes versions that support in-place Pod resize, without modifying Deployment manifests, so your GitOps repository remains the source of truth. Raising requests to prevent contention is as much a part of its function as lowering them to reduce cost.
Node-level sizing depends on data as much as on the algorithm. It requires high-resolution, per-container usage history linked to node, workload and runtime, retained long enough to capture weekly patterns. groundcover collects that history automatically: its sensor runs as a DaemonSet on every node and uses eBPF to read CPU, memory and other resource data directly from the Linux kernel, at high resolution and without application code changes, SDKs or exporters. groundcover's BYOC deployment stores that data in your own cloud account, which makes full retention affordable without sampling. When an incident occurs, the groundcover investigator can record a noisy neighbor as a hypothesis and test it against the same data, citing which workloads overlapped on which node and at what time. Platform teams can use that evidence to verify that each cost reduction did not degrade stability.
Next: changing requests on running Pods
CPU and memory requests, not limits, determine how a node's resources are divided among competing workloads. CPU requests set each container's share of CPU time under contention, and memory requests influence which Pods are evicted first under memory pressure. Both reduce cost safely only when they reflect how co-located workloads use resources together. Until recently, however, a Pod's requests were fixed for its lifetime, and changing them required restarting the Pod. Even an accurate forecast of the 9am peak could not lower requests at 3am and raise them before 9am without restarts. In-place Pod resize, which reached GA in Kubernetes 1.35, removes that restriction, and it is the subject of the next post.








