Kubernetes
November 18, 2026 • 11 min read

In-place Pod resize removes the restart, not the sizing problem

Until Kubernetes 1.35, the CPU and memory requests of a running Pod could not be changed. The API server rejected any change to an existing Pod's container resources, so changing them required deleting the Pod and creating a replacement, and a Pod kept the requests it started with until it was deleted. In-place Pod resize, which reached general availability (GA) in Kubernetes 1.35 (2025), removes that restriction. It does not determine what the requests should be, and that decision still determines whether a cluster can reduce cost without losing stability.

In-place Pod resize removes the restart, not the sizing problem
Anais Dotis
Anais Dotis
November 18, 2026
11 min read
Kubernetes

Until Kubernetes 1.35, the CPU and memory requests of a running Pod could not be changed. The API server rejected any change to an existing Pod's container resources, so changing them required deleting the Pod and creating a replacement, and a Pod kept the requests it started with until it was deleted. In-place Pod resize, which reached general availability (GA) in Kubernetes 1.35 (2025), removes that restriction. It does not determine what the requests should be, and that decision still determines whether a cluster can reduce cost without losing stability.

This post uses two examples to make these concepts concrete. The first is the service from the previous post, which uses 1Gi of memory for most of the day and 4Gi at its 9am traffic peak. It shows why fixed requests force a choice between paying for idle capacity and risking OOM kills. The second is a Java service whose heap size depends on its container memory limit, and it shows why changing resources on a running Pod can change how the application behaves. Using those examples, the post explains why requests were immutable, what in-place resize changes, what it leaves unsolved, and how groundcover's Autonomous Infrastructure uses it to reduce cost while keeping workloads stable.

Why requests could not change on a running Pod

Kubernetes originally treated a container's CPU and memory resources as immutable once its Pod was created. To apply a new request, a controller had to delete the Pod and let its Deployment or ReplicaSet create a replacement with the new values. Every resize was therefore a restart, with the loss of in-memory state, the cold caches and the startup time that a restart brings. That cost shaped how sizing tools were built and how teams used them.

The Vertical Pod Autoscaler (VPA) shows the effect most clearly. VPA is a Kubernetes add-on that analyzes a workload's historical CPU and memory usage and recommends container requests. Its update mode controls what it does with each recommendation:

  • Off: VPA publishes recommendations and changes nothing.
  • Initial: VPA applies recommendations only when a Pod is created.
  • Recreate: VPA evicts running Pods whose requests differ significantly from the recommendation, and the workload's controller recreates them with the new requests.
  • InPlaceOrRecreate: VPA attempts an in-place resize first and evicts the Pod only if the resize cannot be applied.

Before in-place resize existed, Recreate was the only mode in which VPA could change the requests of running workloads, so every adjustment cost an eviction.

Because each change required an eviction, many teams ran VPA in Off mode in production, or did not run it at all, and set requests for the peak instead. For the 1Gi/4Gi service, the only request that avoided OOM kills at 9am was 4Gi, held all day. That request kept the service stable and left 3Gi reserved and unused for about 23 hours a day. Fixed requests made stability and cost savings mutually exclusive for any workload with a daily peak, and percentile-based sizing was the attempt to find a middle ground.

The percentile trap, and why prediction alone did not fix it

Percentile-based sizing sets a request from a percentile of observed usage, such as P90, P95 or P99. A request at the peak avoids failures but reserves capacity that stays idle most of the time. A request at P95 reduces that idle capacity but leaves the workload short during the remaining 5% of the time. For many services, that 5% is the morning traffic peak or a business-critical event. Neither choice reduces cost and assures stability at the same time, which is why the Wand team, now part of groundcover, calls this the percentile trap.

Predictive scaling tools try to avoid the trap by forecasting peaks, and they help with recurring patterns. They have three limits. First, forecasts model recurring patterns such as daily cycles, and they do not anticipate retry storms or sudden traffic spikes. Second, horizontal scaling responds in minutes, because new replicas need scheduling, image pulls, application startup and sometimes a new node, while a memory spike causes an OOM kill in milliseconds. Third, even an accurate vertical forecast could only be applied by restarting the Pod before the peak. In-place resize removes the third limit.

What in-place resize changes

In-place Pod resize makes CPU and memory requests and limits mutable on a running Pod, often without restarting the container. The feature was introduced as alpha in Kubernetes 1.27, graduated to beta in 1.33 and reached GA in 1.35. It separates desired resources from actual resources: spec.containers[*].resources now holds the desired values, and status.containerStatuses[*].resources holds the values the kubelet has applied to the running container. Changes are submitted through a dedicated resize subresource rather than through a regular Pod update.

For the 1Gi/4Gi service, raising the memory request before the 9am peak takes a single API call. Assume the container's memory limit is 6Gi, and that its resizePolicy declares that both resources can change without a restart:

resizePolicy:
- resourceName: cpu
  restartPolicy: NotRequired
- resourceName: memory
  restartPolicy: NotRequired

The resize itself is a patch to the resize subresource:

kubectl patch pod orders-api-6f8c9-x7k2q --subresource resize --patch \
  '{"spec":{"containers":[{"name":"app","resources":{"requests":{"memory":"4Gi"}}}]}}'

Until the kubelet applies the change, the Pod's spec and status report different values:

spec:
  containers:
  - name: app
    resources:
      requests:
        memory: 4Gi   # desired
status:
  containerStatuses:
  - name: app
    resources:
      requests:
        memory: 1Gi   # applied

When the kubelet finishes, the status reports 4Gi. The service then enters its peak with 4Gi reserved on the node, so the scheduler does not place other Pods into that capacity, and the kubelet treats the Pod's usage as within its request if the node comes under memory pressure. No restart was required.

Several details of the GA release matter for anyone automating resizes. Memory limits can now be decreased, but the kubelet allows a decrease only if current usage is below the new limit, and that check is best-effort, not guaranteed. A resize that does not fit in the node's available capacity is marked Deferred, and the kubelet retries Deferred resizes in order of PriorityClass, then QoS class, then how long each has waited. A resize cannot change a Pod's QoS class, and resize is not supported on nodes that use swap, the static CPU Manager or the static Memory Manager. Kubernetes 1.37 also added scheduler preemption for in-place resize as an alpha feature. These rules make resize a dependable mechanism, and they also show which decisions it leaves to the caller.

What in-place resize does not solve

The first limit is node capacity. A resize can only use capacity that is free on the node where the Pod already runs, because the kubelet does not move the Pod to another node. If two replicas of the 1Gi/4Gi service share a node and both request 4Gi at 8:55am, the node needs 6Gi of unreserved memory. Otherwise at least one resize is marked Deferred, and that replica enters its peak with less memory reserved than it needs. An increase is only as reliable as the headroom on that node, which is the node-level problem described in the previous post.

The second limit is runtime behavior, which the Java example shows. By default, the JVM sets its maximum heap to 25% of the container's memory limit, and it reads that limit once, when the JVM starts. An in-place change to the limit does not change the heap of a running JVM. When the container later restarts with a lower limit, the heap shrinks, and garbage collection runs more often and uses more CPU, or the application fails with out-of-memory errors. A sizing tool that then observes lower memory usage and lowers the limit again is responding to behavior its own earlier change produced. The loop persists because the knowledge it depends on is split across teams. Many developers never set JVM heap flags at all, so the JVM defaults to a heap of only 25% of whatever limit the container receives. The platform team or an autoscaler changes that limit later, and no single team owns the relationship between the heap and the limit. Go programs that derive GOMEMLIMIT from the container limit, and Node.js programs that set a V8 heap limit, can produce similar feedback loops.

The third limit is sizing policy. In-place resize applies a change, but it does not decide which change to make or when to make it. VPA in InPlaceOrRecreate mode, which is now beta, removes the eviction from each adjustment, but it still calculates each workload's recommendation from that workload's own usage percentile. It does not account for which co-located workloads peak at the same time.

The fourth limit is template drift. After a resize, the running Pod's spec no longer matches the Pod template in its Deployment. When the Deployment scales out or rolls out, new Pods are created from the template's original values, and any adjustment made in place is lost. Keeping new Pods consistent with the current sizing requires applying the current values at the moment each Pod is created. Each of these four limits requires context from outside the Pod: node capacity, runtime behavior, co-located workloads and the Pod's controller.

How Autonomous Infrastructure uses in-place resize

groundcover's Autonomous Infrastructure uses in-place resize as its mechanism and supplies the decisions that resize leaves open, so that it can reduce cost while assuring stability. It adjusts CPU and memory requests in both directions. It lowers them on over-reserved workloads to reduce cost, and it raises them on workloads that need more capacity to keep them stable. Each change is based on node-level usage, so a request is raised only when the node has capacity for the increase and lowered only when the peaks of co-located workloads leave room for the decrease.

Autonomous Infrastructure changes requests only. It treats each declared limit as a ceiling and does not modify it, so the JVM's limit-based heap sizing in the Java example stays the same and the feedback loop described above does not start. A mutating admission webhook applies the current values to every new Pod of a managed workload, so replicas created by a scale-out or rollout do not revert to the template's original values. Deployment manifests in Git are not modified, so GitOps tooling sees no drift. Recommendations reach each cluster through a pull model, so a cluster that loses its connection to groundcover keeps using the last recommendation it received.

These decisions depend on data that per-Pod tooling does not keep. Node-level sizing needs high-resolution, per-container usage history linked to node, workload and runtime, retained long enough to capture daily and weekly patterns. groundcover collects that history automatically: its sensor runs as a DaemonSet on every node and uses eBPF to read CPU, memory and other resource data directly from the Linux kernel, at high resolution and without application code changes, SDKs or exporters. groundcover's BYOC deployment keeps that history in the customer's own cloud account, which makes full retention affordable. The groundcover investigator can use the same data during an incident, for example to test whether an OOM kill followed a request reduction. That gives platform teams evidence that each cost reduction kept workloads stable.

Next: how scaling layers interact

In-place resize removes the restart that made vertical scaling impractical in production, and it leaves the sizing decision to whatever calls it. Making that decision safely requires node capacity, runtime behavior and co-located usage, and Autonomous Infrastructure supplies those inputs to reduce cost while keeping workloads stable. Vertical resize is also only one of several automated layers acting on the same cluster. The Horizontal Pod Autoscaler (HPA) and KEDA change replica counts, and Karpenter adds and removes nodes, each from its own view of the cluster. The next post explains how those layers interact and how their interactions create fragmentation.

Anais Dotis
Anais Dotis
 

8 min read |
Published on: Nov 18, 2026

Latest posts

Explore related posts

Sign up for Updates

Keep up with all things cloud-native observability.

We care about data. Check out our privacy policy.

No items found.
No items found.
No items found.
No items found.
No items found.
No items found.
No items found.