NUMA in Kubernetes: Performance, Scheduling & Best Practices

Key Takeaways

• NUMA improves performance by keeping CPUs, memory, and devices local for latency-sensitive Kubernetes workloads.
• Kubernetes requires features like Topology Manager and Guaranteed QoS to achieve NUMA-aware resource placement.
• Strict NUMA policies boost performance but reduce scheduling flexibility and can increase pod rejections.
• Dedicated node pools, careful sizing, and observability help teams get the benefits of NUMA without unnecessary complexity.

Modern Kubernetes clusters are doing more than running standard web services. Many now support AI inference, real-time analytics, telecom systems, databases, packet processing, and accelerator-backed workloads. These applications can push servers close to their hardware limits, which makes placement details inside the machine more important than they are for ordinary stateless services.

That is where NUMA in Kubernetes becomes worth understanding. NUMA stands for Non-Uniform Memory Access. The basic idea is simple: not every CPU has the same cost to access every part of memory. A CPU can usually reach memory attached to its own NUMA node faster than memory attached to another NUMA node. Kubernetes can run on NUMA hardware without special tuning, but performance-sensitive workloads may behave inconsistently when CPU, memory, and devices are spread across multiple NUMA nodes.

What Is NUMA in Kubernetes?

NUMA in Kubernetes describes how pods run on servers that contain multiple NUMA nodes. A NUMA node is a hardware locality zone. It usually includes a group of CPU cores, a portion of memory, and sometimes nearby PCIe devices such as NICs, GPUs, or accelerators.

A simplified worker node might look like this:

Worker Node
├── NUMA Node 0
│   ├── CPU cores: 0-15
│   ├── Memory: 128 GiB
│   └── Local NIC
└── NUMA Node 1
    ├── CPU cores: 16-31
    ├── Memory: 128 GiB
    └── GPU / accelerator

From Kubernetes’ point of view, a pod may simply ask for 8 CPUs and 16 GiB of memory. From the hardware’s point of view, the important question is where those resources come from. If the pod receives CPUs from NUMA Node 0 but frequently accesses memory on NUMA Node 1, the workload has to cross the interconnect between nodes. That can add latency, reduce effective bandwidth, and make performance less predictable.

Not every workload will notice this. A small API service probably will not care much. A machine learning inference service, high-throughput database, packet processing workload, telecom function, or GPU-backed application might care a lot.

Kubernetes Resource Management Basics

Kubernetes lets teams define CPU and memory requests and limits for containers. Requests tell Kubernetes what a pod expects to need. Limits define how much it can use at runtime. These values affect scheduling and resource control, but they do not automatically guarantee NUMA locality.

For NUMA-sensitive workloads, the Guaranteed QoS class is usually the cleanest starting point. A pod is Guaranteed when every container has CPU and memory requests that match its limits.

apiVersion: v1
kind: Pod
metadata:
  name: numa-sensitive-app
spec:
  containers:
  - name: app
    image: example/numa-sensitive-app:latest
    resources:
      requests:
        cpu: "4"
        memory: "8Gi"
      limits:
        cpu: "4"
        memory: "8Gi"

This pod gives kubelet a clear resource shape: four CPUs and 8 GiB of memory. That does not solve NUMA placement on its own, but it gives CPU Manager, Memory Manager, and Topology Manager something predictable to work with.

How Kubernetes Handles NUMA Alignment

NUMA-aware placement in Kubernetes happens in two parts. First, the Kubernetes scheduler chooses a worker node based on aggregate capacity and scheduling rules. Then kubelet evaluates whether the pod can actually be admitted on that node under local resource policies.

That second step matters. A node may have enough total CPU and memory, but not enough available CPU and memory on the same NUMA node. In that case, a strict topology policy may reject the pod even though the node looked suitable at the scheduler level.

Three kubelet components matter most here:

• CPU Manager controls CPU assignment. With the static policy, eligible Guaranteed pods that request whole CPU cores can receive exclusive CPUs.
• Memory Manager helps make memory allocation NUMA-aware, so memory can be placed closer to the CPU allocation where possible.
• Topology Manager coordinates hints from CPU Manager, Memory Manager, and device managers. It then decides whether the pod can be admitted under the configured topology policy.

A simplified scheduling flow looks like this:

1. A user creates a pod with CPU and memory requests.
2. The scheduler picks a worker node with enough aggregate capacity.
3. kubelet starts local admission.
4. CPU Manager, Memory Manager, and device managers provide topology hints.
5. Topology Manager merges those hints.
6. kubelet admits or rejects the pod.
7. Resource managers allocate CPU, memory, and devices based on the accepted alignment.

This is why NUMA issues can feel different from normal Kubernetes scheduling problems. The cluster may have capacity overall, but not in the right hardware shape.

Topology Manager Policies for NUMA in Kubernetes

Topology Manager policies control how strict kubelet should be when aligning CPU, memory, and device resources.

The single-numa-node policy is the strictest common option. It is useful when a pod needs CPU, memory, and devices from the same NUMA node. It can also create friction. If a pod is too large to fit inside one NUMA node, or if smaller workloads have already fragmented the node, kubelet may reject the pod.

That is not always a bad thing. Strict rejection can protect performance-sensitive workloads from poor placement. But it also means teams need better capacity planning and clearer workload sizing.

Configuring NUMA in Kubernetes

Configuring NUMA in Kubernetes is usually a node-level task. It is not something teams solve with a single pod manifest. A common approach is to create a dedicated node pool for NUMA-sensitive workloads, configure kubelet policies on those nodes, and keep general-purpose workloads away from them.

A kubelet configuration might look like this:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cpuManagerPolicy: static
memoryManagerPolicy: Static
topologyManagerPolicy: single-numa-node
reservedMemory:
- numaNode: 0
  limits:
    memory: "2Gi"
- numaNode: 1
  limits:
    memory: "2Gi"

In this example, cpuManagerPolicy: static allows eligible pods to receive exclusive CPUs. memoryManagerPolicy: Static enables NUMA-aware memory behavior. topologyManagerPolicy: single-numa-node tells kubelet to require eligible resources to align on the same NUMA node.

Before using strict NUMA policies, check the basics:

• The worker nodes actually expose multiple NUMA nodes.
• Critical pods use Guaranteed QoS.
• CPU requests use whole numbers when exclusive CPUs are needed.
• Pod CPU and memory requests fit within the target NUMA node.
• systemReserved, kubeReserved, and reservedMemory are configured.
• NUMA-sensitive workloads run on dedicated node pools.
• Monitoring is already in place before rollout.

Performance Benefits and Trade-Offs

NUMA-aware placement can improve performance when a workload depends heavily on memory locality, CPU consistency, or local device access. The potential benefits include lower memory access latency, better throughput, more stable p95 and p99 latency, less cross-node interconnect traffic, and better device locality for NICs, GPUs, or accelerators.

For example, a packet processing pod may perform better when its CPUs are close to the NIC queues it uses. A database with a large in-memory working set may behave more consistently when CPU and memory are placed on the same NUMA node.

The cost is flexibility. Strict policies reduce the number of valid placements. They can also create stranded capacity, where the node has free resources in total but not enough resources in the right NUMA layout. That is why NUMA tuning is usually best for a focused set of workloads rather than the entire cluster.

Common Challenges and Limitations

Resource fragmentation is one of the easiest NUMA problems to run into. A node may have 10 CPUs and 72 GiB of memory free, but those resources might be split across two NUMA nodes. If a pod needs 8 CPUs on one NUMA node and the largest available NUMA node only has 6, a strict policy can reject the pod.

Pod sizing is another common issue. If each NUMA node has 16 CPUs and 128 GiB of memory, a pod requesting 20 CPUs cannot fit under single-numa-node. That is a workload design problem, not a Kubernetes bug.

Teams usually address this by resizing the pod, sharding the workload, using larger nodes, or choosing a less strict policy. Dedicated node pools also help because they reduce interference from unrelated workloads. Without that isolation, a small burstable pod can consume just enough local CPU or memory to block a larger NUMA-sensitive pod later.

Best Practices for NUMA-Aware Workloads

Start by creating dedicated node pools for NUMA-sensitive workloads. Label those nodes clearly and use taints to keep unrelated pods away.

kubectl label node worker-1 workload-profile=numa-optimized

kubectl taint node worker-1 workload-profile=numa-optimized:NoSchedule

Then configure the pod with matching selectors and tolerations so it lands only on the tuned nodes.

Use Guaranteed QoS for critical workloads. That means CPU and memory requests should match limits, and CPU requests should use whole numbers when CPU Manager static policy matters.

Size pods around NUMA boundaries. A pod that fits the whole worker node may still fail if it cannot fit one NUMA node after reserved resources are considered. Leave headroom for the operating system, kubelet, and system daemons.

Benchmark before enforcing strict policies. Compare default placement with NUMA-aware placement using p99 latency, throughput, CPU usage, memory behavior, pod rejection rate, and node utilization. If strict alignment does not improve the workload, it may not be worth the operational trade-off.

Monitoring and Debugging NUMA Performance

NUMA issues often show up as performance instability instead of obvious errors. A team may see latency spikes, lower throughput, CPU stalls, or inconsistent benchmark results without a clear message that says “this is NUMA.”

Start by checking where the pod is running:

kubectl get pod trading-engine -o wide
kubectl describe pod trading-engine

Look at the QoS class, CPU and memory requests, node selector, tolerations, and recent events. If the pod is not Guaranteed, CPU Manager and Memory Manager may not behave the way the team expects.

Then inspect the node topology:

lscpu | grep -i numa
numactl --hardware

These commands show how many NUMA nodes exist, which CPUs belong to each NUMA node, and how memory is distributed. If pods are failing to start, kubelet logs can help explain whether Topology Manager, CPU Manager, Memory Manager, or a device plugin rejected the placement.

At the application layer, compare p50, p95, and p99 latency, throughput, error rate, CPU throttling, memory working set, and network round-trip time. A useful test is to run the same workload with default placement and then with stricter NUMA alignment. If the aligned version produces more consistent latency or better throughput, NUMA was probably part of the problem.

Unified eBPF Observability for NUMA in Kubernetes With groundcover

NUMA-related issues can be hard to troubleshoot because they cross several layers at once. The application may show a latency regression, but the root cause could involve pod placement, CPU scheduling, memory allocation, device locality, network behavior, or node pressure.

groundcover provides full-stack observability through an eBPF sensor that requires no code changes, no sidecar proxies, and no manual instrumentation. That gives teams a clearer way to connect application symptoms with infrastructure behavior across the entire Kubernetes environment. It runs as a BYOC (Bring Your Own Cloud) deployment inside your own AWS, GCP, or Azure environment, so telemetry and performance data stay within your infrastructure.

For NUMA troubleshooting, groundcover can help teams:

• Compare latency, throughput, and error patterns across pods and nodes.
• Correlate telemetry with Kubernetes metadata such as node, namespace, deployment, pod, and workload type.
• Use eBPF-based runtime visibility to observe system and network behavior without agent overhead or code changes.
• Review traces, metrics, logs, and events together during root-cause analysis.
• Validate whether stricter NUMA alignment actually improves performance.

A practical workflow might start with a workload that has unstable p99 latency. From there, teams can compare pods across nodes, confirm whether affected pods are running on NUMA-optimized node pools, correlate changes with rescheduling or node pressure, and test whether stricter alignment makes performance more consistent.

Conclusion

NUMA in Kubernetes matters when workloads need predictable CPU, memory, and device locality. Kubernetes can run on NUMA systems without special tuning, but default scheduling does not guarantee that a pod’s resources will come from the same NUMA node. For latency-sensitive and high-throughput workloads, that can lead to remote memory access, inconsistent performance, and harder troubleshooting.

The practical approach is selective tuning. Use dedicated node pools, Guaranteed QoS, whole CPU requests, realistic pod sizing, and topology policies that match the workload’s needs. Then measure the result. With eBPF-powered visibility, groundcover can help teams see whether NUMA tuning improves performance and keep critical Kubernetes workloads running more reliably.