NUMA and virtual machines: keep large VMs fast

On servers with several CPU sockets (and some single-socket CPUs with multiple dies), memory is attached to specific CPUs. Accessing "remote" memory is slower. This is NUMA (Non-Uniform Memory Access). Large VMs that straddle NUMA nodes can lose noticeable performance.

Before you start

• Root on a multi-socket or multi-NUMA KVM host.
• Background: CPU pinning and overcommit.

Inspect the host topology

lscpu | grep -i numa

numactl --hardware

You will see each node's CPUs, memory size and the distance matrix (higher = slower). virsh capabilities also lists NUMA cells.

When NUMA matters

• VMs with many vCPUs or large RAM (close to or larger than one node).
• Latency-sensitive workloads: databases, in-memory caches, high packet rates.
• Small VMs (a few vCPUs and GBs) rarely notice; the Linux scheduler and automatic NUMA balancing handle them.

Fit VMs inside one node when possible

Size VMs so their vCPUs and memory fit within a single NUMA node, then pin them there:

<vcpu placement='static' cpuset='0-7'>8</vcpu>
<numatune>
  <memory mode='strict' nodeset='0'/>
</numatune>

Choose CPUs that belong to node 0 according to numactl --hardware.

Expose NUMA to very large VMs

If a VM must span nodes, give it a virtual NUMA topology that mirrors the host, so the guest OS can place memory sensibly:

<cpu mode='host-passthrough'>
  <numa>
    <cell id='0' cpus='0-7' memory='32' unit='GiB'/>
    <cell id='1' cpus='8-15' memory='32' unit='GiB'/>
  </numa>
</cpu>

Combine with <numatune><memnode cellid='0' mode='strict' nodeset='0'/>... and vCPU pinning so each virtual node maps to a physical node.

Check inside the guest

numactl --hardware

Databases like PostgreSQL and MySQL can then be tuned for NUMA if needed.

Automatic NUMA balancing

Linux migrates memory toward the CPUs using it (kernel.numa_balancing). It helps unpinned VMs; with strict manual pinning some admins disable it to avoid overhead. Measure before changing.

Troubleshooting

• VM fails to start with strict numatune: not enough free memory on that node; use preferred mode or another node.
• Performance worse after pinning: vCPUs pinned to sibling threads or across nodes; re-check topology.
• Hugepages unavailable: allocate them per node; see huge pages.

Frequently asked questions

Do single-socket servers have NUMA?

Some CPUs expose several NUMA nodes per socket depending on BIOS settings; check numactl.

Should VPS customers care?

Usually not; it is a host-side concern unless you buy very large VMs.

Does Proxmox support NUMA?

Yes, there is a NUMA option per VM plus CPU affinity settings.

Next steps

• CPU pinning
• Huge pages and memory
• KVM CPU models