Ceph is open-source software-defined storage that spreads data across many servers. For virtualization its block service, RBD (RADOS Block Device), gives every host access to every VM disk, which enables live migration and automatic recovery from host failure without a central SAN. This guide explains Ceph storage for virtualization at a planning level.
Before you start
- At least three servers. Ceph needs a quorum and replicas on separate hosts.
- Fast, ideally dedicated, networking: 10 Gbit/s minimum, 25 Gbit/s or more for NVMe clusters.
- Familiarity with VM disk formats and storage options.
Ceph components
| Component | Role |
|---|---|
| MON (monitor) | Holds the cluster map and quorum; run an odd number, usually 3 |
| MGR (manager) | Metrics, dashboard and modules |
| OSD | One daemon per disk, stores data and handles replication |
| Pool | Logical group with a replication or erasure-coding rule |
| RBD image | A virtual block device inside a pool, used as a VM disk |
How VMs use RBD
QEMU talks to Ceph directly through librbd, so no filesystem sits between the VM and storage. Every hypervisor in the cluster can open any image, so a VM can be live migrated without copying its disk, and restarted elsewhere if its host dies. RBD supports thin provisioning, snapshots and clones.
Replication and capacity
- A replicated pool with
size=3andmin_size=2keeps three copies and stays writable with one lost. - Usable capacity is roughly raw capacity ÷ 3, minus headroom. Keep the cluster well below full so it can rebalance after a failure.
- Erasure-coded pools save space but are slower for VM workloads.
Hardware and network tips
- Use enterprise SSDs/NVMe with power-loss protection; consumer drives perform badly under Ceph's sync writes.
- Separate public and cluster (replication) networks if possible.
- Plan RAM for each OSD (several GB per OSD) plus the VMs.
- Use identical or similar nodes to keep data balanced.
Deploying Ceph
- Proxmox VE: built-in Ceph installer under Datacenter → node → Ceph, the easiest route for hyperconverged clusters.
- cephadm: the upstream container-based deployment tool for standalone clusters.
- Then add the pool as RBD storage in your hypervisor or VPS panel.
When Ceph is worth it
- Yes: three or more hosts, need for HA and live migration, growth by adding nodes.
- No: a single server or two, or no fast network. Local ZFS or LVM is simpler and faster; see ZFS for VMs.
Frequently asked questions
Is Ceph slower than local NVMe?
Usually yes for single-VM latency, because writes cross the network to several replicas. It trades some latency for resilience and flexibility.
Can I run Ceph on two nodes?
Not safely for production; quorum and replica placement need at least three.
Does Ceph replace backups?
No. Replication protects against hardware failure, not deletion or corruption.