Use Ceph RBD block storage with libvirt and KVM

Ceph RBD gives every KVM host access to the same VM disks over the network, with replication and snapshots built in. This guide connects plain libvirt hosts to an existing cluster to use Ceph RBD with libvirt and KVM.

Before you start

• A healthy Ceph cluster (ceph -s shows HEALTH_OK) reachable from the KVM hosts; see Ceph for virtualization.
• Admin access to Ceph and root on the KVM hosts.
• QEMU with RBD support (qemu-block-extra on Debian/Ubuntu, included on many RHEL-family builds).

Create a pool and client key on Ceph

ceph osd pool create vms

rbd pool init vms

ceph auth get-or-create client.libvirt mon 'profile rbd' osd 'profile rbd pool=vms'

Copy the printed key and the cluster's ceph.conf (monitor addresses) to each KVM host.

Store the key as a libvirt secret

On each host create secret.xml:

<secret ephemeral='no' private='no'>
  <uuid>6f0c1d2e-0000-4000-8000-000000000001</uuid>
  <usage type='ceph'><name>client.libvirt secret</name></usage>
</secret>

Use the same UUID on all hosts (generate one with uuidgen):

virsh secret-define secret.xml

virsh secret-set-value 6f0c1d2e-0000-4000-8000-000000000001 --base64 'AQB...yourkey...=='

Define an RBD storage pool

<pool type='rbd'>
  <name>ceph-vms</name>
  <source>
    <name>vms</name>
    <host name='10.10.0.21' port='6789'/>
    <host name='10.10.0.22' port='6789'/>
    <host name='10.10.0.23' port='6789'/>
    <auth username='libvirt' type='ceph'>
      <secret uuid='6f0c1d2e-0000-4000-8000-000000000001'/>
    </auth>
  </source>
</pool>

virsh pool-define rbd-pool.xml && virsh pool-start ceph-vms && virsh pool-autostart ceph-vms

Newer Ceph releases may use msgr2 on port 3300; list monitors as your cluster advertises them.

Create and use VM disks

virsh vol-create-as ceph-vms web01-disk0 40G

Attach with virt-install: --disk vol=ceph-vms/web01-disk0,bus=virtio,cache=none. Import an existing image with qemu-img convert -O raw image.qcow2 rbd:vms/web02-disk0:id=libvirt (requires ceph.conf and keyring on that host).

Live migrate

Because all hosts reach the same RBD image, migration moves only memory:

virsh migrate --live --persistent --undefinesource web01 qemu+ssh://kvm2/system

See KVM live migration.

Troubleshooting

• "error connecting: Permission denied": wrong key or capabilities; check ceph auth get client.libvirt.
• Pool will not start: monitors unreachable; check firewall and addresses.
• Slow VMs: network or OSD bottleneck; check ceph -s and latency, and use cache=none or writeback as recommended for RBD.

Frequently asked questions

qcow2 on RBD?

Use raw; RBD already provides thin provisioning and snapshots.

Can Proxmox do this for me?

Yes, Proxmox has built-in Ceph and RBD storage support.

Do VM snapshots work?

RBD snapshots work at the storage level; libvirt snapshot support for RBD disks is more limited.

Next steps

• Ceph for virtualization
• libvirt storage pools
• Hyperconverged infrastructure