Ceph

As we just configured our first CEPH[1] cluster, we needed to move our current virtual machines (using raw images stored on standard filesystem) so they use the RBD block device provided by CEPH.

We use Libvirt[2] and Kvm[3] to manage our virtual machines.

Libvirt

Migration with virtual machine downtime




This step can be done offline:

  • stop the virtual machine
 virsh shutdown vmfoo
  • convert the image to rbd
 qemu-img convert -O rbd /var/lib/libvirt/images/vmfoo.img rbd:libvirt-pool/vmfoo
  • update the VM configuration file
 virsh edit vmfoo
 <disk type='file' device='disk'>
   <driver name='qemu' type='raw' cache='none'/>
   <source file='/var/lib/libvirt/images/vmfoo.img'/>
   <target dev='vda' bus='virtio'/>
   <alias name='virtio-disk0'/>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
 </disk>

devient:

 <disk type='network' device='disk'>
   <driver name='qemu'/>
   <auth username='libvirt'>
     <secret type='ceph' uuid='sec-ret-uu-id'/>
   </auth>
   <source protocol='rbd' name='libvirt-pool/vmfoo'>
     <host name='10.0.0.1' port='6789'/>
     <host name='10.0.0.2' port='6789'/>
     ...
   </source>
   <target dev='vda' bus='virtio'/>
   <alias name='virtio-disk0'/>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
 </disk>
  • restart the virtual machine
virsh start vmfoo

Migration without downtime

The trick here is to use migration support in libvirt/kvm and the ability to provide a different xml definition for the target virtual machine:

  • get the current vm disk informations
 qemu-img info /var/lib/libvirt/images/vmfoo.img
  • create an empty rbd of the same size
 qemu-img create -f rbd rbd:libvirt-pool/vmfoo XXG
  • get the current vm configuration
 virsh dumpxml vmfoo > vmfoo.xml
  • edit this configuration to replace the on disk image by the rbd one
 <disk type='file' device='disk'>
   <driver name='qemu' type='raw' cache='none'/>
   <source file='/var/lib/libvirt/images/vmfoo.img'/>
   <target dev='vda' bus='virtio'/>
   <alias name='virtio-disk0'/>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
 </disk>

devient:

 <disk type='network' device='disk'>
   <driver name='qemu'/>
   <auth username='libvirt'>
     <secret type='ceph' uuid='sec-ret-uu-id'/>
   </auth>
   <source protocol='rbd' name='libvirt-pool/vmfoo'>
     <host name='10.0.0.1' port='6789'/>
     <host name='10.0.0.2' port='6789'/>
     ...
   </source>
   <target dev='vda' bus='virtio'/>
   <alias name='virtio-disk0'/>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
 </disk>
  • start the migration process
virsh migrate --live --persistent --copy-storage-all --verbose --xml vmfoo.xml vmfoo qemu+ssh://target_node/system
  • wait until the process finished. The time to wait depends of your cluster performances and your VM size, but there is no interruption of the virtual machine!
  • you're done, your virtual machine is now running over rbd and once checked you can safelly archive or destroy your old disk image.

Notes:

  • of course you have to use libvirt/kvm with rbd support on target node
  • you have to use a recent version of kvm, we had memory exhaustion problems on the hypervisor during the migration process with debian wheezy version

Notes

[1] http://ceph.com/

[2] http://libvirt.org/

[3] http://www.linux-kvm.org/