OpenStack: Integrating Ceph as Storage Backend

22 minute read

cephopenstack

Overview

In this article we will discuss why Ceph is Perfect fit for OpenStack. We will see how to integrate three prominent OpenStack use cases with Ceph: Cinder (block storage), Glance (images) and Nova (VM virtual disks).

Integrating Ceph with OpenStack Series:

Ceph provides unified scale-out storage, using commodity x86 hardware, that is self-healing and intelligently anticipates failures. It has become the defacto standard for software-defined storage. Ceph being an OpenSource project has enabled many vendors the ability to provide Ceph based software-defined storage systems. Ceph is not just limited to Companies like Red Hat, Suse, Mirantis, Ubuntu, etc. Integrated solutions from SanDisk, Fujitsu, HP, Dell, Samsung and many more exist today. There are even large-scale community built environments, Cern comes to mind, that provide storage services for 10,000s of VMs.

Ceph is by no means limited to OpenStack, however this is where Ceph started gaining traction. Looking at latest OpenStack user survey, Ceph is by a large margin the clear leader for OpenStack storage. Page 42 in OpenStack April 2016 User Survey reveals Ceph is 57% of OpenStack storage. The next is LVM (local storage) with 28% followed by NetApp with 9%. If we remove LVM, Ceph leads any other storage company by 48%, that is incredible. Why is that?

There are several reasons but I will give you my top three:

  • Ceph is a scale-out unified storage platform. OpenStack needs two things from storage: ability to scale with OpenStack itself and do so regardless of block (Cinder), File (Manila) or Object (Swift). Traditional storage vendors need to provide two or three different storage systems to achieve this. They don't scale the same and in most cases only scale-up in never-ending migration cycles. Their management capabilities are never truly integrated across broad spectrum of storage use-cases.
  • Ceph is cost-effective. Ceph leverages Linux as an operating system instead of something proprietary. You can choose not only whom you purchase Ceph from, but also where you get your hardware. It can be same vendor or different. You can purchase commodity hardware, or even buy integrated solution of Ceph + Hardware from single vendor. There are even hyper-converged options for Ceph that are emerging (running Ceph services on compute nodes).
  • Ceph is OpenSource project just like OpenStack. This allows for a much tighter integration and cross-project development. Proprietary vendors are always playing catch-up since they have secrets to protect and their influence is usually limited in Opensource communities, especially in OpenStack context.

Below is an architecture Diagram that shows all the different OpenStack components that need storage. It shows how they integrate with Ceph and how Ceph provides a unified storage system that scales to fill all these use cases.

ceph-openstack

source: Red Hat Summit 2016

If you are interested in more topics relating to ceph and OpenStack I recommend following: http://ceph.com/category/ceph-and-openstack/

Ok enough talking about why Ceph and OpenStack are so great, lets get our hands dirty and see how to hook it up!

If you don't have a Ceph environment you can follow this article on how to set one up quickly.

Glance Integration

Glance is the image service within OpenStack. By default images are stored locally on controllers and then copied to compute hosts when requested. The compute hosts cache the images but they need to be copied again, every time an image is updated.

Ceph provides backend for Glance allowing images to be stored in Ceph, instead of locally on controller and compute nodes. This greatly reduces network traffic for pulling images and increases performance since Ceph can clone images instead of copying them. In addition it makes migrating between OpenStack deployments or concepts like multi-site OpenStack much simpler.

Install ceph client used by Glance.

[root@osp9 ~]# yum install -y python-rbd

Create Ceph user and set home directory to /etc/ceph.

[root@osp9 ~]# mkdir /etc/ceph
[root@osp9 ~]# useradd ceph
[root@osp9 ~]# passwd ceph

Add ceph user to sudoers.

cat << EOF >/etc/sudoers.d/ceph
ceph ALL = (root) NOPASSWD:ALL
Defaults:ceph !requiretty
EOF

On Ceph admin node.

Create Ceph RBD Pool for Glance images.

[ceph@ceph1 ~]$ sudo ceph osd pool create images 128

Create keyring that will allow Glance access to pool.

[ceph@ceph1 ~]$ sudo ceph auth get-or-create client.images mon 'allow r' osd 'allow class-read object_prefix rdb_children, allow rwx pool=images' -o /etc/ceph/ceph.client.images.keyring

Copy the keyring to /etc/ceph on OpenStack controller.

[ceph@ceph1 ~]$ scp /etc/ceph/ceph.client.images.keyring root@osp9.lab:/etc/ceph

Copy /etc/ceph/ceph.conf configuration to OpenStack controller.

[ceph@ceph1 ~]$ scp /etc/ceph/ceph.conf root@osp9.lab:/etc/ceph

Set permissions so Glance can access Ceph keyring.

[root@osp9 ~(keystone_admin)]# chgrp glance /etc/ceph/ceph.client.images.keyring
[root@osp9 ~(keystone_admin)]#chmod 0640 /etc/ceph/ceph.client.images.keyring

Add keyring file to Ceph configuration.

[root@osp9 ~(keystone_admin)]# vi /etc/ceph/ceph.conf
[client.images]
keyring = /etc/ceph/ceph.client.images.keyring

Create backup of original Glance configuration.

[root@osp9 ~(keystone_admin)]# cp /etc/glance/glance-api.conf /etc/glance/glance-api.conf.orig

Update Glance configuration.

[root@osp9 ~]# vi /etc/glance/glance-api.conf
[glance_store]
stores = glance.store.rbd.Store
default_store = rbd
rbd_store_pool = images
rbd_store_user = images
rbd_store_ceph_conf = /etc/ceph/ceph.conf

Restart Glance.

[root@osp9 ~(keystone_admin)]# systemctl restart openstack-glance-api

Download Cirros images and add it into Glance.

[root@osp9 ~(keystone_admin)]# wget http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img

Convert QCOW2 to RAW. It is recommended for Ceph to always use RAW format.

[root@osp9 ~(keystone_admin)]#qemu-img convert cirros-0.3.4-x86_64-disk.img cirros-0.3.4-x86_64-disk.raw

Add image to Glance.

[root@osp9 ~(keystone_admin)]#glance image-create --name "Cirros 0.3.4" --disk-format raw --container-format bare --visibility public --file cirros-0.3.4-x86_64-disk.raw
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | ee1eca47dc88f4879d8a229cc70a07c6 |
| container_format | bare |
| created_at | 2016-09-07T12:29:23Z |
| disk_format | qcow2 |
| id | a55e9417-67af-43c5-a342-85d2c4c483f7 |
| min_disk | 0 |
| min_ram | 0 |
| name | Cirros 0.3.4 |
| owner | dd6a4ed994d04255a451da66b68a8057 |
| protected | False |
| size | 13287936 |
| status | active |
| tags | [] |
| updated_at | 2016-09-07T12:29:27Z |
| virtual_size | None |
| visibility | public |
+------------------+--------------------------------------+

Check that glance image exists in Ceph.

[ceph@ceph1 ceph-config]$ sudo rbd ls images
a55e9417-67af-43c5-a342-85d2c4c483f7
[ceph@ceph1 ceph-config]$ sudo rbd info images/a55e9417-67af-43c5-a342-85d2c4c483f7
rbd image 'a55e9417-67af-43c5-a342-85d2c4c483f7':
 size 12976 kB in 2 objects
 order 23 (8192 kB objects)
 block_name_prefix: rbd_data.183e54fd29b46
 format: 2
 features: layering, striping
 flags:
 stripe unit: 8192 kB
 stripe count: 1

Cinder Integration

Cinder is the block storage service in OpenStack. Cinder provides an abstraction around block storage and allows vendors to integrate by providing a driver. In Ceph, each storage pool can be mapped to a different Cinder backend. This allows for creating storage services such as gold, silver or bronze. You can decide for example that gold should be fast SSD disks that are replicated three times, while silver only should be replicated two times and bronze should use slower disks with erasure coding.

Create Ceph pool for cinder volumes.

[ceph@ceph1 ~]$ sudo ceph osd pool create volumes 128

Create keyring to grant cinder access.

[ceph@ceph1 ~]$ sudo ceph auth get-or-create client.volumes mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rx pool=images' -o /etc/ceph/ceph.client.volumes.keyring

Copy keyring to OpenStack controllers.

[ceph@ceph1 ~]$ scp /etc/ceph/ceph.client.volumes.keyring root@osp9.lab:/etc/ceph

Create file that contains just the authentication key on OpenStack controllers.

[ceph@ceph1 ~]$ sudo ceph auth get-key client.volumes |ssh osp9.lab tee client.volumes.key

Set permissions on keyring file so it can be accessed by Cinder.

[root@osp9 ~(keystone_admin)]# chgrp cinder /etc/ceph/ceph.client.volumes.keyring
[root@osp9 ~(keystone_admin)]# chmod 0640 /etc/ceph/ceph.client.volumes.keyring

Add keyring to Ceph configuration file on OpenStack controllers.

[root@osp9 ~(keystone_admin)]#vi /etc/ceph/ceph.conf

[client.volumes]
keyring = /etc/ceph/ceph.client.volumes.keyring

Give KVM Hypervisor access to Ceph.

[root@osp9 ~(keystone_admin)]# uuidgen |tee /etc/ceph/cinder.uuid.txt

Create a secret in virsh so KVM can access Ceph pool for cinder volumes.

[root@osp9 ~(keystone_admin)]#vi /etc/ceph/cinder.xml

<secret ephemeral="no" private="no">
 <uuid>ce6d1549-4d63-476b-afb6-88f0b196414f</uuid>
 <usage type="ceph">
 <name>client.volumes secret</name>
 </usage>
</secret>
[root@osp9 ceph]# virsh secret-define --file /etc/ceph/cinder.xml
[root@osp9 ~(keystone_admin)]# virsh secret-set-value --secret ce6d1549-4d63-476b-afb6-88f0b196414f --base64 $(cat /etc/ceph/client.volumes.key)

Add Ceph backend for Cinder.

[root@osp9 ~(keystone_admin)]#vi /etc/cinder/cinder.conf

[rbd]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = -1
glance_api_version = 2
rbd_user = volumes
rbd_secret_uuid = ce6d1549-4d63-476b-afb6-88f0b196414f

Restart Cinder service on all controllers.

[root@osp9 ~(keystone_admin)]# openstack-service restart cinder

Create a cinder volume.

[root@osp9 ~(keystone_admin)]# cinder create --display-name="test" 1
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2016-09-08T10:58:17.000000 |
| description | None |
| encrypted | False |
| id | d251bb74-5c5c-4c40-a15b-2a4a17bbed8b |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | test |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | dd6a4ed994d04255a451da66b68a8057 |
| replication_status | disabled |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | 783d6e51e611400c80458de5d735897e |
| volume_type | None |
+--------------------------------+--------------------------------------+


List new cinder volume

[root@osp9 ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| d251bb74-5c5c-4c40-a15b-2a4a17bbed8b | available | test | 1 | - | false | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+

List cinder volume in ceph.

[ceph@ceph1 ~]$ sudo rbd ls volumes
volume-d251bb74-5c5c-4c40-a15b-2a4a17bbed8b
[ceph@ceph1 ~]$ sudo rbd info volumes/volume-d251bb74-5c5c-4c40-a15b-2a4a17bbed8b
rbd image 'volume-d251bb74-5c5c-4c40-a15b-2a4a17bbed8b':
 size 1024 MB in 256 objects
 order 22 (4096 kB objects)
 block_name_prefix: rbd_data.2033b50c26d41
 format: 2
 features: layering, striping
 flags:
 stripe unit: 4096 kB
 stripe count: 1

Integrating Ceph with Nova Compute

Nova is the compute service within OpenStack. Nova stores virtual disks images associated with running VMs by default, locally on Hypervisor under /var/lib/nova/instances. There are a few drawbacks to using local storage on compute nodes for virtual disk images:

  • Images are stored under root filesystem. Large images can cause filesystem to fill up, thus crashing compute nodes.
  • A disk crash on compute node could cause loss of virtual disk and as such a VM recovery would be impossible.

Ceph is one of the storage backends that can integrate directly with Nova. In this section we will see how to configure that.

[ceph@ceph1 ~]$ sudo ceph osd pool create vms 128

Create authentication keyring for Nova.

[ceph@ceph1 ~]$ sudo ceph auth get-or-create client.nova mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=vms, allow rx pool=images' -o /etc/ceph/ceph.client.nova.keyring

Copy keyring to OpenStack controllers.

[ceph@ceph1 ~]$ scp /etc/ceph/ceph.client.nova.keyring root@osp9.lab:/etc/ceph

Create key file on OpenStack controllers.

[ceph@ceph1 ~]$ sudo ceph auth get-key client.nova |ssh osp9.lab tee client.nova.key

Set permissions on keyring file so it can be accessed by Nova service.

[root@osp9 ~(keystone_admin)]# chgrp nova /etc/ceph/ceph.client.nova.keyring
[root@osp9 ~(keystone_admin)]# chmod 0640 /etc/ceph/ceph.client.nova.keyring

Ensure the required rpm packages are installed.

[root@osp9 ~(keystone_admin)]# yum list installed python-rbd ceph-common
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Installed Packages
ceph-common.x86_64 1:0.94.5-15.el7cp @rhel-7-server-rhceph-1.3-mon-rpms
python-rbd.x86_64 1:0.94.5-15.el7cp @rhel-7-server-rhceph-1.3-mon-rpms

Update Ceph configuration.

[root@osp9 ~(keystone_admin)]#vi /etc/ceph/ceph.conf

[client.nova]
keyring = /etc/ceph/ceph.client.nova.keyring

Give KVM access to Ceph.

[root@osp9 ~(keystone_admin)]# uuidgen |tee /etc/ceph/nova.uuid.txt

Create a secret in virsh so KVM can access Ceph pool for cinder volumes.

[root@osp9 ~(keystone_admin)]#vi /etc/ceph/nova.xml

<secret ephemeral="no" private="no">
<uuid>c89c0a90-9648-49eb-b443-c97adb538f23</uuid>
<usage type="ceph">
<name>client.volumes secret</name>
</usage>
</secret>
[root@osp9 ~(keystone_admin)]# virsh secret-define --file /etc/ceph/nova.xml
[root@osp9 ~(keystone_admin)]# virsh secret-set-value --secret c89c0a90-9648-49eb-b443-c97adb538f23 --base64 $(cat /etc/ceph/client.nova.key)

Make backup of Nova configuration.

[root@osp9 ~(keystone_admin)]# cp /etc/nova/nova.conf /etc/nova/nova.conf.orig

Update Nova configuration to use Ceph backend.

[root@osp9 ~(keystone_admin)]#vi /etc/nova/nova.conf
force_raw_images = True
disk_cachemodes = writeback

[libvirt]
images_type = rbd
images_rbd_pool = vms
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = vms
rbd_secret_uuid = c89c0a90-9648-49eb-b443-c97adb538f23

Restart Nova services.

[root@osp9 ~(keystone_admin)]# systemctl restart openstack-nova-compute

List Neutron networks.

[root@osp9 ~(keystone_admin)]# neutron net-list
+--------------------------------------+---------+-----------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------+-----------------------------------------------------+
| 4683d03d-30fc-4dd1-9b5f-eccd87340e70 | private | ef909061-34ee-4b67-80a9-829ea0a862d0 10.10.1.0/24 |
| 8d35a938-5e4f-46a2-8948-b8c4d752e81e | public | bb2b65e7-ab41-4792-8591-91507784b8d8 192.168.0.0/24 |
+--------------------------------------+---------+-----------------------------------------------------+

Start ephemeral VM instance using Cirros image that was added in the steps for Glance.

[root@osp9 ~(keystone_admin)]# nova boot --flavor m1.small --nic net-id=4683d03d-30fc-4dd1-9b5f-eccd87340e70 --image='Cirros 0.3.4' cephvm
+--------------------------------------+-----------------------------------------------------+
| Property | Value |
+--------------------------------------+-----------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | wzKrvK3miVJ3 |
| config_drive | |
| created | 2016-09-08T11:41:29Z |
| flavor | m1.small (2) |
| hostId | |
| id | 85c66004-e8c6-497e-b5d3-b949a1666c90 |
| image | Cirros 0.3.4 (a55e9417-67af-43c5-a342-85d2c4c483f7) |
| key_name | - |
| metadata | {} |
| name | cephvm |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | dd6a4ed994d04255a451da66b68a8057 |
| updated | 2016-09-08T11:41:33Z |
| user_id | 783d6e51e611400c80458de5d735897e |
+--------------------------------------+-----------------------------------------------------+

Wait until the VM is active.

[root@osp9 ceph(keystone_admin)]# nova list
+--------------------------------------+--------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------+--------+------------+-------------+---------------------+
| 8ca3e74e-cd52-42a6-acec-13a5b8bda53c | cephvm | ACTIVE | - | Running | private=10.10.1.106 |
+--------------------------------------+--------+--------+------------+-------------+---------------------+

List images in Ceph vms pool. We should now see the image is stored in Ceph.

[ceph@ceph1 ~]$ sudo rbd -p vms ls
8ca3e74e-cd52-42a6-acec-13a5b8bda53c_disk

Troubleshooting

Unable to delete Glance Images stored in Ceph RBD

[root@osp9 ceph(keystone_admin)]# nova image-list
+--------------------------------------+--------------+--------+--------+
| ID | Name | Status | Server |
+--------------------------------------+--------------+--------+--------+
| a55e9417-67af-43c5-a342-85d2c4c483f7 | Cirros 0.3.4 | ACTIVE | |
| 34510bb3-da95-4cb1-8a66-59f572ec0a5d | test123 | ACTIVE | |
| cf56345e-1454-4775-84f6-781912ce242b | test456 | ACTIVE | |
+--------------------------------------+--------------+--------+--------+
[root@osp9 ceph(keystone_admin)]# rbd -p images snap unprotect cf56345e-1454-4775-84f6-781912ce242b@snap
[root@osp9 ceph(keystone_admin)]# rbd -p images snap rm cf56345e-1454-4775-84f6-781912ce242b@snap
[root@osp9 ceph(keystone_admin)]# glance image-delete cf56345e-1454-4775-84f6-781912ce242b

Summary

In this article we discussed how OpenStack and Ceph fit perfectly together. We discussed some of the use cases around glance, cinder and nova. Finally we went through steps to integrate Ceph with those use cases. I hope you enjoyed the article and found the information useful. Please share your feedback.

Happy Cephing!

(c) 2016 Keith Tenzer