Storage for Containers Using Ceph RBD – Part IV

Shipping containers

Overview

In this article we will look at how to integrate Ceph RBD (Rados Block Device) with Kubernetes and OpenShift. Ceph is of course a scale-out software-defined storage system that provides block, file and object storage. It focuses primarily on cloud-storage use cases. Providing storage for Kubernetes and OpenShift is just one of many use cases that fit very well with Ceph.

Ceph integrates with Kubernetes and OpenShift using the kubernetes.io/rbd driver or provisioner. As you will see this enables dynamic provisioning, allowing storage to be provisioned automatically when developers or users request it. Essentially a developer or OpenShift user, requests a PVC (Persistent Volume Claim) against a storage-class where the the kubernetes.io/rbd provisionier is configured. A PVC will create a RBD image in a specific RBD pool on Ceph and map it to a PV (Persistent Volume) in OpenShift. Once a container starts, the PV pointing to an RBD image is mounted. RBD client then maps the RBD image to the appropriate OpenShift node and mounts it using desired filesystem (ext4). The wonderful thing is everything happens automatically and the Ceph storage administrators only need to manage OpenShift project quotas for storage. Since each storage-class maps to an RBD pool it is possible to create various storage SLAs (gold, silver, bronze, etc).

Prerequisites

Before going any further an OpenShift 3.4 or higher cluster is required as well as a Ceph 1.3 or higher cluster. Not to worry, you can set this all up using below guides in a few hours and learn more about OpenShift and Ceph. I ran everything required OpenShift (all-in-one) and Ceph (3 node cluster) on my laptop with 12GB RAM.

Configuring Ceph

In Ceph we need to create an RBD pool for OpenShift and also create a Ceph authx keyring to access the Ceph cluster from OpenShift.

[Ceph Monitor]

Create RBD Pool

[ceph@ceph1]$ sudo ceph osd pool create ose 128

Note: if you don’t have enough free PGs (placement groups) you can go with 32 or even 16 for lab setups.

Create Keyring

[ceph@ceph1]$ sudo ceph auth get-or-create client.ose mon 'allow r' \
osd 'allow class-read object_prefix rdb_children, allow rwx pool=ose' \
-o /etc/ceph/ceph.client.ose.keyring

Copy Keyring file to all OpenShift Nodes

[ceph@ceph1]$ scp /etc/ceph/ceph.client.ose.keyring \
root@192.168.122.60:/etc/ceph

Convert Ceph key to base64 for client.admin user

[ceph@ceph1]$ sudo ceph auth get client.admin
 exported keyring for client.admin
 [client.admin]
 key = AQA8nJBYZAQDKxAAuSX4mzY1YODtPU8gzmIufQ==
 caps mds = "allow *"
 caps mon = "allow *"
 caps osd = "allow *"
[ceph@ceph1]$  echo AQA8nJBYZAQDKxAAuSX4mzY1YODtPU8gzmIufQ== |base64
QVFBOG5KQllaQVFES3hBQXVTWDRtelkxWU9EdFBVOGd6bUl1ZlE9PQo=

Note: save the new base64 key you will need it later.

Convert Ceph key to base64 for client.ose user

[ceph@ceph1]$ sudo ceph auth get client.ose
 exported keyring for client.ose
 [client.ose]
 key = AQDvU+ZYooHxHBAANGVNCfRpA24iYiTtMgt/tQ==
 caps mon = "allow r"
 caps osd = "allow class-read object_prefix rdb_children, allow rwx pool=ose"
[ceph@ceph1]$ echo AQDvU+ZYooHxHBAANGVNCfRpA24iYiTtMgt/tQ== |base64
 QVFEdlUrWllvb0h4SEJBQU5HVk5DZlJwQTI0aVlpVHRNZ3QvdFE9PQo=

Note: save the new base64 key you will need it later.

Configure Ceph Storage Class in OpenShift

[OpenShift Master]

Install ceph-common on all OpenShift nodes

[root@ose3-master ~]# yum install -y ceph-common

Create /var/run/ceph directory

[root@ose3-master ~]# mkdir /var/run/ceph

Create New Project

[root@ose3-master ~]# oc login -u admin
[root@ose3-master ~]# oc new-project ceph

Create Secret for Ceph client.admin user

The key in the secret should be the Ceph authx key converted to base64.

[root@ose3-master ~]# vi /root/ceph-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret
  namespace: default
data:
  key: QVFBOG5KQllaQVFES3hBQXVTWDRtelkxWU9EdFBVOGd6bUl1ZlE9PQo=
[root@ose3-master ~]# oc create -f /root/ceph-secret.yaml

Note: Ceph admin secret should be in default project

[root@ose3-master ~]# oc get secret ceph-secret -n default
 NAME TYPE DATA AGE
 ceph-secret Opaque 1 25s

Create Secret for Ceph client.ose user

The key in the secret should be the Ceph authx key converted to base64.

[root@ose3-master ~]# vi /root/ceph-secret-user.yaml
apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret
data:
  key: QVFEdlUrWllvb0h4SEJBQU5HVk5DZlJwQTI0aVlpVHRNZ3QvdFE9PQo=
[root@ose3-master ~]# oc create -f /root/ceph-secret-user.yaml

Note: Ceph admin secret must be in the project requesting Ceph storage.

Create Storage Class for Ceph

[root@ose3-master ~]# vi /root/ceph-rbd-storage-class.yaml
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: ceph
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/rbd
parameters:
  monitors: 192.168.122.81:6789,192.168.122.82:6789,192.168.122.83:6789
  adminId: admin
  adminSecretName: ceph-secret
  adminSecretNamespace: default
  pool: ose
  userId: ose
  userSecretName: ceph-secret-user
[root@ose3-master ~]# oc create -f /root/ceph-rbd-storage-class.yaml

Dynamic Provisioning using Ceph RBD

Create PVC (Persistent Volume Claim) 

Using the storage-class for Ceph we can now create a PVC.

[root@ose3-master ~]# vi /root/ceph-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: ceph-claim
  annotations:
    volume.beta.kubernetes.io/storage-class: ceph
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
[root@ose3-master ~]# oc create -f /root/ceph-pvc.yaml

Examine PVC and PV

The PVC will create a RBD image in the ose RBD pool and map it to a PV.

[root@ose3-master ~]# oc get pvc
 NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
 ceph-claim Bound pvc-792ec052-1ae6-11e7-9752-52540057bf27 2Gi RWO 3s
[root@ose3-master ~]# oc get pv
 NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
 pvc-792ec052-1ae6-11e7-9752-52540057bf27 2Gi RWO Delete Bound ceph/ceph-claim 50s

List RBD Images

[root@ose3-master ~]# rbd list -p ose --name client.ose \
--keyring /etc/ceph/ceph.client.ose.keyring
 kubernetes-dynamic-pvc-7930af93-1ae6-11e7-9752-52540057bf27

Dynamic Provisioning using OpenShift UI

In OpenShift UI login in and go to the project ceph. Under “resources->storage” you can view the already created PVC.

ceph-pvc

Request New PVC

Clicking “create storage” you can choose storage class and create a PVC. This is repeating what we already did with PVC.

ceph-provision

Under storage we now see both PVCs.

ceph-pvc2

Create MariaDB Persistent Database

In order to use a PVC we need to mount it’s PV in a running Pod. In this case we will click “add to project”, search for MariaDB and select the MariaDB persistent template from the OpenShift services catalog.

ceph-prov-mariadb

On the next screen accept the defaults and create the MariaDB persistent service. Once the MariaDB Pod is deploying, under “resources->storage” you will see the newly created PVC for mariadb.

ceph-mariadb-provisioned

Finally we see that the MariaDB persistent Pod is started and passed health as well as readiness checks. At this point any data written to this database will be saved in an RBD image residing on a Ceph storage cluster.

ceph-mariadb-complete

Summary

In this article we discussed how Ceph RBD integrates with OpenShift and Kubernetes. We saw how to configure Ceph and OpenShift to use RBD through a storage-class. Finally we observed how developers or users can easily consume Ceph storage within OpenShift. Dynamic storage in OpenShift is a huge shift in how we consume storage. I view this as a new beginning in storage. Finally we are really providing storage-as-a-service!

Happy Cephing in OpenShift!

(c) 2017 Keith Tenzer

 

 

One thought on “Storage for Containers Using Ceph RBD – Part IV

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s