Getting Started with Volume Snapshots in OpenShift 4 and OpenShift Container Storage

openshift-logotype-svgheavy-plus-signimage11

Overview

Volume snapshots are the ability to create snapshots of persistent volumes in kubernetes using the container storage interface (csi) driver. The csi driver allows storage solutions to integrate into kubernetes and expose their technologies. Snapshots of course, have been and are a key technology when discussing data workloads because they enable backup/restore seamlessly, on-demand and in a split second. Even though volume snapshots are in the alpha stage, several storage providers already have integrations, including one that is very interesting, Ceph RDB.

Ceph is a software-defined storage platform that is rock solid, very mature and provides file, block as well as object storage. In this case we will using block storage. In Ceph block is provided through rbd (Rados Block Device). A container-native storage solution called Rook, based completely on kubernetes operators provides a container-native deployment of Ceph on Kubernetes or OpenShift. In this article we are using OpenShift 4.2 GA and a beta release of OpenShift Container Storage (based on Rook).

How it Works

The CSI driver allows for a snapshotter to be implemented. The snapshotter runs as a side-car container and watches the kubernetes API for snapshot related events from the CSI driver. In order to create a volume snapshot a volume snapshot class must exist. This is similar to the storage class but defines the snapshotter and access to the appropriate CSI driver. Once a volume snapshot class exists a volume snapshot can be created for a given persistent volume claim (pvc). The volume snapshot will then trigger the snapshot operation on the storage device, in this case Ceph RBD. The volume snapshot allows the snapshotter to provide metadata about the snapshot contents to the end-user.

snapshotter

Volume Snapshot Backup

Now that we understand the basics we will see how to actually create a backup. We will deploy a mariadb database on persistent storage using Ceph RBD, create a table in the database with some records, setup a volume snapshot class and finally perform a volume snapshot of the database.

Deploy Mariadb database

We deployed a persistent mariadb database using ceph rbd provisioner.

mariadb

Add table and records to database

Connect to database.

$ oc rsh -n databases mariadb-6-rswv6
sh-4.2$ mysql -u root
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 20
Server version: 10.2.22-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> use sampledb;

Create table authors and add some records.

MariaDB [(none)]> CREATE TABLE authors (id INT, name VARCHAR(20), email VARCHAR(20));

MariaDB [(none)]> INSERT INTO authors (id,name,email) VALUES(1,"Keith","keith@domain.com");

MariaDB [(none)]> INSERT INTO authors (id,name,email) VALUES(2,"John","john@domain.com");

MariaDB [(none)]> INSERT INTO authors (id,name,email) VALUES(3,"Bob","bob@domain.com");

Show records for table authors.

MariaDB [books]> select * from authors;
+------+-------+-----------------+
| id   | name  | email           |
+------+-------+-----------------+
|    1 | Keith | keith@domain.com|
|    2 | John  | john@domain.com |
|    3 | Bob   | bob@domain.com  |
+------+-------+-----------------+
3 rows in set (0.00 sec)

Get the Ceph RBD Image

First get the persistent volume name for our mariadb.

$ oc get pv -o 'custom-columns=NAME:.spec.claimRef.name,PVNAME:.metadata.name,STORAGECLASS:.spec.storageClassName,VOLUMEHANDLE:.spec.csi.volumeHandle'
NAME                      PVNAME                                     STORAGECLASS                  VOLUMEHANDLE
ocs-deviceset-0-0-cc8nr   pvc-08b360c9-0214-11ea-a05b-005056814c6f   thin                          <none>
ocs-deviceset-1-0-bntqm   pvc-08b53c69-0214-11ea-a05b-005056814c6f   thin                          <none>
ocs-deviceset-2-0-g65tl   pvc-08b70a3e-0214-11ea-a05b-005056814c6f   thin                          <none>
db-noobaa-core-0          pvc-27708237-0214-11ea-a05b-005056814c6f   ocs-storagecluster-ceph-rbd   0001-0011-openshift-storage-0000000000000001-2f81ea18-0214-11ea-aa51-0a580a81000d
rook-ceph-mon-a           pvc-9dd37deb-0213-11ea-a05b-005056814c6f   thin                          <none>
rook-ceph-mon-b           pvc-a12c254a-0213-11ea-a05b-005056814c6f   thin                          <none>
mariadb                   pvc-a370d1ed-021b-11ea-bca7-005056818b87   ocs-storagecluster-ceph-rbd   0001-0011-openshift-storage-0000000000000001-a380eb7b-021b-11ea-aa51-0a580a81000d
rook-ceph-mon-c           pvc-a42a79c9-0213-11ea-a05b-005056814c6f   thin

Next get the Ceph rbd image mapped to our persistent volume.

[ktenzer@keith-laptop ~]$ oc get pv pvc-a370d1ed-021b-11ea-bca7-005056818b87 -o jsonpath='{.spec.csi.volumeHandle}' | cut -d '-' -f 6- | awk '{print "csi-vol-"$1}'
csi-vol-a380eb7b-021b-11ea-aa51-0a580a81000d

Create Volume Snapshot Class

In order to create the volume snapshot class some information is needed. We need to know the snapshotter, the provisioner secret and the secret namespace.

$ oc get sc ocs-storagecluster-ceph-rbd -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2019-11-08T10:35:52Z"
  name: ocs-storagecluster-ceph-rbd
  ownerReferences:
  - apiVersion: ocs.openshift.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: StorageCluster
    name: ocs-storagecluster
    uid: 893852b9-0213-11ea-bca7-005056818b87
  resourceVersion: "30740"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/ocs-storagecluster-ceph-rbd
  uid: 89666c5f-0213-11ea-bab7-00505681830f
parameters:
  clusterID: openshift-storage
  csi.storage.k8s.io/fstype: ext4
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage
  imageFeatures: layering
  imageFormat: "2"
  pool: ocs-storagecluster-cephblockpool
provisioner: openshift-storage.rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

Create volume snapshot class.

$ vi snapshot-class.yaml 
apiVersion: snapshot.storage.k8s.io/v1alpha1
kind: VolumeSnapshotClass
metadata:
  name: ocs-storagecluster-ceph-rbd-snapshot
snapshotter: openshift-storage.rbd.csi.ceph.com
parameters:
  clusterID: openshift-storage
  csi.storage.k8s.io/snapshotter-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/snapshotter-secret-namespace: openshift-storage
$ oc apply -f snapshot-class.yaml

Create Volume Snapshot

Now that we have our volume snapshot class we can snapshot a persistent volume claim.

$ vi create-snapshot.yaml 
apiVersion: snapshot.storage.k8s.io/v1alpha1
kind: VolumeSnapshot
metadata:
  name: mariadb-1
spec:
  snapshotClassName: ocs-storagecluster-ceph-rbd-snapshot
  source:
    name: mariadb
    kind: PersistentVolumeClaim
$ oc apply -f create-snapshot.yaml -n databases

Show volume snapshot.

[ktenzer@keith-laptop ~]$ oc get volumesnapshot 
NAME AGE 
mariadb-1 35s

Show the volume snapshot content. The snapshotter provides metadata about snapshots to kubernetes and stores that information in the snapshot content.

$ oc get volumesnapshotcontent 
NAME                                               AGE
snapcontent-fdf13bf1-0221-11ea-bab7-00505681830f   3m2s
$ oc get volumesnapshotcontent snapcontent-fdf13bf1-0221-11ea-bab7-00505681830f -o yaml
apiVersion: snapshot.storage.k8s.io/v1alpha1
kind: VolumeSnapshotContent
metadata:
  creationTimestamp: "2019-11-08T12:19:22Z"
  finalizers:
  - snapshot.storage.kubernetes.io/volumesnapshotcontent-protection
  generation: 1
  name: snapcontent-fdf13bf1-0221-11ea-bab7-00505681830f
  resourceVersion: "74329"
  selfLink: /apis/snapshot.storage.k8s.io/v1alpha1/volumesnapshotcontents/snapcontent-fdf13bf1-0221-11ea-bab7-00505681830f
  uid: fef777ae-0221-11ea-a05b-005056814c6f
spec:
  csiVolumeSnapshotSource:
    creationTime: 1573215562000000000
    driver: openshift-storage.rbd.csi.ceph.com
    restoreSize: 1073741824
    snapshotHandle: 0001-0011-openshift-storage-0000000000000001-fe517800-0221-11ea-aa51-0a580a81000d
  deletionPolicy: Delete
  persistentVolumeRef:
    apiVersion: v1
    kind: PersistentVolume
    name: pvc-a370d1ed-021b-11ea-bca7-005056818b87
    resourceVersion: "55833"
    uid: a3ef7f08-021b-11ea-bab7-00505681830f
  snapshotClassName: ocs-storagecluster-ceph-rbd-snapshot
  volumeSnapshotRef:
    apiVersion: snapshot.storage.k8s.io/v1alpha1
    kind: VolumeSnapshot
    name: mariadb-1
    namespace: databases
    resourceVersion: "74316"
    uid: fdf13bf1-0221-11ea-bab7-00505681830f

Verify Snapshot on Ceph RBD

OpenShift Container Storage provides a tools pods for easily accessing the Ceph cluster. Save the pod name to a vairable.

TOOLS_POD=$(oc get pods -n openshift-storage -l app=rook-ceph-tools -o name)

Using tools pod show information about mariadb Ceph rbd image.

$ oc rsh -n openshift-storage $TOOLS_POD  rbd -p ocs-storagecluster-cephblockpool info csi-vol-a380eb7b-021b-11ea-aa51-0a580a81000d                                       
rbd image 'csi-vol-a380eb7b-021b-11ea-aa51-0a580a81000d':
	size 1 GiB in 256 objects
	order 22 (4 MiB objects)
	snapshot_count: 0
	id: 5617782da596
	block_name_prefix: rbd_data.5617782da596
	format: 2
	features: layering
	op_features: 
	flags: 
	create_timestamp: Fri Nov  8 11:33:52 2019
	access_timestamp: Fri Nov  8 11:33:52 2019
	modify_timestamp: Fri Nov  8 11:33:52 2019

List all snapshots for mariadb Ceph rbd image.

$ oc rsh -n openshift-storage $TOOLS_POD rbd snap ls ocs-storagecluster-cephblockpool/csi-vol-a380eb7b-021b-11ea-aa51-0a580a81000d
SNAPID NAME                                          SIZE  PROTECTED TIMESTAMP                
     6 csi-snap-fe517800-0221-11ea-aa51-0a580a81000d 1 GiB yes       Fri Nov  8 12:19:22 2019

Volume Snapshot Restore

Now that we have performed and validated a backup lets do a restore. We will drop the table to simulate a loss of data and then perform a restore from our snapshot and hopefully get access to our data again.

Connect to database

$ oc rsh -n databases mariadb-6-rswv6
sh-4.2$ mysql -u root
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 20
Server version: 10.2.22-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> use sampledb;

Drop table

MariaDB [books]> drop table authors;
Query OK, 0 rows affected (0.03 sec)

Verify table is dropped

MariaDB [books]> select * from authors;
ERROR 1146 (42S02): Table 'books.authors' doesn't exist

At this point we have simulated a restore scenario, our data is gone!

Scale Deployment Config to 0

Before we can do a snapshot restore we need to shutdown the database by stopping the pod.

[ktenzer@keith-laptop ~]$ oc scale dc/mariadb --replicas=0
deploymentconfig.apps.openshift.io/mariadb scaled

Perform Snapshot Restore

$ oc rsh -n openshift-storage $TOOLS_POD rbd snap rollback ocs-storagecluster-cephblockpool/csi-vol-a380eb7b-021b-11ea-aa51-0a580a81000d@csi-snap-76040419-0228-11ea-aa51-0a580a81000d
Rolling back to snapshot: 100% complete...done.

Scale Deployment Config to 1

After snapshot restore start database.

oc scale dc/mariadb --replicas=1
deploymentconfig.apps.openshift.io/mariadb scaled

Connect to database

[ktenzer@keith-laptop ~]$ oc rsh -n databases mariadb-6-rswv6
sh-4.2$ mysql -u root
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 20
Server version: 10.2.22-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> use sampledb;

Show records for table authors.

MariaDB [books]> select * from authors;
+------+-------+-----------------+
| id   | name  | email           |
+------+-------+-----------------+
|    1 | Keith | keith@domain.com|
|    2 | John  | john@domain.com |
|    3 | Bob   | bob@domain.com  |
+------+-------+-----------------+
3 rows in set (0.00 sec)

The restore using our snapshot succeeded and as you can see our data is back…magic!

Summary

In this article we discussed how volume snapshots work and are implemented in kubernetes. Using OpenShift 4.2 and OpenShift Container Storage we demonstrated a backup and recovery of a mariadb database using volume snapshots in kubernetes. While it is early, this is a game changer for data workloads. Using volume snapshot we finally have a mechanism to integrate storage snapshots into kubernetes and as such provide proper backup and recovery solutions. I have no doubt this will lead to a large adoption of data workloads on kubernetes so get ready!

Happy Snapshotting!

(c) 2019 Keith Tenzer

 

 

 

 

 

 

 

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s