In this article we will look at how to integrate Ceph RBD (Rados Block Device) with Kubernetes and OpenShift. Ceph is of course a scale-out software-defined storage system that provides block, file and object storage. It focuses primarily on cloud-storage use cases. Providing storage for Kubernetes and OpenShift is just one of many use cases that fit very well with Ceph.
In this article we will configure OpenStack Swift to use Ceph as a storage backend. Object of cloud storage is one of the main services provided by OpenStack. Swift is an object storage protocol and implementation. It has been around for quite a while but is fairly limited (it uses rsync to replicate data, scaling rings can be problematic and it only supports object storage to just mention a few things). OpenStack needs to provide storage for many use cases such as block (Cinder), block (Glance), file (Manila), block (Nova) and object (Swift). Ceph is a distributed software-defined storage system that scales with OpenStack and provides all these use cases. As such it is the defacto standard for OpenStack and is why you see in OpenStack user survey that Ceph is 60% of all OpenStack storage.
OpenStack uses Keystone to store service endpoints for all services. Swift has a Keystone endpoint that authenticates OpenStack tenants to Swift providing object or cloud storage on a per-tenant basis. As mentioned, Ceph provides block, file and object access. In the case of object Ceph provides S3, Swift and NFS interfaces. The RADOS Gateway (RGW) provides object interfaces for Ceph. S3 and Swift users are stored in the RGW. Usually you would want several RADOS GWs in an active/active configuration using a load balancer. OpenStack tenants can be given automatic access and their Keystone tenant ids are automatically configured in the RADOS GW when Swift object storage is accessed from given tenant.
Using Ceph with OpenStack for object storage provides tenants access to cloud storage, integrated with OpenStack using swift and automatically handles authentication of OpenStack tenants. It also provides advantage that external users or tenants (outside of OpenStack) such as application developers can access object storage directly with protocol of choice: S3, Swift or NFS.
Integrating Ceph with OpenStack Series:
In order to integrate OpenStack Swift with Ceph you need to first follow below prerequisites:
- Configure OpenStack environment here
- Configure Ceph cluster here
In this article we will configure OpenStack Manila using CephFS as a storage backend. OpenStack Manila is an OpenStack project providing file services. Manila is storage backend agnostic and you can have many different kinds of storage backends, similar to Cinder. CephFS is a POSIX-Compliant file system that uses the Ceph storage cluster to store data. CephFS works by providing a Metadata Server (MDS) that collectively manages filesystem namespaces. It also coordinates access to Ceph Object Storage Damones (OSDs). Ceph MDS has two modes: active or passive. There are several documented active/passive MDS configurations and multi-mds or active/active MDS that can be configured when a single MDS becomes a bottleneck. Clients can mount CephFS filesystems using the ceph-fuse client or kernel kernel driver.
Integrating Ceph with OpenStack Series:
The following are required to configure OpenStack Manila with CephFS:
- Already configured Ceph cluster (Jewel or higher). See here to setup Ceph cluster.
- Already configured OpenStack (Mitaka or higher). See here to setup OpenStack.
Ceph has become the defacto standard for software-defined storage. Ceph is 100% opensource, built on open standards and as such is offered by many vendors not just Red Hat. If you are new to Ceph or software-defined storage, I would recommend the following article before proceeding to understand some high-level concepts:
Ceph – the future of storage
In this article we will configure a Red Hat Ceph 2.0 cluster and set it up for object storage. We will configure RADOS Gateway (RGW), Red Hat Storage Console (RHCS) and show how to configure the S3 and Swift interfaces of the RGW. Using python we will access both the S3 and Swift interfaces.
If you are interested in configuring Ceph for OpenStack see the following article:
OpenStack – Integrating Ceph as Storage Backend
As of today there are over eleven OpenStack services and more are coming. Each service has complete isolation from other services and that allows OpenStack to scale far beyond the reach of current computing platforms. However due to all these independent services, OpenStack can be very complicated to operationalize in enterprise environments.