Ceph 1.3 Lab Installation and Configuration Guide

Overview

In this article we will setup a Ceph 1.3 cluster for purpose of learning or a lab environment.

 

Ceph Lab Environment

For this environment you will need three VMs (ceph1, ceph2 and ceph3). Each should have 20GB root disk and 100GB data disk. Ceph has three main components: Admin console, Monitors and OSDs.

Admin console - UI and CLI used for managing Ceph cluster. In this environment we will install on ceph1.

Monitors - Monitor health of Ceph cluster. One or more monitors forms a paxos part-time parliment, providing extreme reliability and durability of cluster membership. Monitors maintain the various maps: monitor, osd, placement group (pg) and crush. Monitors will be installed on ceph1, ceph2 and ceph3.

OSDs - Object storage daemon handles storing data, recovery, backfilling, rebalancing and replication. OSDs sit on top of a disk / filesystem. Bluestore enables OSDs to bypass filesystem but is not an option in Ceph 1.3. An OSD will be installed on ceph1, ceph2 and ceph3.

On All Cephs nodes.

#subscription-manager repos --disable=*

#subscription-manager repos --enable=rhel-7-server-rpms

#subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-rhceph-1.3-calamari-rpms --enable=rhel-7-server-rhceph-1.3-installer-rpms --enable=rhel-7-server-rhceph-1.3-tools-rpms

Configure firewalld.

sudo systemctl start firewalld
sudo systemctl enable firewalld

sudo firewall-cmd --zone=public --add-port=80/tcp --permanent
sudo firewall-cmd --zone=public --add-port=2003/tcp --permanent
sudo firewall-cmd --zone=public --add-port=4505-4506/tcp --permanent

sudo firewall-cmd --zone=public --add-port=6789/tcp --permanent

sudo firewall-cmd --zone=public --add-port=6800-7300/tcp --permanent

sudo firewall-cmd --reload

Configure NTP.

yum -y install ntp

systemctl enable ntpd.service

systemctl start ntpd

Ensure NTP is scychronozing.

ntpq -p
remote refid st t when poll reach delay offset jitter
 ==============================================================================
 +privatewolke.co 131.188.3.222 2 u 12 64 1 26.380 -2.334 2.374
 +ridcully.episod 148.251.68.100 3 u 11 64 1 26.626 -2.425 0.534
 *s1.kelker.info 213.172.96.14 2 u 12 64 1 26.433 -6.116 1.030
 sircabirus.von- .STEP. 16 u - 64 0 0.000 0.000 0.000

Create ceph user for deployer.

#useradd ceph
#passwd ceph

#cat << EOF >/etc/sudoers.d/ceph
ceph ALL = (root) NOPASSWD:ALL
Defaults:ceph !requiretty
EOF

#chmod 0440 /etc/sudoers.d/ceph

#su - ceph
#ssh-key-gen
#ssh-copy-id ceph@ceph1
#ssh-copy-id ceph@ceph2
#ssh-copy-id ceph@ceph3

Set SELinux to permissive. Ceph 2.0 now supports SELinux but for 1.3 it was not possible out-of-box.

#vi /etc/selinux/config

SELINUXTYPE=permissive

Create ceph-config dir.

mkdir ~/ceph-config
cd ~/ceph-config

On Monitors.

#subscription-manager repos --enable=rhel-7-server-rhceph-1.3-mon-rpms

#yum update -y

On OSD Nodes.

#subscription-manager repos --enable=rhel-7-server-rhceph-1.3-osd-rpms

#yum update -y

 

On admin node (ceph1).

Setup Admin Console and Calamari.

#sudo yum -y install ceph-deploy calamari-server calamari-clients

#sudo calamari-ctl initialize

#su - ceph

[ceph@ceph1 ceph-config]$cd ~/ceph-config

Create Ceph Cluster.

#ceph-deploy new ceph1 ceph2 ceph3

Deploy Ceph monitors and OSDs.

[ceph@ceph1 ceph-config]$sudo ceph-deploy install --mon ceph1 ceph2 ceph3

[ceph@ceph1 ceph-config]$sudo ceph-deploy install --osd ceph1 ceph2 ceph3

[ceph@ceph1 ceph-config]$sudo ceph-deploy mon create ceph1 ceph2 ceph3

[ceph@ceph1 ceph-config]$sudo ceph-deploy gatherkeys ceph1

Connect Ceph monitors to Calamari.

[ceph@ceph1 ceph-config]$sudo ceph-deploy calamari connect --master ceph1.lab ceph1 ceph2 ceph3

[ceph@ceph1 ceph-config]$sudo ceph-deploy install --cli ceph1

[ceph@ceph1 ceph-config]$sudo ceph-deploy admin ceph1

Check Ceph quorum status.

[ceph@ceph1 ceph-config]$sudo ceph quorum_status --format json-pretty

{
 "election_epoch": 6,
 "quorum": [
 0,
 1,
 2
 ],
 "quorum_names": [
 "ceph1",
 "ceph2",
 "ceph3"
 ],
 "quorum_leader_name": "ceph1",
 "monmap": {
 "epoch": 1,
 "fsid": "188aff9b-7da5-46f3-8eb8-465e014a472e",
 "modified": "0.000000",
 "created": "0.000000",
 "mons": [
 {
 "rank": 0,
 "name": "ceph1",
 "addr": "192.168.0.31:6789\/0"
 },
 {
 "rank": 1,
 "name": "ceph2",
 "addr": "192.168.0.32:6789\/0"
 },
 {
 "rank": 2,
 "name": "ceph3",
 "addr": "192.168.0.33:6789\/0"
 }
 ]
 }
}

Set crush tables to optimal.

[ceph@ceph1 ceph-config]$sudo ceph osd crush tunables optimal

Configure OSDs

Prepare and Active OSDs together

[ceph@ceph1 ceph-config]$sudo ceph-deploy osd create ceph1:vdb ceph2:vdb ceph3:vdb

OR

[ceph@ceph1 ceph-config]$sudo ceph-deploy disk zap ceph1:vdb ceph2:vdb ceph3:vdb

[ceph@ceph1 ceph-config]$sudo ceph-deploy osd prepare ceph1:vdb ceph2:vdb ceph3:vdb

[ceph@ceph1 ceph-config]$sudo ceph-deploy osd activate ceph1:vdb1 ceph2:vdb1 ceph3:vdb1

Connect Calamari to Ceph nodes.

[ceph@ceph1 ceph-config]$sudo ceph-deploy calamari connect --master ceph1.lab ceph1 ceph2 ceph3

Tips and Tricks

Remove OSD from Ceph

[ceph@ceph1 ~]$sudo ceph osd out osd.0
[ceph@ceph1 ~]$sudo ceph osd crush remove osd.0
[ceph@ceph1 ~]$sudo ceph auth del osd.0
[ceph@ceph1 ~]$sudo ceph osd down 0
[ceph@ceph1 ~]$sudo ceph osd rm 0

Ceph Placement Group Calculation for Pool

• OSDs * 100 / Replicas
• PGs should always be power of two 62, 128, 256, etc

Re-deploy Ceph

In case at anytime you want to start over you can run below commands to uninstall Ceph. This of course deletes any data so be careful.

[ceph@ceph1 ~]$sudo service ceph restart osd.3

[ceph@ceph1 ~]$sudo ceph-deploy purge <ceph-node> [<ceph-node>]

Summary

In this article we installed a Ceph cluster on virtual machines. We deployed the cluster, setup monitors and configured OSDs.  This environment should provide the basis for a journey into software-defined storage and Ceph. The economics of scale have brought down barriers and paved the way for a software-defined world. Storage is only the next logical boundry. Ceph being an OpenSource project is already the defacto software-defined standard and is in position to become the key beneficiary of software-defined storage. I hope you found the information in this article of use, please share your experiences.

Happy Cephing!

(c) 2016 Keith Tenzer