OpenStack Tips and Tricks

Overview

In this article we will look at some common OpenStack Kilo configuration optimizations and other tricks. This is by no means a comprehensive guide, just things I have stumbled across that if documented would have saved me time. I continue to update this blog with things I learned. If you have some valuable tips or tricks then let me know so I can add those?

Nested Virtualization

Many run OpenStack on KVM for testing, learning, training or even demos. In order to get acceptable performance, the Hypervisor and guest must be configured to support Nested Virtualization.

Ensure KVM is enabled

#lsmod | grep -i kvm
kvm_intel 148081 6 
kvm 461126 1 kvm_intel

#modinfo kvm_intel | grep -i nested
parm: nested:bool

Unload KVM kernel module

#modprobe -r kvm_intel

Enable nested virtualization in KVM hypervisor

#modprobe kvm_intel nested=Yes

Create a guest for running OpenStack and edit the configuration to enable VMX.

#virsh edit osp7.lab.com

 <cpu mode='custom' match='exact'>
 <model fallback='allow'>SandyBridge</model>
 <feature policy='require' name='vmx'/>
 </cpu>

Start guest and verify that nested virtualization support is enabled.

#ps -ef |grep qemu-kvm |grep vmx
qemu 8557 1 29 09:04 ? 00:03:52 /usr/libexec/qemu-kvm -name osp7.lab.com -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -cpu SandyBridge,+vmx

Change Libvirt type to KVM in Nova.

#vi /etc/nova/nova.conf

virt_type=kvm

Fixing OpenStack Inconsistencies

Sometimes things just don't work as expected, that is life and software. In OpenStack sometimes objects in the database and the actual resource can be inconsistent. This means the resource exists in the database but not anywhere else. I have seen this happen with Cinder volumes when deleting Heat stacks for example. If this occurs, your last resort if the *force* delete commands fail is to go into database and remove resource. It goes with out saying that you need to use extreme caution, as you can cause data loss or even corruption within OpenStack.

Delete Cinder Volume

Sometimes it can take a really long time to delete cinder volumes. The default behavior is to zero blocks. You can change this by setting volume_clear=none in /etc/cinder/cinder.conf. If you want to determine why delete is taking so long you should see what processes are using the volume. For LVM backends you can follow these steps. First check lvdisplay to see if logical volume exists.

# lvdisplay

Get the major and minor number for volume.
# dmsetup info -c
Name Maj Min Stat Open Targ Event UUID
rhel-swap 253 0 L--w 2 1 0 LVM-I057BifDXT5pFxJ69IQuaLXouyIN6DDbltQCdDpuXeoSi3tFgBpYFQiETsCKO3CG
rhel-root 253 1 L--w 1 1 0 LVM-I057BifDXT5pFxJ69IQuaLXouyIN6DDbSP3WbTBmapCefb1mQLbdSzqw8drUculQ
cinder--volumes-volume--a052879b--9bbd--4285--8557--7c16337560c5 253 2 L--w 1 1 0 LVM-nnajQLl8dA7KqOsFJgYajjFVQrkM0wjfcB9fNeV0PL1R9TuuJIX3dNHqYbfmBvL7

Using lsof check processes running on volumes

# lsof | grep "253,2"
dd 5526 root 1w BLK 253,2 0x35f600000 15042 /dev/dm-2

If processes are running (in this case dd), kill them and remove volume using lvremove. Once this is complete set volume status to available in cinder.

 cinder reset-state --state available <volume id>

Finally try deleting the volume. If all else fails you can go into database as last resort and delete things there.

#mysql cinder
> update volumes set deleted=1,status='deleted',deleted_at=now(),updated_at=now() where deleted=0 and id='$volume_uuid';

Detach a Volume from Cinder

#mysql nova
> delete from block_device_mapping where not deleted and volume_id='$volume_uuid' and project_id='$project_uuid';

Delete an Instance

#mysql nova_db
> update instances set deleted='1', vm_state='deleted', deleted_at='now()'' where uuid='$vm_uuid' and project_id='$project_uuid';

Change provision state of Ironic nodes

# mysql ironic

> UPDATE nodes SET provision_state="available", target_provision_state=NULL, reservation=NULL WHERE uuid=<uuid>;

Delete Ironic nodes

# mysql ironic

> delete from ports where uuid="0867df16-82c9-4358-9bc9-a36933c361e1";

> delete from nodes where uuid="92b6477c-d556-4958-9950-5c11ca57e459";

Fixing Horizon Re-login issue

There is an issue in OpenStack Kilo with re-login because of bad cookie session. Here is how to fix the issue.

#vi /etc/openstack-dashboard/local_settings

AUTH_USER_MODEL = 'openstack_auth.User'

Heat Topology Images Broken

service openstack-cinder-volume restart

There is an issue in OpenStack Kilo with the Heat topology images being broken. Here is how to fix it.

#vi /etc/httpd/conf.d/openstack-dashboard.conf

Alias /static/dashboard /usr/share/openstack-dashboard/static/dashboard

systemctl restart httpd

Adding Cinder Volume for LVM backend

By default RDO will use a loopback device for the Cinder LVM backend. In order to change this you can follow the procedure below assuming disk is called /dev/vdb1.

#openstack-config --set /etc/cinder/cinder.conf DEFAULT lvm_type thin

#systemctl restart openstack-cinder-volume

#fdisk /dev/vdb1

#fdisk /dev/vdb1

#pvcreate /dev/vdb1

#vgcreate cinder_storage /dev/vdb1

#vgcreate cinder_storage /dev/vdb1

#vi /etc/cinder/cinder.conf

[lvm]
volume_group=my_new_cinder_storage
volume_driver=cinder.volume.drivers.lvm.LVMVolumeDriver

Force Deleting Keystone Endpoints

# mysql keystone

MariaDB [keystone]> delete from endpoint where id="07d77cefad0049b1ae5e1eb6692ebfa1";

Adding NFS as Cinder Backend

Cinder can use many different backends and using an NFS backend provides a lot of flexibility in addition to removing compülexity with ISCSI/LVM.

If SELinux is enabled allow NFS access

setsebool -P virt_use_nfs on

Create map file to make Cinder aware of NFS shares

#vi /etc/cinder/nfs_share
192.168.0.22:/usr/share/openstack

chown root:cinder /etc/cinder/nfs_share

chmod 0640 /etc/cinder/nfs_share

Configure NFS backend in Cinder

openstack-config --set /etc/cinder/cinder.conf nfs nfs_shares_config /etc/cinder/nfs_share

openstack-config --set /etc/cinder/cinder.conf nfs volume_driver cinder.volume.drivers.nfs.NfsDriver

openstack-config --set /etc/cinder/cinder.conf nfs volume_backend_name nfsbackend

Optionally you can add any required mount options

openstack-config --set /etc/cinder/cinder.conf nfs nfs_mount_options MOUNTOPTIONS

# vi /etc/cinder/cinder.conf
enabled_backends = lvm, nfs

Restart Cinder volume service

openstack-service restart cinder-volume

Configure NFS volume type so that is uses the correct backend in Cinder

cinder type-create nfstype 

cinder type-key nfstype set volume_backend_name=nfsbackend

Configuring RHEV for OpenStack

If you are using RHEV or any virtualization platform under OpenStack then you need to enable nested virtualization and ensure MAC Address Spoofing filters are disabled. Otherwise since OpenStack instance MAC address differs from that of the virtual machine packets will be dropped.

On RHEV-M

#engine-config -s "UserDefinedVMProperties=macspoof=(true|false)"
#service ovirt-engine restart

Edit VM and enable macspoof by setting parameter to 'true'

On Hypervisor Hosts

#yum install -y vdsm-hook-macspoof

#wget http://mirrors.ibiblio.org/ovirt/pub/ovirt-3.5/rpm/el7/noarch/vdsm-hook-nestedvt-4.16.30-0.el7.centos.noarch.rpm

#rpm -Uvh vdsm-hook-nestedvt-4.16.30-0.el7.centos.noarch.rpm

#systemctl reboot

On OpenStack Instance check to ensure nested virtualization active

#egrep 'svm|vmx' /proc/pcuinfo

Remove Packstack (RDO)

If you want to upgrade or change OpenStack deployment and you are using RDO it may be necessary to remove the installation to start cleanly. The below process can be used to accomplish that.

Delete any VMs that may be running or configured

for x in $(virsh list --all | grep instance- | awk '{print $2}') ; do
    virsh destroy $x ;
    virsh undefine $x ;
done ;

Reconfigure network interfaces

Hopefully you saved your original network configuration. You need to replace /etc/sysconfig/network-scripts/ifcfg-* with your original configs or just set IP addresses on those interfaces.

#cp /root/ifcfg-eth0 /etc/sysconfig/network-scripts

#rm /etc/sysconfig/network-scripts/ifcfg-br-ex

Remove packages

yum remove -y nrpe openvswitch "*nagios*" puppet ntp ntp-perl ntpdate "*openstack*" \
"*nova*" "*keystone*" "*glance*" "*cinder*" "*swift*" \
mysql mysql-server httpd "*memcache*" scsi-target-utils \
iscsi-initiator-utils perl-DBI perl-DBD-MySQL ;

Ensure swift processes arent running

ps -ef | grep -i repli | grep swift | awk '{print $2}' | xargs kill ;

Remove configuration data. Note if you are using NFS backend you need to unmount it.

rm -rf /etc/nagios /etc/yum.repos.d/packstack_* /root/.my.cnf \
/var/lib/mysql/ /var/lib/glance /var/lib/nova /etc/nova /etc/swift \
/srv/node/device*/* /var/lib/cinder/ /etc/rsync.d/frag* \
/var/cache/swift /var/log/keystone ;

Remove LVM volume

vgremove -f cinder-volumes ;

Delete SSL certs

find /etc/pki/tls -name "ssl_ps*" | xargs rm -rf ;

Unmount any leftover mounts

for x in $(df | grep "/lib/" | sed -e 's/.* //g') ; do
    umount $x ;
done

#systemctl reboot

Summary

This was a quick article focused on tips and tricks around OpenStack Kilo. I will continue to update this article with new tips and tricks. If you have anything you came across in OpenStack Kilo, please share.

Happy OpenStacking!

(c) 2015 Keith Tenzer