Advanced Deployment with TripleO and OpenStack Director

Overview

In this article we will look at some advanced deployment scenarios using TripleO and the OpenStack Director. This article builds on a previous article: HOWTO: OpenStack Deployment using TripleO and the Red Hat OpenStack Director. It assumes you have a working OpenStack environment built with Red Hat OpenStack Director and complete with undercloud as well as overcloud. We will tear down existing OpenStack environment, create a new deployment using custom OpenStack Ironic profiles, customize post-installation and show how to add additional compute resources.

Tear Down Existing Environment

One of the advantages of TripleO is that it uses OpenStack to deploy OpenStack. This makes it of course very easy to tear down and rebuilt environments. The basic idea is that immutable and reproducible infrastructure leads to much higher stability. Production environments can be tested properly because they are reproducible. Production is never changed, it is re-deployed and only the deployment configuration is ever changed. These are common practices for any software engineering group but only recently have been applied to Infrastructure itself.

Delete ironic baremetal nodes.

[stack@undercloud ~]$ for i in $(ironic node-list | grep -v UUID | awk ' { print $2 } '); do ironic node-delete $i; done

Import the baremetal node configuration.

[stack@undercloud ~]$ openstack baremetal import --json instackenv.json

Undeploy overcloud by deleting the Heat stack.

[stack@undercloud ~]$ heat stack-delete overcloud

Ironic Profiles

In our original deployment we used a single Ironic profile. Normally however you would probably have different resource expectations for controller, compute and even ceph nodes. Using AHC-Tools (Automated Health Check) it is possible to build Ironic profiles that match specific hardware.

Install AHC-Tools.

[stack@undercloud ~]$ sudo yum install -y ahc-tools

Configure AHC-Tools.

[stack@undercloud ~]$ sudo su -
undercloud# sed 's/\[discoverd/\[ironic/' /etc/ironic-discoverd/discoverd.conf > /etc/ahc-tools/ahc-tools.conf 

[stack@undercloud ~]$ unset DIB_YUM_REPO_CONF
[stack@undercloud ~]$ unset DIB_LOCAL_IMAGE
[stack@undercloud ~]$ sudo systemctl stop bootif-fix 
[stack@undercloud ~]$ source ~/stackrc 

Enable discovery using AHC-Tools and re-deploy the undercloud.

[stack@undercloud ~]$ openstack-config --set ~/undercloud.conf DEFAULT discovery_runbench true
[stack@undercloud ~]$ openstack undercloud install
[stack@undercloud ~]$ sudo systemctl start bootif-fix
[stack@undercloud ~]$ source ~/stackrc

[stack@undercloud ~]$ openstack baremetal import --json instackenv.json

[stack@undercloud ~]$ openstack baremetal configure boot 

Ironic nodes should be set to available.

[stack@undercloud ~]$ ironic node-list
+--------------------------------------+------+---------------+-------------+-----------------+-------------+
| UUID | Name | Instance UUID | Power State | Provision State | Maintenance |
+--------------------------------------+------+---------------+-------------+-----------------+-------------+
| 88d5901b-8527-468b-8c7c-852d79096bcc | None | None | power off | available | False |
| d1f47f58-3b29-4dc6-a16c-2999f3422b36 | None | None | power off | available | False |
| 9e6f26df-22dd-4b2b-a862-78b29a284825 | None | None | power off | available | False |
+--------------------------------------+------+---------------+-------------+-----------------+-------------+

Change Ironic node provision state to managed. Manual introspection requires nodes being set to managed.

[stack@undercloud ~]$ ironic node-set-provision-state 88d5901b-8527-468b-8c7c-852d79096bcc manage
[stack@undercloud ~]$ ironic node-set-provision-state d1f47f58-3b29-4dc6-a16c-2999f3422b36 manage
[stack@undercloud ~]$ ironic node-set-provision-state 9e6f26df-22dd-4b2b-a862-78b29a284825 manage

Start introspection for each Ironic node.

[stack@undercloud ~]$ openstack baremetal introspection start 88d5901b-8527-468b-8c7c-852d79096bcc
[stack@undercloud ~]$ openstack baremetal introspection start d1f47f58-3b29-4dc6-a16c-2999f3422b36
[stack@undercloud ~]$ openstack baremetal introspection start 9e6f26df-22dd-4b2b-a862-78b29a284825

Verify introspection completes. Check to ensure nodes power-on and in addition monitor introspection to ensure it completes.

[stack@undercloud ~]$ ironic node-list | grep "power on" | awk '{print $2" ("$8" "$9")";}' 

To check introspection status view journalctl logs.

[stack@undercloud ~]$ sudo journalctl -u openstack-ironic-discoverd -f 

Check introspection status using CLI.

[stack@undercloud ~]$ openstack baremetal introspection status 88d5901b-8527-468b-8c7c-852d79096bcc 
+----------+-------+ 
| Field    | Value | 
+----------+-------+ 
| error    | None  |  
| finished | False | 
+----------+-------+ 

Wait until introspection completes for all nodes.

[stack@undercloud ~]$ openstack baremetal introspection status 88d5901b-8527-468b-8c7c-852d79096bcc 
+----------+-------+ 
| Field    | Value | 
+----------+-------+ 
| error    | None  | 
| finished | True  | 
+----------+-------+

Once introspection completes we need to set the state of each node to 'available'. Again this is automatically done for us when doing automatic introspection.

[stack@undercloud ~]$ for i in $(ironic node-list | grep -v UUID | awk ' { print $2 } '); do ironic node-set-provision-state $i provide; done

Generate report of benchmarking and introspection using ahc-tools.

undercloud$ ahc-report --full

We can view indentical systems as well as systems that are not consistent with our configuration

undercloud$ ahc-report --categories | grep -A3 "3 identical systems"
undercloud$ ahc-report --outliers | grep -i inconsistent | head -n5 

Ironic hardware specs are stored in various configuration files. Once exists for compute, controller and ceph. In addition there is a configuration file that depicts the OpenStack overcloud topology.

Controller specs are stored in following file:

[stack@undercloud ~]$ cat /etc/ahc-tools/edeploy/control.specs
[
('disk', '$disk', 'size', 'gt(4)'),
('network', '$eth', 'ipv4', 'network(192.0.2.0/24)'),
('memory', 'total', 'size', 'ge(4294967296)'),
]

Compute specs are stored in following file:

[stack@undercloud ~]$ cat /etc/ahc-tools/edeploy/compute.specs
[
('disk', '$disk', 'size', 'gt(4)'),
('network', '$eth', 'ipv4', 'network(192.0.2.0/24)'),
('memory', 'total', 'size', 'ge(4294967296)'),
]

OpenStack overcloud topology is stored in following file:

[stack@undercloud ~]$ cat /etc/ahc-tools/edeploy/state
[('control', '1'), ('compute', '*')]

Change the compute and controller specs as follows:

undercloud$ sudo su - 
undercloud# cat << EOF > /etc/ahc-tools/edeploy/control.specs
[
('cpu', 'logical', 'number', 'ge(4)'),
('disk', 'vda', 'size', 'gt(25)'),
]
EOF

undercloud# cat << EOF > /etc/ahc-tools/edeploy/compute.specs
[
('cpu', 'logical', 'number', 'ge(2)'),
('memory', 'total', 'size', 'ge(3000000)'),
]
EOF

Run AHC and have it match profiles with discovered hardware.

undercloud# ahc-match

Node roles can now be viewed. In our case we have 1 controller and two compute nodes that were classified by AHC.

[stack@undercloud ~]$ for i in $(ironic node-list | grep -v UUID | awk '{print $2;}' \
| sed -e /^$/d); do ironic node-show $i | grep -A1 properties; \
echo "======="; done;

| properties | {u'memory_mb': u'4096', u'cpu_arch': u'x86_64', u'local_gb': u'1', |
| | u'cpus': u'4', u'capabilities': u'profile:control,boot_option:local'} |
=======
| properties | {u'memory_mb': u'4096', u'cpu_arch': u'x86_64', u'local_gb': u'1', |
| | u'cpus': u'4', u'capabilities': u'profile:compute,boot_option:local'} |
=======Fgener
| properties | {u'memory_mb': u'4096', u'cpu_arch': u'x86_64', u'local_gb': u'1', |
| | u'cpus': u'4', u'capabilities': u'profile:compute,boot_option:local'} |

Create new Ironic flavors for controller and compute nodes.

[stack@undercloud ~]$ openstack flavor create --id auto --ram 4096 --disk 58 --vcpus 4 control

[stack@undercloud ~]$ openstack flavor create --id auto --ram 4096 --disk 58 --vcpus 2 compute

Update flavors and map them to the Ironic profiles.

undercloud$ openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="control" control 

undercloud$ openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="compute" compute 

Post Installation Customization

Since the basis for TripleO is Heat, customization is thankfully quite simple. In this example we will install Nagios monitoring package to all OpenStack overcloud nodes as part of post-install phase.

Create post installation yaml files.

undercloud$ cat << EOF > ~/templates/firstboot-environment.yaml 
resource_registry: 
  OS::TripleO::NodeUserData: /home/stack/my_templates/firstboot-config.yaml 
EOF 

undercloud$ cat << EOF > ~/templates/firstboot-config.yaml
heat_template_version: 2014-10-16

resources:
  userdata:
    type: OS::Heat::MultipartMime
    properties:
      parts:
      - config: {get_resource: repo_config}

  repo_config:
    type: OS::Heat::SoftwareConfig
    properties:
      config: |
        #!/bin/bash
        yum install -y nagios

outputs:
  OS::stack_id:
    value: {get_resource: userdata}
EOF

Deploy overcloud using custom Ironic profiles and post-installation customization.

[stack@undercloud ~]$ openstack overcloud deploy --templates --control-flavor control --compute-flavor compute --control-scale 1 --compute-scale 1 --neutron-tunnel-types vxlan --neutron-network-type vxlan -e ~/templates/firstboot-environment.yaml
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates
 /home/stack/.ssh/known_hosts updated.
 Original contents retained as /home/stack/.ssh/known_hosts.old
 PKI initialization in init-keystone is deprecated and will be removed.
 Warning: Permanently added '192.168.126.104' (ECDSA) to the list of known hosts.
 The following cert files already exist, use --rebuild to remove the existing files before regenerating:
 /etc/keystone/ssl/certs/ca.pem already exists
 /etc/keystone/ssl/private/signing_key.pem already exists
 /etc/keystone/ssl/certs/signing_cert.pem already exists
 Connection to 192.168.126.104 closed.
 Overcloud Endpoint: http://192.168.126.104:5000/v2.0/
 Overcloud Deployed

Add Additional Compute Resources

As we have seen we can control hardware through Ironic profiles and easily customize the deployment of OpenStack using Heat. Next we will add an additional compute resource. As you have noticed, we have configured three baremetal nodes but have only deployed two (1 X Compute, 1 X Control). A second compute node can easily be added to the OpenStack environment on-the-fly. Again we see the power of immutable infrastructure and automation.

Re-run deployment.

[stack@undercloud ~]$ openstack overcloud deploy --templates --control-flavor control --compute-flavor compute --control-scale 1 --compute-scale 2 --neutron-tunnel-types vxlan --neutron-network-type vxlan -e ~/templates/firstboot-environment.yaml

After deployment completes you should see an additional compute node.

[stack@undercloud ~]$ nova list
+--------------------------------------+------------------------+--------+------------+-------------+---------------+
| ID | Name | Status | Task State | Power State | Networks                                                          |
+--------------------------------------+------------------------+--------+------------+-------------+---------------+
| 53813f9b-7020-4386-9c10-5818633c21ee | overcloud-compute-0    | ACTIVE | - | Running | ctlplane=192.168.126.106   |
| 76e2c3fd-7278-4ded-8893-e6c9d8b4a76f | overcloud-compute-1    | ACTIVE | - | Running | ctlplane=192.168.126.107   |
| 204dd901-a56d-40b6-9b18-bd8ce3ed75ab | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.168.126.108   |
+--------------------------------------+------------------------+--------+------------+-------------+---------------+

Troubleshooting

Introspection Issues

Sometimes introspection may fail. Unfortunately there is no good way of ending the process. To cancel introspection follow the following steps:

[stack@undercloud ~]$ ironic node-set-power-state [NODE UUID] off

[stack@undercloud ~]$ sudo rm /var/lib/ironic-discoverd/discoverd.sqlite

[stack@undercloud ~]$ sudo systemctl restart openstack-ironic-discoverd

Additionally you can change the discovery timeouty by editing the following file:

[stack@undercloud ~]$ sudo vi /etc/ironic-discoverd/discoverd.conf

Note: ensure you use rtl8139 network driver and not virtio if using KVM. The performance of rtl8139 is much better and using virtio will cause PXE timeouts.

Summary

In this article we have seen how to configure Ironic profiles, customize overcloud deployment and add additional compute resources dynamically. The real value of an OpenStack deployment tool is in its ability to create true environment stability. Anyone can install OpenStack and make it look pretty but maintaining it is a completely different story. I would put my money on the latter. Hopefully you found this article of use. As always feedback is welcome so let's hear it!

Happy OpenStacking!

(c) 2015 Keith Tenzer