OpenShift on OpenStack 1-2-3: Bringing IaaS and PaaS Together



In this article we will explore why you should consider tackling IaaS and PaaS together. Many organizations gave up on OpenStack during it’s hype phase, but in my view it is time to reconsider the IaaS strategy. Two main factors are really pushing a re-emergence of interest in OpenStack and that is containers and cloud.

Containers require very flexible, software-defined infrastructure and are changing the application landscape fast. Remember when we had the discussions about pets vs cattle? The issue with OpenStack during it’s hype phase was that the workloads simply didn’t exist within most organizations, but now containers are changing that, from a platform perspective. Containers need to be orchestrated and the industry has settled in on Kubernetes for that purpose. In order to run Kubernetes you need quite a lot of flexibility at scale on the infrastructure level. You must be able to provide solid Software Defined Networking, Compute, Storage, Load Balancing, DNS, Authentication, Orchestration, basically everything and do so at a click of the button. Yeah we can all do that, right.

If we think about IT, there are two types of personas. Those that feel IT is generic, 80% is good enough and for them, it is a light switch: on or off. This persona has no reason whatsoever to deal with IaaS and should just go to the public cloud, if not already there. In other words, OpenStack makes no sense. The other persona feel IT adds compelling value to their business and going beyond 80% provides them with distinct business advantages. Anyone can go to public cloud but if you can turn IT into a competitive advantage then there may actually be a purpose for it. Unfortunately with the way many organizations go about IT today, it is not really viable, unless something dramatic happens. This brings me back to OpenStack. It is the only way an organization can provide the capabilities a public cloud offers while also matching price, performance and providing a competitive advantage. If we cannot achieve the flexibility of public cloud, the consumption model, the cost effectiveness and provide compelling business advantage then we ought to just give up right?

I also find it interesting that some organizations, even those that started in the public cloud are starting to see value in build-your-own. Dropbox for example, originally started using AWS and S3. Over last few years they built their own object storage solution, one that provided more value and saved 75 million over two years. They also did so with a fairly small team. I certainly am not advocating for doing everything yourself, I am just saying that we need to make a decision, does IT provide compelling business value? Can you do it for your business, better than the generic level playing field known as public cloud? If so, you really ought to be looking into OpenStack and using momentum behind containers to bring about real change.

OpenShift and the case for OpenStack

OpenShift of course is infrastructure independent. You can run it on public cloud, virtualization, baremetal or anything that can boot Red Hat Enterprise Linux. All organizations definitely want and will use the public cloud but likely will also want to maintain control, avoiding lock-in. OpenShift is the only way to truly get multi-cloud, enterprise Kubernetes. The idea here with OpenStack is to deliver the on-premise portion of multi-cloud, with the same capabilities as public cloud. Today organizations have an incredible investment in their on-premise IT. Even if you don’t see IT as a value generator, it is clear you most likely won’t want to divest all those resources at once. Growth will most likely be augmented by public cloud as opposed to a complete migration.

To the next point, what is the right infrastructure to actually run on? Certainly over the years a vast majority of applications have moved to virtualization platforms but not all. I expect this also remains. Why? Well beyond 16 vCPUs, VMs start getting into the law of diminishing returns. You end up getting less value out of hyperthreading and usually needing to limit vCPUs to number of cores. Baremetal may also have advantages in certain container use cases like large scale computing. With emergence of AI and also need for large data crunching, baremetal could actually be gaining steam as a future platform. Regardless the point here is you may want your containers to run in VMs (smaller OpenShift Nodes) or baremetal (larger OpenShift nodes) and this is highly dependent on application or workload. Finally, there are other factors that could make baremetal play important role that won’t be covered, cost/performance or isolation/security.

If we stick to virtualization technology we have one and only one choice. This again is where OpenStack shines, at least Red Hat OpenStack. One of the components shipped is ironic (metal-as-a-service). Ironic allows us to manage baremetal just like a virtual machine, in fact in OpenStack there is no difference and why OpenStack refers to compute units as instances, because it could be either. OpenStack can provide OpenShift with VM or baremetal based nodes and much, much more.

OpenShift integration with OpenStack

OpenShift and OpenStack fit perfectly together. Below is a list of the major integration points.

  • Keystone provides identity and can be used to authenticate OpenShift or LDAP users.
  • Ceilometer provides telemtry of IaaS allowing correlation using CloudForms between container, node and instance.
  • Multi tenant could help if running many OpenShift clusters.
  • Heat provides orchestration enabling dynamic scale-up or scale-down of OpenShift cluster.
  • Nova provides OpenShift nodes as a VM or baremetal instance.
  • Neutron provides SDN and through Kuryr (starting with Red Hat OpenStack 13) will allow neutron SDN to be consumed in OpenShift directly allowing single SDN to serve both container and non-container workloads.
  • Cinder provides dynamic storage and provisioning for containers running in OpenShift.
  • LBaaS provides load balancer for API across masters and for application traffic across infrastructure nodes running OpenShift router.
  • Designate provides DNS and OpenShift needs either dynamic DNS or to use wildcard for application domains.
  • Ironic plugs into Nova via ironic conductor and allows provisioning of baremetal systems.


OpenShift on OpenStack Architectures

Important to any underlying architecture discussion is how to group OpenShift masters, infrastructure and application nodes. OpenStack provides two different possibilities.

Resource vs AutoScaling Groups

Resource groups allow us to group instances together and apply affinity or anti-affinity policies via the OpenStack scheduler. AutoScaling groups allow us to group instances and based on alarms, scale-up or scale-down those instances automatically. At first glance, you would think for masters and infra nodes use resource groups and app nodes autoscaling groups. While autoscaling sounds great, especially for app nodes, there are a lot of possibilities that can lead to scaling either happening or not happening when desired. My experience is this can work well with simple wordpress type applications but not something more complex, like a container platform or OpenShift. Also another disadvantage with autoscaling groups is they don’t support an index. Indexes within groups are used to increment the instance name: master0, master1 and so on. A final point is that you can easily scale resource groups, it just needs to be triggered by an update to the Heat stack. The nice thing is you can also control scaling and if it is to be automated, you have more flexibility than relying on alarms in Ceilometer. For all of these reasons I recommend creating three resource groups: masters, infras and nodes.

Two common OpenShift architectures for OpenStack are non-ha and ha within single tenant.


In this architecture we will have one master, one infra node and x application nodes. While certainly application availability can be achieved by deploying across multiple nodes, the master presents a single point of failure for the data plane. The infra node runs the OpenShift router and as such a failure here would mean incoming traffic to applications would be interrupted.



The HA architecture typically has three masters, two infra nodes and x app nodes. There are variations where you could have 3 infra nodes if you are running metrics and logging services that require a third node. In addition you could also split etcd and run it independently, on three additional nodes. If east/west traffic is not allowed between network zones, then you would likely require two infra nodes in each zone, to handle incoming traffic for app nodes. There are many variations of course, but for now let us keep it simple.

openshift_on_openstack_ha (2)

Deploying OpenStack

In order to deploy OpenShift on OpenStack we obviously need OpenStack. Here are some guides to help.

Once OpenStack is deployed you need to ensure a few things are in place.

Create Flavors

# openstack flavor create --ram 2048 --disk 30 --ephemeral 0 --vcpus 1 --public ocp.bastion
# openstack flavor create --ram 8192 --disk 30 --ephemeral 0 --vcpus 2 --public ocp.master
# openstack flavor create --ram 8192 --disk 30 --ephemeral 0 --vcpus 1 --public ocp.infra
# openstack flavor create --ram 8192 --disk 30 --ephemeral 0 --vcpus 1 --public ocp.node

Create RHEL Image

Download RHEL 7.4 Cloud (qcow2) Image

# openstack image create --disk-format qcow2 \
--container-format bare --public \
--file /root/rhel-server-7.4-x86_64-kvm.qcow2 "rhel74"

Create Private Key

# openstack keypair create admin

Save Private Key

# vi /root/admin.pem
# chmod 400 /root/admin.pem

Create Public Floating IP Network

# openstack network create --provider-network-type flat \
--provider-physical-network extnet2 --external public

Create Public Floating IP Subnet

# openstack subnet create --network public --allocation-pool \
start=,end= --no-dhcp \
--subnet-range public_subnet

Create Router

# openstack router create --no-ha router1

Set Router Gateway

# openstack router set --external-gateway public router1

That is it! Everything else will be created automatically by the deployment of the OpenShift infrastructure. If you want to include more or less you can also easily update the Heat templates provided.

Deploying OpenShift on OpenStack 1-2-3

Once you have OpenStack environment configured, deploying OpenShift will be done using a simple three step phased approach.

  • Step 1 Deploy OpenShift Infrastructure using Heat and Ansible.
  • Step 2 Install OpenShift using Ansible.
  • Step 3 Configure OpenShift and additional services using Ansible.

The Heat templates, all playbooks and a README is provided in the following Github repository:


This step is responsible for deploying OpenShift infrastructure. Ansible will be used to call Heat to deploy the infrastructure in OpenStack. The heat templates will create a private network, load balancers, cinder storage, connect to existing public network, boot all instance and prepare the bastion host. The bastion host is used to deploy and manage the OpenShift deployment.

[OpenStack Controller]

Clone Git Repository

# git clone

Checkout release branch 1.0

# git checkout release-1.0

Change dir to repository

# cd openshift-on-openstack-123

Configure Parameters

# cp sample-vars.yml vars.yml
# vi vars.yml
### OpenStack Setting ###
domain_name: ocp3.lab
dns_forwarders: [,]
external_network: public
image: rhel74
ssh_user: cloud-user
ssh_key_name: admin
stack_name: openshift
openstack_version: 12
contact: admin@ocp3.lab
heat_template_path: /root/openshift-on-openstack-123/heat/openshift.yaml

### OpenShift Settings ###
openshift_version: 3.7
docker_version: 1.12.6
openshift_ha: true
registry_replicas: 2
openshift_user: admin

### Red Hat Subscription ###

### OpenStack Instance Count ###
master_count: 3
infra_count: 2
node_count: 2

### OpenStack Instance Group Policies ###
### Set to 'affinity' if only one compute node ###
master_server_group_policies: "['anti-affinity']"
infra_server_group_policies: "['anti-affinity']"
node_server_group_policies: "['anti-affinity']"

### OpenStack Instance Flavors ###
bastion_flavor: ocp.bastion
master_flavor: ocp.master
infra_flavor: ocp.infra
node_flavor: ocp.node

Authenticate OpenStack Credentials

# source /root/keystonerc_admin

Disable host key checking


Deploy OpenStack Infrastructure for OpenShift

# ansible-playbook deploy-openstack-infra.yml \n
--private-key=/root/admin.pem -e @vars.yml


This step is responsible for preparing OpenShift environment. The hostnames will be set,  OpenShift inventory file dynamically generated, systems will be registered to rhn, required packages installed and docker, among other things properly configured.

Get IP address of the bastion Host

# openstack stack output show -f value -c output_value openshift ip_address

  "masters": [
      "name": "master0",
      "address": ""
      "name": "master1",
      "address": ""
      "name": "master2",
      "address": ""
  "lb_master": {
    "name": "lb_master",
    "address": ""
  "infras": [
      "name": "infra0",
      "address": ""
      "name": "infra1",
      "address": ""
  "lb_infra": {
    "name": "lb_infra",
    "address": ""
  "bastion": {
    "name": "bastion",
    "address": ""
  "nodes": [
      "name": "node0",
      "address": ""
      "name": "node1",
      "address": ""

SSH to the Bastion Host using cloud-user and Private Key

# ssh -i /root/admin.pem cloud-user@

[Bastion Host]

Change Directory to Cloned Git Repository

# cd openshift-on-openstack-123

Authenticate OpenStack Credentials

[cloud-user@bastion ~]$ source /home/cloud-user/keystonerc_admin

Disable Host Key Checking

[cloud-user@bastion ~]$ export ANSIBLE_HOST_KEY_CHECKING=False

Prepare the Nodes for Deployment of OpenShift

[cloud-user@bastion ~]$ ansible-playbook prepare-openshift.yml \n
--private-key=/home/cloud-user/admin.pem -e @vars.yml

PLAY RECAP *****************************************************************************************
bastion                    : ok=15   changed=7    unreachable=0    failed=0
infra0                     : ok=18   changed=13   unreachable=0    failed=0
infra1                     : ok=18   changed=13   unreachable=0    failed=0
localhost                  : ok=7    changed=6    unreachable=0    failed=0
master0                    : ok=18   changed=13   unreachable=0    failed=0
master1                    : ok=18   changed=13   unreachable=0    failed=0
master2                    : ok=18   changed=13   unreachable=0    failed=0
node0                      : ok=18   changed=13   unreachable=0    failed=0
node1                      : ok=18   changed=13   unreachable=0    failed=0


This step is responsible for configuring a vanilla OpenShift environment. By default only the OpenShift router and registry will be configured. OpenShift will be deployed based on the dynamically generated inventory file in step 2. You can certainly edit the inventory file and make any changes. After deployment of OpenShift, there is a small post deployment playbook which will configure dynamic storage to use OpenStack Cinder. Optional steps are defined as well to configure metrics and logging if that is desired.

[Bastion Host]

Deploy OpenShift

[cloud-user@bastion ~]$ ansible-playbook -i /home/cloud-user/openshift-inventory --private-key=/home/cloud-user/admin.pem -vv /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml
PLAY RECAP *****************************************************************************************
infra0.ocp3.lab            : ok=183  changed=59   unreachable=0    failed=0
infra1.ocp3.lab            : ok=183  changed=59   unreachable=0    failed=0
localhost                  : ok=12   changed=0    unreachable=0    failed=0
master0.ocp3.lab           : ok=635  changed=265  unreachable=0    failed=0
master1.ocp3.lab           : ok=635  changed=265  unreachable=0    failed=0
master2.ocp3.lab           : ok=635  changed=265  unreachable=0    failed=0
node0.ocp3.lab             : ok=183  changed=59   unreachable=0    failed=0
node1.ocp3.lab             : ok=183  changed=59   unreachable=0    failed=0

INSTALLER STATUS ***********************************************************************************
Initialization             : Complete
Health Check               : Complete
etcd Install               : Complete
Master Install             : Complete
Master Additional Install  : Complete
Node Install               : Complete
Hosted Install             : Complete
Service Catalog Install    : Complete

Run Post Install Playbook

[cloud-user@bastion ~]$ ansible-playbook post-openshift.yml --private-key=/home/cloud-user/admin.pem -e @vars.yml

PLAY RECAP **************************************************************************************************************************
infra0                     : ok=4    changed=2    unreachable=0    failed=0
infra1                     : ok=4    changed=2    unreachable=0    failed=0
localhost                  : ok=7    changed=6    unreachable=0    failed=0
master0                    : ok=6    changed=4    unreachable=0    failed=0
master1                    : ok=6    changed=4    unreachable=0    failed=0
master2                    : ok=6    changed=4    unreachable=0    failed=0
node0                      : ok=4    changed=2    unreachable=0    failed=0
node1                      : ok=4    changed=2    unreachable=0    failed=0

Login in to UI


Configure Admin User

[cloud-user@bastion ~]$ ssh -i /home/cloud-user/admin.pem cloud-user@master0

Authenticate as system:admin User

[cloud-user@master0 ~]$ oc login -u system:admin -n default

Make User OpenShift Cluster Administrator

[cloud-user@master0 ~]$ oadm policy add-cluster-role-to-user cluster-admin admin

Install Metrics

Set Metrics to true in Inventory

[cloud-user@bastion ~]$ vi openshift_inventory

Run Playbook for Metrics

[cloud-user@bastion ~]$ ansible-playbook -i /home/cloud-user/openshift-inventory --private-key=/home/cloud-user/admin.pem -vv /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-metrics.yml
PLAY RECAP **************************************************************************************************************************
infra0.ocp3.lab            : ok=45   changed=4    unreachable=0    failed=0
infra1.ocp3.lab            : ok=45   changed=4    unreachable=0    failed=0
localhost                  : ok=11   changed=0    unreachable=0    failed=0
master0.ocp3.lab           : ok=48   changed=4    unreachable=0    failed=0
master1.ocp3.lab           : ok=48   changed=4    unreachable=0    failed=0
master2.ocp3.lab           : ok=205  changed=48   unreachable=0    failed=0
node0.ocp3.lab             : ok=45   changed=4    unreachable=0    failed=0
node1.ocp3.lab             : ok=45   changed=4    unreachable=0    failed=0

INSTALLER STATUS ********************************************************************************************************************
Initialization             : Complete
Metrics Install            : Complete

Install Logging

Set logging to true in Inventory

[cloud-user@bastion ~]$ vi openshift_inventory

Run Playbook for Logging

[cloud-user@bastion ~]$ ansible-playbook -i /home/cloud-user/openshift-inventory --private-key=/home/cloud-user/admin.pem


In this article we discussed the important role infrastructure plays when deploying a container platform such as OpenShift. We also discussed the basis for various infrastructure decisions, containers, public cloud vs on-premise. There are very compelling reasons why IaaS and PaaS fit so well together and why it is important to tackle both, not just one of them. OpenStack is a perfect fit for OpenShift from an infrastructure perspective and many of the integration points were discussed in detail. Finally a hands-on guide was provided to deploy OpenStack and OpenShift on OpenStack in an automated, easy three step process. Many try to avoid tackling infrastructure and just focus on OpenShift with their applications. I certainly cannot fault anyone for taking that approach, but I think you will get the most value, out of tackling both and if you use containers as an excuse to do so, then so be it.

Happy OpenShifting on OpenStack!

(c) 2018 Keith Tenzer

11 thoughts on “OpenShift on OpenStack 1-2-3: Bringing IaaS and PaaS Together

  1. Hello,
    I am using your manual to deploy Openshift comtainer platform on openstack. While doing last command in step 1 with ansible-playbook, I am getting following error:

    Error! no action detected in task
    The error appears to have been in ‘/home/hmagnusson/openshift-on-openstack-123/deploy-openstack-infra.yml’: line 8, column 5, but may be elsewhere in the file depending on the exact syntax problem
    The offending line appears to be:
    – import_role:
    ^ here
    I am not understanding what causing this error. Can you please help me?
    Thank you.


    • Make sure you are using stuff from master branch, do “git checkout master”. There shouldn’t be playbook errors so if that is case it could be you have old version of ansible that doesn’t support import command. If you installed red hat software and enabled correct repos it should be fine. If you trying with centos then never tried so you on your own there.

      Liked by 1 person

      • Hello,
        Thank you for replying. I have OpenStack on remote bare metal server and I am deploying OpenShift on Fedora while I am connected to OpenStack via VPN.
        I will try doing what you suggested. Also, do all these steps need to be executed on the controller node?


  2. Hello,
    Thank you again for previous help.
    For now, I am getting following error:

    TASK [osp-stack-create : set_fact] *********************************************
    fatal: [localhost]: FAILED! => {“msg”: “the field ‘args’ has an invalid value ([]), and could not be converted to an dict.The error was: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)\n\nThe error appears to have been in ‘/openshift-on-openstack-123/roles/osp-stack-create/tasks/main.yml’: line 40, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- set_fact:\n ^ here\n”}
    to retry, use: –limit @/openshift-on-openstack-123/deploy-openstack-infra.retry

    I am working with centos VM in which I installed ansible (version 2.5.1). I did some google search but in vain.
    I am doing it all by myself with not so much experience. It would be really helpful if you could guide me.
    Thank you.


      • Hello,
        I managed to get through with above mentioned problem (while trying some solutions from Google). Currently I am facing similar kind of issue.
        I am on the release-1.0.
        The error I am getting now is:

        TASK [osp-inventory : Add the nodes to the inventory] **************************
        fatal: [localhost]: FAILED! => {“msg”: “The task includes an option with an undefined variable. The error was: ‘None’ has no attribute ‘address’\n\nThe error appears to have been in ‘/home/stack/openshift-on-openstack-123/roles/osp-inventory/tasks/main.yml’: line 48, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Add the nodes to the inventory\n ^ here\n”}

        Thank you.


    • Hi,

      I am running this on RHEL7.4 and and an getting this exact error. How did you solve it? (I am using release-1.0)
      rpm -q ansible


  3. I solved the error by changing this –



    openstack stack output show -f value -c output_value {{ stack_name }} ip_address


    openstack stack output show -f json -c output_value {{ stack_name }} ip_address


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s