Automated Infrastructure in the On-Premise Datacenter - OpenShift 4.2 on OpenStack 15 (Stein)

27 minute read

OpenShift-LogoType.svg heavy-plus-signopenstack-logo


In this article we start a new journey, automated infrastructure in the on-premise datacenter. We will deploy OpenShift 4.2 on OpenStack. As I am sure you are aware, OpenShift is Red Hat's enterprise kubernetes platform. Kubernetes is of course the brains but by itself is not a platform. OpenShift brings with kubernetes, monitoring, aggregate logging, container registry, security, automated deployment/upgrade, developer experience, huge middleware tooling built around JBoss, serverless frameworks, ISTIO (service mesh), CI/CD integration and the key word "ENTERPRISE".

OpenShift 4 brings with it, one thing that I feel is a breakthrough in our industry. It is the single feature I am most excited about and one of the reasons OpenShift 4 was re-designed, Installer Provision Infrastructure (IPI). Using IPI, OpenShift is capable of not only installing itself but deploying as well as managing the underlying infrastructure needed, from compute, to storage and network. Further using IPI OpenShift can also update, even upgrade itself over the air. This is a huge step forward in platform intelligence and where we need to be to deal with today's complexity.

Users can also still opt for User Provisioned Infrastructure (UPI), if they want to bring their own infrastructure, deal with micro-managing everything and waste a lot of time but why? Yeah I thought so too.

Up until now IPI has only supported public clouds: AWS, Azure and Google. Now with OpenShift 4.2 it is supporting OpenStack. For the first time we can bring IPI into the on-premise Datacenter where it is IMHO most needed. This single feature has the potential to revolutionize on-premise environments and bring them into the cloud-age with a single click and that promise is truly something to get excited about!


Before we start the installation it is important to understand some pre-requisites and how to properly configure them.

Deploy OpenStack

I have provided a guide for deploying OpenStack Stein (RHOSP 15) using packstack on Hetzner root servers here. Even if you aren't using a Hetzner root server this guide can help you spin up a demo OpenStack environment using RDO packstack. If you have Red Hat OpenStack Platform (RHOSP) 15 even better!

Configure External DNS

If you have problems later, this will likely be the cause, trust me from experience, get your DNS sorted from the beginning.

You will need a DNS domain, mine is Once you have a domain you need to manage it somewhere. I can recommend cloudflare, they even have a free tier. Once you have that you need to create two A records under the domain (


The OpenShift installer will ask for a floating ip for API traffic. This is required to install OpenShift. The OpenShift installer will configure an ingres port in OpenStack and use keepalived across the masters.


A DNS wildcard is used to send all application traffic to another IP. An additional IP is needed because the OpenShift router operator (ha-proxy) is responsible for handling application traffic not the API operator. The installer will also not automatically configure this floating ip on the ingress port like it does for API. That needs to be done after the installation.

The diagram below shows my DNS configuration.


OpenStack Configuration

To deploy OpenShift we will need an external (floating ip) network, a compute flavor and setup a project properly.

In order to create an floating ip network you need a subnet of physical IPs that can communicate with the outside world. They don't have to be public but need to be routable to whatever is the "outside world", meaning outside of OpenStack.

In this example we have a flat internet routable network

Authenticate to OpenStack

# source /root/keystonerc_admin

Create public network

(keystone_admin)]# openstack network create --provider-network-type flat \
--provider-physical-network extnet2 --external public

Create public subnet

(keystone_admin)]# openstack subnet create --network public \
--allocation-pool start=,end= --no-dhcp \
--subnet-range public_subnet

Create OpenStack flavor for OpenShift nodes

(keystone_admin)]# openstack flavor create --ram 16384 --disk 25 \
--vcpu 4 --public ocp.controller

Setup Project

(keystone_admin)]# openstack project create openshift_4
(keystone_admin)]# openstack quota set --ram -1 --cores -1 openshift_4
(keystone_admin)]# openstack quota set --secgroups 3 --secgroup-rules 60 \
(keystone_admin)]# openstack user create --password '<password>' \
(keystone_admin)]# openstack role add --project openshift_4 --user \
openshift_admin admin
(keystone_admin)]# openstack role add --user openshift_admin --project \
openshift_4 swiftoperator

Create project RC file

# vi keystonerc_openshift_admin
export OS_USERNAME=openshift_admin
export OS_PASSWORD='<password>'
export OS_REGION_NAME=RegionOne
export OS_AUTH_URL=
export PS1='[\u@\h \W(keystone_openshift_admin)]\$ '

export OS_PROJECT_NAME=openshift_4
export OS_USER_DOMAIN_NAME=Default

Authenticate to OpenShift user

(keystone_admin)]# source /root/keystonerc_openshift_admin

Set temporary key for Swift

The Container OS ignite configuration file used for bootstrapping is stored in Swift (OpenStack Object Store). This allows for URL access using a temporary key.

(keystone_openshift_admin)]# openstack object store account set --property \

Download Red Hat OpenStack Container OS image

You can download the Red Hat Container OS (rhcos) qcow2 image from or this link for the 4.2 release. Once you have the image copy it over to your environment.

Create RHOS Image

First make sure you get the latest RHOS image. Check if the image is a compressed file once downloaded.

(keystone_openshift_admin)]# file rhcos-4.2.0-x86_64-openstack.qcow2
rhcos-4.2.0-x86_64-openstack.qcow2: gzip compressed data
(keystone_openshift_admin)]# mv rhcos-4.2.0-x86_64-openstack.qcow2 \
(keystone_openshift_admin)]# gunzip rhcos-4.2.0-x86_64-openstack.qcow2.gz
(keystone_openshift_admin)]# file rhcos-4.2.0-x86_64-openstack.qcow2
rhcos-4.2.0-x86_64-openstack.qcow2: QEMU QCOW Image (v3)
(keystone_openshift_admin)]# openstack image create rhcos \
--container-format bare --disk-format qcow2 --public --file \

Download OpenShift Installer, Command Line Tools and Pull Secret

You can download the installer, command line tools and pull secret at following URL: You will need the pull secret later.

Select your provider, in this case OpenStack.


Download the installer, CLI and your pull secret.


Once you have the installer and CLI tools downloaded, extract them into /usr/bin.

(keystone_openshift_admin)]# tar xvfz openshift-install-linux-4.2.0.tar.gz \
-C /usr/bin/
(keystone_openshift_admin)]# tar xvfz openshift-client-linux-4.2.0.tar.gz \
-C /usr/bin/

Create floating-ip for API

(keystone_openshift_admin)]# openstack floating ip create \
--floating-ip-address public
(keystone_openshift_admin)]# openstack floating ip list
| ID | Floating IP Address | Fixed IP Address | Port | Floating Network | Project |
| fd98ca80-613f-4d36-819f-c5be66522439 | | | e17b2845-a4a2-44f1-8388-549a352a9fff | 0b8817e8-ffcc-4403-956c-eaadfb785e57 | 4e4c6f30c2c843258239efc5e699bec6 |

Configure OpenShift 4.2

Once OpenStack environment is setup properly and we have our external DNS sorted, the installation configuration for OpenShift can begin.

Get Project ID

The OpenStack project id is needed for the next section.

(keystone_openshift_admin)]# openstack project show openshift_4 -c id \
-f value

Configure clouds.yaml

OpenShift IPI can configure and manage multiple OpenShift on OpenStack clouds. In this section we will just configure one. All the information needed except the project id is available in the keystonerc file.

(keystone_openshift_admin)]# vi clouds.yaml
      project_name: openshift_4
      username: openshift_admin
      password: <password>
      user_domain_name: Default
      project_domain_name: Default
      project_id: c36f0b1258fe447fbbbdcf6e2d2e4e0a

Create Install Configuration

OpenShift IPI will create a directory with the installation configuration. The manifests and ignition configs can be optionally created, you should not have to change them but you could nevertheless.

(keystone_openshift_admin)]# openshift-install create install-config \
? SSH Public Key /root/.ssh/
? Platform openstack
? Cloud openstack
? ExternalNetwork public
? APIFloatingIPAddress
? FlavorName ocp.controller
? Base Domain
? Cluster Name ocp4
? Pull Secret [? for help] ************************************************

Update the installation

We will reduce the number of workers from three to two, set the region, disable octaviaSupport and trunkSupport.

(keystone_openshift_admin)]# vi ocp_install/install-config.yaml
- hyperthreading: Enabled
name: worker
platform: {}
replicas: 2
octaviaSupport: "0"
region: "RegionOne"
trunkSupport: "0"

Create Ignition files (optional)

In case you want to change something in the ignite configuration.

(keystone_openshift_admin)]# openshift-install create ignition-configs --dir

Create Installation Manifests (optional)

In case you want to change something in one of the manifests.

(keystone_openshift_admin)]# openshift-install create manifests --dir ./ocp_install/

Run the Installer

(keystone_openshift_admin)]# time openshift-install --log-level=debug \
create cluster --dir ocp_install
INFO Consuming "Install Config" from target directory
INFO Waiting up to 30m0s for the Kubernetes API at
INFO API v1.14.6+2e5ed54 up
INFO Waiting up to 30m0s for bootstrapping to complete...
DEBUG Still waiting for the cluster to initialize: Working towards 4.2.0: 97% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.2.0: 98% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.2.0: 98% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.2.0: 99% complete, waiting on authentication, console, image-registry, ingress, monitoring, operator-lifecycle-manager-packageserver
DEBUG Still waiting for the cluster to initialize: Working towards 4.2.0: 99% complete
DEBUG Still waiting for the cluster to initialize: Working towards 4.2.0: 100% complete, waiting on authentication
DEBUG Cluster is initialized
INFO Waiting up to 10m0s for the openshift-console route to be created...
DEBUG Route found in openshift-console namespace: console
DEBUG Route found in openshift-console namespace: downloads
DEBUG OpenShift console route is created
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/root/ocp_install/auth/kubeconfig'
INFO Access the OpenShift web-console here:

Once the installation completes we still need to bind the floating-ip for application traffic to the ingres port in OpenStack. In OpenShift 4, everything is an operator, including the console. Accessing any applications will go through the OpenShift router using the wildcard domain. In our case we created * with IP

Create Floating-ip for Application Traffic

(keystone_openshift_admin)]# openstack floating ip create \
--floating-ip-address public

Bind Floating-ip to Ingres Port

The port will be called <cluster name>-<clusterID>-ingress-port or in our case ocp4-lq2ww-ingress-port.

(keystone_openshift_admin)]# openstack floating ip set --port \

At this point you should be able to connect to the console using the kubeadmin account and the password generated by the installer.


Destroy the cluster

Destroying the cluster is just as easy as creating one.

(keystone_openshift_admin)]# time openshift-install --log-level=debug \
destroy cluster --dir ocp_install
(keystone_openshift_admin)]# rm -rf ocp_install/

Validate the Environment

Using the CLI tools we will now authenticate and validate the environment.


To authenticate we simply need to export the kubeconfig created by the installer.

(keystone_openshift_admin)]# export KUBECONFIG=/root/ocp_install/auth/kubeconfig

List all nodes

We can list all controllers and workers and view their status.

(keystone_openshift_admin)]# oc get nodes
NAME                      STATUS   ROLES    AGE    VERSION
ocp4-lq2ww-master-0       Ready    master   166m   v1.14.6+c07e432da
ocp4-lq2ww-master-1       Ready    master   166m   v1.14.6+c07e432da
ocp4-lq2ww-master-2       Ready    master   166m   v1.14.6+c07e432da
ocp4-lq2ww-worker-kxnjh   Ready    worker   155m   v1.14.6+c07e432da
ocp4-lq2ww-worker-zhmfl   Ready    worker   155m   v1.14.6+c07e432da

Get Cluster Version

Shows not only the version but the cluster state.

(keystone_openshift_admin)]#  oc get clusterversion
version   4.2.0     True        False         149m    Cluster version is 4.2.0

Get Cluster Operators

Remember in OpenShift 4, everything is an operator. Here we can view their individual status.

(keystone_openshift_admin)]# oc get clusteroperator
authentication                             4.2.0     True        False         False      150m
cloud-credential                           4.2.0     True        False         False      166m
cluster-autoscaler                         4.2.0     True        False         False      158m
console                                    4.2.0     True        False         False      153m
dns                                        4.2.0     True        False         False      165m
image-registry                             4.2.0     True        False         False      155m
ingress                                    4.2.0     True        False         False      156m
insights                                   4.2.0     True        False         False      166m
kube-apiserver                             4.2.0     True        False         False      164m
kube-controller-manager                    4.2.0     True        False         False      163m
kube-scheduler                             4.2.0     True        False         False      163m
machine-api                                4.2.0     True        False         False      166m
machine-config                             4.2.0     True        False         False      165m
marketplace                                4.2.0     True        False         False      158m
monitoring                                 4.2.0     True        False         False      154m
network                                    4.2.0     True        False         False      163m
node-tuning                                4.2.0     True        False         False      162m
openshift-apiserver                        4.2.0     True        False         False      161m
openshift-controller-manager               4.2.0     True        False         False      164m
openshift-samples                          4.2.0     True        False         False      158m
operator-lifecycle-manager                 4.2.0     True        False         False      165m
operator-lifecycle-manager-catalog         4.2.0     True        False         False      165m
operator-lifecycle-manager-packageserver   4.2.0     True        False         False      156m
service-ca                                 4.2.0     True        False         False      166m
service-catalog-apiserver                  4.2.0     True        False         False      162m
service-catalog-controller-manager         4.2.0     True        False         False      162m
storage                                    4.2.0     True        False         False      159m

List All Pods

Operators of course in the end, deploy pods which run containers. Using this command you can get the status of all pods in the cluster.

(keystone_openshift_admin)]# oc get pods -A

Subscribe Cluster

New in OpenShift 4 is a portal management for OpenShift clusters. Here you can see the overall status, manage subscriptions and even access the console.

Simply login to using your credentials.


Here we can see that our newly provisioned cluster is not subscribe. To subscribe the cluster login to Here you will see the compute nodes and can simply attach the OpenShift subscription.


Configure Htpasswd Identity Provider

OpenShift integrates with many identity providers. In the case of OpenStack, normally you would likely want to use Keystone. For test environments using htpasswd is also an option. To configure htpasswd we will create a user/password, store it in a secret and then configure the identity provider by creating a custom resource. In OpenShift 4, custom resource definitions are how you integrate other technologies with the platform in an API consistent way.

Create htpasswd file

# htpasswd -c -B -b /root/ocp.htpasswd admin <password>

Create Secret

# oc create secret generic htpass-secret \
--from-file=htpasswd=/root/ocp.htpasswd -n openshift-config

Create Custom Resource

# vi htpasswd-cr.yaml
kind: OAuth
  name: cluster
  - name: my_htpasswd_provider
    mappingMethod: claim
    type: HTPasswd
        name: htpass-secret
# oc apply -f htpasswd-cr.yaml

Add role cluster-admin to user

The admin user exists in our htpasswd identity provider but needs to be given permission in OpenShift.

# oc adm policy add-cluster-role-to-user cluster-admin admin

Login using admin user

# oc login -u admin

OpenShift 4 support multiple identity providers. Once we have added our htpasswd provider we need to select it when logging into the web console.


Adding Nodes and Changing Flavor

The beauty of the OpenStack Installer Provisioned Infrastructure (IPI) is you can easily scale nodes and even change their flavors.

First we can see what machine sets exist in OpenShift.

$ oc get machinesets -n openshift-machine-api
ocp4-qchw8-worker   2         2         1       1           8d

Next we can see details of our machine set.

 oc get machinesets ocp4-qchw8-worker -n openshift-machine-api -o yaml
kind: MachineSet
  creationTimestamp: "2020-01-20T12:52:43Z"
  generation: 1
  labels: ocp4-qchw8 worker worker
  name: ocp4-qchw8-worker
  namespace: openshift-machine-api
  resourceVersion: "3797711"
  selfLink: /apis/
  uid: c0183898-3b83-11ea-9ccb-fa163e99d432
  replicas: 2
    matchLabels: ocp4-qchw8 ocp4-qchw8-worker
      creationTimestamp: null
      labels: ocp4-qchw8 worker worker ocp4-qchw8-worker
        creationTimestamp: null
          cloudName: openstack
            name: openstack-cloud-credentials
            namespace: openshift-machine-api
          flavor: ocp.controller
          image: rhcos
          kind: OpenstackProviderSpec
            creationTimestamp: null
          - filter: {}
            - filter:
                name: ocp4-qchw8-nodes
                tags: openshiftClusterID=ocp4-qchw8
          - filter: {}
            name: ocp4-qchw8-worker
            Name: ocp4-qchw8-worker
            openshiftClusterID: ocp4-qchw8
          - openshiftClusterID=ocp4-qchw8
          trunk: true
            name: worker-user-data
  availableReplicas: 1
  fullyLabeledReplicas: 2
  observedGeneration: 1
  readyReplicas: 1
  replicas: 2

Notice our machine set has two replicas, meaning we have two worker nodes in our OpenShift cluster. If we want to scale our cluster to three workers we just run a simple command. OpenShift takes care of everything, including provisioning the infrastructure.

$ oc scale --replicas=3 machineset ocp4-qchw8-worker -n openshift-machine-api

If we want to change our flavor we can just update the machine set configuration.

$ oc edit machineset ocp4-qchw8-worker -n openshift-machine-api
  name: openstack-cloud-credentials
  namespace: openshift-machine-api
flavor: ocp.controller2
image: rhcos
kind: OpenstackProviderSpec

In order to change the flavor you need to scale the worker machine set to 0. In order to do that you must relocate the router pods in the openshift-ingress project. You can also kill the router pods after scaling to 0 as simple workaround. OpenShift will not delete the last node until the router pod is killed.

Configuring Retention for Prometheus

In OpenShift 4 prometheus is used to store metrics used in monitoring and alerting. By default prometheus will store data locally using 'empty dir'. It's default retention is 15 days but you can change it to your desire. You can also add a persistent volume so the data will survive node restarts.

First create a new config map.

$ oc -n openshift-monitoring create configmap \

Edit config map and add retention.

$ oc -n openshift-monitoring edit configmap cluster-monitoring-config
  config.yaml: |
      retention: 24h


In this article we discussed OpenShift 4 and the value of installer provisioned infrastructure. It goes well beyond installation, that is just the starting point. OpenShift 4 is capable of operating itself to a certain degree, including automated upgrades. This will not only increase availability but also give operations teams a lot of time back to do work on more innovative, productive projects. We showed how to properly prepare for an OpenShift 4.2 installation on OpenStack using IPI. Finally, using IPI we successfully deployed and validated an OpenShift 4.2 cluster running on OpenStack. I look forward to hearing about your feedback as we explore this brave new world together.

Happy OpenShifting!

(c) 2019 Keith Tenzer