Enterprise Container Platform in the Cloud: OpenShift on Azure secured by Azure AD

msazurelogo plus_sign openshiftlogo

Overview

This article is a collaboration from Rolf Masuch (Microsoft) and Keith Tenzer (Red Hat). It is based on our work together in the field with enterprise customers.

In this article we will explore how to deploy a production ready OpenShift enterprise container platform on the Microsoft Azure Cloud. The entire deployment is completely automated using Ansible and ARM (Azure Resource Manager). Everything is template driven using APIs. The bennefit of this approach is the ability to build-up and tear-down a complete OpenShift environment in the Azure cloud before your coffee gets cold.

Since OpenShift already uses Ansible as its installation and configuration management tool, it made sense to stick with Ansible as opposed to using other tools such as Power Shell. A Red Hat colleague, Ivan McKinley created an Ansible playbook that builds out all the required Azure infrastructure components and integrates the existing OpenShift installation playbook. The result is an optimally configure OpenShift environment on the Azure Cloud. We have used this recipe to deploy real production Environments for customers and it leverages both Microsoft as well as Red Hat best practices.

You can access and contribute improvements to the Ansible playbook under Ivan’s Github repository:

https://github.com/ivanthelad/ansible-azure

The following, related articles might also be of Interest in case you want a basic understanding of OpenShift.

The pre-requisites for deploying OpenShift on Azure are a valid OpenShift subscription and a valid Azure subscription.

  • If you don’t already have a OpenShift Subscription you can purchase one or get an eval by talking to your partner or Red Hat account manager.
  • If you don’t already have a Microsoft Azure Subscription you can start one here.

Deploying to Azure

Install Fedora 24 workstation for use as deployment workstation. You need very recent versions of python 2.7 (2.7.12) and unfortunately it isn’t available in RHEL or CentOS at writing of this article so we used Fedora.

Install Python and Dependencies

# sudo yum install python
# sudo yum install python-pip
# sudo dnf install python-devel
# sudo dnf install redhat-rpm-config
# sudo dnf install openssl-devel

Install Azure CLI

# sudo dnf install npm
# sudo npm install azure-cli -g
# sudo pip install --upgrade pip
# sudo pip install azure==2.0.0rc5

Authenticate Azure CLI

[ktenzer@ktenzer ansible-azure]$ azure login
 info: Executing command login
 \info: To sign in, use a web browser to open the page https://aka.ms/devicelogin. Enter the code CB8P5ZCKP to authenticate.
 -info: Added subscription Pay-As-You-Go
 info: Added subscription ITS - RedHat Openshift
 info: Setting subscription "Pay-As-You-Go" as default
 +
 info: login command OK

List Azure Resource Groups

In order to list resource groups you need your Azure subscription id. You can view this by logging into Azure portal with your user.

[ktenzer@ktenzer ansible-azure]$ azure group list --subscription <subscription id>
 info: Executing command group list
 + Listing resource groups
 data: Name Location Provisioning State Tags:
 data: ------------- ---------- ------------------ -----
 data: OpenShift_POC westeurope Succeeded null
 data: Shared westeurope Succeeded null
 info: group list command OK

Install Ansible Core

# sudo dnf install ansible

Clone OpenShift Azure Ansible Playbooks

# git clone https://github.com/ivanthelad/ansible-azure.git

Update Playbook parameters

# cd ansible-azure
# cp group_vars/all_example group_vars/all
# vi group_vars/all
resource_group_name: <new resource group name>
## Azure AD user.
ad_username: <Azure user e.g. keith.tenzer@domain.onmicrosoft.com>
### Azure AD password
ad_password: <Azure Password>
#### Azure Subscription ID
subscriptionID: "<subscription id from Azure>"
## user to login to the jump host. this user will only be created on the jumphost
adminUsername: <username e.g. ktenzer>
## user pwd for jump host
## Password for the jump host
adminPassword: <password>
##### Public key for jump host
### Access to environment only allowed through jumphost
sshkey: <ssh key e.g. cat /home/ktenzer/.ssh/id_rsa.pub>

# see https://azure.microsoft.com/en-us/documentation/articles/cloud-services-sizes-specs/
### Size for the master
master_vmSize: Standard_DS3_v2
#master_vmSize: Standard_D2_v2
#master_vmSize: Standard_D1_v2

### Size for the nodes
node_vmSize: Standard_DS3_v2
#node_vmSize: Standard_D2_v2
#node_vmSize: Standard_D1_v2

#### Region to deploy in
region: westeurope

## docker info
docker_storage_device: /dev/sdc
create_vgname: docker_vg
filesystem: 'xfs'
create_lvsize: '80%FREE'
#create_lvsize: '2g'

#### subscription information
rh_subscription_user: <Red Hat Subscription User>
rh_subscription_pass: <Red Hat Subscription Password>
openshift_pool_id: <Red Hat Subscription Pool Id>

########### list of node ###########
### Warning, you currently cannot create more infra nodes ####
### this will change in the future
### You can add as many nodes as you want
#####################################
jumphost:
 jumphost1:
 name: jumphost1
 tags:
 region: westeurope
 zone: jumphost
 stage: jumphost

masters:
 master1:
 name: master1
 tags:
 region: westeurope
 zone: infra
 stage: none
 master2:
 name: master2
 tags:
 region: westeurope
 zone: infra
 stage: none
 master3:
 name: master3
 tags:
 region: westeurope
 zone: infra
 stage: none

infranodes:
 infranode1:
 name: infranode1
 tags:
 region: westeurope
 zone: infra
 stage: dev
nodes:
 node1:
 name: node1
 tags:
 region: westeurope
 zone: app
 stage: dev
 node2:
 name: node2
 tags:
 region: westeurope
 zone: app
 stage: dev

Run Ansible Playbook

# ansible-playbook --forks=50 -i inventory.azure playbooks/setup_multimaster.new.yml

Connect to OpenShift environment

In order to connect to OpenShift environment you need to access jump box. The public IP of jumpbox will be set during playbook run, simply look at outputs to get the public IP for jumpbox.

Connect to jumphost

# ssh -i /home/ktenzer/.ssh/id_rsa.pub ktenzer@<jumphost public IP>

Connect to master1

There are three masters and you can connect and manage environment from any of them.

[ktenzer@jumphost1 ~]$ ssh master1

Login as built-in system:admin user

[ktenzer@master1 ~]$ oc login -u system:admin

List OpenShift nodes

[ktenzer@master1 ~]$ oc get nodes
NAME                                                       STATUS AGE
infranode1.KgsZ98734738nshjdsj2.ax.internal.cloudapp.net   Ready  34d
master1.KgsZ98734738nshjdsj2.ax.internal.cloudapp.net      Ready 34d
master2.KgsZ98734738nshjdsj2.ax.internal.cloudapp.net      Ready 34d
master3.KgsZ98734738nshjdsj2.ax.internal.cloudapp.net      Ready 34d
node1.KgsZ98734738nshjdsj2.ax.internal.cloudapp.net        Ready 34d
node2.KgsZ98734738nshjdsj2.ax.internal.cloudapp.net        Ready 34d

The deployment defaults to a highly available three master node OpenShift cluster. It also deploys three nodes, one for infrastructure and other two for applications. Ideally you would want a second infrastructure node so infrastructure services such as routing, image registry, logging and metrics are also highly available. Adding additional nodes is simply a matter of updating the playbook and re-running it. In addition changing OpenShift configuration is also just matter of updating playbook and re-running it.

In order to access the UI get the public URL from the master configuration file. When accessing the public URL traffic is balanced across all three OpenShift master servers.

[root@master1 ~]# cat /etc/origin/master/master-config.yaml |grep masterPublicURL 
masterPublicURL: https://master-<Resource Group>.westeurope.cloudapp.azure.com:8443

Azure AD Authentication

OpenShift like most platforms has two layers that govern access to the environment. Authentication grants a user access to the platform and authorization gives a user privileges within the environment. OpenShift supports many authentication providers. In the case of Azure AD openId is used for authentication and OpenShift offloads authentication entirely to Azure AD. Any user permitted to access the OpenShift authentication application in Azure AD is permitted to login to the OpenShift environment. Once a user in Azure AD authenticates to OpenShift, privileges can be given at project level to allow the user access within the environment. We will first configure Azure AD by setting up an authentication application and then allowing access within Azure. Once that is configured we will go through steps to allow OpenShift to use the Azure AD authentication application. Finally we will show how user management works within OpenShift when using Azure AD as authentication provider.

Configure Azure AD Application for OpenShift

The following steps describe how to create an Azure WebApp as your authentication barrier, before accessing the OpenShift Admin portal.

Since you already have installed OpenShift into Azure the normal disclaimers about getting a subscription do not apply here. In case you are just reading through and want to do more in Azure this is the link: http://www.Azure.com

Create Application

To create the application, log in to the Azure portal https://portal.azure.com and click on the plus symbol (1), then on Web + mobile (2) and finally on Web App (3).

azure_1

This will open another part on the screen that needs to be filled with further details about the new Web App. These parts are called blades and are used in the Azure portal to present details and configuration options. The first box takes (1) the application name that will become part of the DNS Name of the application. It needs to be unique within the namespace of azurewebsites.net. Your subscription in the following dropdown should be pre-filled. The next option, Resource Group, should be set to“Use existing” from the dropdown (2) and should point to the resource group where your OpenShift installation resides. Special attention is needed for the App Service plan and its location (3). The default location may need adjustment. Click on the tile to configure your service plan. After that click on create at the bottom of the blade to create your Web App. You may want to tick the checkbox “Pin to dashboard” for easier access later.

azure_2

Enable Authentication for Application

When your application has been deployed click on it to access its blade and continue with the configuration by clicking on Authentication/Authorization in the settings.

azure_3

The right side of the blade will change and you can click on the button for App Service Authentication (1) to turn it on. The dropdown for the Authentication providers should show “Log in with Azure Active Directory” (2). Click on the tile below (3) to configure the Azure Active Directory details.

azure_4

In the Azure Active Directory Settings dialog select the Management mode option (1) “Express”. The Current Active Directory (2) should read your own Azure Active Directory name. In the lower part of the dialog the button “Create New AD App” is pre-selected and you can provide an additional application name under “Create App”. By default it can be the same as the Web App name. After that click on OK at the bottom of the blade.

azure_5

azure_6

Don’t forget to click on Save at the top of the blade.

azure_7

Now that the Web App part is finished you can close all the blades and return to the dashboard. The next steps are done in the new Azure Active Directory blade.

azure_8

Configure Azure AD for Application

When you click on it the new blade should open with your existing Azure Active Directory details and the tile App registrations should read “1”. Click on that tile to continue.

azure_9

Click on Endpoints in order to retrieve the Azure Tenant Id. It is the number after the https://login.windows.net part. Take note of the tenant id as it is needed in the OpenShift configuration. Click on the Application you configured in the step before to access its details and configuration options.

azure_10

In the settings blade of your app click on Properties (1) to access the applications Application ID. You need this information as well to configure OpenShift to make use of the Azure AD authentication.

azure_11

azure_12

Close the Properties blade and click on Keys (2) in the Settings blade to create an individual access key. This key is also needed for the OpenShift configuration. Add a key description (1), select a duration of one or two years or even unlimited live time under (2) and click on save. Pay special attention to the warning displayed about the key that is now shown in the value (4). Copy that key and verify that it is stored properly somewhere else before leaving that blade!

azure_13

azure_14

In general, when you add the gathered information into OpenShift, all access to the URL of the installation will now be verified through Azure Active Directory authentication. In case you want to limit the usage to a certain group of users or even individual users you can configure this also but currently only in the Classic Azure portal under https://manage.windowsazure.com.

You may have to sign out and sign in again to access the portal. Once you have signed in, scroll to the Azure Active directory icon and click on the name of your Azure AD to access the dashboard.

azure_15

Enable User Access to Applications Azure AD

Click on APPLICATIONS (1), search (2) for your application in case there are more than one and click on your applications name (3) to go to its properties.

azure_16

In the properties click on “CONFIGURE” and toggle the button “USER ASSIGNMENT REQUIRED TO ACCESS APP” to “YES” and click on “SAVE” at the bottom of the page.

azure_17

azure_18

Now that the application is toggled users and/or groups need to be assigned to it. You configure that under “USERS AND GROUPS” on the same page.

azure_19

Select one of your users, click on assign at the bottom of the page and confirm the dialog. Verify the access by opening a new browser window in private mode and navigate to the OpenShift URL. You should be prompted with an authentication dialog from your Azure AD.

azure_20

azure_21

azure_22

Configure OpenShift for Azure AD Authentication

OpenShift supports many identity providers. In this case we are using OpenId to authenticate users against Azure AD. In order to configure Azure AD authentication in OpenShift the following information from Azure AD is required:

  • Tenant Id – The tenant id from any of the endpoint URLs
  • Client Id – The Application Id from Azure AD Application
  • Client Secret Key – The key that was created for the Application in Azure AD

Configure OpenId Provider

This information should have been captured from the steps above. On all three OpenShift masters edit the master configuration and add openId provider.

[root@master1 ~]# vi /etc/origin/master/master-config.yaml
identityProviders:
 - name: openId
 challenge: false
 login: true
 mappingMethod: claim
 provider:
 apiVersion: v1
 kind: OpenIDIdentityProvider
 clientID: <client id>
 clientSecret: <client secret key>
 claims:
 id:
 - sub
 preferredUsername:
 - preferred_username
 name:
 - name
 email:
 - email
 urls:
 authorize: https://login.microsoftonline.com/<tenant id>/oauth2/authorize
 token: https://login.microsoftonline.com/<tenant id>/oauth2/token

Restart OpenShift on all masters.

[root@master1 ~]# systemctl restart atomic-openshift-master-api
[root@master1 ~]# systemctl restart atomic-openshift-master-controllers
[root@master2 ~]# systemctl restart atomic-openshift-master-api
[root@master2 ~]# systemctl restart atomic-openshift-master-controllers
[root@master3 ~]# systemctl restart atomic-openshift-master-api
[root@master3 ~]# systemctl restart atomic-openshift-master-controllers

Login as Azure AD user

Using the GUI

Verify the access by opening a new browser window in private mode and navigate to the OpenShift URL https://master-Resource-Group.westeurope.cloudapp.azure.com:8443. You should be prompted with an authentication dialog of your Azure AD.

Using the CLI

[root@master1 ~]# oc login -u keith.tenzer@mydomain.onmicrosoft.com -n default
Login failed (401 Unauthorized)
You must obtain an API token by visiting https://master-Resource_Group.westeurope.cloudapp.azure.com:8443/oauth/token/request

[root@master1 ~]# oc login --token=0h9tKLlTicyAr5dYIgW5xiejiMVHIvltEuX6LLVW8CY --server=https://master-Resource-Group.westeurope.cloudapp.azure.com:8443
 Logged into "https://master-<resource group>.westeurope.cloudapp.azure.com:8443" as "G374shPb01h2jwssdjhGhghsghGHJSK" using the token provided.

Notice that the user is actually known to OpenShift is “G374shPb01h2jwssdjhGhghsghGHJSK” not  keith.tenzer@mydomain.onmicrosoft.com

List OpenShift users

Once Azure AD users login they are automatically added to OpenShift.

[root@master ~]# oc get users
 NAME                                        UID                                   FULL NAME    IDENTITIES
 G374shPb01h2jwssdjhGhghsghGHJSK             1245fec9f-598f-1656-12as-123g4g936742 Keith Tenzer openId:G374shPb01h2jwssdjhGhghsghGHJSK

List OpenShift identities

[root@master ~]# oc get identity
NAME                                               IDP    NAME                                       IDP USER NAME                   USER NAME                                   USER UID
openId:G374shPb01h2jwssdjhGhghsghGHJSK             openId FGnyqZlyGFzNCI99feUypSdEoNapBAJ03tg5MWlDSZ G374shPb01h2jwssdjhGhghsghGHJSK 1245fec9f-598f-1656-12as-123g4g936742

Give Azure AD user cluster-admin permission

[root@master ~]# oadm policy add-cluster-role-to-user cluster-admin "G374shPb01h2jwssdjhGhghsghGHJSK"

Give user permission to project

[root@master ~]# oadm policy add-role-to-user admin "G374shPb01h2jwssdjhGhghsghGHJSK" -n test

Remove Azure AD user from OpenShift

Delete user

[root@master ~]# oc delete user G374shPb01h2jwssdjhGhghsghGHJSK

Delete identity

[root@master ~]# oc delete identity openId:G374shPb01h2jwssdjhGhghsghGHJSK

Summary

In this article we have detailed how to standup a production-ready OpenShift environment automatically, before your coffee gets cold in the Azure Cloud. We have seen how to integrate OpenShift into Azure AD for authentication. There is no doubt, container platforms are the future for applications. Container technology enables application to be developed and deployed at much faster speeds, greatly improving a organizations ability to innovate and connect to it’s market segment. Many enterprises are interested in getting their hands on these technologies and evaluating them rather quickly. There is no faster way then standing up OpenShift in the Azure cloud! We hope you found this article interesting and look forward to your feedback.

Happy OpenShifting in the Azure Cloud!

(c) 2016 Keith Tenzer and Rolf Masuch

3 thoughts on “Enterprise Container Platform in the Cloud: OpenShift on Azure secured by Azure AD

  1. Pingback: Enterprise Container Platform in the Cloud: OpenShift on Azure secured by Azure AD | asksven

  2. Hi Keith, great blog… I have a questions.

    I notice this playbook does not exist anymore in the github location, It is now looks a bit different

    ansible-playbook –forks=50 -i inventory.azure playbooks/setup_multimaster.new.yml

    [root@osboxes playbooks]# ls
    init.yml postinstall.yml setupeverything.retry setup_multimaster.new.retry
    install_openshift.yml provision_infrastructure.yml setupeverything.yml start_all.yml
    perform_prereqs.yml roles setup_multimaster.avail.yml stop_all.yml
    [root@osboxes playbooks]#

    https://github.com/ivanthelad/ansible-azure

    The instruction on the github page is not ver details (like your’s), they talk about running the following

    ansible-playbook –forks=50 -i inventory playbooks/setupeverything.yml

    Just wondering if ther are any importaint updates I am missing or it you have a fork of the github page when you build this blog – looks like it received some updates since you built the environment.

    Thanks
    Kyle –

    Like

    • Hi Kyle,

      Yes the playbooks have been updated and changed since writing of blog. The version we worked off for example in blog is based off following fork: https://github.com/j1cken/ansible-azure. You can use this but I recommend using the new setupeverything.yml. The playbooks have also been updated to rollout latest OpenShift which is version 3.3. I haven’t yet tried the new playbooks but if something doesnt work you can open github ticket or let me know?

      Regards,

      Keith

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s