CI/CD with Ansible Tower and GitHub

Overview

Over the last few years CI/CD (Continuous Integration/Continuous Deployment) thanks to new technologies has become a lot easier. It should no longer be a major thorn in the side of developers. Many are moving to cloud platforms which has CI/CD built-in (Azure DevOps for example), others are using Kubernetes which clearly reduces a lot of the complexity around CI/CD. Still at many organizations I see Jenkins or other complex and often homegrown tooling. I certainly recognize this tooling was needed but in 2019 there are better, more streamlined options. Now I get it, our butler Jenkins has served us well, for many years, he has become part of our family. But just like the famous Butler, Alfred from Batman, he has gotten old and likely it is time to look into retirement.

In this article we will discuss and demonstrate how to use Ansible Tower and GitHub for CI/CD.

A video presentation and demonstration is available at following URL: https://youtu.be/lyk-CRVXs8I

Why Ansible Tower for CI/CD?

Most organizations are adopting an infrastructure-as-code approach, that is to say everyone, even operations are becoming developers or at least working and thinking like developers. The automation tool of choice or even standard if you will is clearly Ansible. However Ansible is just an automation runtime, it has no API, RBAC, Credentials and does not enable collaboration between teams. This is where Ansible Tower (or AWX the opensource project) shines. Here are some advantages Ansible Tower bring to CI/CD:

• Reuse infrastructure and operations playbooks in CI/CD
• Better collaboration between devops and infrastructure teams
• Simple CI/CD, just Ansible, no DML, no Groovy, no god knows what
• Ansible Tower is a enterprise product not a project like Jenkins
• Streamlined CI/CD instead of every devops team on their own
• Self organizing and governing because that is what Ansible does best

Now clearly Ansible Tower cannot do everything. It does not have for example a built-in CI/CD approval process and does not specifically focus on CI/CD. Instead Ansible Tower is an automation platform. It views the world through job templates (a playbook) and workflows (group of playbooks). In this case a workflow is a CI/CD pipeline with an API in front of it of course. Once you have your workflows or pipelines and an API in front of them you are 90% done. The rest is just how the process should flow and you might not even need any extra tooling to do that, certainly not complex tooling like Jenkins.

Ansible Tower CI/CD Architecture

The point of this article is to demonstrate that CI/CD can work very simply and with just Ansible Tower and GitHub. The diagram below illustrates what we will be building. This focuses only really on CI but CD would just be additional workflows.

The main purpose of CI is of course to protect the master branch so it always builds. The only way you do that of course is to check code into another branch (like a feature branch), test that code, review code and only merge to master when you are sure all tests have passed. The above architecture accomplished exactly that and does so with a very simplified approach that leverages Ansible Tower as our CI engine. We won't be going into the CD details in this article, but that would just be additional workflows to deploy artifacts generated by the CI process into dev->test->production. Using this architecture one could use GitHub releases to store artifacts. GitHub has ability to then trigger a webhook when latest release is updated which in turn could trigger an Ansible Tower CD workflow.

Ansible Tower Configuration

First we will want to configure a workflow in Ansible Tower that will be a CI/CD pipeline. I think it is good practice to have one workflow per pipeline. In this example we have a single workflow which handles CI.

Configure Credentials

We need machine credentials for Fedora since we will be accessing a fedora host to execute our build. In Ansible Tower credentials allow you to access systems without needing to connect to them directly, like you would with normal Ansible. Users don't even need to know credentials.

Configure Inventory

Inventories in Ansible Tower are generated dynamically. This is important because when we provision a VM we don't at the time know what it's IP will be. Of course after we provision we want to access the VM. The solution is to perform an inventory update after provisioning and then parameterize provisioning and execution playbooks so they create/use same hostnames.

Here we are using OpenStack but this could be any public or private cloud. For doing CI we created an inventory called development and this is what we will be later using in our job templates.

Configure Projects

Projects in Ansible Tower are simply SCM repositories where playbooks reside. In this case of course a GitHub repository. Here notice the developer playbooks to do the build are located with the source code where the provisioning playbooks are in a different repository. This is where we get collaboration between infrastructure operations teams and devops. Something that rarely existed in CI/CD.

Links to go-hello-world and paas-and-iaas repositories.

Configure Job Templates

For our CI pipeline we will need to run three playbooks. Each playbook is a job template. In Ansible Tower a job template is a playbook, it's inventory, survey, credentials and various other things that go way beyond plain old Ansible. The three job templates are one to provision a fedora instance (from infrastructure / operations team), a playbook to perform the go build / tests (from devops team) and a playbook to cleanup (from infrastructure / operations team).

Configure Ansible Tower Workflow

As mentioned a workflow will bring all the playbooks/job templates together. Here we configure a workflow called GO CI/CD Workflow. Notice the below workflow. We are using our three job templates provision fedora, run go build and remove instance. In addition the second step performs an inventory update for the development inventory. A workflow allows you to perform granular inventory updates wherever needed.

We aren't quite done, the last step is to add a survey to the Workflow. Each of the playbooks of course accept parameters. These need to be added in form of survey. A survey provides a user friendly interface for parameterizing workflows or job templates in Ansible Tower.

Ansible Tower CI/CD Setup

The first thing to do is setup tower-cli. This is a much better tool than curl to trigger Ansible Tower. From the webhook we will use tower-cli to update projects in Ansible Tower and launch workflows.

Install tower-cli

My fellow Colleague, Andreas Neeb created a role to install tower-cli.

https://github.com/andyneeb/ansible-demo-infra/tree/master/roles/tower-cli-setup

Simply create playbook that uses role on localhost or host in your inventory where you want to install tower-cli.

Fork go-hello-world

 

Configure GitHub Webhook Token

In order to update commit status in GitHub we need permissions and a webhook token.

In GitHub under your user->settings go to developer settings.

Select personal access tokens and create a new one.

Add a note and select repo:status. This is the minimum permission in GitHub needed to update a commit status. For CI/CD you need to be able in GitHub update commit status.

Configure GitHub Webhook

In GitHub go to the go-hello-world project. Under settings create a new webhook.

Make sure content-type is application/json and set a secret, this is a password that will be used to authenticate to the webhook listener we will deploy next.

Install Webhook Listener on Ansible Tower

The folks at Arctiq created a role and playbook for setting up a webhook listener on Ansible Tower.

https://github.com/ArctiqTeam/tower-webhook

Simply clone the repository, update the vars.yml  and hosts file according.

# vi hosts
[webhook]
webhookserver ansible_host=46.4.207.248 ansible_ssh_port=22

# vi vars.yml
---
projects:
- id: 7
name: go-hello-world

# export GH_SECRET=
# export TOWER_PASSWORD=
# ansible-playbook install.yml

The playbook will create a systemctl service called webhook so it can easily be started and stopped. The webhook will run on port 9000.

 Configure Webhook Listener

In order to use webhook for CI/CD some changes need to be made. Update the webhook.conf as follows.

# vi /etc/webhook/webhook.conf
[
  {
    "id": "go-hello-world",
    "execute-command": "/usr/local/bin/go-hello-world.sh",
    "pass-environment-to-command":
    [
      {
        "source": "payload",
        "name": "head_commit.id",
        "envname": "COMMIT_ID"
      },
      {
        "source": "payload",
        "name": "pusher.name",
        "envname": "PUSHER_NAME"
      },
      {
        "source": "payload",
        "name": "pusher.email",
        "envname": "PUSHER_EMAIL"
      },
      {
        "source": "payload",
        "name": "head_commit.id",
        "envname": "COMMIT_ID"
      },
      {
        "source": "payload",
        "name": "pusher.name",
        "envname": "PUSHER_NAME"
      },
      {
        "source": "payload",
        "name": "ref",
        "envname": "REF"
      }
    ],
    "trigger-rule":
    {
      "match":
      {
        "type": "payload-hash-sha1",
        "secret": "<password>",
        "parameter":
        {
          "source": "header",
          "name": "X-Hub-Signature"
        }
      }
    }
  }
]

Restart webhook

# systemctl restart webhook

Next update the shell script go-hello-world.sh. This will be triggered each time a push request is sent to GitHub. It will also receive environment parameters needed for CI from GitHub.

# vi /usr/local/bin/go-hello-world.sh
#!/bin/sh
printenv >/tmp/env.out
BRANCH=`echo $REF |sed -e 's/refs\/heads\///g'`

### Set Github commit to pending CI/CD
curl "https://api.GitHub.com/repos/ktenzer/go-hello-world/statuses/$COMMIT_ID?access_token=79dc2a78c4c009c3c08e6cbc8233559c535fc8dc" -H "Content-Type: application/json" -X POST -d "{\"state\": \"pending\",\"context\": \"continuous-integration/go-hello-world\", \"description\": \"Ansible_Tower\", \"target_url\": \"https://46.4.207.248\"}"

### Update Project in Tower
tower-cli project update 7 -v -h 46.4.207.248 --wait
if [ $? != 0 ]; then
        curl "https://api.GitHub.com/repos/ktenzer/go-hello-world/statuses/$COMMIT_ID?access_token=<token>" -H "Content-Type: application/json" -X POST -d "{\"state\": \"error\",\"context\": \"continuous-integration/go-hello-world\", \"description\": \"Ansible_Tower\", \"target_url\": \"https://46.4.207.248\"}"
        exit 1
fi

### Execute CI/CD Workflow in Tower
tower-cli workflow_job launch --workflow-job-template=18 -v -h 46.4.207.248 -e openstack_user=admin -e openstack_passwd=<password> -e openstack_auth_url=http://85.10.236.4:5000/v3 -e openstack_project=development -e instance_name=gobuilder -e instance_security_group=dev_base -e instance_image=fedora_28 -e instance_ssh_key_name=admin -e ssh_user=fedora -e instance_network=development -e instance_flavor=m1.small -e repo_url=https://github.com/ktenzer/go-hello-world.git -e repo_branch=$BRANCH -e app_path=hello/src -e app_name=hello --wait

### Update Github commit based on success or failure of CI/CD workflow
if [ $? != 0 ]; then
        curl "https://api.GitHub.com/repos/ktenzer/go-hello-world/statuses/$COMMIT_ID?access_token=<token>" -H "Content-Type: application/json" -X POST -d "{\"state\": \"error\",\"context\": \"continuous-integration/go-hello-world\", \"description\": \"Ansible_Tower\", \"target_url\": \"https://46.4.207.248\"}"
        exit 1

else
        curl "https://api.GitHub.com/repos/ktenzer/go-hello-world/statuses/$COMMIT_ID?access_token=<token>" -H "Content-Type: application/json" -X POST -d "{\"state\": \"success\",\"context\": \"continuous-integration/go-hello-world\", \"description\": \"Ansible_Tower\", \"target_url\": \"https://46.4.207.248\"}"
        exit 0
fi

Run CI/CD Pipeline using Ansible Tower

Clone go-hello-world repository

# git clone https://github.com/ktenzer/go-hello-world.git

Create feature branch

# git branch patch-1
# git checkout patch-1

Add broken code

# vi src/hello/main.go
...
func main() {
        <broken>
	router := NewRouter()
...

Commit broken code

# git commit -a -m "broken code"

# git push origin patch-1

Create pull request in GitHub

Under branches you will now see the patch-1 branch.

You can now create a pull request in GitHub.

Notice that CI/CD is in progress and the source is Ansible Tower. Of course the CI will fail since we broke the code. In Tower you can see looking at the go build job.

Looking at our pull request in GitHub it also now shows error.

Notice the CI/CD process spawned a VM to perform build and run tests.

In this case the instance is not removed since the test failed. The idea is likely the developer would want to fix it and since we are using Ansible, once a fix is committed we will start where we left off, saving potentially a lot of time.

Fix broken code

# vi src/hello/main.go
...
func main() {
	router := NewRouter()
...

Commit broken code

# git commit -a -m "fixed code"

# git push origin patch-1

This will start the CI/CD process again except this time of course our code builds and CI is successful.

Looking at GitHub we see the pull request has gone through CI and all tests are successful.

In Ansible Tower we also see the jobs completed.

At this point a developer could review the code and merge it. In GitHub we will now merge the pull request.

The CI process is started again this time for the master branch. Once master builds we have a release and can start the CD process. We would as mentioned above, push the release to GitHub releases and that would in turn trigger another workflow or several workflows to deploy the release in various dev, test and production environments. We will not be doing this but I think you get the picture.

Summary

In this article we discussed a simpler solution for CI/CD by using just Ansible Tower and GitHub. We saw the advantages of making Ansible reusability the cornerstone of CI/CD, not only in simplicity but bringing infrastructure/operations teams together with development. We went through the configuration and saw a demonstration of how the solution works. If you are interested further or have any questions/feedback please feel free to reach out.

Happy CI/CD’ing!

(c) 2019 Keith Tenzer