In this article we will setup and configure an Ansible Tower cluster on Red Hat Enterprise Linux (RHEL). If you are interested in a single all-in-one deployment, I have already documented this here.
Ansible Tower clustering replaces the traditional active/passive with an active/active configuration. It provides not only HA but scalability as well. Ansible Tower has two critical components: Tower instances running API/Scheduler and the database. RabbitMQ is used for communication between the Tower instances.
Tower instances can be assigned to instance groups and those can be further assigned to resources (Organizations, Inventories and Job Templates). Jobs executed against a resource are assigned to instance group for execution. Isolated instance groups that overcome physical network boundaries are also possible. In this case a minimal Ansible configuration is used on isolated instances so that a Tower instance can execute playbooks in a network zone where there is no direct access to Ansible client nodes. Ansible Tower accesses isolated groups is via ssh.
Ansible Tower uses PostgreSQL for it's internal database. The options are to use a built-in database, installed via playbook during Tower installation or use external PostgreSQL database. The PostgreSQL database can be clustered in active/passive configuration. If this is desired then it is recommended to setup a standalone external PostgreSQL database.
If you are installing the database using the ansible playbook it will be installed in the default location, /var/lib/pgsql/. You need to make sure /var is backed by fast storage and also large enough.
A load balancer is recommended. You can either terminate SSL at the load balancer and go http from there or pass SSL through load balancer to Tower instances.
In this configuration we will setup three Ansible Tower instances and a single Tower database using the built-in installer (playbook). The environment will sized to handle 300 forks (total number of parallel processes that can be run).
Tip: Sizing guidelines suggest 1vCPU per 10 forks and 4GB RAM per 100 forks.
Tower Instance Sizing: 10 vCPUs, 4GB RAM, 25GB Local Storage (>= 750 IOPS)
Note: 10GB of 25GB Local Storage must be dedicated to /var
Tower Database Sizing: 10 vCPUs, 16GB RAM, 25GB Local Storage and 150+ Database Storage (>= 1000 IOPS)
Note: for a demo environment you can optionally exclude database storage.
Before starting you need to deploy four hosts or VM with RHEL 7.3 or 7.4. Once the hosts are configured and on the network, you can register them with subscription-manager and enable the required repositories. For Ansible Tower you need rhel-7-server-rpms, rhel-7-server-extras-rpms and EPEL.
Ports and instances used by Tower are as follows:
- 80, 443 (normal Tower ports)
- 22 (ssh)
- 5432 (database instance - if the database is installed on an external instance, needs to be opened to the tower instances)
- 4369, 25672 (ports specifically used by RabbitMQ to maintain a cluster, needs to be open between each instance)
- 15672 (if the RabbitMQ Management Interface is enabled, this port needs to be opened (optional))
# subscription-manager register
# subscription-manager list --available
# subscription-manager attach --pool=<pool id>
# subscription-manager repos --disable=*
# subscription-manager repos --enable=rhel-7-server-rpms \ --enable=rhel-7-server-extras-rpms
Install and configure EPEL repositories.
# yum install -y http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
# yum install -y ansible
Install Ansible Tower
Optional [On towerdb.lab]
Create LVM Volume and Mountpoint for Database.
Postgres writes in 8k pages so we will set blocksize to 8k.
# pvcreate /dev/sdb1 # vgcreate vg_postgres /dev/sdb1 # lvcreate -L 150G vg_postgres -n postgres_disk1 # mkfs.xfs -d su=8k /dev/vg_postgres/psotgres_disk1
Create Invetory File.
We will use ansible to allow our user (ktenzer) to sudo without password. This is needed for the installer. You can also of course use root to install Tower.
# vi inventory_tower_hosts [servers] tower1.lab tower2.lab tower3.lab towerdb.lab
Allow User Privileged Permissions.
# ansible -m lineinfile -a "dest=/etc/sudoers.d/ktenzer \ line='ktenzer ALL=(ALL) NOPASSWD: ALL' create=yes state=present" -u root \ --ask-pass -i ./inventory_tower_hosts servers
Setup ssh keys.
We need to create ssh key and copy it to all nodes for user who will install Tower (in this case ktenzer).
# su - ktenzer
ktenzer$ for p in tower1.lab tower2.lab tower3.lab towerdb.lab; \ do ssh-copy-id $p; done
ktenzer$ for p in tower1.lab tower2.lab tower3.lab towerdb.lab; \ do ssh $p; exit done
Download Ansible Tower.
[ktenzer@tower1 ~]$ curl -O http://releases.ansible.com/ansible-tower/setup/ansible-tower-setup-latest.tar.gz
Extract Tower tarball.
[ktenzer@tower1 ~]$ gunzip ansible-tower-setup-latest.tar.gz
[ktenzer@tower1 ~]$ tar xvf ansible-tower-setup-latest.tar
Configure Tower Installation Inventory File.
[ktenzer@tower1 ~]$ cd ansible-tower-setup-3.2.1/
[ktenzer@tower1 ~]$ vi inventory [tower] tower1.lab tower2.lab tower3.lab [database] towerdb.lab [all:vars] ansible_become=true admin_password='redhat01' pg_host='towerdb.lab' pg_port='5432' pg_database='tower' pg_username='tower' pg_password='redhat01' rabbitmq_port=5672 rabbitmq_vhost=tower rabbitmq_username=tower rabbitmq_password=redhat01 rabbitmq_cookie=rabbitmqcookie # Needs to be true for fqdns and ip addresses rabbitmq_use_long_name=true
[ktenzer@tower1 ~]$ ./setup.sh PLAY RECAP ***************************************************************************************** tower1.lab : ok=118 changed=54 unreachable=0 failed=0 tower2.lab : ok=112 changed=52 unreachable=0 failed=0 tower3.lab : ok=112 changed=52 unreachable=0 failed=0 towerdb.lab : ok=46 changed=18 unreachable=0 failed=0
Configure Ansible Tower
Ansible Tower Provides a RESTful API, CLI and UI. To connect to the UI simply open browser using https and point to your Ansible Tower IP or hostname.
https://<Ansible Tower IP or Hostname>
Login using the user you configured in the inventory file, in this case admin/redhat01.
Once you are logged in, you need to install Tower license. Tower license comes in a file, so simply browse to the file and accept the terms. If you don't have a license you can get a trial here.
Verify Tower instances under settings (next to the user, upper right).
In this article we explained a bit about Tower clustering and a general way to approach sizing. In addition we went through the easy installation steps to deploy a cluster with three Tower instances and a database. Regardless of if you are new to Ansible or experienced, Tower is definitely worth looking at as it provides a solid management platform for Ansible, making Ansible even easier and more powerful!
(c) 2017 Keith Tenzer