Building Custom Dashboards in OpenStack Horizon

Overview

Horizon is an OpenStack project responsible for providing a dashboard. It brings together all OpenStack projects in a single-pane-of-glass. The below diagram illustrates the connectivity between the Horizon dashboard and other OpenStack services.

Administrators typically don't like to have many different management interfaces and Horizon provides a framework for extending its dashboard services. By building custom dashboards it is possible to seamlessly integrate external components or services with OpenStack.

I have been working on integrating a powerful automation framework called Integra in OpenStack Horizon to allow tighter coupling of OpenStack and enterprise infrastructure. Integra takes automation to a new level by providing a powerful workflow engine that consumes provider exposed actions and allows users to automate without creating any technical debt. If you are interested in Integra you can read more about it at http://integra.emitrom.com.

Horizon Components

Horizon is built on Django which is a web application framework in Python. Django's primary goal is to ease creation of complex database-driven websites. Django emphasizes reusability and pluggability . Before getting into more detail about the code it is important to understand some basic terminology within the Horizon framework.

Terminology

Dashboard - This is the top level UI component, dashboards contains panel groups and panels. They are configured using the dasboard.py file.

Panel Groups - In Horizon panel groups organize similar panels together and provide a top-level drop-down. Panel groups are configured in the dashboard.py file.

Panel - The main UI component, each panel has its own directory and standardized directory structure. Panels are configured in the dashboard/panel/panel.py file.

Tab Groups - A tab group contains one or more tabs. A tab group can be configured per panel using the tabs.py file.

Tabs - Tabs are units within a tab group. It represents one view of the data.

Workflows - A workflow is a series of steps that allow for collecting user inputs. Workflows are created under dashboard/panel/workflows/workflow.py.

Workflow Steps - A workflow consists of one or more steps. A step is a wrapper around an action that understands its context within a workflow. Using workflows and steps we can build multiple input forms that guide a user through a complex configuration process.

Actions - An action allows us to spawn a workflow step. Actions are typically called from within a data table. Two of the most common actions are the DeleteAction and LinkAction.

Tables - Horizon includes a componetized API for dynamically creating tables in the UI. Every table renders correctly and consistently. Data tables are used for displaying information to the user. Tables are configured per panel in the tables.py file.

URLs - In Horizon URLs are needed to track context. At minimum a URL is required to display the main view but any LinkAction or actions that leave the main view will also require a URL. URLs are configured in the urls.py file.

Views - A view displays a data table and encompasses the main panel frame. Views are configured per panel in the views.py file.

Horizon Dashboard Directory Structure

Horizon requires a standard directory structure and strict file naming conventions. Below is an example of the directory structure for a dashboard called Integra that has five panels (actions, jobs, providers, workflows and schedules). As you can see each panel has its own sub-directory.

Below is an example of the directory structure for a panel called providers that belongs to dashboard Integra.

Note: the static and templates directories are always the same you just need to change name of the directory e.g. "providers" and update path in the _scripts.html, base.html as well as index.html. The __init__.py and models.py are never changed, just leave them as is.

Setup Development Environment

Since coding in "vim" is not very fun with Python, it is important to setup an IDE. In addition leveraging devstack provides an OpenStack development environment for Horizon that makes testing Horizon code much simpler. In fact Horizon provides some tools that not only help with mundane tasks but also provide a lightweight Django server for testing. You can use either Fedora or Ubuntu for your development environment, I went with Fedora.

• Install Java 7

  #yum install -y openjdk-7-jre

Java is required to run PyCharm Python IDE

• Download and install PyCharm

• https://www.jetbrains.com/pycharm/

Note: PyCharm is a good IDE but feel free to use a different IDE if you desire.

• Install git

  #yum install -y git

• Clone the devStack git repository

  #git clone https://github.com/openstack-dev/devstack.git

• Run stack.sh

  #cd devstack;./stack.sh

You will be prompted for several passwords. To make things easy just use the same password for each component. The installation will take 5 - 10 minutes and when it completes you should see the below message.

• Horizon is now available at http://192.168.2.211/
  Keystone is serving at http://192.168.2.211:5000/v2.0/
  Examples on using novaclient command line is in exercise.sh
  The default users are: admin and demo
  The password: integra
  This is your host ip: 192.168.2.211
  2015-02-14 18:06:31.434 | stack.sh completed in 387 seconds.

Note: each time you want to shutdown devstack you run unstack.sh and each time you want to start devstack stack.sh.

Horizon Development Tool

One of the great things about devstack is that it includes an important development tool for Horizon. The run_tests.sh script located under /opt/stack/horizon starts the development server and creates default directory structure for a dashboard/panel.

• Running the development server

  #./run_tests.sh --runserver 0.0.0.0:8877

• Creating default dashboard and panel directory structure

  #mkdir openstack_dashboard/dashboards/mydashboard

  #./run_tests.sh -m startdash mydashboard --target openstack_dashboard/dashboards/mydashboard

  #mkdir openstack_dashboard/dashboards/mydashboard/mypanel

  #./run_tests.sh -m startpanel mypanel --dashboard=openstack_dashboard.dashboards.mydashboard --target=openstack_dashboard/dashboards/mydashboard/mypanel

Horizon Start Scripts

Dashboards are loaded through start scripts located under the horizon/openstack_dashboard/enabled directory. In this case I created a _50_integra.py. The number has to do with the order in which dashboards are loaded and rendered. It is similar to the pre-systemd concept of init scripts.

• #vi /opt/stack/horizon/openstack_dashboard/enabled/_50_integra.py

• DASHBOARD = 'integra'

• DISABLED = False

• ADD_INSTALLED_APPS = [ 'openstack_dashboard.dashboards.integra', ]

Code Examples

At this point let us dissect the code behind the dashboard itself and one of the dashboard panels. The code examples are derived from a project (Integra OpenStack UI ) that I am currently working on and are available in Github. This code is changing and influx so you might want to fork the repository from a known state.

Dashboard

Below is an example of a dashboard called Integra that contains four panels (Providers, Workflows, Schedules and Jobs).

[code language="python"]
from django.utils.translation import ugettext_lazy as _

import horizon

class Providers(horizon.PanelGroup):
slug = "providers"
name = _("Providers")
panels = ('providers',)

class Workflows(horizon.PanelGroup):
slug = "workflows"
name = _("Workflows")
panels = ('workflows',)

class Schedules(horizon.PanelGroup):
slug = "schedules"
name = _("Schedules")
panels = ('schedules',)

class Jobs(horizon.PanelGroup):
slug = "jobs"
name = _("Jobs")
panels = ('jobs',)

class Integra(horizon.Dashboard):
name = _("Integra")
slug = "integra"
panels = (Providers, Workflows, Schedules, Jobs)
default_panel = 'providers'

horizon.register(Integra)
[/code]

The code is pretty straight forward. You have panel groups, panels and the horizon dashboard. Panels are organized under panel groups and then attached to the dashboard.

The above code will create the following dashboard and panel structure in Horizon. The providers table is generated from code we will discuss next.

Panel

Now lets dive into the panel "providers" that we have displayed above. In this case both the panel group and panel itself have the same name, but they don't have to and you can also of course have many panels.

The providers panel renders a data table from a REST API endpoint. If we look closely at the image above we can not only see a list of providers from Integra but we can take actions such as deleting a provider or adding a new provider.

panel.py

The panel.py defines the panel and registers it with the dashboard, in this case Integra.

[code language="python"]
from django.utils.translation import ugettext_lazy as _

import horizon
from openstack_dashboard.dashboards.integra import dashboard

class Providers(horizon.Panel):
name = _("Providers")
slug = "providers"

dashboard.Integra.register(Providers)
[/code]

views.py

The views.py is responsible for dynamically rendering the panel frame. It is also responsible for rendering any actions that spawn a new frame. In this case we have the default view ProvidersIndexView that loads a table ProviderTable. The table then is defined in the tables.py.

We also have an action in order to add a new provider that spawns a new frame. When the view AddProviderView is invoked, a workflow instead of a table class will be called. The workflow is defined under integra/providers/workflows/add_provider.py.

[code language="python"]
from horizon import exceptions, tables, workflows, forms, tabs

from openstack_dashboard.dashboards.integra.providers.tables import ProviderTable
from openstack_dashboard.dashboards.integra.providers import utils
from openstack_dashboard.dashboards.integra.providers.workflows.add_provider import AddProvider

class ProvidersIndexView(tables.DataTableView):
table_class = ProviderTable
template_name = 'integra/providers/index.html'

def get_data(self):
return utils.getProviders(self)

class AddProviderView(workflows.WorkflowView):
workflow_class = AddProvider

def get_initial(self):
initial = super(AddProviderView, self).get_initial()
return initial
[/code]

urls.py

The urls.py defines context URLs. A URL is required for the main panel frame and any new frames that are launching workflows. In our case we have two, the INDEX AND ADD_PROVIDER URLs. Notice that each URL is correlated with a view that is defined in the views.py.

[code language="python"]
from django.conf.urls import patterns, url
from openstack_dashboard.dashboards.integra.providers import views

INDEX_URL = r'^$'
ADD_PROVIDER_URL = r'^add'

urlpatterns = patterns('openstack_dashboard.dashboards.integra.providers.views',
url(INDEX_URL, views.ProvidersIndexView.as_view(), name='index'),
url(ADD_PROVIDR_URL, views.AddProviderView.as_view(), name='add'),
)
[/code]

tables.py

The tables.py is responsible for the data table and providing user outputs. Our table displays Integra providers and allows for a couple of actions. It lets us add and delete a provider. It also lets us filter the provider list that is displayed.

The AddTableData and DeleteTableData classes are both LinkActions. The AddTableData will launch a workflow that gathers user inputs. This is how we can add a new provider.

The DeleteTableData class removes a provider from the provider table. Here we use the DeleteAction and call a method in the utils class. This method in turn makes a REST call to Integra in order to delete the specified provider.

At the bottom the table structure is created and the actions are embedded into the providers table. The meta class is a special inner-class for Django data tables that allow us to configure various table options. Finally notice how the filter is added to the table, this is pretty standard and found in many places within Horizon.

[code language="python"]
from django.utils.translation import ugettext_lazy as _

from horizon import tables

from openstack_dashboard.dashboards.integra.providers import utils

class AddTableData(tables.LinkAction):
name = "add"
verbose_name = _("Add Provider")
url = "horizon:integra:providers:add"
classes = ("btn-launch", "ajax-modal")

class DeleteTableData(tables.DeleteAction):
data_type_singular = _("Provider")
data_type_plural = _("Providers")

def delete(self, request, obj_id):
utils.deleteProvider(self, obj_id)

class FilterAction(tables.FilterAction):
def filter(self, table, providers, filter_string):
filterString = filter_string.lower()
return [provider for provider in providers
if filterString in provider.title.lower()]

class UpdateRow(tables.Row):
ajax = True

def get_data(self, request, post_id):
pass

class ProviderTable(tables.DataTable):
id = tables.Column("id",
verbose_name=_("Id"))

name = tables.Column("name",
verbose_name=_("Name"))

description = tables.Column("description",
verbose_name=_("Description"))

hostname = tables.Column("hostname",
verbose_name=_("Hostname"))

port = tables.Column("port",
verbose_name=_("Port"))

timeout = tables.Column("timeout",
verbose_name=_("Timeout"))

secured = tables.Column("secured",
verbose_name=_("Secured"))

class Meta:
name = "integra"
verbose_name = _("Providers")
row_class = UpdateRow
table_actions = (AddTableData,
FilterAction)
row_actions = (DeleteTableData,)
[/code]

utils.py

The utils.py is a utility class. You can call it whatever you want, it is not required but in this case it is nice to separate the Integra REST calls from the rest of our Horizon application.

Three methods are defined in order to delete a provider, add a provider and get a list of all providers. We have already talked about adding and deleting a provider. The getProviders method returns a list of providers from Integra through the REST API. We have created a Provider model class that understands the structure of a provider object. One really nice thing about Python is that it natively handles JSON marshaling and since Integra returns JSON things are in this case quite simple.

[code language="python"]
import traceback
import time
from time import mktime
from datetime import datetime
from requests.auth import HTTPBasicAuth

from django.template.defaultfilters import register
from django.utils.translation import ugettext_lazy as _
import requests

from horizon import exceptions

requests.packages.urllib3.disable_warnings()

integra_url = "https://localhost:8443/rest"
json_headers = {'Accept': 'application/json'}

class Provider:
"""
Provider data
"""

def __init__(self, id, name, description, hostname, port, timeout, secured):
self.id = id
self.name = name
self.description = description
self.hostname = hostname
self.port = port
self.timeout = timeout
self.secured = secured

def getProviders(self):
try:
r = requests.get(integra_url + "/providers", verify=False, auth=HTTPBasicAuth('admin', 'integra'), headers=json_headers)

providers = []
for provider in r.json()['providers']:
providers.append(Provider(provider[u'id'], provider[u'name'], provider[u'description'], provider[u'hostname'], provider[u'port'], provider[u'timeout'], provider[u'secured']))

return providers

except:
exceptions.handle(self.request,
_('Unable to get providers'))
return []

# request - horizon environment settings
# context - user inputs from form
def addProvider(self, request, context):
try:

name = context.get('name')
description = context.get('description')
hostname = context.get('hostname')
port = context.get('port')
timeout = context.get('timeout')
secured = context.get('secured')

payload = {'name': name, 'description': description, 'hostname': hostname, 'port': port, 'timeout': timeout, 'secured': secured}
requests.post(integra_url + "/providers", json=payload, verify=False, auth=HTTPBasicAuth('admin', 'integra'), headers=json_headers)

except:
print "Exception inside utils.addProvider"
print traceback.format_exc()
exceptions.handle(self.request,
_('Unable to add provider'))
return []

# id is required for table
def deleteProvider(self, id):
try:

requests.delete(integra_url + "/providers/" + id, verify=False, auth=HTTPBasicAuth('admin', 'integra'), headers=json_headers)

except:
print "Exception inside utils.deleteProvider"
print traceback.format_exc()
exceptions.handle(self.request,
_('Unable to delete provider'))
return False
[/code]

add_provider.py

The add_provider.py is a workflow that contains a single workflow step. This is the workflow that is called when we add a new provider. It is responsible for getting user input.

The AddProvider class executes when the workflow is called. It calls the SetAddProviderDetails class which then calls the SetAddProviderDetailsAction class and returns the user inputs within the context object.

[code language="python"]
import traceback

from horizon import workflows, forms, exceptions
from django.utils.translation import ugettext_lazy as _

from openstack_dashboard.dashboards.integra.providers import utils

class SetAddProviderDetailsAction(workflows.Action):

name = forms.CharField(
label=_("Name"),
required=True,
max_length=80,
help_text=_("Name"))

description = forms.CharField(
label=_("Description"),
required=True,
max_length=120,
help_text=_("Description"))

hostname = forms.CharField(
label=_("Hostname"),
required=True,
max_length=120,
help_text=_("Hostname"))

port = forms.IntegerField(
label=_("Port"),
required=True,
min_value=1,
max_value=65535,
help_text=_("Port"))

timeout = forms.IntegerField(
label=_("Timeout"),
required=True,
min_value=1,
max_value=100000,
help_text=_("Timeout"))

secured = forms.BooleanField(
label=_("Secured"),
required=False,
help_text=_("Secured"))

class Meta:
name = _("Details")

def __init__(self, request, context, *args, **kwargs):
self.request = request
self.context = context
super(SetProviderDetailsAction, self).__init__(
request, context, *args, **kwargs)

class SetAddProviderDetails(workflows.Step):
action_class = SetAddProviderDetailsAction
contributes = ("name", "description", "hostname", "port", "timeout", "secured")

def contribute(self, data, context):
if data:
context['name'] = data.get("name", "")
context['description'] = data.get("description", "")
context['hostname'] = data.get("hostname", "")
context['port'] = data.get("port", "")
context['timeout'] = data.get("timeout", "")
context['secured'] = data.get("secured", "")
return context

class AddProvider(workflows.Workflow):
slug = "add"
name = _("Add")
finalize_button_name = _("Add")
success_message = _('Added provider "%s".')
failure_message = _('Unable to add provider "%s".')
success_url = "horizon:integra:providers:index"
failure_url = "horizon:integra:providers:index"
default_steps = (SetAddProviderDetails,)

def format_status_message(self, message):
return message % self.context.get('name', 'unknown provider')

def handle(self, request, context):
try:
utils.addProvider(self, request, context)
return True
except Exception:
print traceback.format_exc()
exceptions.handle(request, _("Unable to add provider"))
return False
[/code]

Below we can see the above code in action.

Dynamic Inputs

So far we have seen how we can build static input forms using CharField, IntegerChield or BooleanField. Next lets look at how to create dynamic inputs within workflows using ChoiceField. In our utils.py we already have an method getProviders that returns a list of provider objects. In addition we will add a new method for returning a list of provider actions.

utils.py

[code language="python"]
def getProviders(self):
try:

r = requests.get(integra_url + "/providers", verify=False, auth=HTTPBasicAuth('admin', 'integra'), headers=json_headers)

providers = []
for provider in r.json()['providers']:
providers.append(ProviderAction(provider[u'id'], provider[u'name'], provider[u'description']))

return providers

except:
exceptions.handle(self.request,
_('Unable to retrieve list of posts.'))
return []

def getProviderActions(self):
try:

r = requests.get(integra_url + "/provider_actions/" + id, verify=False, auth=HTTPBasicAuth('admin', 'integra'), headers=json_headers)

providerActions = []
for providerAction in r.json()['providerActions']:
providerActions.append(ProviderAction(providerAction[u'id'], providerAction[u'name'], providerAction[u'description']))

return providerActions

except:
exceptions.handle(self.request,
_('Unable to retrieve list of posts.'))
return []
[/code]

In our workflow action we can get a list of provider action objects and display them to the user using a ChoiceField. Notice the ChoiceField requires the id and name. Only the name is displayed but the id is required for mapping purposes.

add_workflow_action.py

[code language="python"]

class SetAddDetailsAction(workflows.Action):

providerActionsChoices = [(providerAction.id, providerAction.name) for providerAction in providerActions]
providerChoices = [(provider.id, provider.name) for provider in providers]

name = forms.CharField(
label=_("Name"),
required=True,
max_length=80,
help_text=_("Name"))

description = forms.CharField(
label=_("Description"),
required=True,
max_length=120,
help_text=_("Description"))

provider = forms.ChoiceField(
label=_("Providers"),
choices=providerChoices,
required=True,
help_text=_("Providers"))

action = forms.ChoiceField(
label=_("Provider Actions"),
choices=providerActionsChoices,
required=True,
help_text=_("Provider Actions"))
[/code]

Below we can see two dynamic fields being generated. Both populate the ChoiceField from dynamic data that is recieved from the Integra REST API.

Conclusion

OpenStack Horizon is a powerful web-framework built on Django that is easily extended and Integra is a powerful provider based automation framework. We have seen how easy it is to create our own Horizon dashboard and interface with services outside of OpenStack through RESTful APIs. The above examples can be followed to accomplish virtually anything. Horizon is a glimpse into the future of infrastructure single-pane-of-glass management. This has been something we have been promised for years from the proprietary vendors and only now with OpenStack Horizon do we have some real hope.

If you are working on Horizon dashboards or have feedback I would love to hear about it?

Happy Stacking!

(c) 2015 Keith Tenzer