graphing openstack performace with graphite

By Bill Ward | April 9, 2017

This article will cover how to monitor your Openstack cluster’s performace using Graphite. Why would we want to do this? This gives us data that is easily read and interpreted so that we can make informed business decisions about our openstack cloud. This will be useful for decisions about when to add another compute node or if it is time to add more bandwidth to our networking.

Graphite is arguably the standard in graphing applications and we will use it to graph performance data for our openstack cluster. This post will not be going into detail about how graphite works and all the wonderful things you can do with it. If you like a good resource for learning more about Graphite then I highly suggest the below book “Monitoring with Graphite: Tracking Dynamic Host and Application Metrics at Scale” from Amazon:

Prerequisites

  • Ubuntu 14.04

Graphite Server

Our first task is going to be setting up our graphite server.

Install Graphite

The first thing we will do is install the graphite packages.

sudo apt update
sudo apt install graphite-web graphite-carbon

Select “No” when it asks you to purge the database.

Configure Database

Graphite uses postgres as a database so we need to install and configure it.

sudo apt install postgresql libpq-dev python-psycopg2

Now login to the postgres client and create the database and graphite database user. Make sure to change the password.

sudo -u postgres psql
CREATE USER graphite WITH PASSWORD 'change_me';
CREATE DATABASE graphite WITH OWNER graphite;
\q

Graphite Web Application

We need to change some settings in the web application settings file /etc/graphite/local_settings.py.

sudo vim /etc/graphite/local_settings.py

Look for the SECRET_KEY entry and change the salt value. I used openssl to generate a random hash:

ubuntu@graphing:~$ openssl rand -hex 10
a6a252fcb42f6d8ad21a

Which would give us:

SECRET_KEY = 'a6a252fcb42f6d8ad21a'

Next update the TIME_ZONE value to your correct timezone

TIME_ZONE = 'America/Chicago'

Make sure that USE_REMOTE_USER_AUTHENTICATION is set to True”

USE_REMOTE_USER_AUTHENTICATION = True

Finally, update the DATABASES section to look like this:

DATABASES = {
    'default': {
        'NAME': 'graphite',
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'USER': 'graphite',
        'PASSWORD': 'change_me',
        'HOST': '127.0.0.1',
        'PORT': ''
    }
}

Make sure that you update the password to the same password you entered earlier in the Postgres database.

Save the file and exit.

Synchronize the graphite database

Senc the database so that it has the correct structure and data to begin.

sudo graphite-manage syncdb

During the synchronize process you will be asked to create a superuser account. Enter any username and password you want for the new user. This will be the login information you will use to login to the graphite web interface at the end of this post.

Now that we have the application backend ready we can configure Carbon the storage backend for Graphite.

Configure Cargon

Configure Carbon to start on boot.

sudo vim /etc/default/graphite-carbon

Change CARBON_CACHE_ENABLED to true.

CARBON_CACHE_ENABLED=true

Save and exit.

Storage Schemas

Edit the storage schemas file.

sudo vim /etc/carbon/storage-schemas.conf

Add an entry for collectd data.

[collectd]
pattern = ^collectd.*
retentions = 10s:1d,1m:7d,10m:1y

Save and exit.

Storage Aggregation

Copy the example storage-aggregation.conf file to enable.

sudo cp /usr/share/doc/graphite-carbon/examples/storage-aggregation.conf.example /etc/carbon/storage-aggregation.conf

Start Carbon

Finally, we need to start the carbon service.

sudo systemctl start carbon-cache

That concludes installing and configuring carbon. Now we need to install and configure the web interface.

Configure the Graphite Web Interface

Install the packages

sudo apt install apache2 libapache2-mod-wsgi

Enable the web interface.

sudo a2dissite 000-default
sudo cp /usr/share/graphite-web/apache2-graphite.conf /etc/apache2/sites-available
sudo a2ensite apache2-graphite
sudo service apache2 reload

You can now access the web interface.

http://graphite_server_IP

Our Graphite server is now complete. We can now start configuring our Openstack nodes to send performance metrics to it.

Configuring Openstack Nodes

Perform the next steps on all of your Openstack nodes. This will send the performance data from those nodes to Graphite.

Install the packages

sudo apt update
sudo apt install collectd collectd-utils

Configure Collectd

Edit the configuration file.

sudo vim /etc/collectd/collectd.conf

Update the hostname to the hostname of the current node you are enabling. In this example, I am on my only controller node with a hostname of ‘controller’.

Hostname "controller.admintome.local"

Comment the line underneath for FQDNLookup.

Next we need to select the plugins that we will be using for performance monitoring. You can choose any combination of plugins you want just uncomment the ones you want and comment the ones you don’t. Just for the purposes of this tutorial make sure at least the plugins below are enabled.

LoadPlugin cpu
LoadPlugin df
LoadPlugin entropy
LoadPlugin interface
LoadPlugin load
LoadPlugin memory
LoadPlugin processes
LoadPlugin rrdtool
LoadPlugin users
LoadPlugin write_graphite

Next configure the plugins. Find each section below and make sure they are uncommented and change accordingly.

interface plugin

<Plugin interface>
    Interface "eth0"
    IgnoreSelected false
</Plugin>

Make sure it has the correct name for your provider network interface.

write_graphite plugin_

<Plugin write_graphite>
    <Node "graphing">
        Host "graphite_server_IP"
        Port "2003"
        Protocol "tcp"
        LogSendErrors true
        Prefix "collectd."
        StoreRates true
        AlwaysAppendDS false
        EscapeCharacter "_"
    </Node>
</Plugin>

Make sure you update the graphite_server_IP.

Restart the services.

sudo systemctl restart collectd

Repeat these steps for each of your Openstack nodes. In my case I only have two nodes: controller and compute.

Configure a Graphite Dashboard for Openstack

Now we have metrics being pumped into graphite and we need to have a nice and easy way to view this data. We will be creating a custom dashboard that will have all our metrics in one place.

Login with the username and password you configured during the database synchronization at the beginning of this post. In the upper-right click on the Dashboard link. The screen will be split horizontally. The top half allows us to select what metrics we want to graph. The bottom half shows the graphs we have selected. At the top select the following data which will add graphs for our controller node.

collectd.controller_admintome_local.memory.used
collectd.controller_admintome_local.cpu-0.cpu-systemct
collectd.controller_admintome_local.cpu-1.cpu-systemct
collectd.controller_admintome_local.cpu-2.cpu-systemct
collectd.controller_admintome_local.cpu-3.cpu-systemct
collectd.controller_admintome_local.entropy.entropy
collectd.controller_admintome_local.load.load.midterm

You can other metrics as you see fit. If you have a multicore processor (and I hope you do for your openstack nodes) then you will see a metric for each processor core. For these graphs just drag and drop each graph onto the first core’s graph. This will combine them all into one graph.

Repeat this for each one of your openstack nodes. Now save the dashboard by clicking Dashboard and selecting Save As.

saving graphite openstack dashboard

Now you have an awesome performace metric dashboard for all your openstack nodes.

graphite openstack dashboard

I hope you enjoyed this post. If it was helpful or if it was way off then please comment and let me know.

Subscribe to our mailing list

indicates required
Email Format

comments powered by Disqus