Serf custom queries

By Bill Ward | June 17, 2017

Serf is a very powerful tool for managing and orchestrating your clusters. The power comes from the ability for you to write customer queries and events to handle almost any situation. In this post we will go through configuring a serf cluster to handle a sample custom query to get average CPU Utilization from each of our cluster member servers in a matter of seconds.

Event Handler Router

For this post we will be building upon the cluster created in my first post HOW TO INSTALL SERF ON A MULTIPLE NODE UBUNTU CLUSTER. It should go without saying you need to have a serf cluster already setup and configured at least to the point were we have it in that post.

On each one of your serf member servers create a new folder structure that matches the following:

/etc/serf/
└── handlers

In the /etc/serf/ directory create a new file called handler.sh with the following contents:

#!/bin/sh
HANDLER_DIR="/etc/serf/handlers"

if [ "$SERF_EVENT" = "user" ]; then
    EVENT="user-$SERF_USER_EVENT"
elif [ "$SERF_EVENT" = "query" ]; then
    EVENT="query-$SERF_QUERY_NAME"
else
    EVENT=$SERF_EVENT
fi

HANDLER="$HANDLER_DIR/$EVENT"
[ -f "$HANDLER" -a -x "$HANDLER" ] && exec "$HANDLER" || :

This will forward any received events to the /etc/serf/handlers/ directory. Custom queries will take the form of ‘query-example’ and user events will take the form of ‘user-test’. So you would have:

/etc/serf/
├── handlers
│   ├── query-example
│   └── user-test
└── handler.sh

This will forward a custom query of example to the query-example script and a user event named test to user-test. In the next section we will write a handler that will get the average CPU load and send it back as a response using a custom query.

CPU Load Query

Using code from this awesome Stackflow answer, we will write our handler to return the CPU load.

On each server create a new file called query_load in the /etc/serf/handlers/ directory with the following contents:

#!/bin/bash

LOAD=`top -bn1 | grep "Cpu(s)" | \
           sed "s/.*, *\([0-9.]*\)%* id.*/\1/" | \
           awk '{print 100 - $1"%"}'`

echo "${LOAD}"

Make the file executable:

# chmod a+x /etc/serf/handlers/query-load

In the next section we will configure our SystemD service to utilize our new event handler.

Updating the Serf SystemD Service

We need to tell our serf SystemD service to use our new event handler. Add this to the end of the ExecStart line:

-event-handler=/etc/serf/handler.sh

So you should now have:

[Unit]
Description=Serf
Documentation=https://www.serf.io/docs/

[Service]
ExecStart=/usr/local/bin/serf agent -node=%H -iface=eth0 -join=192.168.1.51 -event-handler=/etc/serf/handler.sh
ExecReload=/bin/kill -HUP $MAINPID
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Reload everything:

# systemctl daemon-reload
# systemctl restart serf.service

We are now ready to test our custom query.

Sending our query

After you have the handler added to all your serf servers and restarted the serf service you are ready to test sending our query.

Run this command to query our servers and get an average CPU load for each of them:

# serf query load
Query 'load' dispatched
Ack from 'serf-03'
Ack from 'serf-01'
Ack from 'serf-02'
Response from 'serf-03': 1.1%
Response from 'serf-02': 1.9%
Response from 'serf-01': 1.1%

There you have it, you’re CPU load average for all three systems in a matter of seconds. You could write many more of these in the same way. Just make sure that the response of the handler is written to standard out and it will be returned to serf. Also make sure that you get your response back by 15 seconds. This is the default timeout that serf waits for a response.

I hope you enjoyed this post. If it was helpful or if it was way off then please comment and let me know.

Subscribe to our mailing list

indicates required
Email Format

comments powered by Disqus
Google Analytics Alternative