In this post I will you walk you through creating Serf custom queries that will allow you to perform actions on your infrastructure and get a response back.

Serf is a very powerful tool for managing and orchestrating your clusters. The power comes from the ability for you to write customer queries and events to handle almost any situation. In this post we will go through configuring a serf cluster to handle a sample custom query to get average CPU Utilization from each of our cluster member servers in a matter of seconds.

Event Handler Router

For this post we will be building upon the cluster created in my first post HOW TO INSTALL SERF ON A MULTIPLE NODE UBUNTU CLUSTER. It should go without saying you need to have a serf cluster already setup and configured at least to the point were we have it in that post.

On each one of your serf member servers create a new folder structure that matches the following:

└── handlers

In the /etc/serf/ directory create a new file called with the following contents:


if [ "$SERF_EVENT" = "user" ]; then
elif [ "$SERF_EVENT" = "query" ]; then

[ -f "$HANDLER" -a -x "$HANDLER" ] && exec "$HANDLER" || :

This will forward any received events to the /etc/serf/handlers/ directory. Custom queries will take the form of ‘query-example’ and user events will take the form of ‘user-test’. So you would have:

├── handlers
│   ├── query-example
│   └── user-test

This will forward a custom query of example to the query-example script and a user event named test to user-test. In the next section we will write a handler that will get the average CPU load and send it back as a response using a custom query.

CPU Load Query

Using code from this awesome Stackflow answer, we will write our handler to return the CPU load.

On each server create a new file called query_load in the /etc/serf/handlers/ directory with the following contents:


LOAD=`top -bn1 | grep "Cpu(s)" | \
           sed "s/.*, *\([0-9.]*\)%* id.*/\1/" | \
           awk '{print 100 - $1"%"}'`

echo "${LOAD}"

Make the file executable:

# chmod a+x /etc/serf/handlers/query-load

In the next section we will configure our SystemD service to utilize our new event handler.

Updating the Serf SystemD Service

We need to tell our serf SystemD service to use our new event handler. Add this to the end of the ExecStart line:


So you should now have:


ExecStart=/usr/local/bin/serf agent -node=%H -iface=eth0 -join= -event-handler=/etc/serf/
ExecReload=/bin/kill -HUP $MAINPID


Reload everything:

# systemctl daemon-reload
# systemctl restart serf.service

We are now ready to test our custom query.

Sending our query

After you have the handler added to all your serf servers and restarted the serf service you are ready to test sending our query.

Run this command to query our servers and get an average CPU load for each of them:

# serf query load
Query 'load' dispatched
Ack from 'serf-03'
Ack from 'serf-01'
Ack from 'serf-02'
Response from 'serf-03': 1.1%
Response from 'serf-02': 1.9%
Response from 'serf-01': 1.1%

There you have it, you’re CPU load average for all three systems in a matter of seconds. You could write many more of these in the same way. Just make sure that the response of the handler is written to standard out and it will be returned to serf. Also make sure that you get your response back by 15 seconds. This is the default timeout that serf waits for a response.

I hope you enjoyed this post. If it was helpful or if it was way off then please comment and let me know.


I hope you have enjoyed this article, if so please leave a comment below.  For more articles, please signup for the AdminTome Blog below.  Also please feel free to share the article to your friends using the buttons to the left.  Thanks again for reading this post.

Signup for the AdminTome Blog Newsletter

Get weekly updates from AdminTome Blog including the latest articles and special content only for subscribers.

privacy We value your privacy and would never spam you

%d bloggers like this: