Running Jenkins on Mesos

By Bill Ward | April 6, 2017

This post will walk you through configuring your Jenkins server to use a Mesos Cluster to elasticly use Jenkins Slaves running in Mesos. This has the advantage of using your datacenter hardware much more efficiently and you won’t need several jenkins slaves that could potentially be sitteng there doing nothing but costing you money.

I have seen a few posts out there on how to get this done, but they were either copies of the plugin documentation (which leaves much to be desired) or missed some important stuff that I was left to discover on my own.

Assumptions

Configuring Mesos Slaves

In order to begin using Jenkins on your Mesos Cluster, you will need to first add a Jenkins user to all of your slaves and ensure that you have JAVA_HOME set.

root@mesos-slave-2:~# adduser jenkins
Adding user `jenkins' ...
Adding new group `jenkins' (1001) ...
Adding new user `jenkins' (1001) with group `jenkins' ...
Creating home directory `/home/jenkins' ...
Copying files from `/etc/skel' ...
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Changing the user information for jenkins
Enter the new value, or press ENTER for the default
        Full Name []:
        Room Number []:
        Work Phone []:
        Home Phone []:
        Other []:
Is the information correct? [Y/n]
root@mesos-slave-2:~# which java
/usr/bin/java
root@mesos-slave-2:~# su - jenkins
jenkins@mesos-slave-2:~$ echo "JAVA_HOME="/usr/bin/"" > .profile

Do this for all of your Mesos Slaves.

Configuring Jenkins

IP Hosts Configuration

This part sent me for a loop for a while and gave me a huge headache. It isn’t really documented anywhere but luckely I stumbled accross the correct answer. Make sure in your /etc/hosts file on your jenkins server that you DO NOT have an entry for 127.0.1.1. Change this to the IP address of your jenkins server (which is what I did). I suppose you can also just delete the line. I remember having to do this when configuring openstack, but in this case i just replaced it with the actual IP address of the jenkins server (a 192.168.1.x address). While you are at it make sure you do the same for all your mesos masters and slaves as well. If not you will see a bunch of weird stuff happening when you try starting slave processes.

Copy libmesos.so to jenkins

This step can be tricky. Essentially, all we need to do is copy the compiled libmesos.so to your jenkins server under the jenkins user’s home directory /var/lib/jenkins. If you followed my article on building a mesos cluster (link above) then you have a compiled version of this on all of your mesos masters and slaves in the [mesos download dir]/build/src/.libs/ directory. How you get the file from your mesos server to jenkins is up to you. I found it easiest to SCP it over to the stackadmin user on my jenkins box (all my systems have this user) and then copy it over to the jenkins user home directory and finally change ownership of the file to the jenkins user.

Configure Jenkins

Now it is time to configure your Jenkins Server. Log in to your Jenkins server and click on Manage Jenkins then on Manage Plugins. Click on the Available tab at the top. In the filter window search for mesos. Click the Install and restart Jenkins button and check the checkbox to restart your Jenkins server. Now that the plugin is installed we can start configuring the plugin.

Configuring the Mesos Jenkins plugin

After Jenkins restarts, click on Manage Jenkins then click on Configure System at the top of the menu. Scroll all the way down and you will see a new section called Cloud. Click the Add a new cloud button_ then click on Mesos Cloud.

jenkins mesos plugin

Configure the following options:

Mesos native library path - /var/lib/jenkins/libmesos.so

Mesos Master - I set this to the active master. This allowed me to test the connection using the Test Connection button. After you verify that you can connect to your mesos master you can change this to the zookeeper URL (in the exampe case of my previous article this would be zk://192.18.1.30:2181,192.168.1.31:2181,192.168.1.32:2181/mesos).

Jenkins URL - This is the URL of your jenkins server.

Click on Advanced

On-demand framework registration - If you set this to you no then a slave process will start immediatly. If you select yes then a slave process won’t start until you do a build.

The rest of the options you can leave at the default for now.

Click Save

Configuring your jenkins jobs

All you have to do here is select Restrict where this project can be run and enter mesos in the Label Expression text field. Save your job and run it. You will see your job run and you should see your slave process show up under agents in your mesos dashboard.

jenkins mesos plugin

Now you have a jenkin running slaves on-demand using a mesos cluster. Keep in mind this is a very basic setup. You can do java builds and anything that has plugins for jenkins but if you are like me and do a bunch of crazy stuff in your jenkins slaves like installing a specific version of ruby, installing python modules using pip per install, building docker containers, or anything out of the norm this way as configured will not work. Your builds will fail. We can do this however by configuring a jenkins slave docker container the way we need and using the jenkins mesos plugin to spin up our own jenkins slave docker container each time. That will be for a future post though.

Conclusion

I hope you enjoyed this post. This is my first post under the new Admin Tome branding. If it was helpful or if it was way off then please comment and let me know.

Subscribe to our mailing list

indicates required
Email Format

comments powered by Disqus
Google Analytics Alternative