Highly Available Jenkins With Consul Nomad And Terraform

By Bill Ward | July 11, 2017

Jenkins doesn’t come with high-availability out-of-the-box. Using Nomad/Consul we can configure Jenkins to be highly-available. Furthermore, we can use terraform to make deployment a breeze.

Disclaimer: As of this day Jenkins will give you an error about using the same JENKINS_HOME directory between multiple instances. There is no solution to this yet. If you don’t mind seeing the error then continue along. This isn’t really meant for production. Maybe in the near future Jenkins will fix this and make it easier to enable HA.

Architecture Overview

Refer to following crude diagram for this overview:

Highly-Available Jenkins Architecture

This article assumes that you already have a Consul Cluster configured and that you also have a Nomad Cluster configured to use Consul for service discovery. I have several articles on how to get this setup and configured.

We will be configuring a Nomad Job that will spin up Jenkins in a Docker Container. For this to work make sure that you have Docker enabled on your Nomad Agent servers. In addition, you will need some way to persist the volume data for your Jenkins Docker containers. I did this by setting up an NFS server with a share and mounting that share on all three of my Nomad Agent servers (/nfs/general/).

We code our Nomad job configuration into Terraform so that we can provision and destroy it with ease.

Finally, on your development system you will need to install Terraform, Consul, and Nomad. You will need to set a couple of environment variables so that the CLIs know how to talk to your clusters:

export NOMAD_ADDR="http://your_nomad_ip:4646"
export CONSUL_HTTP_ADDR="your_consol_ip:8500"

The Nomad one is used by terraform to deploy our Nomad Job and the consul one is used to check on things later on.

Terraform Configuration

Here is the terraform configuration that we will use to deploy our Jenkins Infrastructure:

jenkins.tf

provider "nomad" {
  address = "http://nomad-server.admintome.local:4646"
}

# Register a job
resource "nomad_job" "jenkins" {
  jobspec = "${file("${path.module}/jenkins.nomad")}"
}

Pretty simple right? Here we just tell it how to find our Nomad cluster then we give it a jobspec which is in a file we call jenkins.nomad with the following contents:

jenkins.nomad

job "jenkins" {

  datacenters = ["dc1"]

  type = "service"

  group "cicd" {
    count = 1

    restart {
      attempts = 10
      interval = "5m"

      delay = "25s"

      mode = "delay"
    }

    ephemeral_disk {
      size = 300
    }

    task "jenkins" {
      driver = "docker"

      config {
        image = "jenkins:latest"
        port_map {
	  http = 8080,
	  slave = 50000
	}
	volumes = [
	  "/nfs/general/jenkins_home:/var/jenkins_home"
	]
      }

      resources {
        cpu    = 1024 # 500 MHz
        memory = 2048 # 256MB
        network {
          mbits = 10
          port "http" {}
        }
      }

      service {
        name = "jenkins-server"
        tags = ["global", "cicd"]
        port = "http"
        check {
          name     = "alive"
          type     = "tcp"
          interval = "10s"
          timeout  = "2s"
        }
      }

    }
  }
}

I won’t go through this file line-by-line but will instead cover the important stuff. This article is already getting pretty long…

Under the tasks section we have our first important part which the volumes stanza.

volumes = [
    "/nfs/general/jenkins_home:/var/jenkins_home"
]

This tells docker to setup a volume located at /nfs/general/jenkins_home on the host and maps it to /var/jenkins_home in the container. This makes sure that all the instances will hit the shared NFS storage and use the same data.

Notice that we also configure the port_maps to 8080 and 50000 so that we can access everything. The port 50000 will be used by jenkins slaves to talk to the masters. You can add the ‘nomad’ plugin to jenkins and it will spin up slaves on the fly using nomad.

Fire it up

Now provision everything using terraform.

terraform apply

This will deploy the jenkins job to nomad and start everything up. We can check the status by using the nomad client we installed on our development system:

$ nomad status
$ nomad status jenkins
$ nomad alloc-status {task-id} jenkins

One quick note: we need to setup one instance first so that we can configure jenkins. After we have it completely installed then we can spin up multiple jenkins servers that will use the same configuration by increasing the count line in our jenkins.nomad file.

Before we access it we need to setup a load balancer. This is because Nomad will use random ports and different agents for the instances. The load balancer will proxy a known IP and Port to the backend jenkins services running on Nomad.

nomad-01:32743   -> jenkins:8080
nomad-02:32653

NGINX Load Balancing Configuration

You will need a seperate NGINX server running that we can configure load-balancing on for our jenkins servers. On your NGINX server, install the consul agent and configure it to join your consul cluster.

Download and install consul-template

# wget https://releases.hashicorp.com/consul-template/0.19.0/consul-template_0.19.0_linux_amd64.zip
# unzip consul-template_0.19.0_linux_amd64.zip
# mv consul-template /usr/local/bin

Create the following file:

jenkins.conf.ctmpl

{{$service := service "jenkins-server"}}
upstream jenkins-server {
  zone upstream-jenkins-server 64k;
  {{range $service}}server {{.Address}}:{{.Port}} max_fails=3 fail_timeout=60 weight=1;
  {{else}}server 127.0.0.1:65535; # force a 502{{end}}
}

server {
  listen 80 default_server;

  location / {
    proxy_pass http://jenkins-server;
  }
}

Run this command on your load balancer:

# consul-template -template="jenkins.conf.ctmpl:/etc/nginx/conf.d/jenkins.conf

You can also configure a systemd service that will run this automatically, but I won’t be convering that here.

It may look like it is frozen but it is still running. Everytime your Jenkins nomad configuration changes consul-template will automatically update our NGNIX configuration!

Restart NGINX and browse to your load balancer’s IP address. If you have internal DNS setup you can setup an alias of jenkins.yourdomain.local to the load balancer’s IP to make things easier.

You should see Jenkins’ configuration page asking you to input the adminstrator token generated during initial configuration. To find this use the nomad commands on your local development system

$ nomad logs {task-id} jenkins

You get the task-id from running:

$ nomad status jenkins

After you get Jenkins configured and plugins installed you can now increase the count in your jenkins.nomad file.

Conclusion

I trimmed a lot out of this to shorten the article but it still turned out to be a large article. If I missed something just leave a comment below and I will update the article.

I hope you enjoyed this post. If it was helpful or if it was way off then please comment and let me know.

Subscribe to our mailing list

indicates required
Email Format

comments powered by Disqus
Google Analytics Alternative