In this post, I will cover logging Jenkins jobs using Elasticsearch, Logstash and Kibana. This is great if you monitor hundreds of deployments.
Jenkins has great console logging but if you have a ton of jobs because you’re using microservices then keeping track of all these logs can be quite a chore. In addition, you might want to analyze these logs to identify cascading failures or environment issues that is causing your continuous delivery pipeline to fail. This post will cover how to setup an Elasticsearch and Kibana server that is a powerful option to analyze and combine your logs into a “single pane of glass” dashboard.
Configuring the Elasticsearch and Kibana Server
Our first step is installing and configuring our Elasticsearch and Kibana Server. For this post we are going to use a single server for both. In production, you would most likely want to run Elasticsearch in a clustered configuration, but that will be outside the scope of this post. To get the most of elasticsearch I recommend reading Elasticsearch Essentials by Packt Publishing.
This post walks you through install Elasticsearch no Ubuntu 16.04.
I have a new article that shows you how to install the latest version of Elasticsearch on Ubuntu 18.04.1: Install Elasticsearch on Ubuntu 18.04.1
If you want to follow those steps then skip this section and continue in the Logging Jenkins Jobs section.
Start with a fresh installation of Ubuntu 16.04. Configure a static IP for it and install all the updates. There are a few different ways to get Elasticsearch installed. In this post we will download the binary release. But before we do that we need to install Java.
sudo apt install openjdk-8-jdk
Now lets download the Elasticsearch binary and untar it. Just in case you are one of those admins that does everything as root, you must do the following commands as a regular user not as root.
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.0.0.tar.gz tar -xvf elasticsearch-5.0.0.tar.gz cd elasticsearch-5.0.0/bin
We have to take care of a few things in order to get our instance of Elasticsearch to work correctly. The first thing we have to do is configure virtual memory mmap counts to a higher number than the default OS limits.
sudo sysctl -w vm.max_map_count=262144
You can set this permanently in /etc/sysctl.conf. Now you can start your Elasticserver service. Be sure to substitute your routable IP address.
./elasticsearch -Enetwork.host=<ip-address> -v > elastic.log 2>&1 &
Check to make sure everything is working fine by browsing to the following URL:
You should see something like this:
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent 1477619952 20:59:12 elasticsearch yellow 1 1 6 6 0 0 6 0 - 50.0%
Kibana is the frunt end GUI that we are going to use to view and analyze our Jenkins logs. We will install it using their Debian package.
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list sudo apt update && sudo apt install kibana
Start and enable the Systemd Service.
sudo systemctl start kibana.service sudo systemctl enable kibana.service
Now you should be able to browse to the URL below and see the Kibana page:
Now on to phase 2: Configuring Jenkins
Logging Jenkins Jobs
Go to your jenkins URL and login. Go to Manage Jenkins -> Manage Plugins -> Available and search for Logstash. Logstash Jenkins Plugin allows us to forward our build job console logs to our new Elastic search server. You will also want to install the ‘Mask Passwords Plugin‘. After it is installed go to Manage Jenkins -> Global Tool Configuration and scroll to the bottom to the Logstash Plugin section. Set Indexer Type to ELASTICSEARCH, Hostname to http://, Port to 9200, and Key to logstash-jenkins/jenkins. The latest version of Jenkins will show ERROR under these fields but I discovered you can safely ignore then, the plugin will still work. Save the settings.
Now go to one of your Jenkins jobs and click on configure. Under Build Environment you should now see a checkbox for Send console log to Logstash. Check this and click on save. Now run your Jenkins job and it will send the logs to your Elasticsearch server.
I only see this option available for Freestyle type Jenkins Jobs. For pipeline type jobs you will need to code this in Groovy.
Viewing your logs in Kibana
Go back to your Kibana web page. Since we have just setup Kibana, it will ask you to input an index. In our case it will be ‘logstash-jenkins’ and we can enter ‘logstash-jenkins*’ in the index pattern box:
Click Next where it will ask you about a filter. Select ‘@buildTimestamp’ as the time filter and finish. Finally, click on ‘discover’ to see our logs:
Kibana is a very powerful tool and to get the most out of it and learn more about it read Kibana Essentials by Packt Publishing.
I hope you have enjoyed this article, if so please leave a comment below. For more articles, please signup for the AdminTome Blog below. Also please feel free to share the article to your friends using the buttons to the left. Thanks again for reading this post.