It is about Jenkins

We need to prepare the hosting environments for our DevOps Pipeline. This will require us to install several different components. The first core component, of course, is Jenkins. We will also install a variety of other tools in this post.

Click image for full size

Figure 1: Software that supports our pipeline

Our CI/CD Pipeline = Jenkins + Python + Supporting Software

What is Jenkins

Jenkins is the open source automation server written in Java that will build, test, and manage the overall integration and deployment pipeline. The important concept to remember is that it automates the non human part of the whole software development process, which include such things as continuous integration, continuous delivery, and more.

Jenkins has several built-in facilities to automate the CI CD pipeline.

This is the pipeline that we will automate fully with Python code as you can see below.

Click image for full size

Figure 2: The pipeline In Python

Ignoring the built-in facilities of Jenkins

But we will ignore most of these and leverage the Jenkins pipeline itself through Python code.

Leverage Python as much as possible

Rather than use plug-ins, we will focus on running script code in Python.

The advantage of using Python to do our tasks is that we have full control over the behavior that is needed, minimizing dependencies on other software components inside of Jenkins that were not well docucumented or unclear how to install and use.

It gives us the ultimate flexibility in defining the workflow that we want. The downside of writing Python code to automate the pipeline is that you must know a little bit about writing software. But you could learn everything you need in just a few days.

The Jenkins plug-in model took too much of my time

Frankly, I had a lot of challenges setting up the various plug-in components to work properly with Jenkins in my Linux environment. There was a lot lacking in the documents.

Productive immediately with Python

The moment I switch to a Python based approach, I started to become productive almost immediately. I knew the tooling from the command line point of view so translating commandline using Python is really easy using the subprocess module.

Jenkins can run almost anywhere

Because Jenkins relies on Java, it can practically run on any flavor of Linux. In my case, I will be using Ubuntu, tested on version 14.04 and above.

You need to have a JDK and JRE installed. openjdk-7-jre and openjdk-7-jdk are suggested.

add-apt-repository ppa:openjdk-r/ppa
apt-get update
apt-get install -y openjdk-8-jdk
dpkg --purge --force-depends ca-certificates-java
apt-get install ca-certificates-java

Here are the instructions to install Jenkins on Ubuntu.

https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+Ubuntu

Click image for full size

Figure 3: installing Jenkins

Jenkins – groupadd

I discovered some issues if I didn’t add the Jenkins user to the appropriate group. You may or not face this issue.

First create the JENKINS_USER. Then,

groupadd jenkins
useradd -g jenkins jenkins

Docker

Because we will be building docker images, running them as containers, and performing tests against them, we will obviously need the docker client tooling installed on our Jenkins host, which is our Linux VM.

sudo apt-get update
sudo apt-get install -y wget
sudo wget -qO- https://get.docker.com/ | sh

Gradle

Gradle will be executed on the Jenkins host.

The pipeline itself is kicked off when the developer checks in his source code.

A web book then notifies Jenkins to download this source code and compile it using Gradle.

Gradle will also perform unit tests against the build.

So Gradle will be needed on the Jenkins host.

They have plug-in support for Gradle but I found that installing it yourself directly on the host worked better.

A more sophisticated environment would create a Jenkins container that has all these necessary dependencies, such as Gradle, Python, Jenkins and the rest of the toolset.

# Download and extract gradle to opt folder

https://docs.gradle.org/current/userguide/installation.html

sudo wget --no-check-certificate --no-cookies https://downloads.gradle.org/distributions/gradle-${GRADLE_VERSION}-bin.zip 
    && unzip gradle-${GRADLE_VERSION}-bin.zip -d /opt 
    && ln -s /opt/gradle-${GRADLE_VERSION} /opt/gradle 
    && rm -f gradle-${GRADLE_VERSION}-bin.zip
# Add executables to path
sudo update-alternatives --install "/usr/bin/gradle" "gradle" "/opt/gradle/bin/gradle" 1 && 
    update-alternatives --set "gradle" "/opt/gradle/bin/gradle"

Python

Python is single-handedly managing our entire Jenkins pipeline.

Python is perhaps the most essential piece of this entire DevOps toolkit, since it is responsible for kicking off all the related processes that make up our entire pipeline.

The idea here is that Python is cross-platform, meaning that we could leverage Python in both Windows and Linux pipeline environments.

It’s a great thing if you could have parity between both your @indows pipelines as well as Linux pipelines.

RUN apt-get install -y libssl-dev openssl
RUN apt-get update && apt-get -y dist-upgrade && 
    apt-get -y install git build-essential

RUN apt-get install gcc
RUN wget https://www.python.org/ftp/python/3.5.0/Python-3.5.0.tgz &&  
    tar xzvf Python-3.5.0.tgz &&  
    cd Python-3.5.0 &&   
    ./configure &&  
    make &&  
    make install

MySQL – How are you going to capture and retain state for your pipeline?

**I debated this quite a bit previously to myself. ** The question becomes, quote how do you track state for the different steps in the pipeline. Obviously if you have the failure in any of the steps, the system needs to react appropriately.

My simple rule was that if there was anything that went wrong, we would write a record into MySQL. That records would then would be read in subsequent steps in the pipeline.

The reason I like this approach is that the data can be somewhat structured and I could easily accessed from anywhere. Debugging the pipeline is a simple SQL statement. The database could point you to the Python script. Because all the details are contained within each Python script, debugging the pipeline could be a breeze.

More options are available for debugging as well. You could have generous print statements, in our Python scripts that then get trapped to Jenkins log files.

Bottom Line – The last thing you want to be debugging is pipeline code.

The point is to resolve as quickly as possible where the pipeline breaks down.

Constantly tracking state in a persistent relational data store is my answer to stay tracking in this situation. you could go really crazy now partying on this relational data. You can use some analytics even machine learning into the whole equation here. You can even use analytics dashboards like Power BI and be able to view all your bills holistically is one big system. Frankly, there will not be much more for you to add after I finish here demonstrating the entire pipeline in all its gory details.

You could also take prolific log files that get generated within Jenkins and persists those to blob storage.

There is a huge amount of intelligence built into the logging mechanism if you do it correctly.

Click image for full size

Figure 4: Tracking pipeline state in a relational database

Installing MySQL

Click image for full size

Figure 5: Installing MySQL

Installing the JDBC Drivers

Click image for full size

Figure 6: x

Hello World

So the next task is to bring Jenkins up and create a project to begin implementation of our pipeline.

There are two aspects to getting Jenkins started.

The first one is to actually run the war file that gets downloaded.

Here you can see the war file is called Jenkins.war.

The command below also assumes that Java is properly installed and available in the path.

Click image for full size

Figure 7: Starting Jenkins on the command line

Starting your browser

The public IP address that you see here in the address bar of the browser is the public IP address of the virtual machine running in Azure.

You can get this public IP address from the Azure portal.

If your VM has a domain name, you could also use that. Notice that I mapped to port 8086.

That’s because I passed in the parameter as you see below to indicate the port that I wanted exposed by Jenkins.

java -jar jenkins.war --httpPort=8086

Click “New Item.”

Click image for full size

Figure 8: New Jenkins Project

Creating our first pipeline project

After clicking a “new item,” the next step is to provide a “name” and select the “pipeline” project type.

Click image for full size

Figure 9: Building our pipeline from the template of Jenkins

The most simple pipeline we could start with is a “hello world” pipeline as seen below.

Click image for full size

Figure 10: Adding Pipeline code

Click image for full size

Figure 11: Build Now

Click image for full size

Figure 12: Viewing the Console Output

Conclusion

The takeaways from this post is that we are up and running with Jenkins on our Ubuntu VM in Azure. A variety of software technologies were leveraged here. And the point was to get them all installed. We also went in and ran a basic hello world pipeline. You also probably notice that we have defined the entire pipeline in terms of Python script. The names chosen were very descriptive so it’s very clear exactly where to go to debugging our pipeline. Over the next few posts on me to reveal the source code that makes this pipeline possible. If you’re interested in that code, just let me know.