A better way to Jenkins

Jump around, Jump around. No maps this time, but Continuous Integration. My current work involves a lot of a software piece called Jenkins. I’ve been swimming with it the last 2 months and I feel compelled to share how we changed our ways.

This post is an overview guide on how to configure Jenkins to generate the jobs that are defined in a git repo.

What is this?

Jenkins is a butler. You ask him to serve you specific builds at specific times and he does the best he can. I really like the idea of a having a butler, even if it’s only imaginary.

You can extend it with plugins, scripting and a big etcetera. Which makes it quite useful, and, more often than not, overcharged with options.

How do you order something?

Everything works via web interface. You go to Mr Jenkins, ask for a new build, and start filling text boxes, combo boxes and whatever else falls in there. Once it is configured, Jenkins will make sure that the build gets delivered in a regular basis.

Plain Jenkins is nothing more than a glorified crontab. But start adding plugins and you get two outcomes.

• It starts doing a lot of things
• The configuration pages get huge

A huge configuration page is a pain in the ass (and a huge XML even more), and when you are managing 400 configuration pages by hand you start to wonder, is there a better way?

Manual work is for pussies

With Jenkins is almost impossible to follow my favourite principle, that is KISS (a kiss, or Keep It Simple Stupid), but after some months working with it, I decided it was time for a change.

And the change is usually to reduce the amount of manual work, in this case, centralizing the configurations in a single place, with an easy to understand syntax.

Entering: Jenkins-job-dsl

Job dsl does something wonderful for us, it lets you define Jenkins jobs programatically with their DSL (Domain Specific Language).

A simple example:

job {
    name 'test-job-1'
    scm {
        git("https://thisisagitrepo.git")
    }
    steps {
        maven('compile')
        shell('make clean')
    }
}

The previous example will generate a job that does the following:

• Clone git sources
• Call maven compile
• Call make clean

With no click in the Jenkins interface! 😀 (well, sort of), isn’t that great?

This DSL is based on a language called Groovy (don’t get me started on it, I’m not a fan). But that means that we can extend it to pleasure our needs if necessary.

Non supported plugins

Job-DSL supports all the basic Jenkins options, plus can manage a lot of the most common plugins: git, maven, environment, you can check the full list in the changelog wiki.

Unfortunately, not all the plugins are supported, and you may find that something useful for your projects is not on job-dsl. We’ll have to do it ourselves, either improving job-dsl to support it, or using the job-dsl XML interface.

Now the cold sweat starts, deep down, each Jenkins job is configured as a set of huge xml files. Somebody said that rm is the best xml parser, and I agree with all my heart. Working with XML is usually a pain.

Job-DSL provides a special block (configure block), to modify the XML directly. So, if the plugin is not supported, you can create an example job manually, check the generated XML and then create your DSL generating that XML part.

Set up a git repo

This is plain simple, and I won’t extend a lot. What we are looking for here is to create a git repo with the following structure
/root
|
|- job1.groovy
|- job2.groovy
|- job3.groovy
...
|- jobN.groovy


As a start, each job will be self-described. With no inherits or includes. But bear in mind that this DSL is groovy based, so you can get creative 😉

Set up a job dsl job

We will use a Jenkins job to generate Jenkins jobs, and totally forget about it. We can create this job manually and then let it create the rest of the jobs by itself.

Create a new item, freestyle-job:

This job will have a very simple configuration:

• Get DSL files from a git repo
• Run all those files

The only thing to set up is the git url (watch for those credentials), and add a DSL build step ( Build step -> Process Job DSLs). This build step has to be set to look for all groovy files on the filesystem *.groovy. Those are the cloned files from our git repo.

With this simple job, every .groovy file that we drop on our repo is going to be translated into a job in Jenkins.

And this is it, let me just drop a infrastructure diagram here for clarity.

Conclusions

I did not comment that previously, but Jenkins is a pain in the ass. When it works it works, but when it fails, get ready for the worst.

Being able to create jobs with a web interface is nice and all, but at the end, when you have to manage more than 10 jobs, you start automatizing the automation system. Which when you think about it is kind of scary, we get higher and higher up.

References

Jenkins CI ⇒GO
Job DSL plugin ⇒GO
Groovy language ⇒GO