Using Jenkins / Hudson remote API to check jobs status

While working on my talk Writing a Hudson / Jenkins plugin (for EclipseCon NA 2014), I wanted to publish blog posts about the ideas mentioned in the talk; in this post I explain how you can interact with your CI server without using the web interface.

The Jenkins / Hudson remote API can be very convenient to quickly gather jobs status (and even to create or launch jobs ! but I won’t cover that aspect in this blog post); let’s have a look at some examples.

Have a look at the CLI to automate Jenkins configuration tasks.

It’s available at http://hudson-or-jenkins/cli : download the hudson-cli.jar or jenkins-cli.jar and get started with this first command :

Recently, I had to replace 50 jobs with a matrix job : once I created the matrix job, we decided to de activate the previous 50 jobs (before removing them for good)

Using the CLI, I could quickly login :

then get the list of jobs :

I saved the list of the 50 jobs I wanted to de activate in a text file, and I looped through this list :

More efficient than going through the UI, isn’t it ?

Every URL can be represented as XML or JSON

As explained in the Jenkins documentation and Hudson documentation, you just need to append /api/xml or /api/json or /api/python to any Jenkins URL to see the corresponding representation

You can configure the responses adding few arguments to the URL :

Depth

The default depth is 0, to get more details about your jobs, builds, etc, set the depth to 1 :

http://ci.jruby.org/api/xml?depth=1

This argument can be harmful to your CI server when bigger than 1 (it gets really deep past 1 …) since the responses tend to become really large.

Tree

The tree argument allows you to select some portion of the response

http://ci.jruby.org/api/xml?depth=1&tree=jobs[displayName,lastBuild[result]]

xpath and exclude (XML only)

Those arguments are probably the most powerful ones, sadly they are only available for xml responses :

If I only want to display jobs that were not successful, I can simply use this url :

http://ci.jruby.org/api/xml?depth=1&tree=jobs[displayName,lastBuild[result]]&exclude=hudson/job[lastBuild[result='SUCCESS']]

And if I only want to see jobs with a name starting with “jruby”, I can apply Xpath functions to the url :

http://ci.jruby.org/api/xml?xpath=hudson/job[starts-with(name,'jruby')]&wrapper=mywrap

Few Jenkins and Hudson public instances URLs

It’s pretty easy to find some public Jenkins and Hudson instances to practice your latest URL filtering (google for “Hudson Dashboard” or “Jenkins Dashboard”), but it’s harder to find up to date and meaningful (few jobs) instances :

Jenkins :

Hudson :

That said, try your local instance first :-)

Jenkins and Hudson remote API scripts

The main interest of the Jenkins/Hudson remote API is to interact with it from your own software : I gather here some examples in few scripting languages

Groovy Script to parse Jenkins Hudson build results

Python Script to parse Jenkins Hudson build results

JavaScript code to parse Jenkins Hudson build results


 

You can find those scripts on Github ! Or even execute them in your browser for the Javascript Jenkins Hudson Build parser script

Developing a plugin for both Jenkins and Hudson

While working on my talk Writing a Hudson / Jenkins plugin (for EclipseCon NA 2014), I wanted to publish blog posts about the ideas mentioned in the talk; in this post I explain what are the differences between Hudson and Jenkins and how you can make your plugin compatible with both of them.

Since the project split in November 2010, Hudson (now under the Eclipse foundation umbrella) and Jenkins (community driven) APIs have followed their own path… What does that mean for plugin development ?

Jenkins and Hudson differences on February 2014

Hudson and Jenkins governance are quite different, and so are their release rhythm (frequent and LTS releases for Jenkins, major /minor / bug fix releases for Hudson)

Let’s have a look at these 2 projects activities (please keep in mind that I just compared the core projects, not the plugins activity, and also that I only selected some axis of comparisons – this is not the most complete comparison you would imagine !) :

Jenkins activity between 2011 and 2014

On February 2nd 2011, was published the first Jenkins release (1.396, latest is 1.550), since then , if you clone https://github.com/jenkinsci/jenkins.git and run few commands, you can find out :

  • How many commits were pushed since February 2011 :

    returns 6785
  • How many commits were pushed since September 2011 :

    return  5001
  • How many contributors committed code since February 2011 :

    returns 325

Most important features added to Jenkins since February 2011 : (from http://jenkins-ci.org/changelog, I just picked the major enhancements)

  • 1.416 : Added a mechanism to write views in Groovy
  • 1.433 : Use of guice for loading extensions
  • 1.442 : Plugins can now be installed without taking Jenkins offline
  • 1.446 : Jenkins now acts as an SSH daemon
  • 1.463 : Publishers can be now reordered by the user
  • 1.509 : Performance improvement in master/slave communication throughput
  • 1.519 : Remoting classloader performance improvement upon reconnection to the same slave
  • 1.520 : Core started relying on Java 1.6
  • 1.532 : Added a new extension point to control where archived artifacts get stored.
  • 1.535 : “java -jar jenkins.war” now runs on Jetty8. Command line options are still compatible.
  • 1.536 : Executor threads are now created only on demand and Upgrade bundled iplugin versions: ssh-slaves to 1.4, ssh-credentials to 1.5.3 and credentials to 1.8.3
  • 1.542 : Allow build queue and executor status panes to be collapsed.
  • 1.548 : Added infrastructure for moving items into or out of folders.

Hudson activity between 2011 and 2014

We can run the same commands on the Hudson git repository : http://git.eclipse.org/gitroot/hudson/org.eclipse.hudson.core.git/ , except that the history anterior to September 2011 is absent.

  • How many commits were pushed since February 2011 :

    returns  1230
  • How many contributors committed code since February 2011 :

    returns 15

Most important features added to Hudson since February 2011 : (from http://hudson-ci.org/changelog.html and http://www.eclipse.org/hudson/changelog.php)

  • 2.1.0 : Maven 3 plugin, GWT support for plugins
  • 2.2.0 : Cascading project settings (inheritance of Job settings)
  • 3.0.0 : Dependencies clean up (IP clean to join the Eclipse foundation)
  • 3.1.0 : Team Concept (team based authentication and authorization model within Hudson)

It’s interesting to note that the Hudson teams provides their coming priorities on their project page.

What about the usage then ?

And the plugins ?

If you can’t choose which tool to develop plugins for, then read on, you may be writing the same plugin for both, with only few modifications.

Developing the same plugin for both Hudson and Jenkins

Since both tools have the same origin API, and avoid making disruptive API changes (to stay compatible with most plugins), there’s a good chance you can keep the same code for your plugin and make it compatible with both.

Let’s see in practice what it looks like : let’s create a new Hudson plugin :

Once it’s created we can see it in action :

(Hudson requires you to go through an initial setup at http://localhost:8080 the first time you run hpi:run on your new plugin, click finish and you’re done)

You may wonder why I override the HUDSON_HOME environment variable : it’s because I’ll want jenkins and hudson to use separate work directories; we could also avoid setting the environment variable on the command line and specify hudsonHome in the pom :

By now, you should be able to see the sample plugin adding a build option after creating a new job : http://localhost:8080/view/All/newJob

Now we know that this plugin is working fine with the latest version of Hudson, what about Jenkins ?

First, make sure you edit your ~/.m2/settings.xml maven configuration file to be able to pull jenkins maven plugins (we didn’t need to do that with Hudson since the Hudson team pushes their plugins to maven central)

Then, in the pom.xml, just replace the parent with :

and build and run with :

and head your browser to : http://localhost:8080/jenkins/view/All/newJob and create  new job, you should see your plugin deployed :

What this demonstrates is that chances are the plugin you’re currently working on does not rely on Hudson  specific or Jenkins specific API, and it should be just a matter of creating a branch and maintaining it to widen your audience.

A real life example : porting the Accelerated Build Now plugin from Jenkins to Hudson

I successfully ported this plugin from Jenkins (targeting 1.523) to Hudson (3.1.2). Here are the main changes between APIs I had to deal with :

  • of course, jenkins.model.Jenkins had to be replaced by hudson.model.Hudson
  • in 2012 Jenkins added the use if QueueTaskFuture instead of java.util.concurrent.Future
  • in 2012 Jenkins added getStartTimeInMillis to the Run class

There still aren’t a lot changes between the 2 code bases, if you’re curious, you can see the details of the changes between the 2 branches, Jenkins (master) and Hudson : https://github.com/Terracotta-OSS/accelerated-build-now-plugin/compare/hudson

Terracotta Summer Plugins

This summer at Terracotta, we’ve been pretty busy working on making our test and build infrastructure more convenient for the developers.
In fact, Ludovic, Louis and myself used some of our innovation days (days offered by Terracotta we can spend working on projects we want) to create (and contribute to) plugins for Nexus, Maven, and also for Jenkins (well, this last one was not developed during innovation days, but still, there it is !)
These plugins are really useful to us, and since we’ve made them open source (Apache 2 license ), here I am giving you a quick introduction to those plugins (source, binaries and docs , hoping they’ll help you too ! (and also hoping you’ll contribute to them if you like them ! ;-) )
They are hosted on Terracotta’s github account , follow their links to get more details about them.

Nexus plugins

If you want to know what are the dependencies of your project, or what depends on your project, your company repository manager is probably the best place to find out; after all, all the binaries are published there; that’s why we chose to develop / contribute to Nexus plugins :

  • Nexus Dependency Management Plugin : a plugin to list the dependencies of a given artifact (and its parent) and also display some metadata such as SCM url, build url (think Jenkins job url) and the maven profiles enabled during the build; you can also use its optional maven plugin companion , the Maven Metadata Properties Plugin that publishes additional metadata in the pom before committing it to the tag, during release.
  • Nexus Dependency Management Plugin

    Nexus Dependency Management Plugin

  • Nexus Artifact Usage Plugin a plugin (from ebay, we mainly contributed to it to make it work with recent Nexus releases) to list the backward dependencies of an artifact; with that plugin you can visualize all the artifacts depending on a given artifact. The index is built in memory at start up and is updated at each new deployment

The Nexus Dependency Plugin functionality is very similar to Sonatype’s Nexus pro Maven Dependency plugin functionality; but the Nexus Dependency Plugin can not only list your artifacts compile dependencies, but also the provided, and test ones (telling you which scope is each dependency in) and can also give you the parent pom of the selected artifact. (and is open source :-) )

Junit Categories Maven Plugin

This maven plugin lists all the Junit tests of a maven module (classes and methods) annotated with a @Category ; sorted by category names.
So for example, if you type in your terminal :

, you should get something like :

Jenkins Accelerated Build Now plugin

When all Jenkins’ slaves are busy, wouldn’t you like to be able to launch the job you’re waiting after right now ? without adding it to a long queue ? without waiting for current jobs to complete ?
If your answer is yes, then the Accelerated Build Now plugin is for you !
The idea behind this plugin is that a user should not wait for automatically scheduled jobs to complete; his/her jobs should be the priority (and in the case the plugin needed to kill a running job scheduled automatically, it re schedules it).

The plugin code is hosted on Github, its binary is available directly from Jenkins update center (Jenkins-> Manage Jenkins -> Manage plugins) , and its wiki page is hosted on Jenkins Wiki

Bonus : Nexus plugins dev tips

Launch Nexus in debug mode

Open wrapper.conf :

Uncomment some lines to have :

Then start nexus with :

And attach your favourite IDE (in the workspace/ project with all the sources matching your nexus version) to port 8000

Test your plugin without deploying it to a local Nexus

It can be pretty boring to re deploy your built nexus plugin (

) to test any change.
From the Nexus team chat room, I got the following tip :
Just add :

to NEXUS_HOME/conf/nexus.properties , so that XmlNexusPluginRepository reloads your plugin at every save ! (provided you don’t change the methods signatures, in that case you will have to restart Nexus)