Last night an suicide attack took place in Manchester killing at least 22 people. I don’t have much to comment on that apart from that everyone’s thoughts are with those who have been injured or lost friends and family to the attack, and to quote a friend of mine:

If you think you can sow disunity in Manchester with a bomb, you don’t know Manchester.

Posted in Uncategorized | 1 Comment

Tracker 💙 Meson

A long time ago I started looking at rewriting Tracker’s build system using Meson. Today those build instructions landed in the master branch in Git!

Meson is becoming pretty popular now so I probably don’t need to explain why it’s such a big improvement over Autotools. Here are some key benefits:

  • It takes 2m37s for me to build from a clean Git tree with Autotools,  but only 1m08s with Meson.
  • There are 2573 lines of files, vs. 5013 lines of, a 2898 line file, and various other bits of debris needed for Autotools
  • Only compile warnings are written to stdout by default, so they’re easy to spot
  • Out of tree builds actually work

Tracker is quite a challenging project to build, and I hit a number of issues in Meson along the way plus a few traps for the unwary.

We have a huge number of external dependencies — Meson handles this pretty neatly, although autodetection of backends requires a bit of boilerplate.

There’s a complex mix of Vala and C code in Tracker, including some libraries that are written in both. The Meson developers have put a lot of work into supporting Vala, which is much appreciated considering it’s a fairly niche language and in fact the only major problem we have left is something that’s just as broken with Autotools: failing to generate a single introspection repo for a combined C + Vala library

Tracker also has a bunch of interdependent libraries. This caused continual problems because Meson does very little deduplication in the commandlines it generates, and so I’d get combinational explosions hitting fairly ridiculous errors like commandline too long (the limit is 262KB) or too many open files inside the ld   process. This is a known issue. For now I work around it by manually specifying some dependencies for individual targets instead of relying on them getting pulled in as transitive dependencies of a declare_dependency target.

A related issue was that if the same .vapi file ends up on the valac commandline more than once it would trigger an error. This required some trickery to avoid. New versions of Meson work around this issue anyway.

One pretty annoying issue is that generated files in the source tree cause Meson builds to fail. Out of tree builds seem to not work with our Autotools build system — something to do with the Vala integration — with the result that you need to make clean before running a Meson build even if the Meson build is in a separate build dir. If you see errors about conflicting types or duplicate definitions, that’s probably the issue. While developing the Meson build instructions I had a related problem of forgetting about certain files that needed to be generated because the Autotools build system had already generated them. Be careful!

Meson users need to be aware that the rpath is not set automatically for you. If you previously used Libtool you probably didn’t need to care what an rpath was, but with Meson you have to manually set install_rpath for every program that depends on a library that you have installed into a non-standard location (such as a subdirectory of /usr/lib). I think rpaths are a bit of a hack anyway — if you want relocatable binary packages you need to avoid them — so I like that Meson is bringing this implementation detail to the surface.

There are a few other small issues: for example we have a Gtk-Doc build that depends on the output of a program, which Meson’s gtk-doc module currently doesn’t handle so we have to rebuild that documentation on every build as a workaround. There are also some workarounds in the current Tracker Meson build instructions that are no longer needed — for example installing generated Vala headers used to require a custom install script, but now it’s supported more cleanly.

Tracker’s Meson build rules aren’t quite ready for prime time: some tests fail when run under Meson that pass when run under Autotools, and we have to work out how best to create release tarballs. But it’s pretty close!

All in all this took a lot longer to achieve than I originally hoped (about 9 months of part-time effort), but in the process I’ve found some bugs in both Tracker and Meson, fixed a few of them, and hopefully made a small improvement to the long process of turning GNU/Linux users into GNU/Linux developers.

Meson has come a long way in that time and I’m optimistic for its future. It’s a difficult job to design and implement a new general purpose build system (plus project configuration tool, test runner, test infrastructure, documentation, etc. etc), and the Meson project have done so in 5 years without any large corporate backing that I know of. Maintaining open source projects is often hard and thankless. Ten thumbs up to the Meson team!

Posted in Uncategorized | Leave a comment

Night Bus: simple SSH-based build automation


My current project at Codethink has involved testing and packaging GCC on several architectures. As part of this I wanted nightly builds of ‘master’ and the GCC 7 release branch, which called for some kind of automated build system.

What I wanted was a simple CI system that could run some shell commands on different machines, check if they failed, and save a log somewhere that can be shared publically. Some of the build targets are obsolete proprietary OSes where modern software doesn’t work out of the box so simplicity is key. I considered using GitLab CI, for example, but it requires a runner written in Go, which is not something I can just install on AIX. And I really didn’t have time to maintain a Jenkins instance.

So I started by trying to use Ansible as a CI system, and it kind of worked but the issue is that there’s no way to get the command output streamed back to you in real time. GCC builds take hours and its test suite can take a full day to run on an old machine so it’s essential to be able to see how things are progressing without waiting a full day for the command to complete. If you can’t see the output, the build could be hanging somewhere and you’d not realise. I discovered that Ansible isn’t going to support this use case and so I ended up writing a new tool: Night Bus.

Night Bus is written in Python 3 and runs tasks across different machines, similarly to Ansible but with the usecase of doing nightly builds and tests as opposed to configuration management. It provides:

  • remote task execution via SSH (using the Parallel-SSH library)
  • live logging of output to a specified directory
  • an overall report written once all tasks are done, which can contain status messages from the tasks
  • parametrization of the tasks (to e.g. build 3 branches of the same thing)
  • a support library of helper functions to make your task scripts more readable

Scheduled execution can be set up using cron or systemd. You can set up a webserver (i’m using lighttpd) on the machine that runs Night Bus to make the log output available over HTTP

You control it by creating two YAML (or JSON) files:

  • hosts describes the SSH configuration for each machine
  • tasks lists the sequence of tasks to run

Here’s an example hosts file:

  user: automation
  private_key: ssh/automation.key

  proxy_user: jenny
  proxy_private_key: ssh/jenny.key

Here’s an example tasks file:

- name: counting-example
  commands: |
    echo "Counting to 20."
    for i in `seq 1 20`; do
      echo "Hello $i"
      sleep 1

You might wonder why I didn’t just write a shell script to automate my builds as many thousands of hackers have done in the past. Basically I find maintaining shell scripts over about 10 lines to be a hateful experience. Shell is great as a “live” programming environment because it’s very flexible and quick to type. But those strengths turn into weaknesses when you’re trying to write maintainable software. Every CI system ultimately ends up with you writing shell scripts (or if you’re really unlucky, some XML equivalent) so I don’t see any point hiding the commands that are being run under layers of other stuff, but at the same time I want a clear separation between the tasks themselves and the support aspects like remote system access, task ordering, and logging.

Night Bus is released as a random GitHub project that may never get much in the way of updates. My aim is for it to fall into the category of software that doesn’t need much ongoing work or maintenance because it doesn’t try to do anything special. If it saves one person from having to maintain a Jenkins instance then the week I spent writing it will have been worthwhile.

Posted in Uncategorized | 4 Comments

GUADEC call for talks ends this Sunday, 23rd April

GUADEC 2017 is just over three months away, which is a very long time in the future and leaves lots of time to organise everything (at least that’s what I keep telling myself).

However, the call for papers is closing this Sunday so if you have something you want to talk about in front of the GNOME community and you haven’t already submitted a talk then please head to the registration site and do so!

Once the call for papers closes, the Papers Committee will fetch their ceremonial robes and make their way to a cave deep in the Peak District for two weeks. There they will drink fresh spring water, hunt grouse on the moors and study your talk submissions in great detail. When two weeks is up, their votes are communicated back to Manchester using smoke signals and by Sunday 7th May you’ll be notified by email if your talk was accepted. From there we can organise travel sponsorship and finalize the schedule of the first 3 days of the conference, which should be available late next month.

We’ll put a separate call out for BoF sessions, workshops, and tutorial sessions to take place during the second half of GUADEC — the 23rd April deadline only applies to talks.

Posted in Uncategorized | 1 Comment

GUADEC accommodation

At this year’s GUADEC in Manchester we have rooms available for you right at the venue in lovely modern student townhouses. As I write this there are still some available to book along with your registration. In a couple of days we have to give final numbers to the University for how many rooms we want, so it would help us out if all the folk who want a room there could register and book one now if you haven’t already done so! We’ll have some available for later booking but we have to pay up front for them now so we can’t reserve too many.

Rooms for sponsored attendees are reserved separately so you don’t need to book now if your attendance depends on travel sponsorship.

If you are looking for a hotel, we have a hotel booking service run by Visit Manchester where you can get the best rates from various hotels right up til June 2017. (If you need to arrive before Thursday 27th July then you can to contact Visit Manchester directly for your booking at

We have had some great talk submissions already but there is room for plenty more, so make sure you also submit your idea for a talk before 23rd April!

Posted in Uncategorized | 2 Comments

Ten years of Codethink

32813704624_b7e3899b9f_zSpring is here and it is the 10th anniversary celebration of Codethink.  Nobody could have orchestrated it this way but we also have GUADEC happening here in Manchester in a few months and it’s the 20th anniversary of GNOME.  All roads lead to Manchester in 2017!

The company is celebrating its anniversary in various ways: cool new green-on-black T-shirts, a 10 years mug for everyone, and perhaps more significantly a big sophisticated party with a complicated cake.

The party was fun with a lot of old faces some who had travelled quite far to be there. The company was and still is a mix of very interesting and weird people and although we spend most of our time in the same room studiously not talking to eachother we do know how to celebrate things sometimes!

It was odd in a way being at a corporate party with fancy food and a function band and 150 guests in an enourmous monastery given that back when I joined the entire Manchester staff could go for lunch together and all sit at the same table. The first company party I went to was in Paul Sherwood’s conservatory, in fact the first few of them were there. It’s a good sign for sure that the company has quadrupled (or more) in size in the ensuing 6 years.

In hindsight I was quite lucky to have a world class open source software house more or less on my doorstep. I spent a long time trying to avoid working in software (and trying to avoiding working at all), but I did do a Summer of Code project back in 2009 or 2010 mentored by Allison Lortie, who then worked for Codethink and noted that I lived about 5000 miles closer to her office than she did.It was an obvious choice to apply to there when I graduated from University and luckily it was just at a time when they were hiring so I didn’t have to spend too long living on 50p a week and eating shoes for dinner. It was very surreal for the first few months of working there as a world which I’d previously only been involved via a computer turned into a world of real people (plus lots of computers), in fact the whole first year was pretty surreal what with also adapting to Manchester life and discovering the how much craziness there is underneath the surface of the technology industry.

I had no idea what the company did beforehand, and even now the Codethink website doesn’t give too much away. I saw contributions to Free Software projects such as Tracker and dconf (and various other things that were happening 7 years ago) but I didn’t know what kind of business model came out of that activity. It turned out that neither did anyone else at that point; the company grew out of consulting work from Nokia, but the Elopcalypse had just happened and so on starting I got involved in all sorts of different things as we looked for work in different areas: everything from boot speed optimizations and hardware control, to compiler testing and bugfixing, build tools, various automated testing setups, and more build tools, to Python and Ruby webapps, data visualisations, OpenStack, systems administration, report writing and more. Just before Christmas 2011 I got offered to go work in Korea, the catch being that I had to go in 2 days time, and the following year I spent another memorable month there (again with about 2 days notice). I also had month long stints in Bulgaria, and Berlin although these were actually planned in advance, plus all sorts of conferences as the company started to sponsor attendance and a couple of days off for such things. Most importantly of course I got involved in rock climbing which is now pretty much my favourite thing.

Since a long time now it’s felt like the company has a solid business model and while the work we do is still all over different sectors I think I can sum it up as bridging the gap between the worlds of corporate software projects and open-source software projects.  We have some great customers who engage us to do work upstream on Free Software projects which is ideal,  but far from everything we work on is Free Software, and we also work in various fields that I’m pretty unexcited about such as automotive and finance. It’s very hard to make money though if you  spend all your time working on something that you then give away so it’s a necessary compromise in my eyes.

And even in entirely closed source projects having knowledge of all the great Free Software that is available gives us an advantage. There are borderline-unusable proprietary tools still being sold by major vendors to do things like version control, there are unreliable proprietary hardware drivers being sold for hardware that has a functional and better open source driver, there are countless projects using medieval kernels, obsolete operating systems and all sorts of other craziness.Working for a company that trusts its employees is also pretty important, I meet operating systems engineers there are working on Linux-based devices whose corporate IT departments force them to use Windows, so right they trust them to maintain the operating system used in millions of cars but they don’t trust them to maintain the operating system on their laptop.

One thing Codethink lacks still is a model for providing engineer time to help with ongoing maintainance and development of different free software projects. There have been attempts at doing so within the company and I acknowledge it’s very difficult because the drop in, drop out nature of a consultant engineer’s time isn’t compatible with the ongoing time commitment required to be a reliable maintainer. Plus good maintenance skills require years to develop and either require someone experienced with a lot of free time to teach them to you, or they require you to maintain a real world project which you mess up continually and learn every lesson the hard way. Of course open source work that comes out of customer projects is highly regarded and if you’re lucky enough to have unallocated time it can sometimes be used to work through the backlog of bug fixes and feature additions for different tools you use that one inevitably develops as a full time software engineer. Again, it amazes me how many companies manage to actively prevent their developers from pushing things upstream.

We have been maintaining Baserock for years now (and many people have learned lots of lessons the hard way from it :-); BuildStream development is ongoing and I’m even still hopeful we can achieve the original goal of making it an order of magnitude easier to produce a high quality Free Software operating system. I should note that Codethink also contributes financially to conferences and projects in various ways.

I should also point out that we are still hiring. This wasn’t intended to be a marketing essay in which I talked up how great the company is, but it kinda has turned out out that way. I guess you take that as a good sign. My real underlying goal was to make it a bit clearer what it’s like to work here which I hope I’ve done a little.

I am quite proud of the company’s approach to hiring, we take in many graduates who show promise but never got involved in community-driven software projects or never really even got into programming except as a module in a science degree or whatever. Of course we also welcome people who do have relevant experience but they can be hard to find and focusing on them can also have an undesired effect of selecting based on certain privileges. I was debating with Tristan last week whether a consultancy is actually a good place for inexperienced developers to be, there is the problem that you don’t get to see the results of your work very often, you often move between projects fairly frequently and so you might not develop the intuition needed for being a good software maintainer, which is a complex topic but boils down to something like: “Is this going to cause problems in 5 years time?” There’s no real way around this, all we can do is give people a chance in an environment with a strong Free Software culture and that is pretty much what we do.

Ideally here I’d end with some photos from the party but I’m terrible at taking photos so it’s just all the back of people’s heads and lurid green lighting. Instead here’s a photo of a stranger taking a photo of me this afternoon while I was out biking round the river Mersey this afternoon.


Cake photo by Robert Marshall

Posted in Uncategorized | 2 Comments

GUADEC 2017: Friday 28th July to Wednesday 2nd August in Manchester, UK

I'm going to GUADEC 2017

The GUADEC 2017 team is happy to officially announce the dates and location of this year’s conference.

GUADEC 2017 will run from Friday 28th July to Wednesday 2nd August. The first three days will include talks and social events, as well as the GNOME Foundation’s AGM. This part of the conference will also include a 20th anniversary celebration for the GNOME project.

The second 3 days (from Monday 31st July to Wednesday 2nd August) are unconference-style and will include space for hacking, project BoF sessions and possibly training workshops.

The conference days will be at Manchester Metropolitan University’s Brooks Building. The unconference days will be in a nearby University building named The Shed.

Registration and a call for papers will be open later this month. More details, including travel and accommodation tips, are available now at the conference website:

We are interested in running training workshops on Monday 31st July but nothing is planned yet. We would like to hear from anyone who interested in helping to organise a training workshop.

Inside view of MMU Brooks Building

Inside view of MMU Brooks Building


Posted in Uncategorized | 2 Comments