Using BuildStream through Docker

BuildStream isn’t packaged in any distributions yet, and it’s not entirely trivial to install it yourself from source. BuildStream itself is just Python, but it depends on a modern version of OSTree (2017.8 or newer at time of writing), with the GObject introspection bindings, which is a little annoying to have to build yourself1.

So we have put some work into making it convenient to run BuildStream inside a Docker container. We publish an image to the Docker hub with the necessary dependencies, and we provide a helper script named bst-here that sets up a container with the current worked directory mounted at /src and then runs a BuildStream command or an interactive shell inside it. Just download the script, read it through and run it: all going well you’ll be rewarded with an interactive Bash session where you can run the bst command. This allows users on any distro that supports Docker to run BuildStream builds in a pretty transparent way and without any major performance limitations. It even works on Mac OS X!

In order to run builds inside a sandbox, BuildStream uses Bubblewrap. This requires certain kernel features, in particular CONFIG_USER_NS which right now is not enabled by default in Arch and possibly in other distros. Docker containers run against the kernel of the host OS so it doesn’t help with this issue.

The Docker images we provide are based off Fedora and are built by GitLab CI from this repo. After a commit to that repo’s ‘master’ branch, a new image wends its way across hyperspace from GitLab to the Docker hub. These images are then pulled when the bst-here wrapper script calls docker run. (We also use these images for running the BuildStream CI pipelines).

More importantly, we now have a mascot now! Let me introduce the BuildStream beaver:

Beavers, of course, are known for building things in streams, are also native to Canada. This isn’t going to be the final logo, he’s just been brought in as we got tired of the project being represented by a capital letter B in a grey circle. If anyone can contribute a better one then please get in touch!

So what can you build with BuildStream now that you have it running in Docker? As recently announced, you can build GNOME! Follow this modified version of the newcomer’s guide to get started. Soon you will also be able to build Flatpak runtimes using the rewritten Freedesktop SDK; or build VM images using Baserock; and of course you can create pipelines for your own projects (although if you only have a few dependencies, using Meson subprojects might be quicker).

After one year of development, we are just a few blockers away from releasing BuildStream 1.0. So it is a great time to get involved in the project!

[1]. Installing modern OSTree from source is not impossible — my advice if you want to avoid Docker and your distro doesn’t provide a new enough OSTree would be to build the latest tagged release of OSTree from Git, and configure it to install into /opt/ostree. Then put something like export GI_TYPELIB_PATH=/opt/ostree/lib/girepository-1.0/ in your shell’s startup file. Make sure you have all the necessary build dependencies installed first.

Advertisements
Posted in Uncategorized | 5 Comments

Manchester transport consulation

Manchester council is running a transport survey at the moment.

There’s nothing I like more than ranting into pointless online forms, and here’s an extract from my response:

This lane (from Mad Cycle Lanes of Manchester) is a favourite.

Manchester is not the worst city to cycle in due to having largely quite
wide roads, but cycle infrastructure is almost always an afterthought
with many issues. Most cycle lanes pop in and out of existence, forcing
cyclists into the road at dangerous points. For example London Road
going south just past the A57(m) overpass has a cycle lane which
suddenly ends forcing cyclists to pull out in front of fast-moving
traffic that has just come off a motorway. Cycle lanes sometimes
coexist with tram lines, which is very dangerous as mountain bike wheels
are just wide enough to get stuck in the tram lines. When wet, tram
lines are also very slippery which makes them dangerous to cross on a
bike. I have been injured twice as a result of cycle lanes that cross
tram lines. There are also cycle lanes which go onto narrow stretches of
pavement, for example the corner by London Road Fire Station and
Munroe’s Hotel which is painted to look like a cycle lane but is also
clearly a walk way and is too narrow to function as both. In some places
there are cycle lanes which are rendered useless by cars parking in them
or next to them.

Walking should be the main mode of transport to get around the city. At
the moment walking around takes longer than it should because so much
time is spent waiting at traffic lights.

I actually don’t mind cycling round the city too much, and the traffic jams are actually great for cycle safety because cars generally don’t hit you if they aren’t going anywhere. But the absurdity of our existing cycle infrastructure needs to be recognised.

There are more great examples from round the country here and here.

Posted in Uncategorized | 1 Comment

GUADEC 2017: timeline

After the statistics perhaps you are interested in reading a timeline of GUADEC 2017! In particular you can compare it to the burn down chart from the GUADEC HowTo and see how that interacts with reality.

Of course lots of details are excised from this overview but it gives a general sense of the timings. In some follow up posts I’ll go in more detail about what I think went well and what didn’t. We also welcome your feedback on the event (if you can still remember it 🙂

Summer 2014: At some point during GUADEC 2014 I start going on about doing a Manchester edition.

August 2015: Alberto and Allan both float the idea of doing a Manchester bid with me; it seems like there’s just about enough of a team to go for it. I was already planning to be away in summer 2016 at this point so we decided to target 2017.

Alberto has a friend working at MIDAS who gives us a good start and we end up meeting with the Marketing Manchester conference bureau, the University of Manchester and Manchester Metropolitan University.

The meeting with University of Manchester was discouraging (to be honest, they seemed to be geared up only for corporate conferences rather than volunteer-driven events) but Manchester Metropolitan were much more promising.

Winter 2015: We lost touch with MMU for a few months (presumably as University started back up), but we eventually got a proper contact in the conferences department and started moving forwards with the bid.

Spring 2016: Our bid is produced, with Marketing Manchester doing most of the content and layout (as you might be able to tell). Normally I would worry to see only one GUADEEC bid on the table but, having been thinking about our bid for almost a year already I was also glad that it looked like we’d be the main option.

Summer 2016: GUADEC 2016 in Karlsruhe; Manchester is selected as the location for 2017. Much rejoicing (although I am on a 9000 mile road trip at the time).

August 2016: Talks begin with venue drawing up contracts for venue and accommodation. The venue was reasonably painless to sort out but we spent lots of time figuring out accommodation; the University townhouses required final numbers and payment 6 months in advance of the event, so we spent a lot of time looking into other options (but ended up deciding that the townhouses would be best even though we would inevitably lose a bit money on them).

September 2016: We begin holding monthly-ish meetings with myself, Alberto, Allan and Javier present. Work begins on sponsorship brochure (which complicated by needing to coordinate with GNOME.Asia and potentially LAS), talks continue with venue.

December 2016: Contracts finally signed for venue and accommodation (4 months later!), conference dates finalized. We apply for a UK bank account as an “unincorporated association”. Discussion begins about the website, we decide to hold off on announcing the dates until we have some kind of website in place.

January 2017: Basic website finished, dates announced. Lots of work on getting the registration system ready. We begin meeting each week on a Monday evening. Initial logo made by Jakub and Allan.

February 2017: Trip to FOSDEM, where we put up a few GUADEC posters. Summer still seems a long way off. Codethink sponsorship confirmed. We start thinking about keynote speakers. Javier and Lene look into social event venues, including somewhere for the 20th birthday party(with hearts already set on MOSI). The search for new Executive Director for GNOME finally comes to a close with Neil McGovern being hired, and he soon starts joining the GUADEC calls and helping out (in particular with the search for sponsors, which up til now has been nearly all Alberto’s work).

March 2017: After 4 months of bureaucracy, our bank account finally approved. After much hacking and design work, we can finally open registration and the call for papers. We have to finalize room numbers at the University already, although most rooms are still unbooked. Investigation into getting GNOME Beer brewed (which ended up going nowhere, sadly). Requests for visa invites begin to arrive.

April 2017: Lots of planning for social events, the talk days and the unconference days. PIA sponsorship confirmed. Posters being designed. Call for papers closes, voting begins and Kat starts putting together the talks schedule.

May 2017: Birthday planning with help from the engagement team (in particular Nuritzi). The University temporarily decide that we’ll have to pay staff costs of £500 per day to have the canteen open; we do a bunch of research into alternatives but then we go back to the previous agreement of having the canteen open with just a minimum spend. Planning of video recording and design. Schedule and social events planning.

June and July 2017: Continual planning and discussion of everything. More sponsors confirmed. Allan does prodigious amounts of graphic design and organizing printing. Travel sponsorship finally confirmed and lots of visa invitation requests start to arrive. Accommodation bookings continue to come in, along with an increasing amount of queries, changes and cancellations that become quite time-consuming to keep track of and respond to. Evening events being booked and finalized, including more planning of the birthday party with Nuritzi. Discussions of how to make sure the conference is inclusive to newcomers. Water bottles, cake and T-shirts ordered. Registrations keep coming in until we actually hit and go over 200 registrations. We contact volunteers and come up with a timetable.

Finally, the day before GUADEC we collect the last of the printing, bring everything to the venue and hole up in a room on the 2nd floor ready to pre-print names on badges and stuff the lanyard pouches with gift bags. We discover two major issues: firstly the ink on the badges gets completely smudged when we run it through the printer to print a name on it; and secondly the emergency telephone number that we’ve printed on the badges has actually been recycled as the SIM card was inactive for a while and now goes through to some poor unsuspecting 3rd party.

guadec-badges.jpgWe lay out all the badges to try and dry the ink out but 3 hours later the smudging is still happening. We realise that the names will just have to be drawn on with marker pens. As for the emergency telephone… if you look closely at a GUADEC 2017 badge you’ll notice that there’s a sticky label with the correct number covering up the old number on the badge. Each one of these was printed onto stickyback paper and lovingly chopped out and stuck on by hand. You’re welcome! (Nobody actually called the emergency phone during the event).

Javier pointed out that we should be at the registration event at least an hour early (it started at 18:00). I said this was nonsense because most people wouldn’t get there til later anyway. How wrong I was !!! I’m used to organizing music events where people arrive about an hour after you tell them to, but we got to Kro Bar about 17:45 and it was already full to bursting with eager GNOME contributors, many of whom of course hadn’t seen each other for months. This was not the ideal environment to try and set up a registration desk for the first time and I mostly just stood around looking at boxes feeling confused and occasionally moving things around. Thankfully Kat and Benjamin soon arrived and made registration a reality leaving me free to drink a beer and remain confused.

And the rest is history!

Posted in Uncategorized | Leave a comment

GUADEC 2017 by numbers

I’m finally getting around to doing a bit of a post-mortem for the 2017 edition of GUADEC that we held in Manchester this year. Let’s start with some statistics!

GUADEC 2017 had…

  • 264 registrations (up from 186 last year)
  • 209 attendees (up from 160 last year)
  • 72 people staying at the University (30 of whom had sponsorship awarded by the travel committee)
  • 7 people who were sadly unable to attend because their visa application was refused at the last minute

We put four optional questions on the registration form asking for your country of residence, your age, your gender identity and how you first heard about GUADEC. The full set of responses (anonymous, of course) is available here.

I don’t plan to do much data mining of this, but here are some interesting stats:

  • 61 attendees said they are resident in the UK, roughly 32%.
  • The most common age of attendees was 35 (the full age range was between 11 years and 65 years)
  • 14 attendees said they heard about the conference through working at Codethink

We asked for an optional, “pay as you feel” donation towards the costs of the conference at registration time and we suggested payments of £15/€15 for students, £40/€40 for hobbyists and £150/€150 for professionals.

  • 47 attendees (22%) chose to donate nothing
  • 29 attendees (13%) chose 1-15
  • 75 attendees (36%) chose 16-40
  • 51 attendees (24%) chose >40
  • 7 attendees somehow chose “NULL” (I think these were on-site registrations, which followed a different process)

Note that we told Codethink staff that they shouldn’t feel required to donate from their company-provided conference budget as Codethink was already sponsoring at Platinum level, which should account for 15 or more of the people who chose to donate nothing with their registration.

The financial side of things is tricky for me to summarize as the sponsor money and registration donations mostly went straight to the Foundation’s bank account, which I don’t have access to. The fluctionation of GBP against the US dollar makes my own budget spreadsheet even less reliable,but I estimate that we raised around $10,000 USD for the GNOME Foundation from GUADEC 2017. This is of course only possible due to the generosity of our sponsors, and through the great work that Alberto and Neil did in this area.

My van did 94 miles around Manchester during the week of GUADEC. My house is only 4 miles from the centre so this is surprisingly high!

 

Posted in Uncategorized | 1 Comment

BuildStream and host tools

It’s been a while since I had to build a whole operating system from source. I’ve mostly been working on compilers so far this year at Codethink in fact, but my new project is to bring up some odd target systems that aren’t supported by any mainstream distros.

We did something similar about 4 years ago using Baserock and it worked well; this time we are using the Baserock OS definitions again but with BuildStream as a build tool. I’ve not had any chance to get involved in BuildStream up til now (beyond observing it) so this will be good.

The first thing I’m getting my head around is the “no host tools” policy. The design of BuildStream is that every build is run in a sandbox that’s isolated from the host. Older Baserock tools took a similar approach too and it makes a lot of sense: it’s a lot easier to maintain build instructions if you limit the set of environments in which they can run, and you are much more likely to be able to reproduce them later or on other people’s machines.

However your sandbox is going to need a compiler and a shell environment in there if it’s going to be able to build anything, and BuildStream leaves open the question of where those come from. It’s simple to find a prebuilt toolchain at least for mainstream architectures — pretty much every Linux distro can provide one so the only question is which one to use and how to get it into BuildStream’s sandbox?

GNOME and Freedesktop base runtime and SDK

The Flatpak project has a similar need for a controlled runtime and build environment, and is producing a GNOME SDK, and a lower level Freedesktop SDK. These are at present built on top of Yocto.

Up to date versions of these are made available in an OSTree repo at http://sdk.gnome.org/repo. This makes it easy to import them into BuildStream using an ‘import’ element and the ‘ostree’ source:

kind: import
description: Import the base freedesktop SDK
config:
  source: files
  target: usr
host-arches:
  x86_64:
    sources:
      - kind: ostree
        url: gnomesdk:repo/
        track: runtime/org.freedesktop.BaseSdk/x86_64/1.4
        gpg-key: keys/gnome-sdk.gpg
        ref: 0d9d255d56b08aeaaffb1c820eef85266eb730cb5667e50681185ccf5cd7c882
  i386:
    sources:
      - kind: ostree
        url: gnomesdk:repo/
        track: runtime/org.freedesktop.BaseSdk/i386/1.4
        gpg-key: keys/gnome-sdk.gpg
        ref: 16036b747c1ec8e7fe291f5b1f667cb942f0267d08fcad962e9b7627d6cf1981

The main downside to using these is that they are pretty large — the GNOME 3.18 SDK weighs in at 1.5 GB uncompressed and around 63,000 files. Creating a hardlink tree using `ostree checkout` takes up to a minute on my (admittedly rather old) laptop. The Freedesktop SDK is smaller but still not ideal. They are also only built for a small set of architectures — I think just some x86 and ARM families at the moment.

Debian in OSTree

As part of building GNOME’s jhbuild modulesets inside BuildStream Tristan created a script to produce Debian chroots for various architectures and commit them to an OSTree repo. The GNOME components are then built on top of these base Debian images, with the idea that in future they can be tested on top of a whole variety of distros in addition to Debian to make us catch platform-specific regressions more quickly.

The script, which uses the awesome Multistrap tool to do most of the heavy lifting, lives here and pushes its results to a repo that is temporarily housed at https://gnome7.codethink.co.uk/repo/ and signed with this key.

The resulting sysroot are 2.7 GB in size with 105,320 different files. This again takes up to a minute to check out on my laptop. Like the GNOME SDK, this sysroot contains every external dependency of GNOME which adds up to a lot of stuff.

Alpine Linux Toolchain

I want a lighter weight set of host tools to put in my build sandbox. Baserock’s OS images can be built with just a C++ toolchain and a minimal shell environment, so there’s no need to start copying gigabytes of dependencies around.

Ultimately the Baserock project could build its own set of host tools, but to save faff while prototyping things I decided to try Alpine Linux, which is a minimal distribution.

Alpine Linux provide “mini root filesystem” tarballs. These can’t be used directly as they contain device nodes (so require privileges to extract) and don’t contain a toolchain.

Here’s how I produced a workable host tools sysroot. I’m using Bubblewrap (the same tool used by BuildStream to create build sandboxes) as a simple container driver to run the `apk` package tool as root without needing special host privileges. This won’t work on every OS; you can use something like Docker or plain old `chroot` instead if needed.

wget https://nl.alpinelinux.org/alpine/v3.6/releases/x86_64/alpine-minirootfs-3.6.1-x86_64.tar.gz
mkdir -p sysroot
tar -x -f alpine-minirootfs-3.6.1-x86_64.tar.gz -C sysroot --exclude=./dev

alias alpine_exec='bwrap --unshare-all --share-net --setenv PATH /usr/bin:/bin:/usr/sbin:/sbin  --bind ./sysroot / --ro-bind /etc/resolv.conf /etc/resolv.conf --uid 0 --gid 0'
alpine_exec apk update
alpine_exec apk add bash bc gcc g++ musl-dev make gawk gettext-dev gzip linux-headers perl e2fsprogs mtools

tar -z -c -f alpine-host-tools-3.6.1-x86_64.tar.gz -C sysroot .

This produces a 219MB host tools sysroot containing 11,636 files. This is not as minimal as you can go with a GNU C/C++ toolchain but it’s around the right order of magnitude and it checks out from BuildStream’s artifact store into the build directory in a matter of seconds.

We include gawk as it is needed during the GCC build (BusyBox awk is not enough), and gettext-dev is needed by GLIBC (at least, libintl.h is needed and in Alpine only gettext provides that header). Bash is needed by scripts/config from linux.git, and bc, GNU gzip, linux-headers and Perl are also needed for building Linux. The e2fsprogs and mtools are useful for creating disk images.

I’ve integrated this into my builds in a pretty lazy way for now:

kind: import
description: Import an Alpine Linux C/C++ toolchain
host-arches:
  x86_64:
    sources:
    - kind: tar
      url: file:///home/sam/src/buildstream-bootstrap/alpine-host-tools-3.6.1-x86_64.tar.gz
      base-dir: .
      ref: e01d76ef2c7e3e105778e2aa849a42d38dc3163f8c15f5b2de8f64cd5543cf29

This element is obviously not something I can share with others — I’d need to upload the tarball somewhere or set up a public OSTree repo that others could pull from, and then have the element reference that.

However, this is just the first step towards some much deeper work which will result in me needing to move beyond Alpine in any case. In future I hope that it’ll be pretty straightforward to obtain a minimal toolchain as a sysroot that can be pulled into a sandbox using OSTree. The work required to produce such a thing is simple enough to automate but it requires a server to host the binaries which then requires ongoing maintenance for security updates, so I’m not yet going to commit to doing it …

Posted in Uncategorized | Leave a comment

GUADEC Talks Schedule Now Available

As mentioned on the GUADEC 2017 blog, we’ve just published the talks schedule for this year’s edition.

Thanks to everyone who submitted a talk this year, we think there’s a fantastic mix of interesting topics on each day. We sadly had to make some tough decisions and turn down some great submissions as well – we received around 70 talk submissions this year which is a lot to fit into a 3 day, 2 track conference.

It’s now less than 6 weeks until the conference starts! If you’re planning on attending and haven’t yet registered, please register now at registration.guadec.org. It will be possible to register on the day, but we have to finalize room allocations, party venues and food orders way in advance so it will cause problems if everyone waits until the last minute to sign up.

If you’ve not yet booked a room, we have a number of rooms still available at the conference venue. You can book these when you register, or if you already registered then just log into registration.guadec.org and click ‘Edit registration’.

We also have hotel rooms available at fixed prices through Visit Manchester’s hotel booking service. The deadline for booking these rooms is Thursday 29th June.

Thanks to everyone who has opted to volunteer during the conference using the “Keep me informed” box on the registration form. We will be in touch with you very soon to discuss how you can get involved. There’s still time to edit your registration to tick this box if you’ve decided you want to help out on the day!

Posted in Uncategorized | 2 Comments

Manchester

Last night an suicide attack took place in Manchester killing at least 22 people. I don’t have much to comment on that apart from that everyone’s thoughts are with those who have been injured or lost friends and family to the attack, and to quote a friend of mine:

If you think you can sow disunity in Manchester with a bomb, you don’t know Manchester.

Posted in Uncategorized | 1 Comment