How BuildStream uses OSTree

I’ve been asked a few times about the relationship between BuildStream and OSTree. The answer is a bit complicated so I decided to answer the question here.

OSTree is a content-addressed content store, inspired in many ways by Git but optimized for storing trees of binary files rather than trees of text files.

BuildStream is an integration tool which deals with trees of binary files, and at present it uses OSTree to help with storing, identifying and transferring these trees of binary files.

I’m deliberately using the abstract term “trees of binary files” here because neither BuildStream or OSTree limit themselves to a particular use case. BuildStream itself uses the term “artifact” to describe the output of a build job and in practice this could be the set of development headers and documentation for library, a package file such as a .deb or .rpm, a filesystem for a whole operating system, a bootable VM disk image, or whatever else.

Anyway let’s get to the point! There are actually four ways that BuildStream directly makes use of OSTree.

The `ostree` source plugin

The `ostree` source plugin allows pulling arbitrary data from a remote OSTree repository. It is normally used with an `import` element as a way of importing prebuilt binaries into a build pipeline. For example BuildStream’s integration tests currently run on top of the Freedesktop SDK binaries (which were originally intended for use with Flatpak applications but are equally useful as a generic platform runtime). The gnome-build-meta project uses this mechanism to import a prebuilt Debian base image, which is currently manually pushed to an OSTree repo (this is a temporary measure, in future we want to base gnome-build-meta on top of the upcoming Freedesktop SDK 1.8 instead).

It’s also possible to import binaries using the `tar` and `local` source types of course, and you can even use the `git` or `bzr` plugins for this if you really get off on using the wrong tools for the wrong job.

In future we will likely add other source plugins for importing binaries, for example from the Docker Registry and perhaps using casync.

Storing artifacts locally

Once a build has completed, BuildStream needs to store the results somewhere locally. The results go in the exciting-sounding “local artifact cache”, which is usually located inside your home directory at ​~/.cache/buildstream/artifacts.

There are actually two implementions of the local artifact cache, one using OSTree and one using .tar files. There are several advantages to the OSTree implementation, a major one being that it deduplicates files that are present in multiple artifacts which can save huge amounts of disk space if you do many builds of a large component. The biggest disadvantage to using OSTree is that it currently relies on a bunch of features that are specific to the Linux kernel and so it can only run on Linux OSes. BuildStream needs to support other UNIX-like operating systems and we found the simplest route for now to solve this was to implement a second type of local artifact cache which stores each artifact as a separate .tar file. This is less efficient in terms of disk space but much more portable.

So the fact that we use OSTree for caching artifacts locally should be considered an implementation detail of BuildStream. If a better tool for the job is found then we will switch to that. The precise structure of the artifacts should also be considered an internal detail — it’s possible to check artifacts out from the cache by poking around in the  ​~/.cache/buildstream/artifacts directory but there’s no stability guarantee in how you do this or what you might get out as a result. If you want to see the results of a build, use the `bst checkout` command.

It’s worth noting that we don’t yet support automated cleanups of the local artifact cache; that is issue #135.

Storing artifacts remotely

As a way of saving everyone from building the same things, BuildStream supports downloading prebuilt artifacts from a remote cache.

Currently the recommended way of setting up a remote artifact cache requires that you use OSTree. In theory, any storage mechanism could be used but that is currently not trivial because we also make use of OSTree’s transfer protocols, as described below.

We currently lack a way to do automated artifact expiry on remote caches.

Pushing and pulling artifacts

Of course there needs to be a way to push and pull artifacts between the local cache and the remote cache.

OSTree is designed to support downloading artifacts over HTTP or HTTPS and this is how `bst pull` works. The `bst push` command is more complex because officially OSTree does not support pushing, however we have a rather intricate push mechanism based off Dan  Nicholson’s ostree-push project which tunnels the data over SSH in order to get it onto the remote server.

Users of the tar cache cannot currently interact with remote artifact shares at all, which is an unfortunate issue that we aim to solve this year. The solution may be to switch away from using OSTree’s transfer protocols but marshalling the data into some other format in order to transfer it instead. We are particularly keen to make use of the Bazel
content-addressable store protocol although there may be too much of an impedence mismatch there.

Indirect uses of OSTree

It may be that you also end up deploying stuff into an OSTree repository somewhere. BuildStream itself is only interested with building and integrating your project — once that is done you run `bst checkout` and are rewarded with a tree of files on your local machine. What if, let’s say, your project aims to build a Flatpak application?

Flatpak actually uses OSTree as well and so your deployment step may involve committing those files into yet another OSTree repo ready for Flatpak to run them. (This can be a bit long winded at present so there will likely be some better integration appearing here at some point).

So, is anywhere safe from the rise of OSTree or is it going to take over completely? Something you might not know about me is that I grew up outside a town in north Shropshire called Oswestry. Is that a coincidence? I can’t say.



Oswestry, from Wikipedia.


About Sam Thursfield

Who's that kid in the back of the room? He's setting all his papers on fire! Where did he get that crazy smile? We all think he's really weird.
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.