Status update, 16/05/2023

I am volunteering a lot of time to work on testing at the moment. When you start out as a developer this seems like the most boring kind of open source contribution that you can do. Once you become responsible for maintaining existing codebases though it becomes very interesting as a way to automate some of the work involved in reviewing merge requests and responding to issue reports.

Most of the effort has been on the GNOME OpenQA tests. We started with a single gnomeos test suite, which started from the installer ISO and ran every test available, taking around 8 minutes to complete. We now have two test suites: gnome_install and gnome_apps.

The gnome_install testsuite now only runs the install process, and takes about 3 minutes. The gnome_apps testsuite starts from a disk image, so while it still needs to run through gnome-initial-setup before starting any apps, we save a couple of minutes of execution. And now the door is open to expand the set of OpenQA tests much more, because during development we can choose to run only one of the testsuites and keep the developer cycle time to a minimum.

Big thanks to Jordan Petridis for helping me to land this change (I didn’t exactly get it right the first time), and to the rest of the crew in the #gnome-os chat.

I don’t plan on adding many more testsuites myself. The next step is to teach the hardworking team of GNOME module maintainers how to extend the openQA test suites with tests that are useful to them, without — and this is very important — without just causing frustration when we make big theming or design changes. (See a report from last month when Adwaita header bars changed colour).

Hopefully I can spread the word effectively at this year’s GUADEC in Riga, Lativia. I will be speaking there on Thursday 27th July. The talk is scheduled at the same time as a very interesting GTK status update so I suppose the talk itself will be for a few dedicated cats. But there should be plenty of time to repeat the material in the bar afterwards to anyone who will listen.

My next steps will be around adding an OpenQA test suite for desktop search – something we’ve never properly integration tested, which works as well as it does only because of hard working maintainers and helpful downstream bug reports. I have started collecting some example desktop content which we can load and index during the search tests. I’m on the lookout for more, in fact I recently read about the LibreOffice “bug documents” collection and am currently fetching that to see if we can reuse some of it.

One more cool thing before you go – we now have a commandline tool for checking openQA test status and generating useful bug reports. And we have a new project-wide label 9. Integration test failure to track GNOME bugs that are detected by the OpenQA integration tests.

〉utils/pipeline_report.py --earlier=1                                   05/16/2023 05:24:30 pm
Latest gnome/gnome-build-meta pipeline on default branch is 528262. Pipeline status: success
Pipeline 1 steps earlier than 528262 is 528172. Pipeline status: success

Project:

  * Repo: gnome/gnome-build-meta
  * Commit: dc71a6791591616edf3da6c757d736df2651e0dc
  * Commit date: 2023-05-16T13:20:48.000+02:00
  * Commit title: openqa: Fix up casedir

Integration tests status (Gitlab):

  * Pipeline: https://gitlab.gnome.org/gnome/gnome-build-meta/-/pipelines/528172
  * test-s3-image job: https://gitlab.gnome.org/gnome/gnome-build-meta/-/jobs/2815294
  * test-s3-image job status: failed
  * test-s3-image job finished at: 2023-05-16T14:46:54.519Z

Integration tests status (OpenQA):

  * gnome_apps testsuite - job URL: https://openqa.gnome.org/tests/1012
    * 3/31 tests passed
    * Failed: gnome_desktop
  * gnome_install testsuite - job URL: https://openqa.gnome.org/tests/1013
    * 4/6 tests passed
    * Failed: gnome_desktop

The format can be improved here, but it already makes it quicker for me to turn a test failure into a bug report against whatever component probably caused the failure. If you introduce a critical bug in your module maybe you’ll get to see one soon 😉

Status update, 18/04/2023

It’s been a long month, thankfully with a nice holiday in the middle. I am divided between lots of things at work which is not helpful for being able to focus on any interesting thing.

The development of large language models is everywhere at the moment, I shared some thoughts on those on the Lines forum. I won’t go in depth here as we’re going to be discussing these things for the next 15 years anyway. Stay safe out there in the dark forest.

I’ve had a little time to poke at GNOME’s OpenQA tests. I made a simple helper tool named ssam_openqa which simplifies the task of running the OpenQA tests locally on your laptop. My current goal is to figure out how to split up the test suite into multiple pieces, as the current cycle time when developing tests is around 5 minutes which is too slow to be fun. And it has to be fun so you will want to contribute 🙂

Here’s some new music!

Also available on Bandcamp.

Status update 17/03/2023

Hello from my parents place, sitting on the border of Wales & England, listening to this excellent Victor Rice album, thinking of this time last year when I actually got to watch him play at Freedom Sounds Festival, which was one of my first adventures of the post-lockdown 2020s.

I have many distractions at the moment, many being work/life admin but here are some of the more interesting ones:

• Playing in a new band in Santiago – Killo Karallo – recording some initial music which is to come out next week
• Preparing new Vladimir Chicken music, also cooked and ready for release in April
• Figuring out how we can grow the GNOME OpenQA tests while keeping them fun to work with. Here’s an experimental commandline tool which might help with that.
• Learning about marketing, analytics, and search engine optimization.
• Trying out the new LLaMA language model and generally trying to keep up with the ongoing revolution in content generation technology.

Also I got to see real snow for the first time in a few years! Thanks Buxton!

Status update, 17/02/2023

This month I attended FOSDEM for the first time since 2017. In addition to eating 4 delicious waffles, I had the honour of presenting two talks, the first in the Testing & Automation devroom on Setting up OpenQA testing for GNOME.

GNOME’s initial OpenQA testing is mostly implemented now and it’s already found its first real bug. The next step is getting more folk interested within GNOME, so we can ensure ongoing maintenance of the tests and infra, and ensure a bus factor of > 1. If you see me at GUADEC then I will probably talk to you about OpenQA, be prepared!! 🙂

My second talk was in the Python devroom, on DIY music recommendations. I intermittently develop a set of playlist generation tools named Calliope, and this talk was mostly aiming to inspire people to start similar fun & small projects, using simple AI techniques that you can learn in a weekend, and taking advantage of the amazing resource that is Musicbrainz. It seemed to indeed inspire some of the audience and led to an interesting chat with Rob Kaye of the Metabrainz Foundation – there is more cool stuff on the way from them.

Here’s a fantastic sketch of the talk by Jeroen Heijmans:

^{Talk summary sketch, CC BY-SA 4.0}

I didn’t link to this in the talk, but apropos of nothing here’s an interesting video entitled Why Spotify Will Eventually Fail.

On the Saturday I met up with Carlos Garnacho and gatecrashed the GNOME docs hackfest, discussing various improvements around search in GNOME. Most of these are now waiting for developer time as they are too large to be done in occasional moments of evening and weekend downtime, get in touch if you want to find out more!

I must also shout out Marco Trevisan for showing me where to get a decent meal near Madrid Chamartín station on the way home.

Meanwhile at Codethink I have been getting more involved in marketing. Its a company that exists in two worlds, commercial software services on one side and community-driven open source software on the other, often trying our best to build bridges between the two. There aren’t many marketing graduates who are experts in open source, and neither many experienced software developers who want to work fulltime on managing social media, so we are still figuring out the details…

Anyway, the initial outcome is that Codethink is now on the Fediverse – follow us here! @codethink@social.codethink.co.uk

A small Rust program

I wrote a small program in Rust called cba_blooper. Its purpose is to download files from this funky looper pedal called the Blooper.

It’s the first time I finished a program in Rust. I find Rust programming a nice experience, after a couple of years of intermittent struggle to adapt my existing mental programming models to Rust’s conventions.

When I finished the tool I was surprised by the output size – initially a 5.6MB binary for a tool that basically just calls into libasound to read and write MIDI. I followed the excellent min-sized-rust guide and got that down to 1.4MB by fixing some obvious mistakes such as actually stripping the binary and building in release mode. But 1.4MB still seems quite big.

Next I ran cargo bloat and found there were two big dependencies:

• the ‘clap‘ argument parser library
• the ‘regex’ library, pulled in by ‘env_logger‘

I got ‘env_logger’ to shrink by disabling its optional features in Cargo.toml:

env_logger = { version ="0.10.0", default_features = false, features = [] }

As for ‘clap’, it seems unavoidable that it adds a few hundred KB to the binary. There’s an open issue here to improve that. I haven’t found a more minimal argument parser that looks anywhere near as nice as ‘clap’, so I guess the final binary will stay at 842KB for the time being. As my bin/directories fill up with Rust programs over the next decade, this overhead will start to add up but it’s fine for now.

It’s thanks to Rust being fun that this tool exists at all. It definitely easier to make a small C program but the story around distribution and dependency management for self-contained C programs is so not-fun that I probably wouldn’t even have bothered writing the tool in the first place.

Status update, 16/01/2023

The tech world is busy building “AI apps” with wild claims of solving all problems. Meanwhile it’s still basically an unsolved problem to get images and text to line up nicely when making presentation slides.

I’m giving a couple of talks at FOSDEM in February so i’ve been preparing slides. I previously used Reveal.js, which has some nice layout options (like r-stretch and r-fit-text), but pretty basic Markdown support such that I ended up writing the slides in raw HTML.

A colleague turned me onto Remark.js, a simpler tool with better Markdown support and a CLI tool (Backslide), but its layout support is less developed than Reveal.js so I ended frustrated at the work necessary to lay things out neatly.

In the end, I’ve built my own tiny tool based around python-markdown and its attr_list extension, to compile Markdown to Reveal.js HTML in a way that attempts to not be hopelessly annoying. It’s a work in progress, but hopefully I can work towards becoming less frustrated while making slides. The code is here, take it if you like it and don’t ask me any questions 🙂

Status update, 15/12/2022

The last update of 2022, and the 14th since I started writing these.

Before we begin it’s vital to check my list of top albums from 2022, which I’m sharing to Mastodon:

• #5 – Fred Thomas – Another Song About Riding The Bus
• #4 – Klawo – Klawo
• #3 – The Slackers – Don’t Let The Sunlight Fool Ya

For #2 and #1 you will have to follow @vladimirchicken@mastodon.art I guess!

I’m commit full time at work to a project, as is normal, and a couple of spare hours a week lets me push forward a few things in GNOME.

(By the way, if you have a few hours to donate towards improving GNOME, Georges has some ideas for you).

I’ve been intermittently looking at OpenQA testing of GNOME since the summer, and I just posted a short progress report about that over on discourse.gnome.org.

I’ve also been looking at search in GNOME since about 2012, and the two things are hopefully about to converge.

When I first started volunteering my time to help maintain the Tracker search engine, CI was a luxury that GNOME couldn’t afford. Testing a patch for Tracker required a maintainer to do the following:

• download it from Bugzilla
• apply it to the local Git tree
• rebuild, which often involved 5+ minutes waiting for Autotools to check if we’d travelled back in time to the 1980’s since the last build
• run the test suite (those bits which worked)
• install and manually test the change

If anything went wrong due to the patch, the maintainer had to reply to the patch author what went wrong, and check back tomorrow to repeat the process. It’s really only due to the dedication of the maintainer team that anything ever got done.

Things have improved dramatically since the project moved to Gitlab, adopted Gitlab CI and resurrected the ‘functional-tests’ test suite which provides a model of the real desktop search process.

The ‘functional-tests’ model is useful but it is just a model, there are many more components involved when you actually start searching in GNOME Shell. The test surface includes: tracker-miner-fs and libtracker-sparql, GNOME Shell, the many search providers, Nautilus, the content apps (Music, Photos, etc), the GNOME platform libraries, the service manager (systemd), right down to the kernel itself. (If you see the background indexer taking more resources than the Shell, the problem lies in part with the kernel’s scheduler wrongly prioritizing background processes).

There are some plans in the air about changing the design and implementation of search in GNOME. This is really exciting to see – I have ideas myself – but the first step should be to better test what we already have.

And so, I recently ran a survey on what people search, and when I next get a little time to spend on GNOME, I hope to start adding some test content and search tests to OpenQA, so we can exercise the whole search stack, and confidently work to make it better.

Status update, 19/11/2022

Audio Developer Conference

I was at ADC 2022 last week – thanks to Codethink as always for covering the cost and allowing me 2 days time off to attend. It was my first time attending in person, and besides the amazing talks (which will appear online here around the end of this month), I had somehow never realized how many players in the music tech world are British. Perhaps because I always hang out in Manchester and further north while all the activity is happening in Cambridge and London.

Indeed the creator of the famous JUCE Framework is a Brit and was busy at the conference announcing a new(ish) language designed for DSP pipelines and plugins, cleverly named Cmajor.

ADC has no lightning talks but instead an “Open mic night”. I naively pictured a concert in a pub somewhere and signed up to play, but in practice it was closer to lightning talks and there were no instruments to play. For better or worse I had the session for Maharajah on a laptop and did an improvised 5 minute version with judicious (or not) or use of timestretching to speed through the slow parts. Even more fun than regular lightning talks.

I highly recommend the event! Despite the surfeit of Mac users!

Microblogging

Every article published during November 2022 must mention the Twitter meltdown and this is no exception! I’ve personally enjoyed using reading Twitter for years, always without Infinite Scroll and with “Latest tweets” instead of “Home” (i.e. content from people I follow, not entertaining clickbait for my monkey-brain). The sites only value was always its enourmous userbase and now there’s a real wave of migration to Mastodon, like, celebrities and all, I’m spending more time as @vladimirchicken@mastodon.art.

I made that account a while back, initially just for half-baked song ideas, and I will try to keep it more playful and positive than my old Twitter account. For work-related content I already have a very serious blog, after all. (That’s this blog!)

Nushell

At work I’m still spending every day deep in JSON and CSV data dumps and using Nushell to make sense of it all.

It’s really good for exploring data, and you can immediately make a guess at what this Nushell pipeline might do:

fetch http://date.jsontest.com/ | get date

I find the syntax a lot easier than jq, where even basic filters quickly start to look like K or Brainfuck.

I wrote my first Nushell scripts, which is is also very nice, shell functions (“custom commands”) can have type annotations for arguments, default values and each script can have a main function where you define the commandline arguments. 400% nicer than Bash scripts.

I don’t have time to get involved enough in the project right now to open useful bug reports, so I’m going to post a list of gripes here instead. Hopefully its useful for other folk evaluating Nushell so you can get an idea of what’s still missing as of version 0.70:

• Writing a long line (one that wraps to the next line) makes cursor movement increasingly slow… at least on WSL
• where can compare a column to a value, e.g. where date < 11-19-2020, but you can’t compare two columns, which is weird, SQL where can do that.
• The CTRL+C behaviour sucks. Run 100 slow python scripts in a loop, and you have to ctrl+c out of every single one.

Status update 18/10/2022

The most important news this week is that my musical collaborator Vladimir Chicken just released a new song about Manchester’s most famous elephant. Released with a weird B-side about a “Baboon on the Moon”, I am not sure what he was thinking with that one.

I posted on discourse.gnome.org already about GNOME OpenQA testing, now that the tests are up to date I’m aiming to keep an eye on them for a full release cycle and see how much ongoing maintenance effort they need. Hopefully at next year’s GUADEC we’ll be able to talk about moving this beyond an “alpha” service. We’ll soon have something like GNOME Continuous back in action after “only” 6 years of downtime.

Other exciting things in this area: Abderrahim Kitouni and Jordan Petridis have updated gnome-build-meta to track exact refs in its Git history; there are some details to work out so that it still provides quick CI feedback but this was basically necessary to ensure build reproducibility. And Tristan Van Berkom already blogged about research to use Recc inside BuildStream, with the eventual goal of unlocking fast incremental builds within the reproducibility guarantees that BuildStream already provides.

There is no direct link between these projects but I think we share the common vision that Colin Walters already laid out 10 years ago when describing Continuous: GNOME contributors need to be able to develop and test system-level changes involving GNOME, using a reliable & documented process with modest hardware requirements. Many issues and bug reports go beyond a single component, and in many cases right down to the kernel. As an example, when a background indexing task causes lagging in the desktop shell, folk blame the background indexer process, but the indexer is not in control of its own scheduling and such an issue can’t be fully reproduced if we don’t control exactly which kernel is running. Hopefully when these streams of work come to fruition, these kinds of bugs will finally become “shallow”.

Outside of volunteer efforts, I’ve been working on a new client project that is essentially a complex database migration. I don’t get to do much database work at Codethink, its nice to have absolutely no legacy Makefiles to deal with for once, and its been a good opportunity to try out Nushell in a bit more depth. My research so far is mostly setting up Python scripts to run database queries and output CSV, then using Nushell to filter and sort the output. When I tried Nushell a few years ago it still lacked some important features – it didn’t even have a way to set variables at that point – now it’s prepared for anything you can throw at it and I look forward to doing more data processing with it.

I’m not yet ready to switch completely from Fish to Nushell, but … who knows? Maybe it’s coming.

Status update 21/09/22

Last week I attended OSSEU 2022 in Dublin, gave a talk about BuildStream 2.0 and the REAPI, and saw some new and old faces. Good times apart from the common cold I picked up on the way — I was glad that the event mandated face-masks for everyone so I could cover my own face without being the “odd one out”. (And so that we were safer from the 3+ COVID-19 cases reported at the event).

Being in the same room as Javier allowed some progress on our slightly “skunkworks” project to bring OpenQA testing to upstream GNOME. There was enough time to fix the big regressions that had halted testing completely since last year, one being an expired API key and the other, removal of virtio VGA support in upstream’s openqa_worker container. We prefer using the upstream container over maintaining our own fork, in the hope that our limited available time can go on maintaining tests instead, but the containers are provided on a “best effort” basis and since our tests are different to openqa.opensuse.org, regressions like this are to be expected.

I am also hoping to move the tests out of gnome-build-meta into a separate openqa-tests repo. We initially put them in gnome-build-meta because ultimately we’d like to be able to do pre-merge testing of gnome-build-meta branches, but since it takes hours to produce an ISO image from a given commit, it is painfully slow to create and update the OpenQA tests themselves. Now that Gitlab supports child pipelines, we can hopefully satisfy both use cases: one pipeline that quickly runs tests against the prebuilt “s3-image” from os.gnome.org, and a second that is triggered for a specific gnome-build-meta build pipeline and validates that.

First though, we need to update all the existing tests for the visual changes that occurred in the meantime, which are mostly due to gnome-initial-setup now using GTK4. That’s still a slow process as there are many existing needles (screenshots), and each time the tests are run, the Web UI allows updating only the first one to fail. That’s something else we’ll need to figure out before this could be called “production ready”, as any non-trivial style change to Adwaita would imply rerunning this whole update process.

All in all, for now openqa.gnome.org remains an interesting experiment. Perhaps by GUADEC next year there may be something more useful to report.

Team Codethink in the OSSEU 2022 lobby

My main fascination this month besides work has been exploring “AI” image generation. It’s amazing how quickly this technology has spread – it seems we had a big appetite for generative digital images.

I am really interested in the discussion about whether such things are “art”, because I this discussion is soon going to encompass music as well. We know that both OpenAI and Spotify are researching machine-generated music, and it’s particularly convenient for Spotify if they can continue to charge you £10 a month while progressively serving you more music that they generated in-house – and therefore reducing their royalty payments to record labels.

There are two related questions: whether AI-generated content is art, and whether something generated by an AI has the same monetary value as something a human made “by hand”. In my mind the answer is clear, but at the same time not quantifiable. Art is a form of human communication. Whether you use a neural network, a synthesizer, a microphone or a wax cylinder to produce that art is not relevant. Whether you use DALL-E 2 or a paintbrush is not relevant. Whether your art is any good depends on how it makes people feel.

I’ve been using Stable Diffusion to try and illustrate some of sound worlds from my songs, and my favourite results so far are for Don’t Go Into The Zone:

And finally, a teaser for an upcoming song release…

Status update, 16/08/2022

Building Wheels

For the first time this year I got to spend a little paid time on open source work, in this case putting some icing on the delicious and nourishing cake that we call BuildStream 2.

If you’ve tried the 1.9x pre-releases you’ll have seen it depends on a set of C++ tools under the name BuildBox. Some hot codepaths that were part of the Python core are now outsourced to helper tools, specifically data storage (buildbox-casd, buildbox-fuse) and container creation (buildbox-run-bubblewrap). These tools implement remote-apis standards and are useful to other build tools, the only catch is that they are not yet widely available in distros, and neither are the BuildStream 2 prereleases.

Separately, BuildStream 2 has some other hot codepaths written with Cython. If you’ve ever tried to install a Python package from PyPI and wondered why a package manager is running GCC, the answer is usually that it’s installing a source package and has to build some Cython code with your system’s C compiler.

The way to avoid requiring GCC in this case is to ship prebuilt binary packages known as wheels. So that’s what we implemented for BuildStream 2 – and as a bonus, we can bundle prebuilt BuildBox binaries into these packages. The wheels have a platform compatibility tag of “manylinux_2_28.x86_64” so they should work on any x86_64 host with GLIBC 2.28 or later.

Connecting Gnomes

I didn’t participate in GUADEC this year for various reasons, I’m very glad to see it was a success. I was surprised to see six writeups of the Berlin satellite event, and only two of the main event in Mexico (plus one talk transcript and some excellent coverage in LWN) – are Europeans better at blogging? 🙂

I again saw folk mention that connections *between* the local, online and satellite events are lacking – I felt this at LAS this year – and I still think we should take inspiration from 2007 Twitter and put up a few TVs in the venue with chat and microblog windows open.

The story is, that Twitter launched in 2006 to nobody, and it became a hit only after putting up screens at a US music festival in 2007 displaying their site where folk could live-blog their activities.

Hey old people, remember this?! Via webdesignmuseum.org

I’d love to see something similar at conferences next year where online participants can “write on the walls” of the venue (avoiding the ethically dubious website that Twitter has become).

On that note, I just released a new version of my Twitter Without Infinite Scroll extension for Firefox.

Riding Trains

I made it from Galicia to the UK overland, actually for the first time. (I did do the reverse journey already by boat + van in 2018). It was about 27 hours of travel spread across 5 days, including a slow train from Barcelona into the Pyrenees, and a night train onwards to Paris, and I guess cost around 350€. The trip went fine, in comparison to the plane I had booked to return which was cancelled by the airline without notification, so the return flight+bus ended up costing a similar amount and taking nearly 15 hours. No further comment on that but I can recommend the train ride!

High speed main line through the French Pyrenees

Status update 18/07/2022

Summer is here!

All my creative energy has gone into wrapping up a difficult project at Codethink, and the rest of the time I’ve been enjoying sunshine and festivals. I was able to dedicate some time to learning the basics of async Rust but I don’t have much to share from the last month. Instead, let me focus on some projects I’m keeping an eye on.

Firstly, in the Tracker search engine, Carlos Garnacho has landed some important features and refactors. The main one being stream-based serializers and deserializers.

This allows more easily backing up and importing data in and out the tracker-store, and cleaning up some cruft like multiple different implementations of Turtle. It seems ideal having a totally stream-based codec so you can process an effectively infinite amount of data, but there is a tradeoff if you serialize data triple-by-triple – the serialized output is much less human-readable and in some cases larger than if you do some buffering and group related statements together. For this reason we didn’t yet land the JSON-LD support.

Carlos also rewrote the last piece of Vala in libtracker-sparql into C. Vala makes some compromises that aren’t helpful for making a long-term ABI stable C library. There are more lines of code now and we particularly miss Vala’s async support, but this makes maintenance easier as we can now be sure that any misbehaving C code was generated by ourselves and not by the Vla compiler.

There are some other fixes for issues reported by various contributors, thanks to everyone that got involved in the 3.3.0 cycle so far 🙂

Meanwhile, in world of BuildStream there is a lot of activity for the final push towards a BuildStream 2.0 release. I’m only a bystander in the process and to me things look promising. The 2.0 API is finalized and frozen, there’s a small list of blockers remaining – any help to resolve these is welcome! – and then the door is open to a 2.0 release. That’s most likely to be preceded by one or more “1.97” release candidates to allow wider testing. Whatever happens the hope is that the upcoming Freedesktop SDK 22.08 release can be rolled with BuildStream “2.0”.

I’ve been using Bst 1.x for a while and finding it a very helpful tool. The REAPI support is particularly cool as it allows organisations to manage a single distributed build infrastructure that multiple tools can make use of – I hope it gains more traction, and I think it’s already helping to “sell” BuildStream to organisations that are looking at doing distributed builds.

Finally, Kate Bush’s music is super popular again, which is great and I want to share this tidbit from my own youth in the early 2000s right out of Sunderland doing an excellent Hounds of Love cover:

Status update, 17/06/2022

I am currently in the UK – visiting folk, working, and enjoying the nice weather. So my successful travel plans continue for the moment… (corporate mismanagement has led to various transport crises in the UK so we’ll see if I can leave as successfully as I arrived).

I started the Calliope playlist toolkit back in 2016. The goal is to bring open data together and allow making DIY music recommenders, but its rather abstract to explain this via the medium of JSON documents. Coupled with a desire to play with GTK4, which I’ve had no opportunity to do yet, and inspired by a throwaway comment in the MusicBrainz IRC room, I prototyped up a graphical app that shows what kind of open data is available for playlist generation.

This “calliope popup” app can watch MPRIS nofications, or page through an existing playlist. In future it could also page through your Listenbrainz listen history. So far it just shows one type of data:

This screenshot shows MusicBrainz metadata for my test playlist’s first track, which happens to be the song “Under Pressure”. (This is a great test because it is credited to two artists :-). The idea is to flesh out the app with metadata from various different providers, making it easier to see what data is available and detect bad/missing info.

The majority of time spent on this so far has been (re-)learning GTK and figuring out how to represent the data on screen. There was some also work involved making Calliope itself return data more usefully.

Some nice discoveries since I last did anything in GTK are the Blueprint UI language, and the Workbench app. Its also very nice having the GTK Inspector available everywhere, and being able to specify styling via a CSS file. (I’ve probably done more web sites than GTK apps in the last 10 years, so being able to use the same mental model for both is a win for me.). The separation of Libadwaita from GTK also makes sense and helps GTK4 feels more focused, avoiding (mostly) having 2 or 3 widgets for one purpose.

Apart from that, I’ve been editing and mixing new Vladimir Chicken music – I can strongly recommend that you never try to make an eight minute song. This may be the first and last 8 minute song from VC 🙂

Status update, 19/05/2022

I had some ambitious travel plans last month – ambitious by 2020’s standards, anyway – and somehow they came off without any major issues or contagions. In Cologne I was amazed to go to Freedom Sounds festival and witness the return of the Singers ATX alongside host of other ska legends. And then my first ever trip to Italy where I attended Linux App Summit. It was a treat to be around people who are as enthusiastic as I am about the rather niche topic of Linux desktop apps… but hopefully it’s a topic which is becoming less niche.

While safely back home I then managed to contract COVID-19, so this status update is going to be fairly short.

Things I spent time on that may be of interest to someone, include

• a minimal “GUI toolkit” for the monome grid, written partly as another Rust exercise, and partly to make experimental beats. Fun to write. A lot of this was developed offline while travelling, and I discovered Rust’s cargo doc feature which is an incredible tool for the disconnected hacker. Code dump here.
• related to the above, packaging the user space driver for the monome grid as a systemd portable service.
• updating Fedora on my partners laptop. This particular laptop was lacking a Wifi driver but that is now solved in Fedora 36. I can provide yet another datapoint that the GNOME 42 screenshot experience creates joy and happiness.
• watching the various new synthesizers unveiled the Superbooth 2022 event.
• a single experimental beat, plus some production on a new song due in a few months

Trying out systemd’s Portable Services

I recently acquired a monome grid, a set of futuristic flashing buttons which can be used for controlling software, making music, and/or playing the popular 90’s game Lights Out.

There’s no sound from the device itself, all it outputs is a USB serial connection. Software instruments connect to the grid to receive button presses and control the lights via the widely-supported protocol Open Sound Control protocol. I am using monome-rs to convert the grid signals into MIDI, send them to Bitwig Studio and make interesting noises, which I am excited to share with you in the future, but first we need to talk about software packaging.

Monome provide a system service named serialosc, which connects to the grid hardware (over USB-serial) and provides the Open Sound Control endpoint. This program is not packaged in by Linux distributions and that is fine, it’s rather niche hardware and distro maintainers shouldn’t have support every last weird device. On the other hand, it’s rather crude to build it from source myself, install it into /usr/local, add a system service, etc. etc. Is there a better way?

First I tried bundling serialosc along with my control app using Flatpak, which is semi-possible – you can build and run the service, but it can’t see the device unless you set the super-insecure “–devices=all” mode, and it still can’t detect the device because udev is not available, so you would have to hardcode the driver name /dev/ttyACM0 and hotplug no longer works … basically this is not a good option.

Then I read about systemd’s Portable Services. This is a way to ship system services in containers, which sounds like the correct way to treat something like serialosc. So I followed the portable walkthrough and within a couple of hours the job was done: here’s a PR that could add Portable Services packaging upstream: https://github.com/monome/serialosc/pull/62

I really like the concept here, it has some pretty clear advantages as a way to distribute system services:

• Upstreams can deliver Linux binaries of services in a (relatively) distro-independent way
• It works on immutable-/usr systems like Fedora Silverblue and Endless OS
• It encourages containerized builds, which can lower the barrier for developing local improvements.

This is quite a new technology and I have mixed feelings about the current implementation. Firstly, when I went to screenshot the service for this blog post, i discovered it had broken:

Fixing the error was a matter of – disabling SELinux. On my Fedora 35 machine the SELinux profile seems totally unready for Portable Services, as evidenced in this bug I reported, and this similar bug someone else reported a year ago which was closed without fixing. I accept that you get a great OS like Fedora for free in return for being a beta tester of SELinux, but this suggests portable services are not yet ready widespread use.

That’s better:

I used the recommended mkosi build to create the serialosc container, which worked as documented and was pretty quick. All mkosi operations have to run as root. This is unfortunate and also interacts badly with the Git safe.directory feature (the source trees are owned by me, not root, so Git’s default config raises an error).

It’s a little surprising the standard OCI container format isn’t supported, only disk images or filesystem trees. Initially I built a 27MB squashfs, but this didn’t work as the container rootfs has to be read/write (another surprise) so I deployed a tree of files in the end. The service container is Fedora-based and comes out at 76MB across 5,200 files – that’s significant bloat around a service implemented in a few 1000 lines of C code. If mkosi supported Alpine Linux as base OS we could likely reduce this overhead significantly.

The build/test workflow could be optimised but is already workable, the following steps take about 30 seconds with a warm mkosi.cache directory:

• sudo mkosi -t directory -o /opt/serialosc --force
• sudo portablectl reattach --enable --now /opt/serialosc --profile trusted
  
  

We are using “Trusted” profile because the service needs access to udev and /dev, by the way.

All in all, the core pieces are already in place for a very promising new technology that should make it easier for 3rd parties to provide Linux system-level software in a safe and convenient way, well done to the systemd team for a well executed concept. All it lacks is some polish around the tooling and integration.

Linux App Summit 2022

Engineers at Codethink get some time and money each year to attend conferences, and part of the deal is we have to write a report afterwards. Having written the report I thought… this could be a bit more widely shared! So, excuse the slightly formal tone of this report, but here are some thoughts on LAS 2022.

Talk highlights

1. Energy Conservation and Energy Efficiency with Free Software – Joseph De Veaugh-Geiss

Video, Abstract

If you follow climate science you’ll know we have some major problems coming. Solving these will take big societal changes towards degrowth, which was not the topic of this talk at all. This talk was rather about what Free Software developers might do to help out in the wider context of reducing energy use.

I recommend watching in full as it was well delivered and interesting. Here are some of the points I noted:

• Aviation is estimated to cause 2.5% of CO² emissions in 2017, and “ICT” between 1.5-2.8%.
  • Speaker didn’t know if that number includes Bitcoin or other “proof-of-work” things
  • This number does include Bitcoin. (Thanks to Joseph for clarifying 🙂
  • Two things that make the number high: short-lifespan electronic devices, video streaming/conferencing
• Software projects could be more transparent about how much energy they use.
  • Imagine 3 word processors which each provide a Github badge showing energy consumption for specific use cases, so you can choose the one which uses least power
• KDE’s PDF reader (Okular) recently achieved the Blauer Angel eco-certification
• There is an ongoing initiative in KDE to measure and improve energy consumption of software – a “measurement party” in Berlin is the next step.
  • Their method of measuring power consumption requires a desktop PC and a special plug that can measure power at reasonably frequency.
• Free software already has a good story here (as we are often the ones keeping old hardware alive after manufacturers drop support), and there’s an open letter asking the EU to legislate such that manufacturers have to give this option.

2. Flathub – now and next

Video, Abstract
There is big growth in number of users. The talk had various graphs including a download chart of something like 10PB of data downloaded.

The goal of Flathub is not to package other people’s apps, rather to reduce barriers to Linux app developers. One remaining barrier is money – as there’s no simple way to finance development of a Linux app at present. So together with Codethink, GNOME and Flathub are working on a way that app users can make donations to app authors.

Notes

There was a larger presence from Canonical than I am used to, and it seems they are once again growing their desktop team, and bringing back the face-to-face Ubuntu Developer Summit (rebranded Ubuntu Summit). All good news.

COVID safety:

• all participants expect speaker were masked during talks
• the seats were spaced and our “green pass” (vaccine or -ve test) was checked on day 1
• in the social events the masks all came off, these were mostly in the outside terrace area.

Online participation:

• maybe 30% of talks were online and these worked well, in fact the Flathub talk had one speaker on stage and one projected behind him as a giant head, and it worked well.
• the online attendees were effectively invisible in the venue, it would be better if there was a screen projecting the online chat so we could have some interaction with them.

The organisation was top notch, the local team did a great job and the venue was also perfect for this kind of event. Personally I met more folks from KDE than I ever have and felt like a lot of important desktop-related knowledge sharing took place. Definitely recommend the conference for anyone with an interest in this area.

I don’t have any good photos to share, but check https://twitter.com/hashtag/las2022 for the latest. Hope to see you there next year!

Status update, 15/04/2022

As i mentioned last month, I bought one of these Norns audio-computers and a grey grid device to go with it. So now I have this lap-size electronic music apparatus.

Its very fun to develop for – the truth is I’ve never got on well with the “standard” tools of Max/MSP and Pure Data, I suspect flow-based programming just isn’t for me. As within an hour of opening the web-based Lua editor on the Norns device I already had a pretty cool prototype of something I hadn’t even intended to make. More on that when I can devote more time to it.

There is some work to make the open-source Norns software run on the Linux desktop in a container. It seems the status is “working but awkward.” I thought to myself – “How hard could it be to package this as a Flatpak?”

As soon as I saw use of Golang and Node.js I knew the answer – very hard.

In the world of Flatpak apps we are rigorous about doing reproducible builds from controlled inputs. The flatpak-builder tool requires that all the Go and JavaScript dependencies are specified ahead of time, and build tools are prevented from connecting to the internet at build time. This is a good thing, as the results are otherwise unpredictable. The packaging tools in the Go and Node.js ecosystems make this kind of attention to detail difficult to achieve.

There is a solution, which would be to build the components using BuildStream and develop plugins to integrate with the Node.js and Go worlds, so it can fetch deps ahead of time. There’s an open issue for Go plugin. For Node.js I am not sure if anything exists.

This led me to thinking – how come the open-source, community developed world of Flatpaks is so much better and doing reliable, reproducible builds than the corporate environments that I see in my day job?

One last cool thing – thanks to instructions here by @infinitedigits I was able to produce this amazing music video!

View this post on Instagram

A post shared by Vladimir Chicken (@vladimirchicken)

Status update, 18/03/2022

This month has had a big focus on music! I just released a 4 track EP called Rust In Peace, which you can listen to on various popular music platforms and (better still) download it from Bandcamp.

The COVID19 pandemic is not over (I can name 5 folk who have been COVID+ just this week), but its effects on society are becoming less, to the point I could even do a launch gig for the EP in the amazing Café Arume in Santiago – my first real “gig” since 2018.

A similar effect is happening at work, in that work-related travel is becoming possible. My current project, on a huge codebase that resembles a famous painting by Hieronymus Bosch, involves making changes that affects 100s or 1000s of developers, who we meet via Slack DMs and audio-only Zoom calls. For the first time in years, on-site visits are a possibility again and while I’m genuinely not keen to make a habit of intercontinental air travel, meeting the organisation face to face at least once would make a huge difference to my day to day work – the issues are much more about managing processes and people than any specific technologies, and this is super hard to do with strangers.

I am also hoping to travel to Italy next month to attend Linux App Summit 2022 in person, so, see you there? Talk submissions for LAS (online + face to face) are open until midnight tonight (18th March), so perhaps its not too late to submit a talk or lightning talk!

A few other interesting steps I’ve taken this month:

• Learn GNOME Shell keyboard shortcuts for moving and resizing windows. You can do a lot with Meta+F7/F8, it turns out.
• Implement the first “Special mix” playlist generator for Calliope; it takes a histogram of my Listenbrainz history, picks a year and finds 60 minutes of songs that I first discovered in that year. Like the old “Mystery years” radio show, but with better music.
• Buy a Norns audio-computer; basically a RPi 3 with a rich ecosystem of experimental audio effects, written Lua and Supercollider. I have been tempted recently by some of the amazing modern guitar pedals like Chase Bliss Mood and Hologram Microcosm, and this is my attempt to avoid buying any of those 🙂

Status update, 16/02/2022

January 2022 was the sunniest January i’ve ever experienced. So I spent its precious weekends mostly climbing around in the outside world, and the weekdays preparing for the enourmous Python 3 migration that one of Codethink’s clients is embarking on.

Since I discovered Listenbrainz, I always wanted to integrate it with Calliope, with two main goals. The first, to use an open platform to share and store listen history rather than the proprietary Last.fm. And the second, to have an open, neutral place to share playlists rather than pushing them to a private platform like Spotify or Youtube. Over the last couple of months I found time to start that work, and you can now sync listen history and playlists with two new cpe listenbrainz-history and cpe listenbrainz commands. So far playlists can only be exported *from* Listenbrainz, and the necessary changes to the pylistenbrainz binding are still in review, but its a nice start.

Status update, 17/01/2022

Happy 2022 everyone! Hope it was a safe one. I managed to travel a bit and visit my family while somehow dodging Omicron each step of the way. I guess you cant ask for much more than that.

I am keeping busy at work integrating BuildStream with a rather large, tricky set of components, Makefiles and a custom dependency management system. I am appreciating how much flexibility BuildStream provides. As an example, some internal tools expect builds to happen at a certain path. BuildStream makes it easy to customize the build path by adding this stanza to the .bst file:

variables:
    build-root: /magic/path

I am also experimenting with committing whole build trees as artifacts, as a way to distribute tests which are designed to run within a build tree. I think this will be easier in Bst 2.x, but it’s not impossible in Bst 1.6 either.

Besides that I have been mostly making music, stay tuned for a new Vladimir Chicken EP in the near future.