I am volunteering a lot of time to work on testing at the moment. When you start out as a developer this seems like the most boring kind of open source contribution that you can do. Once you become responsible for maintaining existing codebases though it becomes very interesting as a way to automate some of the work involved in reviewing merge requests and responding to issue reports.
Most of the effort has been on the GNOME OpenQA tests. We started with a single gnomeos test suite, which started from the installer ISO and ran every test available, taking around 8 minutes to complete. We now have two test suites: gnome_install and gnome_apps.
The gnome_install testsuite now only runs the install process, and takes about 3 minutes. The gnome_apps testsuite starts from a disk image, so while it still needs to run through gnome-initial-setup before starting any apps, we save a couple of minutes of execution. And now the door is open to expand the set of OpenQA tests much more, because during development we can choose to run only one of the testsuites and keep the developer cycle time to a minimum.
Big thanks to Jordan Petridis for helping me to land this change (I didn’t exactly get it right the first time), and to the rest of the crew in the #gnome-os chat.
I don’t plan on adding many more testsuites myself. The next step is to teach the hardworking team of GNOME module maintainers how to extend the openQA test suites with tests that are useful to them, without — and this is very important — without just causing frustration when we make big theming or design changes. (See a report from last month when Adwaita header bars changed colour).
Hopefully I can spread the word effectively at this year’s GUADEC in Riga, Lativia. I will be speaking there on Thursday 27th July. The talk is scheduled at the same time as a very interesting GTK status update so I suppose the talk itself will be for a few dedicated cats. But there should be plenty of time to repeat the material in the bar afterwards to anyone who will listen.
My next steps will be around adding an OpenQA test suite for desktop search – something we’ve never properly integration tested, which works as well as it does only because of hard working maintainers and helpful downstream bug reports. I have started collecting some example desktop content which we can load and index during the search tests. I’m on the lookout for more, in fact I recently read about the LibreOffice “bug documents” collection and am currently fetching that to see if we can reuse some of it.
One more cool thing before you go – we now have a commandline tool for checking openQA test status and generating useful bug reports. And we have a new project-wide label 9. Integration test failure to track GNOME bugs that are detected by the OpenQA integration tests.
〉utils/pipeline_report.py --earlier=1 05/16/2023 05:24:30 pm
Latest gnome/gnome-build-meta pipeline on default branch is 528262. Pipeline status: success
Pipeline 1 steps earlier than 528262 is 528172. Pipeline status: success
Project:
* Repo: gnome/gnome-build-meta
* Commit: dc71a6791591616edf3da6c757d736df2651e0dc
* Commit date: 2023-05-16T13:20:48.000+02:00
* Commit title: openqa: Fix up casedir
Integration tests status (Gitlab):
* Pipeline: https://gitlab.gnome.org/gnome/gnome-build-meta/-/pipelines/528172
* test-s3-image job: https://gitlab.gnome.org/gnome/gnome-build-meta/-/jobs/2815294
* test-s3-image job status: failed
* test-s3-image job finished at: 2023-05-16T14:46:54.519Z
Integration tests status (OpenQA):
* gnome_apps testsuite - job URL: https://openqa.gnome.org/tests/1012
* 3/31 tests passed
* Failed: gnome_desktop
* gnome_install testsuite - job URL: https://openqa.gnome.org/tests/1013
* 4/6 tests passed
* Failed: gnome_desktop
The format can be improved here, but it already makes it quicker for me to turn a test failure into a bug report against whatever component probably caused the failure. If you introduce a critical bug in your module maybe you’ll get to see one soon 😉