Ubuntu Touch, also known as the Ubuntu Phone, is the Ubuntu flavor specifically designed for touch devices such as smartphones and tablets. Although it's based on the same core packages as its desktop counterpart, it follows a completely different release and upgrade model. One of the reasons for that is because the touch devices use an image-based update mechanism on top of its default read-only core filesystem.

This means that a phone user only gets new and upgraded core packages for its system when an image is built and published to the respective stable image channel – notifying the users of the available over-the-air (OTA) upgrade. However, this happens only after each change goes through various stages of development, quality assurance, testing, and coordination.

In this article, I'll take a look at how all this works. An overview of the general release workflow can be seen in Figure 1. I'll start this update journey from the very top – the code.

Figure 1: A diagram showing the general overview of the update release process.

The Code

The first stage of any change is, of course, the code. Almost all Ubuntu upstream projects use the Launchpad and Bazaar for source development and hosting [1]. Each Touch-specific component has its own Launchpad project registered and a trunk branch managed by the project's maintainers.

When a developer wants to submit a fix, enhancement, or any other piece of code, she or he prepares a bzr branch based off trunk, submits the changes on top of it, pushes it to Launchpad, and creates a so-called merge request to trunk. (See the "Launchpad and Bazaar" box for more details.)

This merge request then gets reviewed by the maintainers and approved if everything looks and works fine. However, this doesn't mean that the change is yet merged or released into the world in any physical form – for now, it's still only code. How does this get transformed to binary form to be used on the system? That takes me to the next stage.

CI Train

Both the Ubuntu desktop and its touch equivalent use Debian (.deb) packages to provide core components. By core components, I mean all the base system files, libraries, graphical interfaces, system daemons, tools, etc. Every Ubuntu-owned Touch component carries Debian packaging in their bzr code trunks, allowing building binary packages in an easy way that can be then installed on any Ubuntu system. Building binary packages is a rather trivial task, but sometimes it can be time consuming, because a lot depends on the environment in which the packages are being built.

To make sure the generated binaries are clean and ready to be used on live systems everywhere, special chroot builders are required. Also, in a distributed development environment and given the need for continuous integration, a special tool that could automate at least parts of the process was required. The current approach used by Ubuntu for this is the CI Train [2].

The CI Train is a set of tools that basically helps build, test, and coordinate package releases to the Ubuntu archives (Figure 2). Now for some explanation of the most important terms: It's a database consisting of "landings" – or isolated batches of code changes – which are prepared by "landers," who are developers responsible for releasing those changes.

Figure 2: The landing requests view of the CI Train web tool.

For those interested in technical details, the CI Train is a Jenkins instance running Python 3 scripts for the core functionality, a Flask-based back end, PostgreSQL database, and a JavaScript user front end. Each landing is assigned a "silo" – basically a specific Launchpad PPA that is used for building the end .deb packages for each landing.

All the lander needs to do is list all merges for the projects she wants to build as part of the landing and get it assigned to a silo. The train then takes all the merge branches, merges them locally, and builds Debian binary packages out of them in the assigned silo PPA – informing the lander of success or failure.

For landings targeting the stable phones, there's one additional testing step required. Once the lander is happy with the packages, each silo then requires a QA sign-off – the end packages get double-checked by testers from the QA Team making sure that all the components work and that there are no regressions in existing features (Figure 3).

Figure 3: Diagram showing the general steps of a CI Train landing.

A landing that has been approved by both the landers and the QA team gets published by the Landing Team "trainguards" – that is, the people responsible for doing the final sign-off of the package to the archives. This is also where the package is checked for correctness.

The Archives and Clicks

As mentioned before, Ubuntu Phone uses the same packages as the desktop version – at least for its core components, which are things like the user interface, libraries, and some drivers. All of this is provided by .deb packages fetched from the standard Ubuntu archives.

As of this writing, the current Ubuntu Touch devices are based off the vivid series, extended with additional changes from a special stable phone overlay. This overlay PPA includes Touch-specific packages installed on top of the vivid base system. For those with more advanced knowledge: Phones use the vivid, vivid-updates, vivid-security, and the overlay packages in their root filesystems.

Besides the core components, Ubuntu Phone also introduced the notion of a click package. A click package is a simpler, more confined and secure package alternative specific to Ubuntu. All applications available for download from the Ubuntu Phone store are click applications.

Generally, each image comes with a set of pre-installed click core applications. Although their upgrades are mostly managed through the store itself – appearing on user phones instantly after being published to the store – each OTA update also comes with those packages updated as well. Some are pre-installed on the root filesystem during build, and others come as part of something called the custom tarball.

This tarball can be different for each device, shipping different specific applications. Every phone is different, and every manufacturer can request different things installed by default. The custom tarball is the place for this customization. Each device also has its specific device tarball including all low-platform specifics. Now, using packages from the archive, clicks from the store, the custom and device tarballs, an Ubuntu Phone image is built.

Images and Channels

When an image is built, it gets picked up by the system-image infrastructure and put into an image channel according to the internal configuration. Multiple sets of channels currently are available, with the most important being: stable, rc, rc-proposed, devel, and devel-proposed [3]. Figure 4 shows the general channel layout used currently. Be aware that this layout changes very dynamically. All the daily-built images that can become potential new stable candidates use the rc-proposed channels.

Figure 4: The general Ubuntu Touch image channel layout and hierarchy.

Even though all the changes that land in rc-proposed images have been pretested by the QA team before their release to the Ubuntu archives, it's still not recommended to use this channel in production devices. There's still risk of important features regressing – or even images not booting completely.

The only recommended channels are the stable ones. Some channels automatically pick new images – others require explicit image copy requests. The rc and stable channel sets are manual channels, meaning that they only get new images when someone from the Landing Team performs an image promotion to the next level.

Milestone Preparation and Testing

Ubuntu Phone currently follows a six-week update cadence. This means that every six weeks a new OTA update is being prepared and released to the stable channel. Everything starts with a list of critical bugs that are targeted for the update – this list is created by the Product Team in cooperation with the Landing Team. Generally, the first four weeks are used for development, during which developers write code and release it through the CI Train.

At the fourth week, the release manager also calls out a feature and string freeze, similar to what's happening in the desktop Ubuntu before release. During this freeze period, no new features or changes to UI strings are normally allowed.

Next, in the week of the milestone, the Landing Team calls out the final freeze, closing the all Touch landings. From this point on, nothing except update-blocking fixes can land to the archives. The most recent rc-proposed image containing all changes up to this point is then handed over to the QA team for testing.

The diagram in Figure 5 outlines the testing procedures. The QA team takes the image for each of the production devices and first performs sanity testing. The sanity test suite's purpose is to check whether the basic functions of the phone are working correctly – things like calls, messaging, basic networking, location, etc.

Figure 5: Various stages of image testing done on a release candidate.

Once all production devices look correct, the QA team moves on to regression testing. This is a very detailed and time-consuming manual test suite; its purpose is to make sure no new unplanned user-visible regressions (variations from the usual behavior) are visible on the image. This step usually takes around two to three days, depending on the number of devices that need to be tested. At the same time, the Landing Team also consults the autopilot smoke-testing results, making sure that automated testing works as expected.

For some devices, the team also tries to build in time for manufacturers to perform their own testing before releasing the update for their platforms. This usually happens after the image has passed internal Canonical testing. This is also the time when all the candidate images are copied into the rc channel.

OTA Update

Once the test results are in, all the manufacturers are happy, and no blocking bugs have been found, the Landing and Product Teams make the final decision about image promotion. The release notes are finalized, and the images are prepared for release. Note that the update doesn't have to be published for all devices at once.

Sometimes, when one of the platforms is thought to not have sufficient quality, the Landing and Product Teams can decide to release the update only for selected platforms – with the rest later, after the fixes for the shortcomings are ready and landed. (See the "Product and Landing Teams" box for more information.)

Each standard update uses the so-called "phased upgrade" mechanism. Every new stable image is rolled out to users in phases, meaning not all of them get the update at once. This acts as a final safety net, allowing the team to roll-back the update in case there's something really broken that got missed by all the previous testing steps. In that case, the image only gets to a limited number of users.

The phased upgrade takes usually up to 24 hours, with the new version reaching more and more users with each hour. After that, all users should be able to see the new OTA update notification on their phones. Then, the freeze is lifted, and all the developers start preparations for the next update. And, the story starts all over again (see Figure 6).

Figure 6: An OTA update ready to be installed.

Conclusion

The Ubuntu Touch release process is still not perfect, of course, but the developers are constantly try to modify and improve learning from the shortcomings of each update.

There are many challenges ahead, and with the number of official devices always increasing, coordinated releases will become more and more difficult. However, the Ubuntu Phone keeps on rolling.