Python-based framework for functional testing

Functional testing involves verifying application behavior against given design specifications. For graphical applications, this process usually means performing specific user interactions with its interface, such as moving the mouse, clicking buttons, typing character sequences, and comparing the resulting application state and outputs against the expectations. When Unity – the flagship user interface of Ubuntu – was growing in complexity, the need for a practical automated testing suite appeared.

The first versions of Autopilot were embedded into the Unity source, as its sole purpose was for testing the shell. Now, after many development cycles, Autopilot has blossomed into a multi-purpose functional testing framework for any graphical user interface.

What Exactly Is Autopilot?

Autopilot is a Python-based automated testing tool for graphical interfaces running on GNU/Linux systems [1]. The tool enables developers to perform input operations (with mouse, keyboard, or even touch gestures) on the application just as a normal user would, but automatically. By using application introspection, Autopilot later can assert whether the final application state equals the expectations.

Tests can be written and run in suites with multiple use-case scenarios. For example, one test can automatically be run multiple times in the case of a single monitor or multiple monitors, if those are available. After a test is finished, the environment under test can be cleaned up according to your needs. That way, before the next test is executed, you have a clean state with which to start anew.

Inside, Autopilot is build on the python-testtools module, with its tests frequently making use of the python-testscenarios package. Currently, the framework is able to introspect not only the Unity desktop shell but also any existing Qt 4.x, Qt 5.x, and GTK 3.x application of interest. Introspection lets you access the internal state variables of the interface – such as, which elements are focused, which checkboxes are toggled, or which UI flags are set. For all of these toolkits, Autopilot tests are written in a similar manner with only smaller differences.

Autopilot has no graphical interface, only a command-line interface. In return, it offers features such as failure test case video recordings, which make it more convenient to analyze what went wrong during a failing test.

Autopilot is under constant development, so please take into account that some details may change in the newer releases. This article is based on the current state of Autopilot version 1.2 (based on revision 100). Please consult the current documentation for the most up-to-date details [2].

Installing Autopilot

Autopilot as a standalone package is available in the official Ubuntu "universe" package pool from Ubuntu Quantal 12.10 onwards. But, using the latest Autopilot snapshots from the official Autopilot PPA is recommended, because those are most up to date. All you have to do is add ppa:autopilot/ppa to your repository listing and fetch the python-autopilot package. This package provides the basic essentials for all Autopilot work.

sudo apt-add-repository ppa:autopilot/ppa
sudo apt-get update
sudo apt-get install python-autopilot

If you want to test GTK+ and Qt applications, you will also need to install the platform-specific Autopilot packages (Listing 1).

sudo apt-get install autopilot-gtk libautopilot-gtk0 # For GTK+
sudo apt-get install autopilot-qt                      # For Qt4
sudo apt-get install autopilot-qt5                     # For Qt5

Because Autopilot is purely Python based, it also can be built manually for non-Ubuntu-based GNU/Linux systems, but, in this case, you must resolve dependencies yourself. So, the Python packages for modules such as testscenarios, testtools, dbus, ibus, gobject, xdg, etc. must be installed beforehand. The source code can be downloaded directly from the packages from Launchpad or fetched straight from the repositories source trunk by using Bazaar (lp:autopilot) [3].

If Ubuntu is being used as the reference system, starting out by downloading the Unity source code is best, because it includes good reference test code for Unity itself; it's a good way of learning the test constructs. Every code should be accompanied by tests, either unit-tests or functional-tests using Autopilot, where applicable. So, a lot of good reference code is available for different user behavior.