No distribution provides its users with more packages than Debian. This implies that it also needs a well-developed system for building packages. Like its counterparts on RPM-based systems, it automates both source code compiling and creating packages. This article introduces the Debian build system that creates Debian packages from a program's source code fully automatically. I will explain the powerful tools that are available and provide guidance on improving your distro.

Germ Cell

To use the Debian build system with Ubuntu, you first need to install the build-essential, debhelper, dh-make, quilt, fakeroot, and devscripts packages. Next, export two environmental variables with your email address and name as shown below. It's best to add these to your .bashrc.

export DEBMAIL="<e-mail_address>@<domain>.<tld>"
export DEBFULLNAME="<first_name> <last_name>"

Both Debian and Ubuntu have their source code in their own repositories that you can identify with the keyword deb-src at the beginning of the line in /etc/apt/sources.list. The source repositories are activated by default after installation. The apt-get source <name> command downloads the sources for the name package into the current directory.

Avoid working with root privileges, because you'll be using the <name>_<version>.orig.tar.gz, <name>_<version>.debian.tar.gz, and <name>_<version>.dsc files as a normal user for further processing. Also added to the named files are patches with the .diff.gz extension that clean up errors in the code.

The file with the .orig.tar.gz extension, however, is missing in packages that the Debian maintainers monitor themselves and that don't have an upstream entity. However, if the program's upstream maintainers have provided the sources as "Debianized" (which include a controlling debian/ subdirectory), the <name>_<version>.debian.tar.gz file will be missing.

Unpack the tarball with apt-get source into the <name>_<version> directory. If you have the Launchpad [1] Debian sources, use dpkg-source -x <name>_<version>.dsc instead for manual unpacking. Using dpkg-buildpackage -uc -us from the unpacked source directory is enough to compile the software and integrate it into a dpkg installable binary package. The first use of dpkg-buildpackage often requires installing more dev packages.

Another call after importing the dependencies should always end with a functioning Debian package for sources from the current distribution, as long as the basic system components (libraries and compiler) are in a deliverable state. It pays to test dpkg-buildpackage against a current Linux distribution to see if it returns a message of a successful build in the console window.

You should see a line such as: dpkg-deb: Package <named> is built in "../<name>_<version>_i386.deb" at the end of the console output.

Switching and Swapping

Building packages starts to make a lot of sense as soon as you start building packages that aren't available for installation on your own system. The ffDiaporama package [2], for example, is a program for creating multimedia presentations and isn't included in the Ubuntu 12.04 repositories. Ubuntu 12.10 and 13.04 has packages for the program with associated sources [3]. I was curious whether the Ubuntu 12.10 sources would also compile under Ubuntu 12.04. If not, I wouldn't be able to install the package from the Ubuntu version because of unsatisfiable dependencies.

As a general rule, the requirements for translating from the sources are much less restrictive than the binary requirements of a specifically tailored program. For example, there are no bindings to a certain architecture (i386 or x86_64). Even porting between Debian and Ubuntu and back often works smoothly, because of the fact the lion's share of Ubuntu packages come from the Debian "unstable" sources.

You can get the ffDiaporama sources from the Launchpad site [3] in the Packages in Distributions section (Figure 1). Under "ffdiaporama" source package in Quantal, you will find the ffdiaporama_1.3-1.dsc, ffdiaporama_1.3.orig.tar.gz, and ffdiaporama_1.3-1.debian.tar.gz files that you can download into a common directory. The dpkg-source -x ffdiaporama_1.3-1.dsc command unpacks the source archive, and dpkg-buildpackage -uc -us then builds a package tailored for Ubuntu 12.04 from the created ffdiaporama-1.3 directory. Because this is a rather large package, compiling can take some time.

Figure 1: Launchpad is specifically tailored to Ubuntu development. Almost all hosted projects have Debian sources and often binary packages.

Thus, two commands are all you need to build installable binary packages automatically on your current system. If the sources don't come from your distribution themselves, success is by no means guaranteed. Chances are the program developers didn't test their sources against existing libraries or compiler versions. However, non-system Debian sources often compile just as smoothly as with ffDiaporama.

You can always deinstall the resulting Debian packages if they don't work and replace them with the versions supplied by your distribution. The package manager prevents any overwriting of files from other packages, so you don't need to fear any damages if you install ordinary applications. However, swapping system components can be risky even with the package manager.

Everything New

Unfortunately, the ffDiaporama version 1.3 I used here to create an Ubuntu 12.04 package is not the latest release. However, the Debian build system helps Ubuntu maintainers keep thousands of packages current and can quickly update a packaged program to its newest version. From the program home page, download the ffDiaporama 1.5 sources [4] into the same directory as the version 1.3 sources.

In the debian/ folder, you will find the patches/ subdirectory. You can delete the fixes contained in patches/ because they don't work (and are often unnecessary). How to push a new version upstream is described in the "Debian New Maintainers' Guide" [5]. Next, you can run the following command from ffdiaporama-1.3/:

$ uupdate -v 1.5 ../ffdiaporama_1.5.tar.gz

Change to the ../ffdiaporama-1.5/ directory that the update process creates. This is where you'll find the sources for ffDiaporama 1.5 as well as the debian/ subdirectory in ffdiaporama-1.3/.

If nothing changed significantly in the build process from version 1.3 to 1.5, dpkg-buildpackage -uc -us should run smoothly and build a package for ffDiaporama 1.5.

Things didn't run quite so smoothly in practice, however. The compile broke with the message ../engine/cDeviceModelDef.h:48:38: fatal error: libavdevice/avdevice.h: file or directory not found. If the compiler can't find a header file (with the .h extension), this means that the required development package is missing. Luckily, you can easily find out which one it is.

Install apt-file (if you don't already have it) and, by using apt-file update with root, bring the database to its newest state. Then, use apt-file search libavdevice/avdevice.h to find the package with the missing header file. In this example, it happens to be libavdevice-dev.

Until compiling works, you may find yourself trying to compile and then having to install more and more development library files. In this case, I was lucky and only had to install libavfilter-dev and libqimageblitz-dev. The build package then built a functioning ffDiaporama 1.5 package (Figure 2).

Figure 2: The reward for your efforts. The version 1.5 of ffDiaporama now runs on Ubuntu 12.04 LTS. It allows installation via package manager, and the program can be removed leaving no residue behind.

The missing dev packages were created because uupdate took over for package building (the Build-Depends) from version 1.3. A powerful build system like Automake (known from ./configure; make; make install) or Cmake would have intercepted any errors before compiling with understandable messages, such as Could NOT find <package> or configure: error: Package requirements (<package> >= <version>) were not met. ffDiaporama uses the simpler Qmake instead.

To make sense of the incomprehensible console window gibberish from other errors not specifically related to missing dependencies, it's recommended to search for the first occurrence of the word error. Because other users have likely already run into the same problem, an Internet search for the error message is often helpful.

From Scratch

So far, I've depended on the helper scripts (Figure 3) that characterize the Debian build system: Dpkg-source to unpack the source tarball and dpkg-buildpackage to compile and automate the packaging.

Figure 3: The Debian packaging workflow is based on the four tools (dpkg-source, dpkg-buildpackage, uupdate, and dh_make) that hide the process from the user.

The powerful dh_make helper aids in "Debianizing" the source code of software, in generating the control files in the debian/ subdirectory that automate compiling, and in creating packages. I will soon be testing it on ffDiaporama 1.5 – although I have already arrived at the current version with help by updating an outdated one. It shows how package creation works without much preparation.

To do this, unpack the upstream source code. The files should end up in a subdirectory named <program_name>_<xx>.<yy>. Its name must be lowercase with the underscore between name and version. From this directory – ffdiaporama_1.5/, in this case – the following command creates a control folder:

$ dh_make -p <Name>_<Version> --createorig

The dpkg_buildpackage command, with no or just minimal ado, runs through this folder. Then, dh_make asks what kind of package type it should prepare for the Debian folder. Almost always the s ("single-binary package") option is the right choice; the others are more geared to experienced maintainers.

After calling dh-make, you need to add descriptions for the package in places marked with square brackets in the control file. You should also enter comments for the current version of the package in the changelog. Because even a wrongly placed space character in changelog can bring the package build process to a halt, you should call up the dch tool from the source folder. Like visudo, it encapsulates the EDITOR command-line editor as an environmental variable and checks the syntax before saving. Both uupdate and dh_make add a changelog entry that you just need to expand.

Satisfying Dependencies

Once you've installed the previously mentioned version 1.5 dev package on your system, the buildpackage takes its course. As with the update from version 1.3, a binary package is created without your needing to worry about details at compile time.

The many dh helper scripts combine into a powerful autopilot that compiles the source code of many programs without the manual application of the rules file. The command dh auto_configure automatically determines which build system (Autoconfigure, Cmake or Qmake) the program uses and makes the appropriate calls.

According to the dh_make developers, the success rate is around 90% for all programs. This means that you only need to run dpkg-source, dh_make, and dpkg-buildpackage with the source code and enter simple descriptions in the control file. You then automatically get a Debian package for the automatically compiled program.

That's an impressive achievement – although in practice a success rate of 90% without manual intervention might be a bit ambitious. A blind package development might not always be successful. Debugging errors implies knowing the content of the debian/ control directory and the function of the files contained therein.

Figure 4 shows the content that dh_make automatically creates in the upstream ffDiaporama source code. Fortunately, only three of the many files are absolutely necessary for building the package: control, rules, and changelog. The remaining files are meaningful for Debian policy-compliant packages [6], but play no role in building packages for your own use.

Figure 4: The dh_make tool prepares the software's source code for automatic compiling and packaging. The "debian/" control directory contains numerous template files (on the left), but only three of them (on the right) are strictly necessary.

Nitty-Gritty

I'll now compare the three relevant files in the original ffDiaporama package from the Ubuntu repository with the versions created by dh_make. They clearly show how they are used in automation, even if not all the visible changes for a successful build package process are clear.

The control file is a text file with descriptive data. Figure 5 shows the template that dh_make creates on the left, with the completed version from the Ubuntu package from Launchpad on the right. The short description (the first line after Description:) and the following multiline descriptions can easily be filled with the text from the program website. Added to it are the upstream URL and (specific to Debian) the uploader and the Git repository of the Debian package developers. Packages for home use don't need these fields; you can just replace the text between brackets on the left.

Figure 5: The "control" file contains descriptive data for package building and the subsequent binaries.

For Depends:, the automatically created control file contains only the variables ${shlibs:Depends} and ${misc:Depends} that the automatic build process completes with automatically calculated dependencies. Among them are all the shared libraries that link to the binaries contained in the package but not the standalone background programs. ffDiaporama calls FFmpeg and uses for it the ffmpeg packages as a (manual, exceptionally) registered dependency.

For your own packages, the entries under Build-Depends are unnecessary and are intended for repositories to be built on the distribution servers in a minimal environment. However, if your system provides all packages required for compiling, the buildpackage process can function even with empty Build-Depends. The build dependencies have no effect on the binary package during the build process.

Grayed out in Figure 5 on the right is a second Package: definition. Ubuntu developers divided ffDiaporama into two packages, ffdiaporama and ffdiaporama-data. For this to work, dpkg-buildpackage needs files with the names ffdiaporama.install and ffdiaporama-data.install that contain lists of files belonging to the subpackages. In this way, the distributors separate the 32-bit and the 64-bit installed system files from those used equally across platforms.

This process, however, belongs to a higher level of package building artistry. If the lines after Package: ffdiaporama-data and the .install files are missing, then a single ffdiaporama package is created that contains all files related to the program. It functions flawlessly on the architecture upon which it's built. The Debian build system (unlike RPM) then doesn't need a file list for its package content; it simply packages all the files added by make install.

Control Center

Although control contains only descriptive data, rules determines the course of the build packaging process, including compiling the source code. The rules file is a Makefile, like that used during the program compile. Figure 6 shows the version created by dh_make on the left, with the version from Ubuntu ffDiaporama on the right.

Figure 6: The "rules" file in the debian/ subdirectory is the key Makefile that drives the entire package build process.

I'll stay with the default version for now: Apart from the comments, it consists of the following equally short and cryptic lines:

%:
   dh $@

To understand these lines requires some basic knowledge of the GNU Make macroprocessor. In Makefiles, <name>: is a called a "target."

Thus, make install, for example, executes all shell commands that follow install: in the Makefile. The %: is a wildcard target that Make always executes, regardless of whether you specify make love or make war.

The $@ variable in the Makefile contains the name of the evoked targets. Therefore make -f rules dh_auto_configure calls on dh dh_auto_configure. Because rules is an executable file and the first shebang (#!) line selects Make as the interpreter, the direct call rules dh_auto_configure works. (The entry after the shebang (e.g., #!/bin/sh) refers to the interpreter of the code in this file. This mechanism is exclusive to Unix-like operating systems.)

Through the Makefile indirection that passes the target name with the dh parameter, dpkg-buildpackage then calls dh clean, dh build, and fakeroot dh binary. These three calls constitute the three main phases of package building: cleaning the source directory of possible previous build attempts, compiling the source code (./configure and make), and packaging the compiled files in a Debian.

All three dh calls start further helpers who perform the actual work. Their numbers alone reflect Debian's typical thoroughness: Three scripts remove remnants of any previous compiler run, four configure and compile the source code (./configure, make or their equivalents), and all of 42 are responsible for the make install call and creating the Debian package (Figure 7).

Figure 7: Using the Makefile rules file, dh_buildpackage makes three calls, starting a slew of helper scripts that compile the source code and build the Debian package.

A side note: dh binary and its subordinate helpers run under Fakeroot. Within the Fakeroot environment, they can write files of root ownership without having "real" root privileges. That's essential because files installed system-wide need to be root for security purposes. Fakeroot, however, prevents an erratic make install call from damaging the system.

Bank Shot

The reason for the indirect call to dh <parameter> via Makefile is evident from a look at the right-hand version of rules from the original ffDiaporama package in Figure 6. The targets beginning with override marked in red demonstrate how dh_<xxx> calls are replaced by shell commands: Every override_ target signals the build system to execute it instead of the target's dh_<xxx>.

The dh_auto_configure call replaces the original Ubuntu rules file with the qmake-qt4 ffDiaporama.pro command, the Qmake-specific equivalent of the familiar ./configure.

Maintainers might have considered this safer than relying on automatic creation. However, the automatic version created by dh_make also works: The help script apparently detects Qmake as the correct build system. There are cases, however, where only replacing Debian helpers with shell commands works.

Moreover, the dh_fixperms override combines console commands with Debian helpers instead of completely replacing them – a common approach in practice. In this specific case, an automated check for security vulnerabilities in file privileges does not allow write permissions to be added by hand before calling dh_fixperms.

The third override doesn't replace dh_install, but rather gives the helper script a calling parameter only. Which ones the Debian helpers understand as well as their role in the build system are described in the script manpages [7].

An important practical tip: If dh_auto_configure, for example, calls the correct configure command, but you still want to supply a parameter, then you can write an override_dh_auto_configure: target with the content dh_auto_configure -- --<parameter>. The helper script passes all parameters after the -- unchanged to the called command.

From Layman to Master

You now know the basics of Debian package creation, which is usually enough to build your own packages for many applications. For problem cases, the Debian New Maintainers' Guide [8] can help, especially chapters 2.4 through 2.9, 4, 6, and 9.

If the autopilot fails, you first need to find out how a program compiles at the console. Then, you can put your findings into the auto_configure, auto_make, and possibly auto_install override targets in rules. Mastering all the subtleties of the Debian build system, however, can't be accomplished in a day.

Given the dizzying array of helper scripts, keep in mind that, with the abundance of full Debian repositories, it's seldom necessary to build a package at all. Updating Debian sources with new program versions, especially minor ones, usually goes without a hitch – and usually takes no more than a quarter of an hour.