Systemd is the result of the work of many developers, but, if there's a public face to be given to the software, it's certainly that of Lennart Poettering.

The valiant developer has already checked off a number of important projects on his to-do list, including Avahi and the PulseAudio sound system. With systemd, he's been working since 2010 on getting central services on a modern, uniform basis – emanating from a new approach for the init system.

Poettering doesn't shy away from unpopular decisions, which has earned him the reputation of using dictatorial methods. The decision to develop systemd only for Linux brought criticism, scorn, and insults. For him, meanwhile, the decision has a clearly understandable foundation.

Along with his work on systemd, Poettering is advocating for a unified Linux desktop. His newest project, which he's pursuing together with kernel guru Greg Kroah-Hartman, deals with integrating the Dbus message system as Kdbus in the kernel. His announcement at Google Plus [1] that systemd, together with Kdbus and all userspace tools in Fedora, starts without a problem and, thus, represents a new milestone, resulted in yet another hailstorm of harsh criticism.

Such opposition, however, has never diverted the Red Hat developer from his course (see the "Lennart Poettering Interview" box). He likes to speak his mind (Figure 1) as a free-thinker with enough backbone and support to see his vision through and perhaps remove a few old Linux relics.

Figure 1: Lennart Poettering's t-shirt shows his opinion on the sayings of Mark Shuttleworth.

To the Roots

If you're dealing with an init system, you very quickly recognize why working on this key component can cause so much turmoil. The term comes from "initiating" and encompasses, on Unix-like operating systems, the system startup and orderly shutdown.

Init is the first process the kernel generates, and it starts all other child processes. It's therefore assigned process ID 1 (PID 1). Among init's tasks are essentially starting all necessary services, mounting the filesystem, and setting up the network.

The SysVinit [2] init system used on Linux and other Unix derivatives since the 1990s was developed in 1988, when AT&T needed a start mechanism for its System V operating system in version R4.

In 2004, Solaris replaced SysVinit with its successor, Service Management Facility (SMF) [3], followed for the Mac OS X in 2005 by launchd [4]. Canonical has used Upstart [5], developed in-house based on SysVinit, since 2007. With Gentoo's FreeBSD offshoot, OpenRC [6] has been an alternative since 2007.

Born of Necessity

Systemd essentially comes from Red Hat developers Lennart Poettering and Kay Sievers. Before the two set upon the task of remaking the init system under Linux in 2007, they took a close look at launchd and SMF, because these init systems provided parallel processing startup capabilities that the static SysVinit didn't have.

Because Upstart also had this ability, the original idea of the developers was to enhance Ubuntu's init system. Legal reasons that had to do with Canonical's Contributor License Agreement (CLA) [7] prohibited this – in that neither Poettering nor Sievers was willing to sign the agreement.

Thus, systemd was born of necessity. Two fundamental decisions of the time still raise hackles: Systemd was deliberately and without compromise designed for Linux only. Systems with other kernels such as BSD and Hurd simply stood on the sidelines; a porting to BSD was possible, but expensive.

The other decision grated against the old Unix principle of "one tool for one job" in that it set some of the jobs pretty close together. This may loosen up slightly in the future so that functions are available only individually, as Ubuntu currently does with Upstart in conjunction with logind.

More than init

Systemd is short for system daemon, that is, a background process that provides services for the system. The init process was at the beginning of the development and still gets most of the attention. Other components such as logind and the journal, however, are up there as well.

Systemd's init process is based on a slim PID-1 service that does only the bare minimum. Thanks to parallel processing, the computer's startup and shutdown runs much faster. It's based on a technology that Apple developed for launchd: socket activation.

Socket activation allows all services to start concurrently. The process also got simpler because developers and administrators no longer need to be aware of which service starts at which point during bootup with which dependencies. The trick is that systemd itself creates the communication sockets with the services to be started and buffers the data until the service is running and accepts it. The SysVinit concept rather forces dependent services to wait for each other, because the init system doesn't store any data itself.

Because all services start concurrently, systemd contributes greatly to simplifying the structure of the service files. These replace the SysVinit shell scripts that often went wild. Figures 2 and 3 show the differences based on the Kexec [8] example.

Figure 2: SysVinit: The Kexec startup script elicits constructs at various places where the purpose isn't readily apparent.

Figure 3: Systemd: The Kexec service file needs only a few lines whose function is clearer based on the variable names.

Systemd organizes all tasks pending at startup – such as readying the hardware, mounting the filesystem and starting the services – in units. The service files of the various units have their own filename extensions, which give a clue as to their functions.

Files with the .mount extension take care of mounting and unmounting the filesystems; those with the .service extension handle the background services. These units are stored uniformly in /lib/systemd/system . For a modification, copy the files in question manually to /etc/systemd/system/ and edit them as needed; the system automatically uses them from then on.

An important decision in designing systemd that contributed to its being used only in Linux was the integration of cgroups [9]. These control groups allow allocating and limiting kernel resources, such as CPU time, data throughput, and process group memory usage.

Systemd therefore provided new unit types as of version 205, including scope units and slice units. The former collect the system services or applications and all their dependencies into cgroups and manage them. The slice units provide the units with the required resources.

In the long term, the developers want to hand over the use and control of the cgroups completely to the daemon. In this way, systemd would become a new intermediate layer between the kernel and applications. Poettering himself sees it as a "glue." In any case, the structure thereby acquires some complexity (Figure 4).

Figure 4: The different layers of the systemd infrastructure make it difficult to consider it a simple service.

Journal

Unlike SysVinit, systemd logs the boot messages right from the beginning. Thus, screenshots of the boot process captured by smartphone for support forums become a thing of the past. Systemd instead creates a socket [10] for it at runtime, on which a minimal logging service runs and records the messages from the early boot phase. It later passes the socket to the regular protocol services and the message is journaled.

The journal replaces the long-used Syslog. In a first demonstration, it received much criticism because the journal deviated from the ongoing concept of storing text files in a flat hierarchy. Instead the files are stored as binary.

Apart from that, the journal has some hard-to-ignore advantages. Besides the aforementioned boot process logging, it allows for highly targeted queries. Thus, the command on the first line below shows the messages since the last system startup; whereas the second line shows what the web server put out in messages since midnight. In a blog entry, Poettering shows other more detailed queries [11].

$ journalctl -b
$ journalctl -u httpd --since=00:00 --until=9:30

As the name logind [12] suggests, the daemon takes care of user and session management and replaces the no longer developed ConsoleKit [13]. Thus, systemd begins to enter userspace, which again was not met without criticism.

In the end, Poettering wants to go still further by using the same mechanism for the processes starting Gnome or KDE and its applications. Apple's Mac OS X already does this. The first approaches can already be seen in Gnome 3, which has recently been pulling in parts of systemd by integrating logind.

Neverending Story

Distributions such as Fedora and openSUSE, but also Debian derivatives such as Siduction, Semplice, and soon Tanglu, have already switched over to systemd (Figure 5).

Figure 5: Front ends already exist to exhibit systemd activities.

The dependencies to parts of systemd were the catalyst for hard decisions about the future init system in Debian. The discussion began in October 2003 with news on the Debian developer mailing list [14]: A complaint was filed about the dependency of the Gnome settings daemon on systemd – introduced by the hope that it wouldn't "turn into some kind of flame war."

But, that's just what happened. The discussion became heated and with no result, until it became clear after a week that no solution was possible. Another developer requested that the Debian Technical Committee (CTTE) [15] should take on the problem [16] and come up with a decision [17].

The CTTE familiarized itself with the technical issues and tried to find a solution supported by all its members. The board comprised eight members – of which four soon supported systemd, with the other four supporting Upstart. Of the latter, two were currently working at Canonical, and a third was a former Ubuntu employee.

The Debian project has its particular problems because of its many supported architectures. Thus, it was clear from the beginning that the Hurd and KFreeBSD platforms weren't working with systemd. This proved to be a problem technically and politically for the largest distribution that decides things without any corporate backing.

Of the three candidates that could replace SysVinit, only one could come to the rescue: OpenRC, developed in Gentoo. Although it currently has little chance of establishing itself as the default init system, with some effort, it could run both architectures.

In the case of Upstart, a fork of the software would apply, because Canonical only allows collaboration when the aforementioned CLA is signed (see the "Canonical's Gag" box). Mark Shuttleworth would probably rub his hands if Debian took over the further development of Upstart.

Early February saw a second set of ballot initiatives [21] that committee member Ian Jackson (a Canonical colleague) proposed, without resolution, only to be followed by a third. The third vote was like the first, a simple proposal – with one difference: It included a clause that allowed Debian developers to adopt the ultimate decision-making tool at Debian, the General Resolution (GR) and thereby repeal the decision of the CTTE with a simple majority [22].

Normally, such a decision needs a two-thirds majority at Debian. The GR needs six official Debian developers to agree. In mid-February, the third ballot led to almost four months of discussion. Finally, Bdale Garbee, chair of the committee, broke the stalemate by invoking for the first time a special voting procedure for such situations and declared systemd a winner [23].

Whether the developers tilt the vote again with another GR has yet to be seen. Systemd would probably win again but would have a broader base. Thus, the argument of an arbitrary decision would be removed from the hands of any critics, with Upstart having hardly a chance in the process. The procedure, however, showed that the CTTE statutes might require some reform. Attempts by one or the other members to drag out the process and torpedo it are hard to overlook.

In his blog, Canonical founder Mark Shuttleworth conceded a few days after the decision to be the "graceful loser" and announced that even Ubuntu would make the switch to systemd in the long run, provided developers declared it to be as stable as the previous solution, Upstart.

Thus, with Debian and Ubuntu, the last major distributions are on course with systemd. Together they form a critical mass to influence its development in keeping with Debian. There may also be less contention among the distribution warehouses. Gentoo and Slackware are the sole outsiders; Gentoo continues to rely on OpenRC, while Slackware will probably keep the faith with SysVinit.

Optimizing the Boot Process with Systemd

Systemd provides a few tools to analyze the boot process and optimize it. These tools all work under a user account, which means no root privileges – the corresponding command is systemd-analyze .

In our test, on a workstation with KDE and a good many services and SSD for the whole system, the result was not convincing (Listing 1, line 1). A freshly installed current notebook with KDE started faster (Listing 1, line 2). On a lighter desktop environment, the values were in the three-second range.

01 02 Startup finished in 10.432s (kernel) + 6.287s (userspace) = 16.720s
03 Startup finished in 1.327s (kernel) + 3.116s (userspace) = 4.443s

If the startup takes too long, you can use systemd-analyze blame to test which processes and services are to blame. On an older notebook, the blame option revealed that the NTP service was stuck in a loop for more than 30 seconds. As it turned out, it was trying to contact a server that no longer existed, so a configuration correction fixed the problem.

If necessary, you can create a graph with systemd-bootchart that shows the behavior at boot time (Figure 6). The command is as follows:

Figure 6: Using a boot chart, you can tell at first glance which services are coming up on startup.

$ systemd-analyze plot > test.svg; inkscape test.svg

In this case, the Inkscape application helped in viewing the graph. Alternatives include image viewers such as Gthumb or the Firefox web browser.

Conclusion

The switchover to systemd does not present high hurdles to either distributions or users. In the near future, systemd remains compatible with Sysvinit so that the well-known commands for manipulating services work just as well as those for shutting down, restarting, and changing run levels.

Systemd proves to be a good solution for Debian and Ubuntu, to stand among the others, such as Fedora, openSUSE, Arch Linux, and Mageia, who are already using it. If the developers have any voice, systemd will help standardize processes for all distributions, providing advantages for developers and administration alike. More details on systemd are provided in Lennart Poettering's blog [24].

Some developers are accusing Red Hat of striving for more control over Linux with systemd. Upstart, on the other hand, would pull in a system that is totally under a company's control. Viewed objectively, systemd provides the best technically suited init system for the foreseeable future. Debian would be best to recognize this and be up front about it.l