When it comes to managing system services, a Linux system administrator can choose between the classic init [1] and the relatively new systemd [2]. These utilities form a meeting ground for mature technology and new concepts.

Post-Boot

Directly after booting the kernel, a service will start and then accompany the rest of the system's ongoing operation and shut down. Throughout, this service is assigned the Process ID 1.

You can use ps on a running system to determine which program executes the administration of services. (See the instructions in the "Determining Process 1" box.)Whatever the result, you may be misled because of a possibly existing downward-compatible invocation, which may have convinced you that init is performing this task when it is indeed systemd.

~$ ps -ax | head -2
  PID TTY      STAT   TIME COMMAND
    1 ?        Ss     0:01 /sbin/init

~$ ps -e | head -2
  PID TTY          TIME CMD
    1 ?        00:00:01 systemd

~$ ps -e | head -2
  PID TTY          TIME CMD
    1 ?        00:00:00 init

System States (init)

On Linux and other related systems, a runlevel defines a certain system state, which may include services that are running (daemons).

In Table 1, you will find a list of the possible system states. Except for S , all of the other states are indicated with a numerical value running from 0 to 6. The runlevels between 2 and 5 are not used uniformly by the various distributions.

You can determine what the default runlevel at system start is by looking at the beginning of the /etc/inittab file. Listing 1 shows a relevant snippet. The standard runlevel of the system, initdefault , is indicated in the line beginning with id . The default runlevel is 2 in the example.

01 .
02 .
03 .
04 # The default runlevel.
05 id:2:initdefault:
06
07 # Boot-time system configuration/initialization script.
08 # This is run first except when booting in emergency (-b) mode.
09 si::sysinit:/etc/init.d/rcS
10
11 # What to do in single-user mode.
12 ~~:S:wait:/sbin/sulogin
13 .
14 .
15 .

The single user mode (si: ) must have previously finished, except in emergency mode due to interruptions of the boot process. There's an important entry at the end of the listing. init executes an automatic login for root as a necessary precondition for operating the system in single user mode.

Starting Scripts (init)

The init service invokes the start and stop scripts belonging to the applicable runlevel so that the desired system state can be established. In Debian based systems, such as Ubuntu, you will find the scripts under /etc/init.d .

If you use the package manager of your Linux distribution, you will find that the scripts have already been set up with the installation of the service. It is seldom the case that you will need to do a manual set up. However, if you do find yourself needing a script, then you should use the files that already exist as a model [3].

Listing 2 shows the start and stop scripts for atd , which are set up by the distribution.

01 #! /bin/sh
02 ### BEGIN INIT INFO
03 # Provides:          atd
04 # Required-Start:    $syslog $time $remote_fs
05 # Required-Stop:     $syslog $time $remote_fs
06 # Default-Start:     2 3 4 5
07 # Default-Stop:      0 1 6
08 # Short-Description: Deferred execution scheduler
09 # Description:       Debian init script for the atd
                        deferred executions
10 #                    scheduler
11 ### END INIT INFO
12 #
13 # Author:       Ryan Murray <rmurray@debian.org>
14 #
15
16 PATH=/bin:/usr/bin:/sbin:/usr/sbin
17 DAEMON=/usr/sbin/atd
18 PIDFILE=/var/run/atd.pid
19
20 test -x $DAEMON || exit 0
21
22 . /lib/lsb/init-functions
23
24 case "$1" in
25   start)
26         log_daemon_msg "Starting deferred execution
           scheduler" "atd"
27         start_daemon -p $PIDFILE $DAEMON
28         log_end_msg $?
29     ;;
30   stop)
31         log_daemon_msg "Stopping deferred
           execution scheduler" "atd"
32         killproc -p $PIDFILE $DAEMON
33         log_end_msg $?
34     ;;
35   force-reload|restart)
36     $0 stop
37     $0 start
38     ;;
39   status)
40     status_of_proc -p $PIDFILE $DAEMON atd && exit 0 || exit $?
41     ;;
42   *)
43     echo "Usage: /etc/init.d/atd {start|stop|restart|force-reload|status}"
44     exit 1
45     ;;
46 esac
47
48 exit 0

When you modify the configuration of a service (i.e., a daemon), you can manually stop and restart the service with this script (Figure 1). You can also ask about the daemon's. To execute the desired action, you should include start , stop , and status when you call the script.

Figure 1: Stop, start, and status inquiry of a service using an init script.

Runlevels and Directories (init)

Each runlevel has its own directory where there are links that point to /etc/init.d/[SCRIPTNAME] . The link name begins with an S when it stands for a start script or with K for kill script .

The alphabetical order determines the processing sequence. When a service is installed with a package manager, its links receive appropriate name. When you compile a service, then you need to figure out the order yourself. For example, you must remember a web server can only be started once the network is already running. In the worst case, you can always name your script S99zzzzz .

The runlevel directories rc0.d , rc1.d , rc2.d , rc3.d , rc4.d , rc5.d , rc6.d , and rcS.d are located under /etc . In the latest Debian and Ubuntu versions, the directories rc3.d , rc4.d , and rc5.d have almost identical content to rc2.d and are not usually used. This is an opportunity for the tinkerer; you can, for example, create groupings of server services.

The script for this example atd is located in /etc/init.d . The S15atd link for startup is found in /etc/rc2.d . The links that init calls to end this daemon are found in several directories, namely /etc/rc0.d , /etc/rc1.d , and /etc/rc6.d . In each, the link is named K01atd . This action is presented more clearly in Listing 3.

Within the <I>/etc<I> directory:

init.d/atd
rc0.d/K01atd -> ../init.d/atd
rc1.d/K01atd -> ../init.d/atd
rc2.d/S15atd -> ../init.d/atd
rc3.d/S15atd -> ../init.d/atd
rc4.d/S15atd -> ../init.d/atd
rc5.d/S15atd -> ../init.d/atd
rc6.d/K01atd -> ../init.d/atd

Inittab Configuration (init)

Note the inittab file is not currently used in Ubuntu and hasn't been used since Upstart was included in Ubuntu 9.10 – Karmic Koala. However, old installations and other distros still include the inittab, even those that have passed on to systemd.

Therefore, knowing about this file is still useful.For the record, the inittab file controls what happens whenever a system is rebooted or forced to change run levels.

This file has entries in the following form:

id:Runlevel(s):Action:Command

The first field, id , indicates the entry clearly and uniquely. You can list one or more runlevels in field 2 to which the entry should apply. When there are several runlevels, they must be entered in ascending order. See the excerpt in Listing 3 for the id ca .

The third field of the entry in /etc/inittab tells init how it should behave. You will find some of these applications in Table 2.

You have already seen the beginning of /etc/inittab in Listing 1. The file contains additional possibilities for interesting settings. There is an excerpt from /etc/inittab in Listing 4. At the beginning of Listing 4, you can see an instruction for init telling it to wait for the finish of the /etc/init.d/rc script for the indicated runlevel.

01 l0:0:wait:/etc/init.d/rc 0
02 l1:1:wait:/etc/init.d/rc 1
03 l2:2:wait:/etc/init.d/rc 2
04 l3:3:wait:/etc/init.d/rc 3
05 l4:4:wait:/etc/init.d/rc 4
06 l5:5:wait:/etc/init.d/rc 5
07 l6:6:wait:/etc/init.d/rc 6
08 # Normally not reached, but fallthrough in case of emergency.
09 z6:6:respawn:/sbin/sulogin
10
11 # What to do when CTRL-ALT-DEL is pressed.
12 ca:12345:ctrlaltdel:/sbin/shutdown -t1 -a -r now
13
14 # Action on special keypress (ALT-UpArrow).
15 #kb::kbrequest:/bin/echo "Keyboard Request--edit /etc/inittab to let this work."
16
17 # What to do when the power fails/returns.
18 pf::powerwait:/etc/init.d/powerfail start
19 pn::powerfailnow:/etc/init.d/powerfail now
20 po::powerokwait:/etc/init.d/powerfail stop
21
22 # /sbin/getty invocations for the runlevels.
23 #
24 # The "id" field MUST be the same as the last
25 # characters of the device (after "tty").
26 #
27 # Format:
28 #  <id>:<runlevels>:<action>:<process>
29 #
30 # Note that on most Debian systems tty7 is used by the X Window System,
31 # so if you want to add more getty's go ahead but skip tty7 if you run X.
32 #
33 1:2345:respawn:/sbin/getty 38400 tty1
34 2:23:respawn:/sbin/getty 38400 tty2
35 3:23:respawn:/sbin/getty 38400 tty3
36 4:23:respawn:/sbin/getty 38400 tty4
37 5:23:respawn:/sbin/getty 38400 tty5
38 6:23:respawn:/sbin/getty 38400 tty6

You can also see the instruction for what should happen when the key combination Ctrl+Alt+Del is pressed. The computer start is preset for all runlevels except 0 and 6 . The pf , pn , and po entries specify how to handle a power outage in the presence of an Uninterruptible Power Supply (UPS).

The block beginning with # Note that on most Debian ….. determines console activation. The first console, tty1 , is activated for the runlevel 1 with 5. The virtual consoles tty2 and tty6 are reserved for runlevels 2 and 3. In Debian, the graphical user interface is on tty7 . If you need more virtual consoles, and if you are working on a Debian computer, you can begin the setup of additional consoles starting with tty8 .

Runlevel and Service Administration (init)

You can find out the current and previous runlevels using the command who -r (Figure 2). The commands in Table 3 are used for a runlevel change, including shutdown and restart of a computer.

Figure 2: Determining the current and previous runlevels.

When you use the kill -1 1 command, init reads its configuration again without restarting the computer. You can power down the system using the kill -9 1 command.

systemd

In contrast to init, systemd, used by Ubuntu as from 15.04 – Vivid Vervet, starts services in parallel, making the operating system load faster. To ensure that this works, systemd itself sets up the sockets, which the services will use for communication once they are started. It then buffers any data to be handed to a service until the service has successfully started and can accept them.

If the service crashes, then systemd can restart it. All accesses by the client applications are buffered during the restart and are then processed by the daemon. The sockets set up by systemd can be transferred to other executing programs. These then assume control of the sockets.

It is possible to track processes that use cgroups . This feature is a kernel function, which collects processes together with their child processes into groups with a hierarchical structure.

A cgroup receives its name from the service, and a cgroup is formed for each service under systemd. This process grouping lets you avoid the type of chaos that can arise when there are numerous external processes under one service. Additionally, this approach prevents some of the external processes from surviving when the service crashes.

Not all of the services necessarily need to be started at system start. Instead, many can be executed when only when they are needed for the first time, which speeds up the boot process quite a bit. For this to happen, it is necessary to evaluate the activity on a network or IPC socket or a FIFO buffer. In this regard, systemd partly assumes functionality from inetd .

It is possible now to make a lot of invocations without being the root user. Additionally, systemd takes on the administration of mounts. It is also possible to take snapshots of the state of the system to later restore it if needed.

The runlevels continue to exist in the form of targets. You can figure out the purpose of the .target files from their names – for example, in halt.target , which powers down the system.

As with Upstart, systemd ignores the /etc/inittab file. However, the scripts for administration of services, known as RC scripts, are supported.

Configuring Services and Resources (systemd)

The directions for configuring and executing services are stored in so-called units. These are interchangeably named units or services . The various unit types are described in Table 4. The individual configuration files exist as plain text and are structured like INI files. Thus, it's possible to edit them with any text editor.

Continuing from the example I used previously in the section on init, for the at daemon, you will find the structure of a typical unit file in Listing 5.

01 [Unit]
02 Description=Deferred execution scheduler
03 Documentation=man:atd(8)
04
05 [Service]
06 ExecStart=/usr/sbin/atd -f
07 IgnoreSIGPIPE=false
08
09 [Install]
10 WantedBy=multi-user.target

These files and links to them are always stored in /lib/systemd so that they are accessible when booting. In the case of Debian and Ubuntu, you will find additional unit files stored in /usr/lib/systemd . Modified unit files are located in /etc/systemd/system . Files that the system created at runtime are located in /run/systemd/system .

If you want to create or modify your own service unit files, you can copy files your want to modify from /lib/systemd/…../ to /etc/systemd/system/ . You can also save new unit files there. However, it is not a good idea to ever save modified or new unit files in /lib/systemd/ because they will be overwritten during updates of systemd .

The unit files have various sections which contain specifications and assignments. Table 5 shows a large selection; the entire list can be found in the manpages [4]. A unit file will contain only one section in addition to [Unit] and, if necessary, [Install] .

Runlevels and Targets

The runlevels continue to exist in the form of corresponding targets. However, they differ by distribution. The entries here originate from a Debian 8, but are more or less the same in Ubuntu. Table 6 contains a list of comparisons.

Your Own Service (systemd)

After all that theory, I'll now show how to create your own systemd service. The shell script in Listing 6 collects a list of the hosts found on the network at a given interval. It is saved in executable form under /usr/sbin . You will be able to see the results using tail -f /tmp/netlist.txt .

01 #! /bin/sh
02 while true;
03 do
04
05  echo "List of active network users" > /tmp/netlist
06  echo "------------------------------------" >> /tmp/netlist
07  date +%d.%m.%Y-%H:%M:%S >> /tmp/netlist
08  echo "------------------------------------" >> /tmp/netlist
09
10  # Execute fping and save entire output in log file
11
12  fping -r 0 -g 192.168.0.0/24 > fping.log 2>&1
13
14 # Change 192.168.0.0/24 to whatever works for your network
15
16  # Filter out the unreachable hosts
17
18  cat fping.log | grep "alive" | sort  >> /tmp/netlist
19  echo "------------------------------------" >> /tmp/netlist
20  sleep 120
21 done

You should first create the shell script shown in Listing 6 and save it in the /usr/sbin directory as netshow.sh (for it to work, you may have to install fping first using apt-get ). Remember to make it executable with

sudo chmod 700 netshow.sh

so that systemd can start the program. Then, save the unit file netshow.service (Listing 7) in /etc/systemd/system .

01 [Unit]
02 Description=Listing of active hosts
03 Documentation=man:fping(8)
04
05 [Service]
06 ExecStart=/usr/sbin/netshow.sh
07 IgnoreSIGPIPE=false
08
09 [Install]
10 WantedBy=multi-user.target

Now systemd must be instructed to process the unit file and start the application. To do so, the service must be enabled with sysctl in order to establish a permanent start:

sudo systemctl enable netshow.service

In the process, the required symbolic link will be created in the target directory multi-user.target .

You then start the service with the following command:

sudo systemctl start netshow.service

You can check to see whether the service is running with the following command:

systemctl status netshow.service

You can see the entire installation procedure in Figure 3.

Figure 3: Installing a service.

Additionally, the outcome of the executing service can be found in Figure 4.

Figure 4: The log created by the service.

Systemd Fundamentals

Knowing a few rules can make working with systemd straightforward and successful. Here are some important things to remember:

• A unit must be enabled so that you can start it manually or automatically.
• A unit must be disabled so that it cannot be started again.
• A unit, or a non-modified unit file you created on your own, should be masked to protect it from inadvertently being enabled.
• You should move a unit which has a service file in /etc/systemd/system up one directory level. This ensures the file against inadvertently being enabled.
• A masked unit can only be re-enabled if it is unmasked beforehand. There is no unit file in /etc/systemd/system .
• You should copy or move a file that you have created or modified to /etc/systemd/system .

When masking a unit, a link pointing to /dev/null will be saved in the target directory. This indicates to systemd that the unit file with this designation should not be considered. A concrete example is provided in the following section.

Administration with systemctl

Now that you have become acquainted with the systemctl program for controlling systemd, the most important options are listed in Table 7.

It is possible to fine-tune many options with additional entries. It is not possible, however, to list all of these within the confines of this article. The following examples provide a lot of information for everyday use.

Figure 5 is a complete example of how to terminate and mask a service, PostgreSQL-RDBMS, and then how these measures are reversed.

Figure 5: Working with systemctl.

The unit file is not in /etc/systemd/system . You will find all of the processes belonging to the service for the status request systemctl status …. . Additionally, the status itself and its lifetime is indicated. You can receive a listing of all unit files with systemctl list-unit-files (Figure 6).

Figure 6: List of all unit files.

Alongside the states discussed previously (enabled , disabled and masked ), you will also find the state static . In this state, the unit file is not enabled, but it has no pertinent instructions in its [Install] section. Therefore, it cannot be controlled by systemctl for a variety of reasons: The particular unit might be referenced by other units, such as .wants , .requires . Another reason could be that the unit should be activated via a socket, timer, D-Bus, or udev.

It is possible to refine your state requests by including more conditions, for example --state . Frequently, it is easier to filter with grep . If, as in the example given, you would like to list only the masked unit files then you can use the following command:

systemctl list-unit-files | grep masked

You can also filter according to the type of unit file (Figure 7). The call

systemctl list-units --type=mount

Figure 7: Filtered list of unit files by type.

lists all units of the Mount type.

By the way, systemd without any arguments lists all of the units that have been started after the boot process.

Listing Processes (systemd)

The administration via systemd puts each service into its own kernel control group or cgroup . This lets you determine, at a glance, which processes belong to which unit.

You will find a tree view similar to the one in pstree (Figure 8).

Figure 8: A process tree presented with systemd-cgls.

Creating a Container (systemd)

You can create containers with the systemd-nspawn instruction. This topic is too big for the space restrictions of this article, so that subject matter receives only a mention here. Containers can be queried with the following command:

systemctl list-machines

See Figure 9.

Figure 9: Listing of a container.

Administration with systemadm (systemd)

This GUI program, available on Ubuntu via the systemd-ui package, makes it possible to quickly and easily administer systemd and the services [5][6].

The interface is self-explanatory and the operation itself takes little getting used to. Figure 10 shows a status request.

Figure 10: Status notification with systemadm.

syslog vs. journald

Now that you are using systemd for your services, you should also use systemd-journald instead of syslog . This can be handled as you prefer. For example, you can change from syslog to systemd-journald or vice versa.

When you use journald , the log data will only be saved during runtime. You can modify the configuration by changing entries in /etc/systemd/journald.conf . By using the setting Storage=persistent you can permanently save the journal in /var/log/journal . You should limit the size of the log with something like SystemMaxUse=100M . Listing 8 shows all of the items in this file that have been modified. Note that the log files from systemd/journald are not searchable with customary text tools like find, grep, etc. You will need to use journalctl .

01 [Journal]
02 Storage=persistent
03 ......
04 SystemMaxUse=100M
05 ......

To see the logs using journalctl , you can enter the command without any other option and the program will list a complete journal. Table 8 contains some filtering possibilities.

Figure 11 shows a journal request that was limited to one service, apache.2service .

Figure 11: Journal inquiry for apache2.service.

Conclusion

Now you know quite a bit about both control programs for starting the systems and services. If you are familiar with SystemV Init, then the switch to systemd may take some getting used to. The work you invest will pay off, though, because many more possibilities for configuration are waiting for you.