The ZFS filesystem [1] was developed by Sun Microsystems and released as open source to the public in November 2005 as part of OpenSolaris. Then, as now, it offers features not found in other filesystems. One of the main design aims was to make it as robust and reliable as possible. ZFS stores not only the data but also cryptographic checksums of the data, which allow it to identify data that has been altered because of bit-rot, power failures, or people playing with magnets. Today, after years of development, improvements, and additional features, there is hardly anything ZFS does not do.

Although many features, such as support for various RAID modes, hot-swapping of disks and very large limits on data size, number of files and similar attributes, are geared toward the enterprise user, plenty of others will be of interest to home users. For example, ZFS supports transparent compression, with a choice of different compression algorithms and levels. Data can be de-duplicated automatically, making all those backup copies less space-consuming. Speaking of backups, ZFS supports block-level snapshots that only consume space when files actually change as well as the ability to send and receive those snapshots as filesystem streams over the network. Anybody who has had to run rsync on large numbers of files will appreciate the performance improvement that this brings.

Although ZFS has been the main filesystem on OpenSolaris since 2005 and has been fully supported on FreeBSD and other BSDs for several years, its adoption by the Linux community [2] has been somewhat slower, not because of lack of enthusiasm, but rather because of legal issues. ZFS was released under the Common Development and Distribution License (CDDL). This license is not compatible with the GPL, meaning that ZFS cannot be bundled with the Linux kernel. One way around this restriction was to implement ZFS support through a Filesystem in Userspace (FUSE) module.This approach comes with a number of limitations, most notably, a significant performance penalty is incurred for running through FUSE, and ZFS cannot be used as a root filesystem.

More recently, the ZFS on Linux project has provided another solution to this problem. It implements ZFS support as a kernel module that does not rely on FUSE, but exploits a licensing loophole. As long as the ZFS kernel module is compiled by the user, and the user does not distribute it, there are no legal problems. Thus, ZFS on Linux makes it feasible to use ZFS as a root filesystem on Linux, and an increasing number of distributions are providing ZFS on Linux packages.

Among the alternatives to ZFS are Btrfs for Linux and HAMMER for DragonFly BSD. Both aim to support similar feature sets, but neither is currently as stable and mature as ZFS. Btrfs, in particular, still has a number of stability issues. The mainstream adoption of HAMMER is further inhibited by the fact that it is only fully supported in a niche operating system.

One major feature is, however, missing in the current open source version of ZFS: filesystem encryption. Although it's implemented and available in Solaris, this code has not made its way to the public yet. With the acquisition of Sun Microsystems by Oracle and the subsequent change of attitude toward derivative open source projects, it is questionable whether this code will be released.

Even though native ZFS encryption is not currently available, there is no reason to expose ZFS data to anybody who cares to look. Many block-level disk encryption solutions are available, and they are all compatible with ZFS. PC BSD (a FreeBSD derivative) even offers support for an encrypted ZFS root filesystem in the installer; a simple click is enough to set it up. On Linux, this procedure is not as simple. No installer offers the option to use ZFS at all at the moment. With some additional effort, however, this is not too difficult to achieve.

In the remainder of this article, I will assume that you want to install Ubuntu 12.10. Most of the steps described are not specific to this particular distribution, but it provides packages for all the relevant software (if not already included in the installer). For other distributions, the installation will be similar, but additional or different steps may be required to retrieve, compile, and install software.

Warning

The instructions below describe how to set up the basic system with respect to the filesystem and related matters. To complete the steps, you should be familiar with how Linux works and comfortable with using the command line. You may need to set up some additional items, such as support for specific hardware yourself.

After following the steps in this article, you will have only a very basic installation that will require significant additional setup. If you are not comfortable configuring and installing Ubuntu Linux from scratch, do not try to do this without help from someone who is.

Running a pure ZFS Linux system is still somewhat experimental. Although the implementation of ZFS itself can be considered stable and mature, the interface to the Linux kernel is relatively new. Furthermore, there is very little support for the setup described here. Things may break when upgrading the system and if they do, your choice of tools to recover and fix your system will be limited. So, if you still feel adventurous enough to undertake this installation, read on.

Getting Started

Before starting the actual installation, get the latest Ubuntu 12.10 boot image. Although it's possible to install a 32-bit ZFS system, the 64-bit image is highly recommended. To begin, boot the machine you want to install with the image that you obtained and choose Try Ubuntu at the initial screen. Once the live distribution is up and running, open a terminal and become root.

The first thing you should do here is to set up the network; you will require a network connection during the installation. After that, partition the hard drive on which you want to install Ubuntu. You will need a small partition to boot from. The rest of the disk can be allocated to the encrypted ZFS partition. Remember to set the bootable flag on your boot partition. Then, create a filesystem of your choice on that partition. I will assume that the disk you want to use for the installation is /dev/sda with boot partition /dev/sda1 and the rest of the disk in /dev/sda2 . If the device name differs on your system or you prefer to use UUIDs, change the instructions below accordingly.

Setting Up the Disk and Bootstrapping

The next step is to set up the encrypted disk. In your terminal, type:

# cryptsetup luksFormat -l 512 -c aes-xts-plain64 -h sha512/dev/sda2
# cryptsetup luksOpen /dev/sda2 cryptroot

The first command will set up the disk as an encrypted partition with a keyfile size of 512 bytes using sha512 as a passphrase hash and aes-xts-plain64 as the encryption cipher. You are, of course, free to adapt these options to your liking. If you want to increase the level of security, overwrite /dev/sda2 with random data before setting it up as an encrypted partition. Note, however, that with a large disk this process may take days. The second command opens the encrypted partition that you just created as cryptroot . Here, you will need to enter the password that you gave when the container was set up.

These commands will set up the entire partition for ZFS. If you want to separate it into multiple slices (e.g., to have encrypted but non-ZFS swap), you can create multiple volumes on top of the encrypted container with the commands provided by lvm2 . Note that if you choose this route, you also will need to install lvm2 in the system you are about to set up. In the setup described here, we will create a ZFS volume for swap later.

After creating the encryption layer, you can set up the actual ZFS filesystem. The Ubuntu installer image does not come with ZFS support. Fortunately, you can use a personal package archive (PPA) for ZFS that makes installing the kernel modules and other required software easy. In your terminal, enter the following:

# apt-add-repository --yes ppa:zfs-native/stable
# apt-get update
# apt-get install debootstrap ubuntu-zfs

The debootstrap utility is required to install the system later.

Because the ZFS kernel module cannot be distributed in binary form and needs to be compiled, this step will take a few minutes.

Next, you can set up the ZFS pool:

# zpool create -O mountpoint=none -o ashift=12 rpool/dev/mapper/cryptroot
# zfs create -o mountpoint=/ rpool/root
# zpool set bootfs=rpool/root rpool
# zpool export rpool
# zpool import -R /mnt rpool

The first command sets up a pool named rpool for a disk with a block size of 4KiB (ashift=12 ). The next step creates a single filesystem for the entire Ubuntu installation. Note that this command will cause an error message because zfs will attempt to mount the new filesystem at the specified mount point. This is of no consequence and saves you from having to specify the mount point later.

For the sake of simplicity, we will create only one filesystem for everything. In a real setup, adding other filesystems for the various parts of the system is advisable. A more complex setup allows you to take better advantage of the many features ZFS offers. For each filesystem, you can specify whether to compress and at what level, reserve a minimum amount of space to be available, or restrict the size. Snapshots can be created per filesystem and will immensely simplify common backup tasks. At this point, it is not necessary to anticipate and create all the filesystems you might need. You can easily create them later.

After setting the bootfs property on the root filesystem, you can export the pool. It is reimported with an alternative root of /mnt , meaning that the filesystem will be mounted at that location. Exporting the pool lets you specify the alternate root and makes sure that all metadata of the newly created pool and filesystem are written to disk.

Now, you can proceed with the actual installation.

# debootstrap quantal /mnt

This step will take a while to complete. Make a cup of tea while you wait.

To speed things up a bit, you could have given the appropriate directory on the Ubuntu live image as source for the packages to install. The command shown will fetch them over the network.

Setting Up the System

After the bootstrapping is done, you can set up the newly installed system. To begin, chroot into it:

# mount --bind /dev/ mnt/dev
# chroot /mnt /bin/bash --login

To make it easier to differentiate between commands to be run in the live system and in the newly installed one, in these examples, all commands to be run in the chroot are prefixed with two hashes. In the new system, the first thing to do is to mount all the filesystems you need:

## mount /dev/sda1 /boot
## mount -t proc proc /proc
## mount -t sysfs sysfs /sys
## ln -sf /dev/mapper/cryptroot/dev/cryptroot

Do not mount the special filesystems from the live system as you did for /dev . If you do so, the bootloader will be unable to identify the root filesystem and will not create a boot configuration file. The last command is needed for the same reason. Next, generate the default locale:

## locale-gen en_US.UTF-8

Although this step is not strictly necessary, it will save you from error messages during the rest of the setup. Now you can install some additional software:

## apt-get update
## apt-get install ubuntu-minimal software-properties-common

If you used the boot image as a package source during the installation and it is out of date, you might want to upgrade the already installed software as well. The package software-properties-common contains the apt-add-repository command you'll need to add the ZFS PPA:

## apt-add-repository --yes ppa:zfs-native/stable
## apt-get update
## apt-get install ubuntu-zfs cryptsetup

In addition to installing ZFS support as you did in the live system, cryptsetup is added here so you can handle the encrypted partition.

The last command will install a whole slew of additional packages, including the GRUB boot loader. When asked where to install it, choose the system disk (here /dev/sda ). Next, create a ZFS volume for swap:

## zfs create -V 4G rpool/swap
## mkswap /dev/rpool/swap
## swapon /dev/rpool/swap

In this example, the size of the swap partition is 4GB. You can adapt this to your particular needs.

This also is a good point at which to set up /etc/fstab . You will need entries for /boot and swap .

Making It Boot

The steps described so far are reasonably straightforward and do not deviate significantly from a more or less standard installation. To make the new system boot, however, some more elaborate steps are required.

Booting a ZFS system requires a special initial RAM disk. Although there's a PPA and package for this, the version for 12.10 (Quantal) has broken dependencies and cannot be installed. The reason is that booting used to require a specially patched GRUB. With the inclusion of GRUB 2 in Ubuntu 12.10, this is no longer necessary, but the dependencies have not been updated to reflect that yet. Although it's not the prettiest solution, a feasible workaround is to install the files that this special initramfs package provides manually:

## apt-get install wget
## wget http://ppa.launchpad.net/zfs-native/stable/ubuntu/pool/main/z/zfs-linux/zfs-initramfs_0.6.0.91-0ubuntu1~quantal1_amd64.deb
## dpkg -x zfs-initramfs_0.6.0.91-0ubuntu1~quantal1_amd64.deb zfs-initramfs
## cp -r zfs-initramfs/* /

The only files the zfs-initramfs package contains are hooks for the tool that creates the initial RAM disk. These hooks copy the required binaries and configuration into the initramfs . Next, you can set up the configuration files to tell the system about the encrypted partition on which the system is installed:

## echo 'cryptroot UUID=<UUID> none luks' >> /etc/crypttab

The first component in the configuration file designates the name of the device that provides access to the encrypted container. Choosing the same name you used to set up the ZFS pool and the rest of the system is important.

The UUID of the encrypted partition can be obtained using the blkid command. You could use a filesystem path here instead of the UUID, but because the paths are assigned dynamically by udev and it is possible for them to change, using the UUID is recommended. Similar configuration must be provided for the tool that creates the initial RAM disk:

## echo 'target=cryptroot,source=UUID=<UUID>,key=none,rootdev' >>/etc/initramfs/conf.d/cryptroot

The name of this file is arbitrary; cryptroot was chosen here to be consistent with the rest of the setup. Again, the name given as target must be the same as for the rest of the setup. The UUID is the same as before. This concludes the configuration required to create the initial RAM disk. You can run the following command to create it:

## update-initramfs -c -k all

The other element required to make the system bootable is the bootloader. The GRUB configuration needs to be changed to boot the ZFS system. In /etc/default/grub , find the command line passed to the kernel (GRUB_CMDLINE_LINUX_DEFAULT ), and add the following options to it:

root=ZFS=rpool/root boot=zfs

This step will tell GRUB to boot a ZFS system and where to find the root filesystem. Also at this point, remove the splash option from the command line – the splash screens are not installed in this basic installation and enabling this option will prevent you from seeing the prompt for the passphrase of the encrypted partition. Next, let GRUB generate a new configuration file:

## update-grub

The main configuration is now done. Before you can boot into the new system, just a few loose ends need to be tied up. The symlink to the encrypted device you created earlier in /dev needs to be persistent for future updates of the GRUB configuration to succeed. Tell udev to create it as follows:

## echo 'ENV{DM_NAME}=="cryptroot",SYMLINK+="cryptroot"' >/etc/udev/rules.d/99-local.rules

The particular number at the beginning of the file name does not matter as long as it ensures that this rule file is run after the mappings for the encrypted containers have been set up. The file that does this has number 55 on the image I used. Again, the name given in the file needs to match the name for the encrypted mapping specified earlier.

All that remains to make the new system usable is to set the root password:

## passwd root

Do not forget this step. You will not be able to use the new system unless you do this. Optionally, you can set up network interfaces and other basic system configuration. Finally, unmount the filesystems and exit the chroot :

## umount /boot
## umount /sys
## umount /proc
## exit

In the live system, unmount the chroot 's /dev directory, export the pool and reboot:

# umount /mnt/dev
# zpool export rpool
# reboot

Congratulations! You should now have a pure encrypted ZFS system. You can now set up the rest of the system, add users, and install a desktop environment.

Note that support for this kind of setup by Ubuntu is still somewhat lacking. In particular, after kernel upgrades, you should take care that the ZFS modules and the initial RAM disk are recreated correctly. You will not be able to boot if they are not.

Adding a Cache Device

ZFS pools can have cache devices – usually solid-state disk (SSD) – that are used to speed up the filesystem operations. Many modern systems, and even some laptops, have SSDs in addition to normal hard disks now. Adding a cache device can be done by executing a simple command, but you should, of course, encrypt the cache device.

Having another encrypted device would require you to enter a second passphrase when booting. This would be annoying and is actually unnecessary, because the Ubuntu cryptsetup package ships with a script that allows you to derive a passphrase from data on a disk.

Here, I will use the decrypted main disk as source for the derived passphrase. Although this approach is not as secure as an independent passphrase, it does not constitute a security risk here because the main disk still needs to be decrypted before the passphrase can be derived, and because the second disk is only a cache device.

Assuming that the cache device is /dev/sdb , you can set up the encrypted partition as follows:

# /lib/cryptsetup/scripts/decrypt_derived cryptroot > /tmp/key
# cryptsetup luksFormat -l 512 -c aes-xts-plain64 -h sha512/dev/sdb --key-file /tmp/key
# rm /tmp/key
# /lib/cryptsetup/scripts/decrypt_derived cryptroot | cryptsetupluksOpen /dev/sdb cryptcache

This will set up the new encrypted container as cryptcache with the same parameters as the other one. Adding the encrypted device as cache to the pool is straightforward:

# zpool add rpool cache /dev/mapper/cryptcache

Now, you just need to add the information about the encrypted device to the system.

No additional steps are required to make the ZFS configuration persistent:

# echo 'cryptcache UUID=<UUID> cryptroot luks,keyscript=/lib/cryptsetup/scripts/decrypt_derived' >> /etc/crypttab

You can verify that the cache device is present and obtain usage statistics by typing zpool iostat -V .

Conclusion

The ZFS on Linux project has made it possible to use the advanced ZFS filesystem for the system root on Linux installations. Although this setup is still somewhat exotic and support for it is lacking, the currently available tools are entirely adequate for setting up a pure ZFS Linux system and even throwing full disk encryption into the bargain.

This article demonstrated all that is needed to set up such a system. Some experience with Linux is necessary to perform the steps confidently, but you do not need to be a guru to complete them.

I hope that using ZFS on Linux will become much easier in the future and will be supported by standard installers. Some BSD distributions are one step ahead in this respect, but no doubt Linux will catch up soon.