↩ 005. Low-curse ZFS-on-root for new Debian installations on multi-disk systems ↩

Mon, 14 Sep 2020 00:20:20 +0200, updated Tue, 15 Sep 2020 20:08:12 +0200, Tue, 29 Sep 2020 23:06:46 +0200, Sat, 12 Dec 2020 01:22:52 +0100, Mon, 15 Mar 2021 23:49:31 +0100, with thanks to Pierfrancesco for many pre-bullseye bugs! and Sat, 11 Jun 2022 00:19:37 +0200

In a brief departure from microcomputing saprophagia, imagine you're an american and acquiring a ThinkPad P15, sporting two two-terabyte drives and, naturally, you'd like to have a single continuous volume spanning both of them to do whatever your heart desires.

After the shock at willingly staying in the American empire and paying it for the privilege wears off, two approaches reveal themselves:

1. LVM, with its native debian-installer support, and
2. ZFS, without it, but with the ability to do zfs(8)-send backups, live-mirroring to an external enclosure with a [mirror[internal, external], mirror[internal, external]] topology, and me egging you on to do it.

Now, I also wanted to do a ZFS-on-root for my own nefarious uses, and so it was decided:

Good, do it before I buy the p15 and report back

# Set-up

To minimise how cursed this is, the following restrixions are in place:

• EFI booting, because GRUB's ZFS detexion and handling blows ass, and
• a normal d-i installation and no external computer nor drives required, since doing d-i's job by hand was covered previously in Installing Debian's x32 port in 2020, and it's a similarly good experience (plus building the modules in a chroot is a good time at any rate).

This means that all you need is an EFI-compatible multi-disk platform and some way to EFI boot it into d-i — the bootloader will be fixed, ZFS installed normally, and the rootfs dumped/restored thereonto after normally booting into the target system.

The test setup is QEMU -bios OVMF.fd and two 8G drives, one of which is designated as primary. Filesystem tuning is not covered, encryption is supported, SecureBoot is not covered because I haven't figured it out yet, most-all gotchas are hopefully explained, there's a prior art that was of little help, and zfs-{initramfs,dracut} are full of problems that I try to work around here:

# Installation

The restrixions to the original installation are all during the partitioning phase and as follows:

1. Stick to just the primary drive,
2. pick an EFI partition large enough for a few kernels and initrds, and
3. stuff everything onto the rootfs:

Either installing sid or enabling backports for buster is required, since zfs-mount-generator(8) appeared in 0.8.0; if you don't install GRUB you'll have to boot manually from the EFI shell after copying the kernel and initrd from target/boot/ to target/boot/efi/.

# Bootloader

Using GRUB and a unified boot-on-root is, for better or for worse, not possible; GRUB supports these non-R/O zpool features:

/*
 * List of pool features that the grub implementation of ZFS supports for
 * read. Note that features that are only required for write do not need
 * to be listed here since grub opens pools in read-only mode.
 */
#define MAX_SUPPORTED_FEATURE_STRLEN 50
static const char *spa_feature_names[] = {
  "org.illumos:lz4_compress",
  "com.delphix:hole_birth",
  "com.delphix:embedded_data",
  "com.delphix:extensible_dataset",
  "org.open-zfs:large_blocks",
  NULL
};

Which means that these, as of OpenZFS 0.8.4, are the ones that it doesn't support:

nabijaczleweli@tarta:~$ man zpool-features | grep READ-ONLY.*no -B4 | sed 's/GUID//' | \
                        awk '!/^$/ && !/COMPATIBLE/ && !/--/ \
                             {if(last == "") last = $1; else {print last "\t" $1; last = ""}}' |
                        grep -v 'lz4_compress|hole_birth|embedded_data|extensible_dataset|large_blocks' | column -t
bookmark_v2            com.datto:bookmark_v2
device_removal         com.delphix:device_removal
edonr                  org.illumos:edonr
encryption             com.datto:encryption
large_dnode            org.zfsonlinux:large_dnode
multi_vdev_crash_dump  com.joyent:multi_vdev_crash_dump
sha512                 org.illumos:sha512
skein                  org.illumos:skein

Confer zpool-features(5) for details so as to why this is not acceptable (remember: you couldn't any of these even by accident on your root pool, and most of them can't ever be turned off).

Using a separate /boot on ZFS required a lot of dicking around with the options for GRUB to recognise the filesystem at all (confer prior art, Step 2: Disk Formatting, 4. Create the boot pool, with its 12 lines of options for pool features alone) and GRUB has difficulties generating the right root= cmdline options for a normal pool anyway (but it does work if you convince it hard enough and have no other options, like on EFI-less systems).

Using a separate /boot on ext*/FAT didn't make much sense for booting on EFI, either, so cutting GRUB out was a simple way to berid of a lot of bollocks.

Boot into your freshly installed system, become root, and:

I'd recommend rebooting now to verify that this works, which should look like this:

If not, and sd-boot shows errors or doesn't start at all; boot into the EFI shell, fs0:, and \<MID>\<VER>\linux initrd=\<MID>\<VER>\initrd.img-<VER> root=/dev/sda2 (the shell should support tab-completion, you might need to add a space before completing the initrd) (the root= option assumes you installed to the second partition of the first SCSI drive, as I did; adjust to taste), then write me an e-mail or a DM or whatever else is listed here so I can issue a correxion; thanks in advance, &c.:

Now we can boot without dealing with GRUB. On to the ZFS bit:

# The ZFS bit

And now a brief interlude on the layout of ZFS datasets in the VFS.

# Layout

Prior art, Step 3: System Installation issues a lot of commands and doesn't really explain why.

The final mount tree is as such:

               zoot/
/              zoot/root
├── boot       zoot/boot
├── home       zoot/home
│   └── nab    zoot/home/nab
├── root       zoot/home/root
├── opt        zoot/opt
├── srv        zoot/srv
│              zoot/usr
├── usr/local  zoot/usr/local
│              zoot/var
├── var/cache  zoot/var/cache
│              zoot/var/lib
├── var/log    zoot/var/log
└── var/tmp    zoot/var/tmp

Why?

• The root dataset is not mounted, because the explicit initrd's root= parameter requires a dataset to mount, and the scripts don't behave too well when the detected bootfs is the root; zoot/root is filling that role instead;
• /var, /var/lib, and /usr are all required for the system to boot (I didn't try a split-/usr system, but don't have much hope either way), and the initrd only mounts the root filesystem — things don't go well when these aren't there, but the descendant filesystems are only used after the real init remounts everything;
• additionally, /var/log needs POSIX ACLs for sd-journald and /var/tmp needs to be sticky a+rwx.

And so:

If all went well, the system should now prompt for a password:

dracut might try to use a stored mount option like errors=remount-ro for the rootfs; in that case mount -t zfs -o zfsutil zoot/root /sysroot and regenerating the initrd (then committing it to the ESP!) will help.

This was fixed in OpenZFS 2.0.3 (Fri, 12 Feb 2021 16:30:01 +0800).

Post-login, the mounts should now be all ZFS:

Or:

nab@zoot:~$ findmnt | grep zfs
/                              zoot/root      zfs        rw,relatime,xattr,noacl
├─/home                        zoot/home      zfs        rw,relatime,xattr,noacl
│ └─/home/nab                  zoot/home/nab  zfs        rw,relatime,xattr,noacl
├─/opt                         zoot/opt       zfs        rw,relatime,xattr,noacl
├─/boot                        zoot/boot      zfs        rw,relatime,xattr,noacl
├─/srv                         zoot/srv       zfs        rw,relatime,xattr,noacl
├─/var/cache                   zoot/var/cache zfs        rw,relatime,xattr,noacl
├─/root                        zoot/home/root zfs        rw,relatime,xattr,noacl
├─/var/log                     zoot/var/log   zfs        rw,relatime,xattr,posixacl
├─/usr/local                   zoot/usr/local zfs        rw,relatime,xattr,noacl
└─/var/tmp                     zoot/var/tmp   zfs        rw,relatime,xattr,noacl

And, yes, the chrome of the windows changed from Windows 10 to i3, this is a goddamn mess of a post.

One last part now, to actually use both disks:

And that's it. There ain't much fan-fare to it, since this took only the very best part of a week to get right.

root@zoot:~# zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zoot    15G   777M  14.2G        -         -     0%     5%  1.00x    ONLINE  -
root@zoot:~# zpool status
  pool: zoot
 state: ONLINE
  scan: none requested
config:

        NAME                         STATE     READ WRITE CKSUM
        zoot                         ONLINE       0     0     0
          ata-QEMU_HARDDISK_QM00002  ONLINE       0     0     0
          zoot-root                  ONLINE       0     0     0

errors: No known data errors

Or, as was succinctly put by the instigator of all this mess:

On Tue, 29 Sep 2020 22:16 -0800, after minor hiccups, the deed has finally been done, with a recommendation of the highest order:

Nit-pick? Correction? Improvement? Annoying? Cute? Anything? Don't hesitate to post or open an issue!