A slightly improved Gentoo-Linux/openrc install

Here we present a slightly improved guide on how to install a hardened x86_64 Gentoo Linux musl/openrc/UEFI signed UKI operating system on an encrypted ZFS pool with automatic decryption using TPM. This time we will configure the kernel manually instead of blindly using the distribution kernel supplied by the sys-kernel/gentoo-kernel package. This way we will be able to streamline our operating system even more to our specific needs, and the needs of our hardware.

Provisioning

We will use the Alpine Extended ISO to bootstrap Gentoo. Make sure to boot with secureboot in setup mode or to already have keys ready to deploy.

After booting the Alpine Linux extended ISO, partition the disks. For this action internet is required since zfs, sgdisk and various other necessary packages are not included on the extended ISO, therefore they need to be obtained from the Alpine package repository.

Set it up with setup-interfaces and setup-apkrepos:

sh# setup-interfaces -ar #(1)!
sh# setup-apkrepos -c1

To use Wi-Fi simply run setup-interfaces -r and select wlan0 or similar.

Install the necessary packages:

sh# apk add zfs lsblk sgdisk wipefs dosfstools zlevis

and load the ZFS kernel module:

sh# modprobe zfs

Wipe the existing disk partitions:

sh# zpool labelclear -f /dev/<disk>
sh# wipefs -a /dev/<disk>
sh# sgdisk --zap-all /dev/<disk>

Create on the disk an EFI system partition (ESP) and a Linux filesystem partition (LFP):

sh# sgdisk -n 1:1m:+512m -t 1:ef00 /dev/<disk>
sh# sgdisk -n 2:0:-10m -t 2:8300 /dev/<disk>

Reload the device nodes:

sh# mdev -s

Format the ESP with a FAT32 filesystem:

sh# mkfs.fat -F 32 -n esp /dev/<disk>1

ZFS pool creation

The ZFS system pool is going to be encrypted. First generate an encryption key and save it temporarily to the file /tmp/rpool.key with:

sh# cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 1 > /tmp/rpool.key && cat /tmp/rpool.key

All the while we use zlevis for automatic decryption, this key is required when making changes are made to the BIOS or secureboot, so make sure to save it.

Create the system pool:

sh# zpool create -f \
        -o ashift=12 \
        -O compression=lz4 \
        -O acltype=posix \
        -O xattr=sa \
        -O dnodesize=auto \
        -O encryption=on \
        -O keyformat=passphrase \
        -O keylocation=prompt \
        -m none \
        rpool /dev/<disk>2

Then create the system datasets:

sh# zfs create -o mountpoint=none rpool/root
sh# zfs create -o mountpoint=legacy -o quota=48g rpool/root/gentoo
sh# zfs create -o mountpoint=legacy -o quota=32g rpool/root/gentoo/var
sh# zfs create -o mountpoint=/home -o atime=off -o setuid=off -o devices=off -o quota=<home-quota> rpool/home

Setting the <home-quota> depends on the total size of the pool, generally try to reserve some empty space in the pool.

Write the encryption key to TPM with zlevis:

sh# zlevis encrypt rpool '{"pcr_ids":"0,5,7"}' < /tmp/rpool.key

To check if it worked, perform zlevis decrypt rpool.

Finally, export the zpool:

sh# zpool export rpool

Installation

To install Gentoo Linux on the system, the ESP and the datasets of the system pool have to be mounte d to the live (ISO) environment.

First import and decrypt the system pool:

sh# zpool import -N -R /mnt rpool
sh# zfs load-key -L file:///tmp/rpool.key rpool

Then mount the datasets and the ESP on /mnt:

sh# mount -t zfs rpool/root/gentoo /mnt
sh# mkdir /mnt/var
sh# mount -t zfs rpool/root/gentoo/var /mnt/var
sh# mkdir /mnt/efi
sh# mount -t vfat /dev/disk/by-label/esp /mnt/efi

Now we are going to fetch a stage 3 tarball, the archive containing a minimal Gentoo environment. Which will act as the seed of the Gentoo install. Replace the <release_date> with the latest stage file release:

sh# wget https://distfiles.gentoo.org/releases/amd64/autobuilds/current-stage3-amd64-musl-hardened-openrc/stage3-amd64-musl-hardened-<release_date>.tar.xz #(1) #(2)!

It is also possible to use links instead of wget which provides a small user interface for navigation:

sh# links https://distfiles.gentoo.org/releases/amd64/autobuilds/current-stage3-amd64-musl-hardened-openrc

There are also other mirrors like https://ftp.snt.utwente.nl/pub/os/linux/gentoo/releases/amd64/autobuilds/current-stage3-amd64-musl-hardened/ which might provide a faster download depending on your location. Check out https://www.gentoo.org/downloads/mirrors/ for other mirrors.

Unpack the stage file in the root of the system:

sh# tar -xpf stage3-*.tar.xz --numeric-owner -C /mnt

To have a functional chroot into the system, copy resolv.conf and bind the system process directories:

sh# cp --dereference /etc/resolv.conf /mnt/etc/
sh# for dir in dev proc sys run; do
> mount --rbind --make-rslave /$dir /mnt/$dir
> done
sh# chroot /mnt

Configuring portage

There are various degrees to which you can configure portage (1). We prefer a fine grained approach where every package that is installed into the system is listed explicitly in /etc/portage/sets.

The package manager of Gentoo Linux, which provisions build files from the Gentoo ebuild repository or any additional ebuild repositories.

A base set consisting of the packages that we require in this installation:

/etc/portage/sets/base

# /etc/portage/sets/base
# Set containing system base packages

# kernel
sys-kernel/modprobed-db
sys-power/acpid

# portage
sys-apps/portage
app-portage/gentoolkit
app-portage/cpuid2cpuflags
app-eselect/eselect-repository
app-admin/eclean-kernel

# musl
sys-libs/timezone-data
sys-libs/error-standalone

# init/services
sys-apps/openrc
net-misc/openntpd

# security
app-crypt/jitterentropy
sys-apps/apparmor
sys-apps/apparmor-utils
sec-policy/apparmor-profiles

# shell
app-editors/vim
sys-process/htop

We will extend this base set along the way. (1)

For a full base set, or examples of other types of sets (lang or video) see my sets and Nils' sets.

To obtain the required functionality of the packages that we will be installing (1) we need to set some global USE flags:

And no more than the required functionality of the packages!

/etc/portage/package.use/00glo

# /etc/portage/package.use/00glo
# Global USE flags

# system requirements
*/* hardened initramfs hostonly secureboot modules-sign udev acpi apparmor

# user requirements
*/* networkmanager dbus hwaccel bash-completion man pam pipewire system-pipewire v4l vulkan wayland eme-free jpeg png svg gdk-pixbuf

# remove bloat
*/* -debug -telemetry -modemmanager -ext* -ppp -systemd -elogind -X -kde -gnome -gtk-doc -webengine -vala

To properly compile the packages we require the following compiler flags, options, licensing, logging, kmod and secureboot settings:

/etc/portage/make.conf
# A Gentoo installation is highly personal so diverting from this example is encouraged.
# Please consult /usr/share/portage/config/make.conf.example for a more detailed example.

# compiler flags
COMMON_FLAGS="-march=native -O3 -pipe"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}"
FCFLAGS="${COMMON_FLAGS}"
FFLAGS="${COMMON_FLAGS}"
RUSTFLAGS="${RUSTFLAGS} -C target-cpu=native"
CHOST="x86_64-pc-linux-musl"
LC_MESSAGES=en_US.utf8

# compiler options
MAKEOPTS="-j12 -l14" #(1)!

# portage logging
PORTAGE_ELOG_CLASSES="log warn error"
PORTAGE_LOGDIR="/var/log/portage"
PORTAGE_LOGDIR_CLEAN="find \"\${PORTAGE_LOGDIR}\" -type f ! -name \"summary.log*\" -mtime +7 -delete"

# licensing
ACCEPT_LICENSE="-* @FREE"

# emerge flags
EMERGE_DEFAULT_OPTS="--with-bdeps y --quiet-build y --ask y"

# kmod settings
MODULES_SIGN_KEY="/var/lib/keys/modules-sign.pem"
MODULES_SIGN_HASH="sha512"

# secureboot settings
SECUREBOOT_SIGN_KEY="/var/lib/sbctl/keys/db/db.key"
SECUREBOOT_SIGN_CERT="/var/lib/sbctl/keys/db/db.pem"

The MAKEOPTS defines (-j the number of jobs/processes) and limits (-l the load average) how many parallel make/compilation processes can be launched from portage. The number of parallel processes are limited by the number of available logical CPUs and by RAM, each process can take up to 2 GB of RAM. The load average -l is generally set slighly above the number of jobs -j.

Synchronise the package ebuild repositories and emerge the present base set:

sh# emaint sync
sh# emerge -uDU @base

Configuring the base system

Configure some key aspects of the system:

sh# echo <hostname> > /etc/hostname
sh# echo TZ="/usr/share/zoneinfo/<Region>/<City>" > /etc/env.d/00tz
sh# echo MUSL_LOCPATH="/usr/share/i18n/locales/musl" > /etc/env.d/01locales #(1)!
sh# env-update && source /etc/profile
sh# eselect locale set <locale> #(2)!
sh# rc-update add ntpd default
sh# passwd root #(3)!

Musl does not support locales out of the box. They are not necessary but some programs rely on them to set the language of their application.
The correct <locale> can be found in the locale list:
```
sh# eselect locale list
```
The root password does not really matter because it is going to be locked after a user has been created.

Configuring the firmware

Set the CPU flags by fetching them with cpuid2cpuflags:

sh# echo $(cpuid2cpuflags) > /etc/portage/package.use/01cpu

and format:

/etc/portage/package.use/01cpu

# /etc/portage/package.use/01cpu
# System specific CPU flags

*/* CPU_FLAGS_X86: <your-cpu-flags>

The installation of the hardware firmware on the system, such as the CPU microcode, is hardware specific:

AMD CPUIntel CPU

The microcode updates for systems with an AMD CPU are all contained in sys-kernel/linux-firmware, accept its license:

/etc/portage/package.license/firmware

# /etc/portage/package.license/firmware
# Accept firmware license

sys-kernel/linux-firmware linux-fw-redistributable @BINARY-REDISTRIBUTABLE

add the firmware to the base set:

/etc/portage/sets/base

# firmware
sys-kernel/linux-firmware

The microcode updates for systems with an Intel CPU require alongside sys-kernel/linux-firmware also sys-kernel/intel-microcode, accept their license:

/etc/portage/package.license/firmware

# /etc/portage/package.license/firmware
# Accept firmware license

sys-kernel/linux-firmware linux-fw-redistributable @BINARY-REDISTRIBUTABLE
sys-firmware/intel-microcode intel-ucode

add the firmware to the base set:

/etc/portage/sets/base

# firmware
sys-kernel/linux-firmware
sys-firmware/intel-microcode

and re-emerge:

sh# emerge -uDU @base

Configuring the boot system

Edit the fstab to set the correct mounts at boot time:

/etc/fstab

rpool/root/gentoo       /       zfs     rw,noatime,xattr,posixacl,casesensitive                 0 1
rpool/root/gentoo/var   /var    zfs     rw,noatime,nodev,nosuid,xattr,posixacl,casesensitive    0 2
/dev/disk/by-label/esp  /efi    vfat    defaults,nodev,nosuid,noexec,umask=0077                 0 2
tmpfs                   /tmp    tmpfs   rw,nodev,nosuid,noexec,mode=1777                        0 0
proc                    /proc   proc    nodev,nosuid,noexec,hidepid=2                           0 0

Configure the kernel command-line to be able to import the zpool correctly:

/etc/kernel/cmdline

rw root=ZFS=rpool/root/gentoo rootflags=noatime quiet splash #(1)!

For first boot up we will keep things simple and only enable a minimal set of kernel options.

Add sbctl, sbsigntools and systemd-utils to the boot set:

/etc/portage/sets/boot

# /etc/portage/sets/boot
# Set containing packages for the boot system

# systemd-boot
sys-apps/systemd-utils

# secureboot
app-crypt/sbctl
app-crypt/sbsigntools

Set the required USE flags for systemd-utils such that on emerging the bootloader systemd-boot will be installed and tell installkernel to generate a UKI when installing a kernel:

/etc/portage/package.use/05boot

# /etc/portage/package.use/05boot
# USE flags for boot system

# installkernel
sys-kernel/installkernel dracut ukify uki

# systemd-boot
sys-apps/systemd-utils kernel-install boot ukify

and emerge:

sh# emerge -uDU @boot

Verify that secureboot is in setup mode with sbctl status!

Create and enroll the secureboot keys into the system:

sh# sbctl create-keys
sh# sbctl enroll-keys #(1)!

Whilst enrolling the keys it might be necessary to add the --microsoft flag if you are unable to use custom keys.

Enable dracut to create a UKI with ukify:

/usr/lib/kernel/install.conf

layout=uki
initrd_generator=dracut
uki_generator=ukify

and enable automatic signing with sbsign:

/etc/kernel/uki.conf

[UKI]
SecureBootSigningTool=sbsign
SecureBootPrivateKey=/var/lib/sbctl/keys/db/db.key
SecureBootCertificate=/var/lib/sbctl/keys/db/db.pem

Installing the temporary distribution kernel

In the context of initial boot debugging and kernel configuration it is easier to first compile and install a temporary distribution kernel and from there strip down to obtain your own minimal custom kernel with the help of modprobed-db.

First create the kernel module signing key:

sh# openssl req -new -nodes -utf8 -sha512 -x509 -outform PEM -out /var/lib/keys/modules-sign.pem -keyout /var/lib/keys/modules-sign.pem

which enables us to emerge the temporary distribution kernel:

sh# emerge --oneshot gentoo-kernel

Having the kernel build we are now able to compile and install the zfs kernel module alongside the zfs tool set. We add them to the base set:

/etc/portage/sets/base

# filesystem
sys-fs/zfs
sys-fs/zfs-kmod

and re-emerge:

sh# emerge -uDU @base

Configuring the boot loader

Install the boot loader on the ESP:

sh# bootctl install
Copied "/usr/lib/systemd/boot/efi/systemd-bootx64.efi.signed" to "/efi/EFI/systemd/systemd-bootx64.efi".
Copied "/usr/lib/systemd/boot/efi/systemd-bootx64.efi.signed" to "/efi/EFI/BOOT/BOOTX64.EFI".
Random seed file /efi/loader/random-seed successfully refreshed (32 bytes).
Created EFI boot entry "Linux Boot Manager".

and configure the boot loader:

/efi/loader/loader.conf

timeout 3
editor no

One may verify the signed files by running sbctl verify.

Generate the hostid and reconfigure the Gentoo Kernel:

sh# zgenhostid
sh# emerge --config gentoo-kernel

Finally, add some services for ZFS:

sh# rc-update add zfs-mount sysinit
sh# rc-update add zfs-import sysinit #(1)!
sh# rc-update add zfs-load-key sysinit #(2)!

Omit if a faster boot time is preferred.
Omit if a faster boot time is preferred.

Now exit the chroot, unmount the datasets and reboot:

sh# exit
sh# umount -lf /mnt
sh# reboot

Optional installation

Repositories

GURU is an extra repository containing ebuilds that are not available in the Gentoo repository. Although the packages it contains might not be as well tested as in the main repository they are still necessary for some setups; such as some dependencies required for automatic decryption and graphical user interfaces. Add GURU with:

sh# eselect repository enable guru
sh# emaint sync --repo guru

Portage-ample is an extra extra repository containing ebuilds maintained by us. Add portage-ample with:

sh# eselect repository add portage-ample git https://git.ampel.dev/ampel/portage-ample
sh# emaint sync --repo portage-ample

Automatic decryption

Automatic decryption will be managed with zlevis, which is able to unlock an encrypted ZFS root pool with keys saved in a TPM. Currently it is only available in the portage-ample repository and also has some dependencies in the guru repository.

Add zlevis to the boot set:

/etc/portage/sets/boot

# tpm
app-crypt/zlevis

Set the dracut flag:

/etc/portage/package.use/05boot

# tpm
app-crypt/zlevis dracut

and re-emerge the boot set:

sh# emerge -uDU @boot

Finally, enable the zlevis module for dracut:

/etc/dracut.conf.d/zlevis.conf

nofsck="yes"
add_dracutmodules+=" zlevis "

Swap

We will add swap to the system with zram-init. So add it to the base set:

/etc/portage/sets/base

# init/services
sys-block/zram-init

and re-emerge the base set:

sh# emerge -uDU @base

Configure zram-init to create a swap device of size one fourth of the ram size:

/etc/conf.d/zram-init

load_on_start="yes"
unload_on_stop="yes"
num_devices="1"

type0="swap"
size0=`LC_ALL=C free -m | awk '/^Mem:/{print int($2/4)}'`
maxs0=1
algo0=zstd
labl0=zram_swap

and add zram-init to the default runlevel:

sh# rc-update add zram-init default

Users

To run processes securely, in an environment with fewer privileges, a user is necessary.

Before creating the user, emerge doas, to be able to "do as" root when it is required. Which requires us to set:

/etc/portage/sets/base

# shell
app-admin/doas

and:

/etc/portage/package.use/10base

# /etc/portage/package.use/10base
# USE flags for base packages

# doas
app-admin/doas persist

then re-emerge the base set:

sh# emerge -uDU @base

and configure doas by editing:

/etc/doas.conf

permit persist :wheel as root

Now users who are in the wheel group are allowed to use the doas command to gain root privileges.

We can add a user, set its password and add it to the wheel group (if admin) with:

sh# useradd -m -G wheel -s /bin/bash <username>
sh# passwd <username>

The wheel group should ideally only be assigned to one singular admin account. The users in the group are allowed to use the doas command to gain root privileges. This is necessary for installing packages and changing system files but not for a normal user.

If you have checked that doas works with the user then you can lock the root account because it imposes security risks if it is kept open. This can be done with:

sh# passwd -l root

and by changing its login shell to:

/etc/passwd

root:x:0:0:root:/root:/sbin/nologin

Networking

For desktop use NetworkManager is preferred over dhcpcd as network daemon, due to its versatility, i.e. Wi-Fi and VPN compatibility, MAC randomisation, et cetera.

Add NetworkManager to the base set:

/etc/portage/sets/base

# init/services
net-misc/networkmanager

and set the relevant USE flags:

/etc/portage/package.use/10base

# networkmanager
net-misc/networkmanager -wext

Also make sure the networkmanager USE flag is enabled in your make.conf.

Now re-emerge the base set:

sh# emerge -uDU @base

and configure networkmanager to have MAC randomisation by editing:

/etc/NetworkManager/NetworkManager.conf

[main]
hostname-mode=none
plugins=ifupdown,keyfile

[ifupdown]
managed=true

[device]
wifi.scan-rand-mac-address=yes

[connection-mac-randomization]
ethernet.cloned-mac-address=random
wifi.cloned-mac-address=random

Then stop any other network service, such as dhcpcd running in the dynamic runlevel, and enable NetworkManager:

sh# rc-update add NetworkManager default

For users to be able to modify connections on the system they will have to be added to the plugdev group.

Kernel configuration

We have arrived at kernel configuration which does not need to take place immediately after the installation. Rather it is wiser to configure the kernel only after the system has been fully configured (after possibly a graphical session or other infrastructure) so that we can list all the necessary kernel modules using modprobed-db.

Probing for kernel modules

Particularly when ready, perform:

sh$ modprobed-db store

Try to probe all kernel modules you could possibly need for your system, i.e. plugging in all the devices you may ever need.

You can check for any new or missing modules with:
sh$ modprobed-db debug
and repeat modprobed-db store to store these as well.

Make your config

Once your list of kernel modules (1) includes everything you'll need, you can proceed by selecting the sources of the latest kernel (2):

List your stored kernel modules with:
```
sh$ modprobed-db list
```
Selecting means symlinking /usr/src/linux-<ver-latest-kernel>-gentoo to /usr/src/linux.

sh# eselect kernel set <num-latest-kernel> #(1)!

The number coupled to the latest kernel can be found with:
```
sh# eselect kernel list
```

Make your way to /usr/src/linux and load your modules into the kernel config with localmodconfig:

sh# cd /usr/src/linux
sh# make LSMOD=/home/<user>/.config/modprobed.db localmodconfig

You will notice that make acquires the kernel config from the previous (distribution) kernel and adapts it to your modprobed.db. Which sets you up with quite a good starting point for further configuration. There is still a lot of nonsense to be removed from the kernel at this point, but this is something you should do (and struggle with) yourself:

sh# make MENUCONFIG_COLOR=blackbg menuconfig

I myself followed gentoo/wiki/kernel (1) and gentoo/handbook/kernel. Don't expect a coherent story when reading these pages, their priorities vary quite a bit throughout the wiki. Thus, try to use your neurons for a change!

Scroll down to Managing the kernel to get a nice overview, most worthwile to read/follow are Applying manual configuration and Security.

Some considerations:

Make sure that you enable and enforce signed kernel modules and specify the signing key.
Make sure that you enable signed kernel images for secureboot.
Make sure that you enable TPM 2.0 support for automatic decryption.
Use the information in /sys/devices/system/cpu/vulnerabilities/ to compile the required security vulnerability mitigations.
Using lspci -v, modinfo and modprobed-db list to scope out and check hardware requirements manually is highly recommended.
modprobed-db enabled modules seem to be often in <M> format, so that is how you can identify and differentiate them from the nonsense enabled in the default config.

There is a security benefit to disable kernel modules as a whole and compile everything into the kernel image. Unfortunetely zfs-kmod doesn't allow us to do this due to Sun Microsystems amazing licensing of ZFS :(

Make your kernel

If you are zufrieden with your config it might be time to compile the kernel. Compiling the kernel with root is cringe, instead we add a specific builder user:

sh# useradd -M -s /bin/bash builder 
sh# chown builder /usr/src
sh# passwd builder
sh$ su builder

Compile the kernel with the builder user:

builder@sh$ make -j12 -l14 #(1)!

To repeat: the MAKEOPTS defines (-j the number of jobs/processes) and limits (-l the load average) how many parallel make/compilation processes can be launched. The number of parallel processes are limited by the number of available logical CPUs and by RAM, each process can take up to 2 GB of RAM. The load average -l is generally set slighly above the number of jobs -j.

If the compilation failed for some reason use make V=1 to compile in verbose mode to get more info on the error.

Next rebuild your external modules (1):

In our case zfs-kmod.

sh# emerge @module-rebuild

Install your kernel

Before the installation make sure that /etc/kernel/cmdline is minimal and free of any security measures when you install your own configured kernel for the first time. This way your ability to track your mistakes is greatly enhanced. On that note, install the modules and the kernel:

sh# make modules_install
sh# make install

make install will call installkernel which in turn calls dracut to create a UKI with ukify and sbsign to sign the UKI. Finally installkernel will add the new UKI to the bootloader configuration.

Now reboot and try out your own kernel!

Concluding remarks

We have reached the end of this year's Gentoo Linux install guide. I must admit functionality wise not much has changed with respect to a year ago. Yet I spend an idiotic amount of time on improving things. Nonsense mostly, but perhaps it has now become easier to rebuild a system to a particular spec (1). The OS has become a few MBs lighter due to the change to a custom kernel and well removing attack surface does make it more resilient to attacks. To connect to that, something that I was eager to add but didn't find the time for this summer is SELinux. SELinux is in my eyes quite an important layer missing in my system. Hopefully the next entry will explicitly work with SELinux (2) and App-Armor will be thrown out!

As specified in my configs for example.
Global USE flags will be set, kernel will be compiled with it, etc.