Skip to content

CUDA computations on a musl Gentoo Linux host, is it possible?

I always thought that Nvidia was a lost cause on Linux, especially on a musl Gentoo Linux system. Yet I found that in the particular use case of CUDA computation not all is hopeless. As of writing this we still require glibc for user space Nvidia drivers, but these can be nicely tucked away in a glibc chroot. This glibc requirement apparently leads people to believe that the Nvidia kernel modules also require glibc (see for example the x11-drivers/nvidia-drivers gentoo ebuild), but this is plainly incorrect. The Nvidia kernel modules are independent of a C library and can thus also be build on musl systems.

Installing Nvidia kernel modules

I will try to briefly outline how to actually get CUDA capability with minimal bloat. Actually, I significantly reduced the effort by writing an ebuild for the kernel modules sys-kernel/nvidia-kmod and an ebuild for the kernel module loader sys-kernel/nvidia-modprobe. Which leaves us only with the setup of the chroot and a possible firmware blob that needs to be added to /usr/lib/firmware.

First emerge the just mentioned packages (1):

  1. At the time of writing both packages are only available in the portage-ampel repository, add portage-ampel with:

    sh# eselect repository add portage-ampel git https://git.ampel.dev/repos/portage-ampel
sh# emaint sync --repo portage-ampel

sh# emerge sys-kernel/nvidia-modprobe sys-kernel/nvidia-kmod

and call installkernel to make them loadable.

Configuring the glibc chroot

Setting up the chroot is a little more effort, to keep it simple we will use a Void Linux rootfs which you should unpack in /var/lib/chroots/void.

To have a functional chroot, probe the Nvidia modules, copy resolv.conf, and mount the following system directories:

sh# nvidia-modprobe -c0 -u
sh# cp /etc/resolv.conf /var/lib/chroots/void/etc
sh# for dir in dev proc sys; do
> mount --bind /$dir /var/lib/chroots/void/$dir
> done
sh# chroot /var/lib/chroots/void /bin/bash

Now we will perform the most ugly part of this outline, we will fetch Nvidia's proprietary user space drivers:

sh# wget https://download.nvidia.com/XFree86/Linux-x86_64/<version>/NVIDIA-Linux-x86_64-<version>.run #(1)!
  1. Make sure the <version> matches the version of the host side Nvidia kernel modules!

and extract it:

sh# NVIDIA-Linux-x86_64-595.84.run --extract-only

From the extracted tree we only need to link the following for CUDA capabilities (1):

  1. This list may be incomplete. 
sh# for lib in libcuda libnvidia-ml libnvidia-ptxjitcompiler; do
> ln -s /usr/src/NVIDIA-Linux-x86_64-<version>/${lib}.<version> /usr/lib/${lib}.so.<version>
> ln -s /usr/lib/${lib}.so.<version> /usr/lib/${lib}.so
> ln -s /usr/lib/${lib}.so.<version> /usr/lib/${lib}.so.1
> done
sh# ldconfig
sh# ln -s /usr/src/NVIDIA-Linux-x86_64-<version>/nvidia-smi /usr/sbin/nvidia-smi

Possibly required firmware blobs

Back on the host side, check:

sh# dmesg | grep -iE 'nvidia|nvrm' | grep -i fail

If any fail's show up you possibly need a Nvidia firmware blob. Copy it from the extracted tree:

sh# mkdir -p /usr/lib/firmware/nvidia/<version>
sh# cp /var/libs/chroots/void/usr/src/NVIDIA-Linux-x86_64-<version>/firmware/gsp_<arch>.bin /usr/lib/firmware/nvidia/<version/

Which finalises this outline. As a dessert I present you this script to enter the chroot with minimal effort:

/usr/local/sbin/enter-void
#!/bin/sh
set -e
CHROOT="/var/lib/chroots/void"

/usr/sbin/nvidia-modprobe -c0 -u

mount --bind /dev "$CHROOT/dev"
mount -t proc /proc "$CHROOT/proc"
mount --bind /sys "$CHROOT/sys"

/usr/sbin/chroot "$CHROOT" /usr/sbin/nvidia-smi
exec /usr/sbin/chroot "$CHROOT" /sbin/su - <username>