Using Fly.io as a remote builder for nixpkgs

Offload building of your local nixpkgs to a remote builder on Fly.io

May 13, 2024 4 min

A service I enjoy using, and write about often, is fly.io, as it lets me have compute on demand, without having to worry about provisioning and managing an entire server.

Editor's Note: This was a fun experiment, but you are likely better off using a managed service such as nixbuild.net for anything even resembling a production use case.

One of my latest use cases for it is offloading building various nixpkgs from my low-powered laptop to a remote Fly.io builder. I can give the builder more resources for the period when I need them.

Start by creating the builder machine. The default has 256 MB of RAM, while this example requests 2 GB and four CPUs. Adjust those values for the package you are building. Fly machines can be created from container images, so this setup uses the same nixos/nix image as Docker.

fly launch --name tklk-nixbuilder --region ams -o personal --build-only --image nixos/nix --no-public-ips  --vm-cpus 4 --vm-memory 2048

The --build-only flag keeps Fly from starting the machine before the remaining configuration is in place.

The builder needs additional storage because building a package can pull in more of its dependency graph. Create a volume for the Nix store:

fly volumes create nix_store -a tklk-nixbuilder -s 50 -r ams -y

This example uses a 50 GB volume. Fly bills for the storage regardless of whether the machine is powered on, so choose the size based on the expected store. Set the mount path in fly.toml:

[mounts]
  source="nix_store"
  destination="/data/nix"

Do not mount the empty volume at /nix yet. It would hide the existing Nix store, leaving the machine without the binaries it needs. Mount it at /data/nix first, copy the existing store, and then change the destination to /nix.

Some internal Fly.io processes look for the /bin/sleep binary. NixOS stores binaries under hashed /nix/store paths instead of /bin, so that path does not exist. Either create a symlink or set the command in the configuration. This example sets the command explicitly:

[experimental]
  cmd = ['nix-shell', '-p', 'coreutils', '--run', 'sleep inf']

Deploy the machine after the configuration is in place:

fly deploy -a tklk-nixbuilder

After the machine starts, SSH into it to finish the setup:

fly ssh console -a tklk-nixbuilder

First, copy the contents of the nix store into the mounted volume.

cp -a /nix/. /data/nix/

Back on your host machine, change the destination path of the mounted volume in fly.toml to /nix. Then restart the machine with fly deploy -a tklk-nixbuilder.

Generate an SSH key for the remote connection with fly ssh issue:

fly ssh issue -o personal -u root /home/tklk/.ssh/fly_key

Add the following entry to ~/.ssh/config to avoid repeating the connection options:

Host nixbuilder.fly
	StrictHostKeyChecking no
	UserKnownHostsFile=/dev/null
	Port 2200
	User root
	HostName localhost
	IdentityFile ~/.ssh/fly_key

Editor's Note: This SSH entry disables host key checking for nixbuilder.fly. That avoids prompts for the local proxy endpoint, but it also removes SSH host verification for this entry.

Start a local proxy from port 2200 to port 22 on the remote builder. This avoids configuring a WireGuard VPN connection to Fly.io, which is more involved than this port forward. The trailing & runs the proxy in the background. If the terminal closes, the proxy stops and must be restarted before reconnecting.

fly proxy 2200:22 &

With the proxy running, test the remote builder by adding --store ssh-ng://nixbuilder.fly to a nix build command.

Example:

nix build nixpkgs#mercurial --store ssh-ng://nixbuilder.fly --show-trace

The Fly builder uses x86-64 so if your local machine is aarch64, pass --system x86_64-linux to nix-build so the remote builder can build the package. You can copy the x86-64 output back, but it will not run natively on the Arm machine. For aarch64 builds, nixbuild.net supports aarch64 builders, though not macOS builders as of this writing.