A service I enjoy using, and write about often, is fly.io, as it lets me have compute on demand, without having to worry about provisioning and managing it an entire server.

Editors Note: This was a fun experiment, but you are likely better off using a managed service such as nixbuild.net for anything even resembling a production use case.

One of my latest use cases for it, is to offload building of nixpkgs from my low powered laptop, to a remote builder on fly where I can provision as many resources as I need, for as short of a period of time as needed.

To start off, we’ll need to create the builder machine. You’ll likely need more than the default 256mb of ram that is provisioned, but maybe not the 2gb that I request. You can adjust the number of cpus and memory as needed. Since fly’s machines, are essentially containers (they do some neat things, and you can read about it in one of their many blog posts, but that is outside of scope of this post), you can also use the same container image that you would use with docker, in this case I am using nixos/nix.

fly launch --name tklk-nixbuilder --region ams -o personal --build-only --image nixos/nix --no-public-ips  --vm-cpus 4 --vm-memory 2048

We are not asking fly to start the machine right away, as we’ll need to add some more configuration before powering it on.

First, we’ll need to give the machine more storage space, as storing/building a package may build more of the package’s dependency graph than just the package itself, and there will need to be some place to store it.

fly volumes create nix_store -a tklk-nixbuilder -s 50 -r ams -y

I’m setting 50gb, but since you are billed for space consumed regardless if the machine is powered on or not, you may wish to make this volume smaller based on your needs. You’ll also need to set the path that the volume is mounted with in your fly.toml configuration file.

[mounts]
  source="nix_store"
  destination="/data/nix_store"

Interesting note about this coniguration, even though nix stores most things in /nix, we are mounting to /data/nix_store initially, as otherwise if we mount it directly to /nix, it will be empty, and the machine won’t have access to any binaries. So we will mount to a different directory, copy everything over to the new volume, and then set the path to the correct location.

Some internal Fly.io processes look for the /bin/sleep binary, however in NixOS, due to it being entirely reproduceable, it stores binaries with a specific path name with a cryptographic hash of the package’s build dependency graph. This means that the /bin/sleep binary is not available, so we’ll either need to create a symlink to it, or we can set a path to the binary in the configuration.

[experimental]
  cmd = ['nix-shell', '-p coreutils', '--run', "sleep inf"]

Your machine is now ready to be powered on. You can do this by running:

fly deploy -a tklk-nixbuilder

Your terminal will show you progress of the machine starting up, and in a short period of time it will be all ready, and you can now SSH into it to finish some final setup steps.

fly ssh console -a tklk-nixbuilder

First, copy the contents of the nix store into the mounted volume.

cp -a /nix/store/. /data/nix_store/

Back on your host machine, you can change the destination path of the mounted volume in your fly.toml configuration file to /nix/store and restart the machine with fly deploy -a tklk-nixbuilder

Your remote machine is now all ready to handle building of your remote nixpkgs.

You can now use the fly ssh issue command to generate an SSH key that can be used to SSH into your machine.

fly ssh issue -o personal -u root /home/tklk/.ssh/fly_key

For sake of convenience you can add some minor configuration to your ~/.ssh/config file to reduce CLI options when letting your nix-builder connect to the remote machine.

Host nixbuilder.fly
	StrictHostKeyChecking no
	UserKnownHostsFile=/dev/null
	Port 2200
	User root
	HostName localhost
	IdentityFile ~/.ssh/fly_key

Finally, you’ll need to setup a proxy to forward port 2200 on your local machine to port 22 on your remote builder. This is to avoid having to configure a wireguard VPN connection to fly.io, which is a bit more involved than just adding a port forward. The & in the command below, forces the proxy into the background so you don’t need to have a terminal dedicated to just the proxy command. Note: if you close the terminal then this command will quit and you’ll need to start it again if you want to connect back.

fly proxy 2200:22 &

That’s it! Your remote builder, and local machine are all setup for building.

You can test your remote builder by adding --store ssh-ng://nixbuilder.fly to any nix-build command.

Example:

nix-build --store ssh-ng://nixbuilder.fly -A mercurial --show-trace

Some caveats are that the remote builder will be an x86-64 architecture, and if your local machine is aarch64 (aka Arm), you’ll need to pass --system x86_64-linux to nix-build to ensure that the remote builder will build the package. Sadly this means that the package will be built for x86-64, and not aarch64, so you’ll won’t be able to copy it to your machine. As the editors note says, you are better off using something like nixbuild.net, as they support aarch64 builders (though not macOS ones as of the time of writing).

Continued reading: