TechKnowLogick

Vanity, thy name is Go

Tue, 10 Mar 2026 00:00:00 +0000

Go vanity imports let you put a custom domain in front of your import paths. Instead of go get gitea.com/user/repo/pkg (or in my case go get src.tklk.dev/tklk/x/pkg), it's just go get tklk.dev/pkg. Shorter, and decoupled from wherever the code actually lives.

After moving everything into a monorepo, this became trivial to set up. One repo means a single vanity domain covers all packages under it. I got it working with Bunny.net using nothing but a custom 404 page. No servers, no edge compute, no edge rules. Just a static HTML file.

How Go vanity imports work#

When you run go get tklk.dev/somepkg, the Go toolchain makes an HTTP request to https://tklk.dev/somepkg?go-get=1 and parses the HTML response. It looks for a <meta name="go-import"> tag in the <head>, which tells it where the actual source code lives.

The meta tag has three space-separated values in its content attribute: the import prefix, the VCS type, and the repository URL.

<meta name="go-import" content="tklk.dev git https://src.tklk.dev/tklk/x">

For a monorepo, all subpaths under the import prefix resolve to the same repo root. tklk.dev/somepkg and tklk.dev/otherpkg both point at https://src.tklk.dev/tklk/x. The Go toolchain handles the rest, mapping the subpath to a directory within the repository.

The trick: a custom 404 page#

For most cases, the Go toolchain does not care about the HTTP status code. It fetches the URL and parses the HTML looking for the meta tag. A 200 and a 404 work equally well, as long as the response body contains what it needs.

A redirect to Gitea would've been simpler, but the Go toolchain reads the meta tag from the final response after following redirects. Gitea serves its own go-import meta tag with the repo path as the import prefix, so a redirect would make the toolchain see the Gitea hostname instead of the vanity domain. The 404 page sidesteps this because the response comes from the vanity domain itself.

Bunny.net lets you configure a custom error page for a pull zone. Set that 404 page to an HTML file containing the go-import meta tag, and every path under tklk.dev/anything that doesn't match a real file returns the custom error page. The Go toolchain reads the meta tag and resolves the import.

The full 404.html:

<!DOCTYPE html>
<html>
<head>
  <meta name="go-import" content="tklk.dev git https://src.tklk.dev/tklk/x">
</head>
<body>
  <p>Not found. See <a href="https://src.tklk.dev/tklk/x">source</a>.</p>
</body>
</html>

That's the entire thing. The go-import tag tells Go where the repo is.

For this specific pullzone of mine, it also serves a few real files, such as my Docker registry at /v2/, and a few other things that don't conflict with package names, so when the custom 404 fires for paths that don't match anything else it's still fine.

Setting it up#

If you're already using Bunny.net for static hosting, this is a small addition. I covered the basics of creating storage zones and pull zones in a previous post.

To set it up:

Create a storage zone and pull zone for the vanity domain (or reuse an existing one).
Upload the 404.html to the storage zone. Bunny expects custom error pages at bunnycdn_errors/404.html, or if you wanted it somewhere else, you can specify the path in the storage zone settings.
In the pull zone settings, set the storage zone as the origin.
Add a DNS CNAME record pointing tklk.dev to the pull zone hostname.

Testing it#

A quick curl confirms it's working:

curl -s 'https://tklk.dev/somepkg?go-get=1'

You should see the HTML with the meta tag come back, even though the response status is 404. From there, go get tklk.dev/somepkg resolves to the Gitea repo and pulls the code down.

Clean import paths, zero infrastructure to run, and it costs nothing beyond what the CDN already costs. The monorepo is what makes the whole setup so simple. If the domain fronted multiple independent repos instead, each import path would need its own meta tag pointing at a different repository, and you'd need edge compute or a server to route them. I have a separate project, go-vanity-url, that handles that case by generating a set of static HTML files from a TOML config, one per package, each with the correct go-import meta tag. You can host the output on any static file server or CDN. With a monorepo though, none of that is necessary. One meta tag covers everything, so a static 404 page is enough.

Monorepo! Monorepo! Monorepo!

Wed, 04 Mar 2026 00:00:00 +0000

What's it called? Monorepo! That's right, monorepo!

I've been maintaining many small Go repositories for personal projects, libraries, and a bunch of experiments. The overhead that comes with all of that adds up. After recently switching to a monorepo, I'm a convert: so much toil immediately disappeared.

The inspiration#

Xe Iaso maintains a public monorepo, x, that contains many Go projects, experiments, and libraries. Sometimes full projects, such as Anubis, have even been spun out of it. The repo is well-structured, using idiomatic Go project layouts, with a go.mod at the root, and has repo-wide policies for things like linting, and other helpful items.

The problem#

Whenever I have a new idea, I usually just make a temporary/scratch folder for it, since setting up CI, linting, dependency management, etc. for a one-off experiment feels like more work than it's worth. If the experiment turns out to be interesting and I want to keep it around, then I make a repo for it, and copy the code over. I do this, since I don't want to have to sign up for maintenance on something that might just be a dead end.

Whenever repos were created, they were inconsistent, and sometimes using a shared library meant signing up for potentially having to update multiple different repos that imported it when making a "breaking" change.

Dependency management was the worst of it. I use Renovate to keep things updated, and it does its job well. When you have several repos with overlapping dependencies, "doing its job" means a steady stream of PRs across all of them, especially for repos of low importance. Even with automatic merging on dependency updates, those emails still come in, notifications happen, and hopefully the tests pass. If not, and something needs to be fixed, then I have to decide if I'm going to spend the time to context switch and fix it, or if I'm just going to ignore it and let it rot until I need to use that repo again and notice the problem then. It's a minor annoyance that adds up. Especially as more and more repos get added.

One repo to rule them all#

After looking at how Xe structured their x repo, I moved everything into a single repository. Most of the annoyances I described above just went away.

With a monorepo, I have CI set up right out of the gate for any new CLIs, servers, docker images, linting, etc. I don't have to worry about that for new things, and by having a standard structure, I can just copy and paste from existing projects to get started. The barrier to entry for new experiments is much lower, so I'm more likely to try out new ideas, and have them use a consistent coding style and structure.

I can also have shared utilities across multiple programs, and just have to update them once without needing to bump deps for a bunch of repos. I can also have shared CI and linting rules, so when I want to change a rule, I change it once and it applies everywhere.

Having the infrastructure already set up means I can start new things without worrying about it. More of my experiments end up in source control, giving me better visibility into them if I want to go back and revisit them later.

The freedom of building for one#

The monorepo approach doesn't only work at the scale of one person; some of the largest companies in the world use monorepos for similar reasons. Those companies have the resources to build tooling to manage the complexity of a monorepo at scale, but the core motivation is the same: reduce overhead and friction in maintaining code.

This isn't unique to monorepos, but having a personal one with no expectation of supporting a wide audience gives me the freedom to make choices I wouldn't make if I were building a library for others to use.

This is the opposite of what you'd want for an open source library or a shared codebase at work. In those contexts, the overhead of separate repos with clear versioning and ownership boundaries exists for good reason. But for personal projects, that overhead is solving a problem that doesn't exist.

Give it a try#

If you are maintaining many small repos, possibly more than just Go, and you find yourself spending more time on repo maintenance than on the code itself, it might be worth trying. Xe's monorepo is a good reference for how to structure things.

A serverless Docker registry with Bunny.net

Tue, 03 Mar 2026 00:00:00 +0000

I host all of my Docker images using my personal Gitea instance, but rather than exposing my home network on the internet, I figured I could upload the images to Bunny.net too, and use the CDN there to serve them.

Bunny.net has edge compute (think AWS Lambda@Edge), but you end up paying a per-request cost, so I wanted to see if I could do it entirely with edge rules, which are cheaper. Luckily, the OCI distribution API turns out to be mostly static file lookups (at least for public images since you don't need to consider authentication). A CDN that can rewrite URLs and set response headers can pretend to be a registry.

What a Docker registry actually serves#

The OCI distribution spec boils down to a few GET endpoints:

GET /v2/ returns 200 OK as an auth check (conventionally with a body of {}, though the spec doesn't mandate a specific body). If you return 200, clients will assume auth isn't required.
GET /v2/<name>/manifests/<reference> returns a manifest (by tag or digest)
GET /v2/<name>/blobs/<digest> returns layers and configs

There's a wrinkle with manifests. Since I opted not to use edge compute, where I could easily distinguish an index from a manifest, and set headers accordingly, I had to find another way. The solution was to store platform-specific manifests and multi-arch indexes separately, so the CDN can set Content-Type based on path alone. If a client requests a manifest by digest, the CDN looks for it in the manifest directory first. If it's not there, it must be an index, so the CDN issues a redirect to the index path. This way the client gets the correct Content-Type without the CDN having to inspect file contents.

The responses also need specific headers: the correct Content-Type and Content-Length for each artifact, and Docker-Content-Digest on manifests and blobs. I also set Docker-Distribution-API-Version: registry/2.0 on all /v2/ responses. This is a legacy Docker header that the OCI spec considers optional, but some older clients still expect it. If you serve the right files with the right headers at these paths, Docker doesn't know the difference.

Storage layout#

The file structure in the Bunny storage zone mirrors the API paths, with some extras to help the edge rules dispatch correctly:

/v2/
  _root                              # static {} for the base endpoint
  _blobs/sha256:<hex>                # shared blob store (deduplicated)
  <image>/
    manifests/
      sha256:<hex>                   # platform-specific manifests
      tags/<tag>                     # image index JSON (multi-arch)
    indexes/sha256:<hex>             # image indexes (by digest)

Blobs live in a shared _blobs/ directory, deduplicated across images.

The edge rules#

Eleven edge rules let me make Bunny.net look like a Docker registry. They handle path rewrites, response headers, and the 404-to-redirect trick for manifest vs index dispatch.

Base endpoint#

Three rules handle the /v2/ base path:

Set the Docker-Distribution-API-Version: registry/2.0 header on all /v2/* requests.
Rewrite /v2/ (exact match) to serve the _root file, which contains {}.
Set Content-Type: application/json on that base response.

Tag resolution#

When a client requests /v2/<name>/manifests/<tag> (where the tag is a human-readable name like latest, not a sha256 digest), rule 4 rewrites the path to /v2/<name>/manifests/tags/<tag>. This uses Bunny's triggerMatchAll to match /v2/*/manifests/* while excluding paths that already contain sha256: or tags/.

Rule 5 sets Content-Type: application/vnd.oci.image.index.v1+json on responses from /v2/*/manifests/tags/*. In my setup, tag lookups always return an image index (the multi-arch manifest), since I always build multi-arch with buildx. If you only built single-platform images, a tag could point directly to a platform manifest instead.

Blob requests#

Rule 6 rewrites blob requests from /v2/<name>/blobs/sha256:* to /v2/_blobs/sha256:*, pointing at the shared store. Bunny's path segment variables (%{Path.3}) extract the sha256:<hex> segment to build the rewritten path.

Manifest vs index dispatch#

When a client requests a manifest by digest (/v2/<name>/manifests/sha256:*), the CDN looks in /manifests/ first. If the digest belongs to a platform manifest (the config and layers for a single architecture, like linux/amd64), the file exists there and the CDN serves it directly. Rule 8 sets Content-Type: application/vnd.oci.image.manifest.v1+json on these paths.

If the digest refers to an image index (the document that lists which platform manifest to use for each architecture), it won't exist in /manifests/. It only lives in /indexes/, so the request 404s. Rule 7 catches that 404 and issues a 302 redirect to /v2/<name>/indexes/sha256:*. This has to be a real redirect, not a URL override, so the client makes a fresh request. That way the response header rules evaluate against the /indexes/ path. Rule 9 then sets Content-Type: application/vnd.oci.image.index.v1+json correctly.

Digest headers#

Rules 10 and 11 extract the sha256 digest from the URL path (using the %{Path.3} path segment variable mentioned above) and set it as the Docker-Content-Digest header. One covers manifest paths, the other covers index paths after redirect. Strictly speaking, the OCI spec also requires Docker-Content-Digest on blob responses, but in practice the clients I've tested (docker, crane, skopeo) don't depend on it for blobs, so I haven't added a rule for that.

Publishing images#

Unlike a regular Docker Registry, where you can use docker push, I have to upload files in a specific way since I need manifests and indexes to upload into the directory structure above. I build images with docker buildx in CI, targeting multiple architectures and exporting in OCI format. A publish script transforms the OCI layout into the storage zone structure, then uploads everything using the Bunny.net API for storagezones. After the upload, the script purges the tag manifest URL so docker pull immediately sees the new version.

publish-oci-image <image-name> <tag> <oci-layout-dir-or-tar>

  extract OCI input into a staging directory if it's a tar file

  read index.json to find the top-level index digest
  copy the index blob to:
    /v2/<name>/manifests/tags/<tag>   (tag lookup)
    /v2/<name>/indexes/<digest>       (digest lookup)

  for each platform manifest listed in the index:
    copy the manifest blob to /v2/<name>/manifests/<digest>
    for each blob (config + layers) referenced by that manifest:
      copy to /v2/_blobs/<digest> if not already there

  upload the directory to the Bunny storage zone as it matches the structure from above
  purge the CDN cache for /v2/<name>/manifests/<tag>

The staging directory mirrors the final storage layout exactly, so the upload is a straightforward sync. Even if two platform manifests reference the same layer, the script only copies it once. The cache purge at the end targets just the tag URL, because that's the only path where staleness matters (blob digests are content-addressed, so they never change).

Garbage collection#

With no registry server managing state, storage can balloon quickly, so I have a separate GC command to handle cleanup. It walks each image's tags, resolves every referenced blob, manifest, and index, then checks for unreferenced files in _blobs. If it finds any, it deletes them using the Bunny API. This can be a bit slow since it has to walk the whole graph, but it's good enough for my use case since I only publish a few images and for each of those images, I only use the latest tag.

Pulling images#

docker pull tklk.dev/abc:latest works. I've verified it with docker, crane, and skopeo. Each one sees a real registry because the headers and paths are correct.

Limitations#

There's no docker push. Images go through the publish script, which means CI is the only path to publishing. Images are public only, so this approach only works for content you're happy to expose without authentication. I also haven't implemented tag listing (/v2/<name>/tags/list) or the Docker-specific catalogue endpoint (/v2/_catalog, which isn't part of the OCI distribution spec). I know what images I have because I put them there.

If you need private images, a push API, tag listing, or fine-grained access control, you probably want something like Gitea. This setup works well for personal and public images where the push workflow goes through CI anyway, and it runs on a single pull zone with no servers behind it.

Tricking GitHub Copilot into reviewing PRs in Gitea

Mon, 02 Mar 2026 00:00:00 +0000

Xe Iaso wrote about building a review bot that uses a self-hosted LLM to review GitHub pull requests. It was an interesting idea, and when I found out about SDK's for "AI" CLIs, I wondered if I could do something similar, but using a harness built/maintained by a billion-dollar corporation, instead of handling the loop and execution environment myself.

Disclaimer: I chose Copilot for this because: it was free to use, the novelty of having a GitHub thing do something for something other than their platform, and these companies don't need more money, especially for a silly experiment.

Contrary to the title, I didn't actually "trick" Copilot. Turns out these AI coding CLIs have SDKs that let you interact with them programmatically, kinda like you would with a RESTful API, except in the same non-deterministic way the CLIs behave, plus they can still go through their normal loop and call tools. I happened to use Copilot, but it seems like this is a common pattern amongst these tools. I should also note that I have reservations about AI tooling in general (a topic for another post that's been sitting in my drafts). This post is about a pattern I found interesting, not an endorsement of any particular product, and especially not an endorsement/condoning of the practices of the companies behind them.

The approach#

Xe's setup triggers via a GitHub Actions workflow when someone comments /reviewbot on a PR. It uses an OpenAI-compatible API backed by a self-hosted model running on a DGX Spark, with an agentic loop that has two tools: one for executing Python (to analyze the codebase) and one for submitting the review. The bot also clones the repo so the model has filesystem access through Python execution.

The way I approached this was different, and yet also similar. A webhook listener watches for a specific pattern in PR comments, and when it matches, it fires off a review. The review starts off by the golang process fetching information about the PR such as diff, other existing comments, etc., then spins up the copilot cli, with a crafted prompt based on the collected context, and a set of custom tools. The agent then explores the codebase and calls the tools as needed, and when it's done, the review summary gets posted back to the PR.

The tools#

The agent gets three tools. These are the bridge between the PR and the agent itself. They translate between "AI wants to give feedback" and "Gitea needs specific API calls." The agent has no idea it's talking to Gitea.

post_inline_comment posts a comment on a specific file and line, with a severity level, either blocker (must-fix before merge) or suggestion (recommended improvement). The PR author can triage quickly: address the blockers, consider the suggestions.

note_low_confidence records an observation the agent isn't confident about. More on this one in a moment.

submit_review posts the final summary as a top-level PR comment. The agent calls this exactly once, as its last action.

The note_low_confidence tool came from Angie Jones' post on teaching Copilot to think like a maintainer, where she discusses setting a confidence threshold for AI reviewers. Her key insight: set a confidence threshold (>80%) so the reviewer only comments when it's fairly sure something is wrong. Without that threshold, AI reviewers dump every observation as an equally-weighted comment, and people learn to ignore the bot fast.

Note: While we may describe these machines as "thinking", it is important to recognize our anthropomorphization of what is really statistical analysis being done by the model.

I agree with having a confidence threshold, but my confidence in these models are also not high enough to trust in their weighting. If I were building a product, or if this were a production environment where overloading the developers with noise would be a problem, then I probably would have the bot stay silent. But, for an experiment, I wanted to see all that these models had to offer, to make the evaluation myself.

These low confidence observations are hidden in the main review, in a collapsible section at the bottom, so as not to distract from the main review, but they're still there for anyone who wants to see them. This way, I get to be my true Hannah Montana self, and have the best of both worlds.

The system prompt#

Angie's post was full of great ideas, especially around instructions on what to skip, such as linting, since presumably there are other CI workflows already catching those errors, and they're better and more efficient at finding them.

The prompt also explains the confidence model (high confidence gets an inline comment, low confidence gets a note_low_confidence call) and includes the expected output format for the final summary.

PR context goes in as a structured template: title, author, branches, description, the diff, existing comments (filtered to exclude bot noise). If the diff is too large, it's truncated with a note telling the agent to use git diff on specific files. The agent has the full cloned repo, so it can always look at more than what's in the prompt.

The feedback loop#

Once the agent is done, the review summary comment gets updated. Low-confidence notes are appended in a collapsible <details> section, and usage metadata (token counts, model used, API calls, cost if available) goes in another collapsible section at the bottom.

I include the usage metadata because when someone reads a bot-generated review, they should be able to see what produced it and roughly what it cost. Having it say "claude-sonnet-4.5, 12 API calls, 45k input tokens..." makes sure that the developer is aware of the resources that went into producing the review, and hopefully encourages their future contributions to be focused. If a contribution is too large, it may overflow the context window, causing reviews to be less effective.

The pattern#

So little of this is Copilot-specific. The SDK, the model, the hosting. Those are all interchangeable. The tools are what make it work somewhere it was never built for.

Code is left as an exercise to the reader. Although, if you are a VC, I am happy to send you my routing number in exchange for $10 Billion for this unicorn.

DIY Multi-Hop Boundary Sessions without HCP

Wed, 16 Apr 2025 00:00:00 +0000

A common theme in my posts, is to treat my homelab as if it has the same security requirements as a production environment. Since, in theory, it is a production environment, and I wouldn't want anything to happen to it. For protected access, I normally use something like Tailscale, since I don't need to expose anything to the internet, but sometimes when I am accessing it via a remote network, that network may be locked down and prevent Tailscale/wireguard connections, so I need a break-glass solution.

It is no secret from other posts on this blog, that I use Hashicorp's suite of tools, and the next one I'd like to introduce is Boundary. Boundary is a tool that allows for identity based access management, meaning that no matter the system I want to connect to in my homelab, I can log into Boundary, and connect to it, without having to worry about SSH tunneling, or exposing it to the internet. I also might not have access to a machine that I can install VPN software on, and in the case of Boundary, it is a single binary that doesn't need administrative access to run, so it is a viable option for me.

The Problem#

Sadly, I am budget constrained for my homelab, and the community version of Boundary does not support ingress/egress workers, which are required for multi-hop sessions. This is a feature that is only available in the enterprise version of Boundary, or HCP (HashiCorp Cloud Platform).

I'd like to have these multi-hop sessions, so I can keep some VLAN isolation, and not have the server in the DMZ be able to connect back directly to every server in my homelab.

Here's a visual of what I wanted to achieve:

graph TB subgraph Internet client[Client/Laptop] end subgraph DMZ controller[Boundary Controller] dmz_worker[DMZ Worker] frps[FRP Server] end subgraph Internal_VLAN[Internal VLAN] internal_worker[Internal Worker] internal_resources[Internal Resources] end internal_worker -- 1. Register --> controller client -- 2. Connect --> controller client -- 3. Session --> dmz_worker internal_worker <--> frps dmz_worker -- 4. Connect --> frps internal_worker -- 5. Access --> internal_resources

My DIY Solution#

The Boundary connection flow is key to my solution. Each session goes through the following steps:

The client (my laptop) connects to the controller
The controller provides an authorization token
The client connects to the worker port using this token
The worker establishes the session with the target

With this process in mind, I developed a workaround. First, I had to create an FRP (Fast Reverse Proxy) server on the same server as my DMZ Boundary Controller and Worker. That way the internal workers could reach out to that server, and I wouldn't need to open any firewall rules in the direction of the internal workers. Then, after the workers are registered with Boundary, I also had to create a target for the internal workers which point to a unique port from frps on the Boundary server loopback that aligns with a specific worker.

Here's how the connection flow works with my solution:

sequenceDiagram participant Client participant Controller as Boundary Controller participant DMZ as DMZ Worker participant FRP as FRP Server participant Internal as Internal Worker participant Target as Internal Resource Client->>Controller: 1. Authenticate & request session Controller->>Client: 2. Return session token Client->>DMZ: 3. Connect to internal worker target DMZ->>FRP: 4. Forward to internal worker via FRP tunnel FRP->>Internal: 5. Connection established to internal worker Client->>Controller: 6. Request session for internal resource Controller->>Client: 7. Return session token Client->>Internal: 8. Connect to internal resource via established tunnel Internal->>Target: 9. Connect to target resource

Setting it up#

Using the standard Boundary installation process, you'd need to expose the DMZ Boundary Controller (for API access, and usually 9200), and worker (usually port 9202) to the internet. It could be something like boundary.tklk.dev, and based on your preferences you could use Caddy and have it use real TLS certificates for when you are attempting to access the API. The internal workers will also need to connect to the cluster worker coordinator on port 9201, but that doesn't need to be exposed to the internet, and can be done via the same way that the internal workers connect to the FRP server.

Installing FRP Server#

First, you'll need to install FRP on your DMZ server:

Download the appropriate FRP release from GitHub
Extract the archive and locate the frps binary
Create a configuration file as shown below
Set up a systemd service (optional) to ensure it runs automatically

A sample frps configuration, based on what I am actually using:

bindPort = 7000
auth.method = "token"
auth.token = "your_secure_token"

# meaning the internal workers can only use this range for listening on
allowPorts = [ {start = 9500, end = 9999}  ]

Internal Worker Setup#

Now that you have the brains of the operation setup, you'll need to move onto the internal workers, and follow these steps:

Install and configure a standard Boundary worker

# minimal worker config
# you may wish to add more configuration options, such as using a KMS
listener "tcp" {
  # this is the listener that frpc will connect to over the loop back
  address = "127.0.0.1:9202"
  purpose = "proxy"
}

worker {
  name = "internal-worker-1"
  # this is the address that the boundary controller will announce to the client
  # so this is what the target session should listen to on your local laptop/client
  public_addr = "127.0.0.1:9500"
  description = "Worker in the internal VLAN"
  controllers = ["boundary.tklk.dev:9201"]
}

Install FRP client (frpc)
Configure frpc to establish a reverse tunnel back to the primary server on a loopback interface

Here's a sample frpc configuration, based on what I am actually using:

serverAddr = "boundary.tklk.dev"
serverPort = 7000
auth.method = "token"
auth.token = "your_secure_token"

[[proxies]]
name = "boundary_internal_worker"
type = "tcp"
localIP = "127.0.0.1"
# 9202 is the Boundary worker's default port, but since you may have multiple internal workers, you might want to pick
# a unique port for each worker (which you'll also need to set in the worker config)
# It is strongly recommended that you don't use port 9202 as the DMZ worker will already likely be using this port, causing the frps connection to fail
localPort = 9202
remotePort = 9500

Registering the Internal Worker as a Target#

The last part of the setup is that you'll now need to register the internal worker as a target in Boundary. This is done by creating a target that points to the FRP server's loopback interface, and the port that you set in the frpc configuration. This can be done via terraform, the boundary cli, or the Boundary UI.

A target with 127.0.0.1 as the target host, and the port that you set in the frpc configuration, along with the default client port being the same as the worker port is the bare minimum you'll need to set.

You may also want to set an egress worker filter, to ensure that your DMZ worker is the one that connects to the frps port, as the target uses localhost, the routing might be assigned to a different worker that is not yet accessible.

Using the Setup#

Now comes the fun part, actually using the setup. Go through the standard authentication process to get a session token, and then use the boundary connect command to establish a connection to the internal worker target.

# Authenticate with Boundary (if not already authenticated)
boundary authenticate

# List available targets
boundary targets list -recursive

The workflow to access internal resources involves two steps:

First establish a connection to the internal worker:
```
boundary connect -target-id=ttcp_1234567890  # Your internal worker target ID
```
If nothing else on your local machine is listening on the worker port as defined above, you'll open a connection to the internal worker. In your session list, this connection may be marked as "pending" since no traffic is being sent yet.

Once connected, you can access targets in the internal VLAN:

boundary connect -target-id=ttcp_0987654321  # Internal resource target ID

While this requires an extra step compared to the enterprise solution, it provides the same functionality at no additional cost.

Limitations and Considerations#

You need to manage FRP alongside Boundary
The two-step connection process is less streamlined than the enterprise solution
Nested connections can add a small amount of latency
Security considerations: Be careful with your FRP token to prevent unauthorized access

Conclusion#

While I probably don't need the multi-hop sessions in my homelab, and likely single hop sessions would be sufficient, I wanted to fully explore the capabilities of Boundary to learn more about it in depth. I came out of this experience with a deeper understanding of the connection flow, and techniques to extend and debug various networking issues of it.

This could be extended to use vault and broker SSH credentials with boundary, but you could leverage my previous post on using vault with SSH to do that instead. For now, I am happy with the setup I have.

SSH into your Gitea Actions for debugging

Thu, 10 Apr 2025 00:00:00 +0000

While Gitea Actions allows for local debugging of workflows, sometimes you need to debug directly on the runner. This is especially challenging since the workflows only exist for the duration of the run. Even if you have access to the server and can docker exec into the container, it may terminate while you are debugging.

Since Gitea Actions are compatible with GitHub Actions, you can use action-tmate to SSH directly into the runner environment, and debug your workflow in real-time.

Adding action-tmate to your workflow#

First, edit your Gitea Actions workflow YAML to add the action-tmate step.

name: CI
on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      # Add this step wherever you need debugging access
      - name: Debug with SSH
        uses: mxschmitt/action-tmate@v3

When this step runs, it will pause the workflow and output SSH connection details in the logs. The workflow will wait until you've connected, finished debugging, and exited the session. However, if you place this step after a failing one, the workflow may stop before reaching it.

Connecting to your runner#

Once the action runs, check your workflow logs. You'll see connection details like:

WebURL: https://tmate.io/t/...
SSH: ssh abc123...@...tmate.io

You can connect using either SSH or the web interface. The web interface is especially useful if your network's firewall blocks SSH connections.

Once connected, you'll have full shell access to the runner environment, allowing you to inspect files, run commands, and verify network connectivity.

When you're done debugging, exit the SSH session and then the workflow will continue with the next step after the action-tmate step.

Security: Remember that SSH access is public (if someone gets the link), so don’t leave it idle for too long.

Targeted debugging approaches#

You probably don't want to SSH into every workflow run. Here are some more practical approaches:

Only on failure:#

- name: Debug with SSH
  if: ${{ failure() }}
  uses: mxschmitt/action-tmate@v3

This only activates the SSH session if a step in the workflow failed.

Manual trigger:#

With the introduction of workflow_dispatch in Gitea 1.24, you can manually trigger workflows, and have an option to enable SSH debugging:

on:
  workflow_dispatch:
    inputs:
      debug_enabled:
        type: boolean
        description: 'Enable SSH debugging'
        required: false
        default: false

# Then in your job:
steps:
  - name: Debug with SSH
    if: ${{ github.event_name == 'workflow_dispatch' && inputs.debug_enabled }}
    uses: mxschmitt/action-tmate@v3

Conclusion#

There are a lot of ways to debug failing workflows, but nothing beats direct access to the exact environment where the workflow is running. action-tmate provides many other advanced configuration options as well that you can investigate.

Manage an entire fleet of macOS VMs with HashiCorp Nomad

Wed, 09 Apr 2025 00:00:00 +0000

As an experiment to familiarize myself with Nomad driver plugins, I created a driver plugin for Tart that allows you to manage macOS VMs using Nomad. It allows you to manage macOS VMs with Nomad in a similar way to how you would manage Docker containers.

Disclaimer: This is a WIP and not yet ready for production use. It is a proof of concept and should be treated as such. Please do not use this in production environments. If you need production ready solutions, consider using Orchard which is built by the same developers behind Tart.

What is Tart?#

Tart is a VM Manager that is built to let you manage macOS VMs on Apple Silicon. One of Tart's key features is its integration with OCI-compatible container registries, allowing you to store and distribute VM images just like Docker containers.

Why create the Nomad driver plugin?#

One of the ways I like to learn new things is to build something. Since I wanted to learn more about Nomad and its driver plugin system, I decided to create a driver plugin. I chose Tart, since many others already exist such as Docker, etc.. Since Tart only manages specific VMs on a specific machine, rather than orchestrates across several, I thought it would be a good candidate for a driver plugin. Tart already has an orchestration manager, Orchard, but it is bespoke for Tart itself, and you may want to use Nomad for other workloads as well.

Sample Nomad Job#

Here's a simple example of how you can spin up a macOS VM:

job "tart-demo" {
  datacenters = ["dc1"]
  type        = "service"

  group "demo" {
    task "vm" {
      driver = "tart"

      config {
        image     = "ghcr.io/cirruslabs/macos-sequoia-base:latest"
        cpu       = 4
        memory    = 8192
        disk_size = 50
        command   = "echo 'Hello from Tart VM'"
      }
    }
  }
}

Conclusion#

There are many reasons why you may want to orchestrate macOS VMs, such as CI/CD workflows, ephemeral development environments, or testing infrastructure, and this plugin allows you to do just that.

Trouble with Azure Functions, and Managed Identities

Mon, 31 Mar 2025 00:00:00 +0000

As a part of a project to automate TLS certificates for Azure Application Gateways, I ran into an unexpected issue with Azure Functions and managed identities. Specifically, the tool I was using was attempting to fetch a managed identity token, but wasn't able to. I had to review the code of the tool, and many pages of documentation to discover that Azure Functions provide environment variables that contain the correct endpoint information which differ ever so slightly from the standard endpoint.

Editor's Note: Please don't take any of this in a negative light. The functionality that would normally be provided by an SDK couldn't be used, and the implementation is uncommon enough that it's understandable a tool like acme.sh wouldn't cover this edge case. It is a great tool, and I'm very happy with it.

I firmly believe that all TLS certificates should be automated, and the project I'm working on is to automatically request certificates from Let's Encrypt/ZeroSSL and store them in Azure Key Vault so that Azure Application Gateway can use them. This is the first in what will likely be a series of posts documenting this journey.

Overview#

My idea was to run acme.sh in an Azure Function, using Azure's managed identity to authenticate with Azure Key Vault. This would allow the certificates to be stored directly in Key Vault for App Gateway to use.

Since the Azure Function would run on a schedule, this would ensure certificates are always renewed well before expiration. Fresh certificates, without any manual intervention, and no need to worry about expiration dates.

The Problem#

Installing acme.sh in the azure function was quick enough, but when trying to run it with dns validation using a managed identity, I ran into the issue of being unable to fetch a token.

The error was that it was unable to connect to the managed identity token service. In most Azure environments, services access the managed identity token service via a link-local address (typically 169.254.169.254). This endpoint is well-documented and widely used. In most cases acme.sh script would've been using the correct endpoint.

After SSH'ing into the Azure Function itself, I was able to confirm that it wasn't able to connect to the metadata endpoint. It was only after reading through the python SDK docs that I discovered that I should be using the IDENTITY_ENDPOINT environment variable to get the correct endpoint.

The Solution#

Now that I discovered the correct endpoint (along with the additional header to be used for authentication), I was able to fetch the managed identity token successfully with curl. Then I had to search through acme.sh's code to discover how it was attempting to fetch the token. I discovered that it made the same assumption I had about the hardcoded endpoint, and so I was able to quickly throw together a code change to fix the issue.

In the spirit of open-source, and to thank the authors for their work on acme.sh, I was able to send that change as a PR to fix the issue so that no one else would have to go through the same debugging process.

What's Next?#

In the series of blog posts that I plan to write about the system design for auto-renewal of TLS certs using ACME for Azure App Gateways, I'll dive deeper into the actual implementation of the certificate automation process, and how I deeply integrated it into the Azure ecosystem.

Stay tuned if you're interested in learning more about how to automate TLS certificates for Azure Application Gateways.

SSH Multi-Factor Authentication with SoloKey V2

Sun, 30 Mar 2025 00:00:00 +0000

In a bout of attempting to improve my operational security, I decided to review my current practices and see which areas I could improve. While I already use SSH keys for auth, having a file that is essentially a password to all my servers is a bit of a security risk if it ever is compromised. I already use my SoloKey for 2FA on various websites, so why not for SSH too? This way, even if someone gets my private key file, they can't access my servers without physically having my SoloKey. Still a risk (see the XKCD comic about a $5 wrench), but less so.

Editor's Note: While MFA by using a physical key is a great improvement, you really should look into using SSH Certificates as they are time scoped, and depending on how you have them set up, have automatic rotation. Among other benefits, this is a great way to ensure that even if a key is unknowingly compromised, it can't be used for long.

Also, this same process could be used for other physical keys like YubiKeys, but I have several SoloKeys V2 (cheaper and open-source) so that is what I used.

Diving right into it#

According to the docs, all that is required should only be running ssh-keygen -t ed25519-sk and then touching the SoloKey when prompted. However, when testing this, I was on macOS devices and ran into an issue where the SSH version that ships with macOS doesn't support these types of keys. So, I had to install a newer version of OpenSSH with:

brew install openssh libfido2

After installation, you can check that you have the right version:

ssh -V

You should see OpenSSH_8.2 or later, which supports the FIDO2 keys. If you don't, then you may need to update your $PATH environment variable to point to the Homebrew version of SSH.

Creating a key tied to your SoloKey#

Now that I have the right version of SSH installed, I can finally create the key:

ssh-keygen -t ed25519-sk -f ~/.ssh/id_ed25519_sk

You'll be asked for a passphrase (recommended), and be prompted to touch your SoloKey to confirm. Once done, you'll have two new files:

~/.ssh/id_ed25519_sk - your private key
~/.ssh/id_ed25519_sk.pub - your public key

Making it convenient#

For convenience, I added the key to my SSH agent:

ssh-add ~/.ssh/id_ed25519_sk

If you don't use the SSH agent, you could instead add the key to your SSH configuration file

vi ~/.ssh/config

# Use the security key for all connections
Host *
    IdentityFile ~/.ssh/id_ed25519_sk

Setting up the server side#

Similarly to how you would add a regular SSH key, you can add the public key to your server:

cat ~/.ssh/id_ed25519_sk.pub | pbcopy
ssh user@remote-server "vi ~/.ssh/authorized_keys" # Paste the public key and save

Just like with your local machine, you'll need to make sure the remote server also supports the new key type. You can check this by running:

ssh user@remote-server "ssh -V"

The moment of truth#

Now try connecting:

ssh user@remote-server

If everything's set up correctly, you'll be prompted to touch your SoloKey to authenticate.

Wrapping up#

Setting this up took a bit of tinkering (especially figuring out about macOS and its older SSH version), but the extra security is worth it. You can also use this key for other services, like your Gitea instance.

If you are on a mac, you could also look into the tool Secretive, which allows you to store your SSH keys in the Secure Enclave of your Mac and integrate with the native Touch ID.

Gitea Actions: Forget about YAML, let's use Nix

Sun, 04 Aug 2024 00:00:00 +0000

I maintain many repositories using Gitea Actions, and I've run into a lot of pipeline failures due to the smallest of issues. Most commonly it is whitespace differences in YAML files that cause unexpected parsing or complete failures. Recently, I was helping someone debug a workflow where a copy and paste from a different workflow brought in spaces for some lines instead of tabs leading to the pipeline to fail.

Update: After writing this post, I discovered that a similar approach is already being done with Dhall. If you're interested in this concept, check out github.com/regadas/github-actions-dhall, which has already implemented much of what I discuss here, but using Dhall instead of Nix.

The Problem with YAML#

YAML is extremely sensitive to whitespace, and that can lead to subtle issues that are hard to debug. Especially when you can't see the differences visually between tabs and spaces.

The Solution: Nix#

In my case, as is the case with all my problems, the solution was to use Nix. Recently, I was using Terranix to manage some infrastructure (which is a tool that translates Nix to JSON for Terraform), I realized a similar approach could work for Gitea Actions. Both JSON and YAML represent data as key-value pairs, making them easily convertible between each other, meaning I can use the same approach to convert Nix to JSON to YAML.

Benefits of Using Nix for Workflows#

Elimination of whitespace-related errors
Nix features, like conditionals, loops, and functions
Improved code reuse and maintainability
Ability to split workflows into smaller, manageable components

Implementing Nix-based Workflows#

Step 1: Creating a Basic Step Function#

The most common component of a workflow is a step. So we could start by defining a function that creates a step.

# steps.nix
let
  mkStep = { name ? null, uses ? null, run ? null, with' ? null }:
    let
      step = {
        name = if name != null then name else null;
        uses = if uses != null then uses else null;
        run = if run != null then run else null;
        with = if with' != null then with' else null;
      };
    in
      builtins.filterAttrs (a: v: v != null) step;
in
  mkStep

This mkStep function creates a step object and removes any null fields, resulting in cleaner YAML output.

Note: In Nix, with is a reserved keyword in Nix, so I used with' instead.

Step 2: Creating a Checkout Function#

Building on the mkStep function, the most common step in every workflow is the checkout step. So we can create a function to create a checkout step which uses the mkStep function.

# steps.nix
# ... keep the previous code and add the new `mkCheckout` function
  mkCheckout = { name, uses ? "actions/checkout@v4", with' }:
    mkStep {
      name = name;
      uses = uses;
      with' = with';
    };

# Export both functions
{
    mkStep = mkStep;
    mkCheckout = mkCheckout;
}

This approach can be extended to other common actions, and potentially even generating functions automatically from action.yaml files.

Step 3: Creating a Workflow#

Using the functions above, we can create a complete workflow.

# workflow.nix
let
  steps = import ./steps.nix;

  mySteps = [
    (steps.mkCheckout { with' = { fetch-depth = 0; }; })
    (steps.mkStep { run = "echo 'Hello, World!'" })
    # More steps could be added here
  ];

in
  {
    name = "Example Workflow";
    on = "push";
    jobs = {
      build = {
        "runs-on" = "ubuntu-latest";
        steps = mySteps;
      };
    };
  }

Generating the Gitea Actions YAML Workflow#

To convert our Nix workflow to YAML, we can use the following command:

nix-instantiate --eval --strict --json ./workflow.nix | yq eval -P - > example.yaml

This command generates the YAML file (assuming you have the yq command installed).

Future Possibilities#

The benefits can be immediatly seen, especially for larger projects. For example, in the Gitea project, there are two workflows, one to build release artifacts for a nightly release, and one to build release artifacts for a tagged release. Both of these workflows are very similar, and so arguments and conditionals could be used to reduce output by half.

Nix Flakes could potentially be used to define the inputs for an Action's step, and then be used by the workflow. This would allow for a more declarative and modular approach to workflows.

Conclusion#

By using Nix, I can now write workflows in a more declarative and modular way, and all of my pain points with YAML have been resolved.

Editor's note: The Nix above hasn't been tested, and is just a rough draft of what I'm thinking. If you do attempt this as an approach I'd love to hear, and update this post with any feedback you might have. I am also a maintainer of the Gitea project.

Streamlining My GCP SSH Workflow: Using SSH Configuration Tricks

Thu, 18 Jul 2024 00:00:00 +0000

In my use of Google Cloud Platform (GCP), I often recreate virtual machines (VMs), requiring frequent updates to connection information. This process becomes cumbersome, especially when dealing with stale entries in the known_hosts file and managing DNS entries for a growing number of servers. Additionally, using the Identity-Aware Proxy (IAP), which requires the gcloud CLI for secure access, adds another layer of complexity. What I ended up doing was to leverage my previous experience with SSH configuration files to streamline my workflow.

The Challenge#

There were several issues I needed to solve:

Frequent VM recreation leading to connection information changes
Managing stale known_hosts entries
DNS management for numerous servers
The necessity of using IAP for secure access
The complexity of the gcloud compute ssh command

The Solution#

To implement my solution, I needed two key tools:

gcloud CLI installed locally
nc (netcat) installed on the remote servers to proxy the connection from IAP to VM

The gcloud compute ssh is a wrapper around ssh that helps with authentication, and IP resolution. This command also got a bit unwieldy when attempting to combine it with other ssh based tools such as ansible, or rsync, as if I wanted to rsync between my local machine, and a remote server, I had to pass the command into rsync, and the same for other ssh-based tools. As I previously worked with Vault to adjust the ssh configuration to execute a command per connection I figured I could use the same trick here. I found a helpful stating point on StackOverflow and was able to adapt/extend it for my needs.

To maintain security without opening SSH ports to the public or assigning public IPs to each VM, I utilized GCP's Identity-Aware Proxy for tunneling. The gcloud cli makes this easy by providing the --tunnel-through-iap flag.

The Configuration#

The final addition to my SSH configuration was the following:

Host *.gcloud
    ProxyCommand bash -c 'IFS=. read -r server zone project _ <<< "${1}"; gcloud compute --project "$project" ssh --zone "$zone" --ssh-key-file ~/.ssh/gcp_key --tunnel-through-iap "${2}@$server" --command="nc 0.0.0.0 22"' _ %h %r
    IdentityFile ~/.ssh/gcp_key
    # GCP handles the known hosts, and hostkey checking separately, so we can ignore them here
    StrictHostKeyChecking no
    UserKnownHostsFile=/dev/null

This configuration takes a hostname I provide (e.g., server1.us-west1-b.project1-id.gcloud) and converts it into variables that are then passed to the gcloud CLI.

I also wanted to forward some ports from inside GCP's network to my local machine. So to add customizations for specific host, you can define a more specific host (without the wildcard), and add the options. SSH will then also use the settings defined in the wildcard block.

Host server1.us-west1-b.project1-id.gcloud
    LocalForward 3306 10.20.30.40:3306

# Host *.gcloud...

Troubleshooting#

If you are following along and encounter issues with this setup, ensure that you've added the IAP IP addresses to your GCP network ingress rules. You can find the list of IP addresses in the GCP documentation. Also, you may need to sign into the gcloud cli, if you haven't already.

Conclusion#

This setup has saved me a lot of time. I could manage DNS entries when creating the servers with infra-as-code, but this approach eliminates extra configuration which presents another point of failure, and lets me add extra security by not opening up any ports to the world.

Use Drone plugins in Gitea Actions

Sat, 29 Jun 2024 00:00:00 +0000

Drone.IO is a CI/CD platform that uses Docker Containers for each pipeline step. This makes it easy to use existing Docker containers as steps in your pipeline. You can go futher and create special containers that are designed to be used as steps in your pipeline that accept arguments from the pipeline to run pre-defined tasks.

These plugins can be re-used in Gitea Actions to be able to run the same tasks in Gitea Actions as you would in Drone.

To give an example, a Drone plugin I used was drone-scp, and what it does is to allow me to copy files from the build pipeline to a remote server. I had a step in my Drone pipeline that looked like this:

- name: scp files
  image: appleboy/drone-scp
  settings:
    host: example1.com
    username: ubuntu
    password:
      from_secret: ssh_password
    target: /home/deploy/web
    source:
      - release/*.tar.gz

The workflow step will call a pre-defined binary in the drone-scp Docker image that will copy the files from the release directory to the target directory on the remote server without the need to have that script defined in full in the pipeline.

As this is published as a Docker image, it can be used in Gitea Actions as well. There are two approaches: a quick approach that works for most plugins, and a more involved approach that wraps the plugin as a full Gitea Action.

The simple approach: use the Docker image directly#

The easiest way to use a Drone plugin in Gitea Actions is to reference the Docker image directly with uses: docker:// and pass the plugin's expected PLUGIN_ environment variables via the step's env: block:

name: copy files
on: [push]
jobs:
  build:
    name: Build
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: docker://appleboy/drone-scp
      env:
        PLUGIN_HOST: ${{ secrets.HOST }}
        PLUGIN_USERNAME: ubuntu
        PLUGIN_PASSWORD: ${{ secrets.SSH_PASSWORD }}
        PLUGIN_TARGET: /home/deploy/web
        PLUGIN_SOURCE: release/*.tar.gz

This works because Drone plugins read their configuration from PLUGIN_-prefixed environment variables (derived from the settings: block in Drone), and step-level env: variables in Gitea Actions are passed directly into Docker containers. No repackaging is needed.

The full approach: wrap the plugin as a Gitea Action#

If you want to publish a reusable Action with defined inputs (so consumers use with: instead of env:), you can wrap the Drone plugin in a Dockerfile, entrypoint script, and action.yml. The Dockerfile should look something like this:

FROM appleboy/drone-scp

COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

ENTRYPOINT ["/entrypoint.sh"]

The entrypoint will be a short script that passes all the input arguments for the action to the drone-scp binary. The entrypoint.sh script should look something like this:

#!/bin/sh

set -eu

sh -c "/bin/drone-scp $*"

Note: When using the with: block in Gitea Actions (like GitHub Actions), inputs are passed to Docker containers as environment variables prefixed with INPUT_ (e.g. INPUT_HOST). Drone plugins natively expect the PLUGIN_ prefix (e.g. PLUGIN_HOST). Many popular Drone plugin authors (including Appleboy) have updated their binaries to accept both prefixes. If a plugin does not recognise INPUT_ variables, you may need to translate them in the entrypoint script by mapping each INPUT_* variable to its corresponding PLUGIN_* variable.

Finally, you'll need an action.yml file that lets Gitea Actions know what inputs should be used.

name: 'drone-scp Gitea Action'
# This tells the Gitea Action runner what type of Action this is
# in this case, it's "docker", and the image to use is the Dockerfile
runs:
  using: 'docker'
  image: 'Dockerfile'
inputs:
  host:
    description: 'scp remote host'
# ... define all the remaining inputs that exist for the Drone Plugin

Now, you can reference this action in any Gitea Actions workflow the same way that you would have used the Drone.IO plugin.

An example of this in action:

name: copy files
on: [push]
jobs:
  build:
    name: Build
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: https://gitea.com/techknowlogick/scp-gitea-action@main
      with:
        host: ${{ secrets.HOST }}
...

The author of the SCP Drone plugin, Appleboy, has done exactly this already, and you can use that action from his repo found at https://github.com/appleboy/scp-action/

This is a simple example, but you can convert any Drone plugin to a Gitea Action by following the same steps. This allows you to re-use existing plugins in Gitea Actions without having to re-write the entire plugin, which is especially useful as you can use the programming language of your choice, rather than relying on the ones supported by the runner natively.

Experimenting with Proxmox on Apple Silicon: Turning Your M1 MacBook Air into a Virtual Datacentre

Fri, 28 Jun 2024 00:00:00 +0000

In an effort to create an easy-to-manage VM environment that I can use on the go, even with spotty internet, I decided to try running Proxmox on my Apple Silicon MacBook Air. I could use Parallels or VMware Fusion, but I instead chose Proxmox to support open-source software. Should the need arise to run production-level workloads, I can always purchase a Proxmox license and deploy it on a dedicated server. The challenging part was that Proxmox doesn't distribute itself for arm64/aarch64, so I needed to find a way how to do that.

Getting Started#

Since Proxmox is an OS in and of itself (based on Debian Linux), and doesn't come packaged for arm64, I needed a way to get a VM for it up and running. Since it is based on Debian Linux, a Debian install could be converted to Proxmox, and there are many guides on how to do this for x86_64, so I decided to attempt this route.

UTM has a great interface for working with Apple's Hypervisor, and QEMU. Using QEMU directly was definitely an option, but UTM's user-friendly interface made the setup process much smoother and more intuitive for me. After installing UTM, I created a Debian VM using an image provided directly by UTM from their gallery of several pre-made VM images. This was quick and easy, as the presets were all configured so everything worked out of the box.

Converting Debian to Proxmox#

Now that I had a Debian VM, I needed to convert it to a Proxmox VM. Since Proxmox doesn't support Arm or distribute their packages for Arm I would have to build those myself. Luckily, some other kind soul already did this and published them.

Warning: I'm installing packages from an unofficial source. My risk level is fine with this, but if you are considering doing this, I strongly recommend you evaluate the choice carefully. If you are not comfortable, you can build the packages from source.

To use those pre-made packages I'll need to configure a few things with the VM first. First, you'll need to make sure your VMs hostname can be resolved using /etc/hosts. Since the VM already has debian as the hostname, I figured I'd keep that, and so I added 192.168.64.11 debian to the end of the hosts file. This is so that the VM can resolve its own hostname, and so that the Proxmox installer can resolve the hostname as well.

Update: These packages depended on OpenSSL 1.1, which is now removed from Debian, and so you'll need to build from source and link the packages to OpenSSL 3+

Now that the VM has the prerequisite configuration, we can get started with installing Proxmox.

# Add the 3rd party repository
echo "deb https://global.mirrors.apqa.cn/proxmox/debian/pve bullseye port">/etc/apt/sources.list.d/pveport.list

# Download/Trust the key that signs the 3rd party packages
curl https://global.mirrors.apqa.cn/proxmox/debian/pveport.gpg -o /etc/apt/trusted.gpg.d/pveport.gpg 

# pull the latest updates from the repositories, and update the system
apt update && apt full-upgrade

# install ifupdown2, which is required for Proxmox
apt install ifupdown2

# and finally, install Proxmox
apt install proxmox-ve postfix open-iscsi

After running through all these steps, I rebooted the VM, and then I was able to browse the Proxmox admin interface at https://YOUR_IP_HERE:8006

Running Proxmox#

Now, you are running a standard install of Proxmox and can do anything with it that you would on any other install. The only difference is that you need to ensure any container or VM you run uses an arm64/aarch64-based image.

Alternatives considered#

A non-exhaustive list of options considered, include Parallels and VMWare Fusion as mentioned above, but also just running Asahi Linux and running VMs directly on the host without having to use a hypervisor for Proxmox itself. Using a separate device was also an option, but this would require lugging around that device when I'm out and about. Keeping it as something on my laptop, without disrupting my day-to-day desktop environment, was most preferable for me.

Conclusion#

Running Proxmox on the M1 MacBook Air was surprisingly smooth, even with the base model specifications. The system handled the virtualization tasks well, with minimal noticeable lag. The user experience was better than what I would expect on an x86_64 system with similar specs. I also wasn't running any serious workload, but was able to run several VMs and containers without much of a performance hit.

Future enhancements could include using Terraform to automate environment setup or exploring integrations with the Proxmox API to script operations. For now, I'm happy with the setup and will continue to experiment with it.

Disclaimer#

Editor’s Note: This setup is not recommended for production use. The aarch64/arm64 packages I’m using are not officially supported or provided by Proxmox, and running production workloads on an M1 MacBook Air is not advisable, as it's not meant to be a server.

Bye-bye Docker Hub, time to self-host all of my Docker images

Wed, 26 Jun 2024 00:00:00 +0000

One of the great things about Gitea, is that it comes with a built-in Docker registry. This means that you don't need to be reliant on Docker Hub (or its rate-limits) to host your docker images, or any other OCI compliant image for that matter.

In this post, I'll go over how I migrated some of my docker images from the Docker Hub to my own self-hosted registry in Gitea. I won't go over the setup of the package registry in Gitea itself, as it's enabled by default, and I'd rather focus on the migration of images itself. The approach I took also didn't use any registry specific tools, and so it can be generalized to most other registries.

The MVP ("most valuable player") of this whole thing is a tool called skopeo. It is a command line tool that allows for many different operations on container images, and image registries. It also doesn't need to be run as root, or have an active docker daemon running.

While skopeo is a great tool, it can't fetch the entire list of images in a certain namespace, so I had to use some bash scripting to fetch the list of images, loop over them, and pass each of them to skopeo. Some caveats to be aware of, this script uses skopeo sync instead of skopeo copy, and that means that sync will copy all tags at once. The reason for choosing sync, rather than copy, is that it can handle all of the tags at once, instead of having to do another API to fetch the list of tags.

#!/bin/bash

## Variables to change
# TODO: these shouldn't be hardcoded, but rather passed in as arguments or via env vars
#       but that's a future problem

# Docker Hub credentials
HUB_NAMESPACE="<namespace>" # the user/org namespace on Docker Hub that you want to bring over
HUB_USERNAME="<username>"
HUB_PASSWORD="<password>"

GITEA_DOMAIN="<domain>" # e.g. gitea.example.com
GITEA_PACKAGE_NAMESPACE="<namespace>" # the user/org that you wish to store the images under
GITEA_USERNAME="<username>" # user that has access to the package registry namespace
GITEA_TOKEN="<password>" # token needs the packages:write permission

# Function to check if a command exists
command_exists() {
    command -v "$1" >/dev/null 2>&1
}

# Check if jq and skopeo are installed
if ! command_exists jq || ! command_exists skopeo; then
    echo "either jq or skopeo is missing, please make sure they are both installed" >&2
    exit 1
fi

# login into Docker Hub and fetch an API token
# its possible to do this without the token, but you'll likely run into ratelimits, and it would also only provide public images
TOKEN=$(curl -s -H "Content-Type: application/json" -X POST -d '{"username": "'${HUB_USERNAME}'", "password": "'${HUB_PASSWORD}'"}' https://hub.docker.com/v2/users/login/ | jq -r .token)

# Get list 100 of images that the namespace has, if you have more than that then pagination will need to be added
REPOS=$(curl -s -H "Authorization: JWT ${TOKEN}" "https://hub.docker.com/v2/repositories/${HUB_NAMESPACE}/?page_size=100" | jq -r '.results[].name')
# TODO: The fetching of images above is Docker Hub API specific, and so if your source is non-Docker Hub this will need to be adjusted

# Loop through images and pass them to skopeo
for repo in $REPOS; do
    echo "Syncing ${HUB_NAMESPACE}/${repo} to ${GITEA_DOMAIN}/${GITEA_PACKAGE_NAMESPACE}/${repo}"
    skopeo sync \
        --src docker --src-creds ${HUB_USERNAME}:${HUB_PASSWORD} \
        --dest docker --dest-creds ${GITEA_USERNAME}:${GITEA_TOKEN} \
        "docker.io/${HUB_NAMESPACE}/${repo}" \
        "${GITEA_DOMAIN}/${GITEA_PACKAGE_NAMESPACE}"
done

echo "Finished moving images!"

You can also use this script to copy images from other namespaces, so if you want to have a copy of another namespace, say for example Bitnami, you could use this script to do that as well.

This script has helped me move over a lot of images, and it's been working great for me. It's missing a few things, like passing in the options as arguments, error handling, or pagination for the Docker Hub API, but I didn't have more than 100 images in my namespace, so I didn't need to worry about that. If you do use this script, and modify it, I'd love to hear about it.

If you want to read more about the skopeo sync options, you can check out the skopeo sync documentation.

Disclaimer: I am a maintainer of Gitea. You can use this with other registries as well, so you don't need to use Gitea to use this script.

Let Wireguard clients use your IP Blocks advertised by BGP

Sun, 16 Jun 2024 00:00:00 +0000

As an addendum to my previous post on giving yourself an IPv6 address using wireguard, I wanted to show how, if you are lucky enough to have a IPv4 range before the "great IPv4 exhaustion", you can assign those addresses to clients using wireguard.

Several consumer, prosumer, and enterprise cloud hosts, allow you to advertise your IP range for your cloud servers. Some of these providers are OVH, Vultr, and Equinix Metal, but like last time, I suggest you look into OVH due to their predictable costs around bandwidth, as well you don't need to configure a BGP connection with them which makes setup much less work.

Note: If you followed part 1 and added a MASQUERADE rule for your network interface, you will need to remove it when assigning real public IPs from your own range. The MASQUERADE rule rewrites the source IP of outgoing packets to the server's address, which defeats the purpose of giving clients their own public IP. Remove it with iptables -t nat -D POSTROUTING -o <YOUR_NETWORK_INTERFACE> -j MASQUERADE (and the equivalent ip6tables rule if applicable).

Assuming you setup your wireguard connections as described in the previous post, there are a few adjustments that you'll need to make to give your clients an address from your range. For the sake of simplicity, I'll assume your IPv4 range is 8.8.8.0/24, even though this specific range is used by Google's public DNS service.

# server adjustments to wg0.conf
[Peer]
# change each peer to have a specific "allowed IP" from your block of addresses
AllowedIPs = 8.8.8.1/32  # Let's this specific client use this address publicly, note: if you are using IPv6 you'd give them one here too
#...

and for your client, you'd have their Interface configuration match the address defined on the server:

# client adjustments to wg0.conf
[Interface]
Address = 8.8.8.1/32 # must match the "AllowedIPs" configuration on the server for this peer
# ...

You'll need to ensure that your server can indeed receive/send traffic for these ranges. Depending on if you've setup a BGP connection with your provider, or are using OVH where no BGP connection is required, the way to ensure this different based on your situation. A quick way to do this on OVH would be to create virtual interfaces of eth0/eno1 for each address.

Editors Note: If you don't have your own IPv4 range, you can do this with IPv6 which is much easier to get, and if you are an EU citizen I can assist you with this process (send an email to hello at 'this domain without the text part')

Giving yourself IPv6 with Wireguard

Sat, 15 Jun 2024 00:00:00 +0000

After many years of asking, my Internet Service Provider (ISP) still doesn't support IPv6; the same goes for many other ISPs. This is a problem for many reasons, the first being the exhaustion of IPv4 addresses, leading some ISPs to implement "Carrier-Grade NAT" (CGN), where many customers share the same address—the second being that it is getting more and more expensive for small websites to have their own dedicated IPv4 addresses, including cloud providers instituting a cost to have an address. The obvious solution would be for all ISPs to start supporting IPv6, but sadly, this requires effort on the part of ISPs, and the problem isn't known to many of their customers, so the easy path for them is to keep status-quo.

What I have done to work around this is to rent a server in a datacentre that has an IPv6 address, and to tunnel my traffic through it using Wireguard. This is a moderately involved solution, and does come with a cost, but by showing websites that IPv6 traffic is increasing, perhaps it'll show that IPv6 is important to implement.

You can do this with any server that has an IPv6 address, but I'll be using one from OVH, because they don't have bandwidth costs, and they have a datacentre located near me to reduce any additional latency that may be introduced from tunnelling my traffic. Other providers, such as DigitalOcean, Vultr, Linode, or others could also work, but beware around bandwidth costs.

The way I started was to install Wireguard on both my laptop, and server. Assuming you are using Debian, or Ubuntu, you can run apt install wireguard on both your laptop and server to install all the software you need.

Next, I had to create a public and private key for both my laptop, and server. I did this by using the wireguard tools we just installed.

mkdir ~/.wg
cd ~/.wg
wg genkey | tee privatekey | wg pubkey > publickey

This will create a private key, and a public key in the ~/.wg directory. Now, you'll need to configure the tunnel on the client and server.

# server config
# /etc/wireguard/wg0.conf
[Interface]
Address = 10.0.0.1/24  # Private IPv4 subnet for the VPN
Address = fd42:42:42::1/64  # Private IPv6 subnet for the VPN
PrivateKey = <private_key_from_server>
ListenPort = 51820

[Peer] # your laptop
PublicKey = <public_key_from_laptop>
AllowedIPs = 10.0.0.2/32, fd42:42:42::2/128

# ... you can add many other peers to this file, just make sure to increment their allowed IPs

# client config
# /etc/wireguard/wg0.conf
[Interface]
PrivateKey = <private_key_from_laptop>
Address = 10.0.0.2/24, fd42:42:42::2/64 # addresses that matches "AllowedIPs" from server config
DNS = 1.1.1.1, 2606:4700:4700::1111 # your choice of DNS server, these are cloudflare's but you could choose your own

[Peer]
PublicKey = <public_key_from_server>
AllowedIPs = 0.0.0.0/0, ::/0 # this says to send all traffic over the tunnel
Endpoint = <server_public_ip>:51820

Now, you need to make sure your server can route your IPv6 traffic for you. This is done with some iptables configurations, and other system settings.

# as root run the following commands
# the next three commands will let your system packet forward for IPv4 and IPv6
echo 'net.ipv4.ip_forward = 1' | tee -a /etc/sysctl.conf
echo 'net.ipv6.conf.all.forwarding = 1' | tee -a /etc/sysctl.conf
sysctl -p # this will immediately apply the configuration additions

# now we need to let iptables to route the traffic appropriately
# on my machine, my network interface is eno1, but it may be eth0 for you or something else, you can find this using: ip addr
iptables -A FORWARD -i wg0 -j ACCEPT
iptables -A FORWARD -o wg0 -j ACCEPT
iptables -t nat -A POSTROUTING -o <YOUR_NETWORK_INTERFACE> -j MASQUERADE
ip6tables -A FORWARD -i wg0 -j ACCEPT
ip6tables -A FORWARD -o wg0 -j ACCEPT
ip6tables -t nat -A POSTROUTING -o <YOUR_NETWORK_INTERFACE> -j MASQUERADE

# ensure that your server has the wireguard ports open
iptables -A INPUT -p udp --dport 51820 -j ACCEPT
ip6tables -A INPUT -p udp --dport 51820 -j ACCEPT

# don't allow outside traffic to connect back to your machine that you didn't initiate
# this is to prevent your machine becoming world accessible
# if you plan on running a public service, such as a webserver on your laptop, you may need
# to adjust these
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -j DROP
ip6tables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
ip6tables -A INPUT -j DROP

Warning: The INPUT DROP rules above will drop all traffic that is not already established or explicitly allowed. If you are connected to this server over SSH, make sure you add a rule to allow SSH traffic before adding the DROP rules (e.g. iptables -A INPUT -p tcp --dport 22 -j ACCEPT and the equivalent ip6tables rule). Otherwise, you will lock yourself out.

Note: These iptables rules are not persistent across reboots. To make them survive a reboot, you can either use the iptables-persistent package (apt install iptables-persistent and then netfilter-persistent save), or use WireGuard's PostUp and PostDown directives in wg0.conf to apply and remove the rules when the tunnel starts and stops.

Now, both your laptop and server are ready to go. To start Wireguard on your server, run wg-quick up wg0 on your server. To start Wireguard on your laptop, run wg-quick up wg0 on your laptop. You can check that it's working by running wg show on both your laptop and server.

Editors Note: Instead of using wireguard for this, you could use something like Tailscale and setup/configure an exit-node, but this is left as an exercise for the user.

Instead of Docker-in-Docker, what about Docker-on-Fly?

Fri, 07 Jun 2024 00:00:00 +0000

In my ever-ending quest to do things ~~that are probably not the best idea~~ that aren't a part of a usual use-case, I wanted to see if I could run a docker daemon on Fly.io.

Even though you can think of Fly.io as a "container"-service hosting platform, it's actually using Firecracker micro-VMs to run your service. They transmogrify Docker images into the micro-VMs they run. This means, it's much easier to run a docker daemon, as you don't need to worry about the implications of running the docker daemon inside of a docker container.

After creating a whole Docker image with the docker daemon inside of it, and getting it working on Fly, I found that Fly already made one, and it's much more succinct than mine. So this post will reference their implementation instead, as I suspect mine will never see the light of day.

# first clone the repo
git clone https://github.com/fly-apps/docker-daemon.git docker-on-fly
# then enter the directory
cd docker-on-fly
# --build-only because we need to create a volume
fly launch --build-only
# creating a volume with 50gb of storage
fly volumes create data --size 50 --region ams
# run the service
fly deploy
# ssh into the machine (this example runs a container)
fly ssh console -C "docker run --rm hello-world"

You can now do some interesting things on there such as running entire services with docker compose. So instead of trying to map an existing service into a fly.toml file, you could just use an existing docker-compose.yml.

Next steps could include establishing a wireguard tunnel to your service and having the docker client on your local machine connect to the daemon over the tunnel.

Adding SFTP abilities to Namespace.so's ephemeral instances

Thu, 06 Jun 2024 00:00:00 +0000

Update (February 2026): Namespace has since added native file upload support to their CLI. You can now use nsc instance upload to copy files directly to an instance, making the shell scripts in this post unnecessary. This post is kept for historical purposes.

Namespace.so is a service that provides ephemeral instances (Linux, and macOS) for you to use, primarily used for CI/CD workflows. Similar to Fly.io, it's another way to not have to manage servers.

I'm currently using it to run some of my CI/CD workflows, but I want to play around with it in other ways. One, would to potentially make it a run-on-demand nixpkgs builder, similar to how I have with Fly.io, but with Fly.io I need to manage the instances myself, whereas Namespace will terminate instances after a set period of time. This reduces the monthly hosting spend.

Before I start, I don't begrudge Namespace at all for the (current, as of date of post) inability to SFTP files directly to instances, they are working hard on a lot of things, and this was a fun little thing I wanted to do. Namespace is constantly adding/building new functionality, and I suspect this will eventually be something they add, but I wanted something to play around with now. I happily pay Namespace for their service, and am very excited to see what they add next.

I asked in the Namespace discord, if anyone had any way to do this already, and I was pointed to croc as something others have used before, but I wanted something that I didn't need to run on the remote server in addition to locally. I'm sure there are a thousand other ways to do this, but I like to make fun little tools for myself in bash, and so this was a great excuse to play around.

Unlike with Fly.io, Namespace doesn't support direct SSH access, so I needed to come up with a way to "SFTP" the files up to the server. These files could be configuration, binary tools, and other files needed to run whatever is needed to run what I am working on at that time.

The nsc client allows for pseudo-shell sessions, and also running arbitrary commands, so I thought I could use that to my advantage. I originally did a naive way of copying files with: nsc ssh $machine_id 'echo "hi" > /root/test.txt', but that wouldn't work well with binary files, or large text files. Due to terminal length limitations, I also had to chunk the files up, and then reassemble them on the remote machine. This means that the transfer will take longer than usual, as each chunk will need to re-establish a connection with the server. A way to speed this transfer up would be to parallelize the transfer, but that's not in scope for right now.

#!/bin/bash

# help text
usage() {
    echo "Usage: $0 -l <local_file> -r <remote_file> -m <machine_id> [-c <chunk_size>]"
    echo ""
    echo "This Script uses the Namespace Client 'nsc' to copy files to your instance"
    echo "Note: You'll need to ensure you've logged in with 'nsc login' first"
    echo ""
    echo "Options:"
    echo "  -l <local_file>   Path to the local file to be transferred"
    echo "  -r <remote_file>  Path to the remote file to be created"
    echo "  -m <machine_id>   Machine ID for the nsc ssh command"
    echo "  -c <chunk_size>   Size of the chunks for splitting the base64 encoded file (default: 1k)"
    echo "  -h, --help        Show this help message and exit"
    exit 1
}

chunk_size="1k" # default chunk size

# parse args
while getopts ":l:r:m:c:h" opt; do
    case ${opt} in
        l )
            local_file=$OPTARG
            ;;
        r )
            remote_file=$OPTARG
            ;;
        m )
            machine_id=$OPTARG
            ;;
        c )
            chunk_size=$OPTARG
            ;;
        h )
            usage
            ;;
        \? )
            usage
            ;;
    esac
done

# check for (full) help flag
for arg in "$@"; do
    if [ "$arg" == "--help" ]; then
        usage
    fi
done

# make sure all args are set
if [ -z "${local_file}" ] || [ -z "${remote_file}" ] || [ -z "${machine_id}" ]; then
    usage
fi

#  base64 encode file and split into chunks
cat $local_file | base64 > /tmp/local_file.b64
split -l $chunk_size /tmp/local_file.b64 /tmp/chunk_

# init remote file
nsc ssh $machine_id "echo -n '' > /tmp/remote_file.b64"

# FIXME: transfer chunks individually, then reassemble them on server (this would allow for parallelization)
# loop over chunks and send each one
for chunk in /tmp/chunk_*; do
    chunk_content=$(cat $chunk)
    nsc ssh $machine_id "echo -n '$chunk_content' >> /tmp/remote_file.b64"
done

# decode base64 file
nsc ssh $machine_id "base64 -d /tmp/remote_file.b64 > $remote_file"

# clean up local temp files
rm /tmp/local_file.b64 /tmp/chunk_*

echo "File transferred successfully."

Then you'd run it like:

# ensure you are logged into namespace.so
nsc login

# create a new ephemeral instance (4 cores, 8gb ram)
machine_id=$(nsc create --machine_type 4x8 --bare --output json | jq -r .cluster_id)

# copy file up to server (assuming you have chmod +x the script already)
./transfer_file.sh -l /home/tklk/Photos/nyan_cat.gif -r /root/nyan.gif -m $machine_id

A way I could've avoided this using this hacky bash script would be to perhaps install tailscale in the ephemeral instance, and use tailscale file cp to copy the files up, but that's something for another day.

Update:

I had some extra time after writing this post, and went back to add parallelization to the script. I used gnu's parallel to send the chunks to the remote machine. This sped up the transfer time significantly. You'll need to ensure you have parallel installed on your local machine to use these adjustments as it's not installed by default on several common OSs. Here's an updated section that you can replace parts of the script above:

nsc ssh $machine_id "mkdir -p /tmp/chunks && echo -n '' > /tmp/remote_file.b64"
send_chunk() {
    chunk=$1
    chunk_name=$(basename $chunk)
    chunk_content=$(cat $chunk)
    nsc ssh $machine_id "echo -n '$chunk_content' > /tmp/chunks/$chunk_name"
}
export -f send_chunk
export machine_id

# use gnu parallel to transfer files
# FIXME: accept -j as an argument to be able to adjust the hardcoded number
#        left as an exercise for the reader
find /tmp/chunk_* | parallel -j 4 send_chunk

# reassemble chunks
nsc ssh $machine_id "cat /tmp/chunks/* > /tmp/remote_file.b64 && rm -r /tmp/chunks"

# based64 decode and write to destination path
nsc ssh $machine_id "base64 -d /tmp/remote_file.b64 > $remote_file && rm /tmp/remote_file.b64"

Build your own CDN - Part 3: Allowing bring-your-own domains with automatic TLS certificates

Wed, 05 Jun 2024 00:00:00 +0000

Now that we are able to synch TLS certs and the Caddy configuration across various points-of-presence, we now need to ensure that Caddy will only request certificates for approved names. As we are doing this for Gitea Pages, we won't know what the domain names are in advance and will have to create a validation service to handle this. This approach can also be abstracted to any SaaS that provides the ability for customers to bring their own domain.

Caddy has a built-in way to request if a certain domain name is approved or not. The way this process works, is that in the Caddy configuration you'll define an endpoint that will receive a request for a domain name, and run against your custom validation rules which could be a static list, a database call, or something else. If the name is approved, then the endpoint will return a 2xx status code, otherwise it will deny the request with a non-2xx status code.

The benefits of ensuring that domains are processed against this validation service include not requesting certs for domains outside of your control which could use up your Let's Encrypt quota, and it also prevents malicious actors from pointing their names at your infrastructure and receiving valid TLS certs.

Following the previous tutorial where the Caddy configuration is dynamically generated, the customers domain will be added to the config file, but we don't want users to be able to add any domain of their choice, we want to ensure that they have control over the domain, and that it also doesn't point to a nefarious location.

Using the following configuration as an explanation, we can see how this would work:

{
    on_demand_tls {
        ask https://validation.service.tld/endpoint
    }
}

example.com {
    tls {
        on_demand
    }
    respond "Hello World!"
}

In this example, we are telling Caddy to ask the validation service at https://validation.service.tld/endpoint if the domain name is approved for requesting TLS certificates. If the domain name is approved, then Caddy will request a certificate for the domain name, and serve the content as defined in the configuration, otherwise no certificates will be requested, protecting your Let's Encrypt rate-limit.

The validation service should be as fast as possible, to limit the time it takes for Caddy to request the certificate and respond to the initial request to that domain. The way Caddy will interact with that validation URL would be to make a GET request with a query string of ?domain=example.com where example.com is the domain name that the user is requesting a certificate for. If the validation service returns a 2xx status code, then Caddy will request the certificate, otherwise any other status code will prevent Caddy from making a request for a certificate.

The validation service could be as simple as a Go application that reads a file of approved domain names, or as complex as a database lookup. The important part is that it responds quickly and accurately to the domain name requests. Ideally, if you are running external checks on the domain, such as ensuring that it points to your loadbalancer or that it doesn't attempt to use a nefarious domain, you should perform those checks asynchronously and cache the result in a database, rather than making those checks every time a request to the validation service is made.

I will leave the completion of the validation service up to the reader, as there are many possible ways to implement this, and even many possible use-cases. There is no requirement for which language is used, or how the validation server be setup. You could use a standard http server that responds 200 to every request, to an entire monolithic application that integrates with your billing system, and everything in between.

In this series, we looked at various different topics, and how I used them as building blocks for a DIY CDN, however these topics can each be used independently of each other, and in different ways. I hope you've enjoyed this series, and I look forward to writing more in the future. If I find that something else in building this CDN interesting, I'll be sure to add yet another post in this series.

Build your own CDN - Part 2: Using Nomad to manage Caddy

Fri, 17 May 2024 00:00:00 +0000

In the previous post we discussed how to provision, and keep TLS certs in synch across multiple Caddy instances. In this post we will look at how to dynamically create and distribute Caddy configuration to multiple nodes using Hashicorp's Nomad.

Similar to other orchestrations tools such as Kubernetes, Nomad allows you to schedule and run containers, and non-containerized applications, across many servers with a standardized configuration.

Using nomad, we can manage multiple servers, in many datacentres, to create those Points of Presence (PoPs) for our CDN we discussed in our previous post. We will create configuration that will run Caddy on each edge node, and generate configuration dynamically to act as a reverse proxy based on the applications that nomad is also running.

First, we need to create the nomad job specification that defines the Caddy service we want to run. This will look something like:

job "caddy" {
  # as you with a CDN, you'll want to have datacentres close to where your data is being browsed
  # you can define as many PoPs here as you wish
  datacenters = ["dc1", "dc2", "dc3"]
  constraint {
    # this constraint will ensure that the job will only be run on nodes that have
    # a certain hostname, in this case only those that are loadbalancers
    attribute = "${attr.unique.hostname}"
    value     = "edge-lb-.+"
    operator  = "regexp"
  }
  group "loadbalancer" {
    # by default count is 1, meaning only one container will be created
    # however you can use nomad's autoscaler to dynamically change this
    # otherwise if you know the number of nodes in advance, and they will
    # remain fixed, you can hardcode as you wish
    # count = 1
    constraint {
      # ensure that a Caddy servers in the group are scheduled on separate physical hosts
      operator  = "distinct_hosts"
    }
    task "server" {
      driver = "docker"
      config {
        # we'll use the official caddy image, but if you want to leverage the S3 cert sharing from
        # the previous post, you'll need to build your own image with the caddy plugin included
        image = "caddy:2"
        # to allow Caddy to bind directly to any port for the host, instead of using docker port forwarding
        # you can give caddy access to the host namespace
        network_mode = "host"
      }
    }
  }
}

This job specification will create a Caddy instance on each node that has a hostname that matches edge-lb-.+. This will allow you to have multiple loadbalancers in each datacentre, and have Caddy running on each of them. This example will serve the default Caddy landing page, and not forward any requests to your applications or request TLS certificates.

To go beyond serving the default Caddy landing page, and start serving TLS certificates, we'll need to extend the nomad job, and provide a template for the Caddy configuration file.

The way that template files are loaded into a nomad job, is by mounting them into the container as a file. We can do this with:

...
task "server" {
config {
    ...
    # mount in the generated caddy configuration as a ready only volume
    # the path used, is the path to the generated files from the below template stanzas
    mount {
        type     = "bind"
        source   = "..${NOMAD_ALLOC_DIR}/../server/caddy"
        target   = "/etc/caddy"
        readonly = true
    }
}

template {
        data = <<EOH
# caddy configuration goes here
# this is hardcoded configuration that responds with the hostname of the node that is handling the response
:80 {
  respond "Hello World from {{ env "node.unique.id" }}!"
}
        EOH
        # where to write out the configuration
        destination = "caddy/Caddyfile"
    }
}

As you can see, the templates can take variables, and generate configuration based on them. We can extend this behaviour to have nomad generate configuration based on endpoints of services it is running:

... Caddyfile template

{{range service "nomad-example-app"}}
{{index .ServiceMeta "domain"}} {
  tls {
    on_demand
  }
  header {
    X-Balance "{{ env "node.unique.id" }}"
  }
  reverse_proxy {{.NodeAddress}}:{{.Port}}
}
{{end}}

This will take the list of services from Nomad with the name "nomad-example-app", loop over them using the domain meta information, and have Caddy proxy traffic to the address, and port of each installation. Right now, the connection will happen over whichever network the docker bridge is attached to, and so please be aware that if your IPs are publically accessible, that bypass of the Caddy server may be possible based on network firewall rules.

Minor updates to your application service will be needed. In your application Nomad specification, you'll need to add a port that will be exposed for the container to listen on. In our case we are exposing port 80 of the container, and nomad will publish that to a random high port on the host. In your service meta information, you'll need to define a domain so that nomad will be able to create a configuration block in the generated Caddy configuration.

group "nomad-example-app" {

    network {
      port "http"  { to = 80 }
      mode = "bridge"
    }
    service {
      name = "nomad-example-app"
      port = "http"
      meta {
        domain = "nomad-app.example.com"
      }
    }
    task "server" {
      driver = "docker"

      config {
        image = "traefik/whoami:latest"
        ports = ["http"]
...

Now as new services come online, or old ones go away, nomad will update the caddy configuration as needed.

Nomad will restart the Caddy container as the configuration changes, but as this is possibly undesirable for a production load balancer, in the future you may want to look at using Caddy's automatic reloading capabilities instead. This can be done by setting change_mode = "signal" and change_signal = "SIGUSR1" on the template stanza, which will send a signal to the Caddy process when the configuration changes. Caddy handles SIGUSR1 by performing a graceful config reload, avoiding the downtime of a full container restart.

There are ways to expand on this further, by adding healthchecks, creating services dynamically using gitops or the nomad API, and integrating with a service mesh like Consul connect for mTLS on connection between Caddy, and the application. This covers the basics of using Nomad to dynamically generate and update Caddy configuration based on services defined by Nomad.

In the way I plan to use the above for Gitea.pages, is to have a container exist for each static site, and create nomad services for each site. The static site will have an instance created in each region, and the Caddy server will direct traffic to the container in its region. This allows for regional CDN like capabilities, while keeping the sites loosely coupled and using Nomad for orchestration. When new static sites are built, we can let nomad know a new container is published, and then nomad will handle updating the services globally.

In the next post we will look into how to use Caddy to safely request certificates for custom customer-provided domains on-demand, and how to validate those requests.

Using Fly.io as a remote builder for nixpkgs

Mon, 13 May 2024 00:00:00 +0000

A service I enjoy using, and write about often, is fly.io, as it lets me have compute on demand, without having to worry about provisioning and managing an entire server.

Editors Note: This was a fun experiment, but you are likely better off using a managed service such as nixbuild.net for anything even resembling a production use case.

One of my latest use cases for it, is to offload building of nixpkgs from my low powered laptop, to a remote builder on fly where I can provision as many resources as I need, for as short of a period of time as needed.

To start off, we'll need to create the builder machine. You'll likely need more than the default 256mb of ram that is provisioned, but maybe not the 2gb that I request. You can adjust the number of cpus and memory as needed. Since fly's machines, are essentially containers (they do some neat things, and you can read about it in one of their many blog posts, but that is outside of scope of this post), you can also use the same container image that you would use with docker, in this case I am using nixos/nix.

fly launch --name tklk-nixbuilder --region ams -o personal --build-only --image nixos/nix --no-public-ips  --vm-cpus 4 --vm-memory 2048

We are not asking fly to start the machine right away, as we'll need to add some more configuration before powering it on.

First, we'll need to give the machine more storage space, as storing/building a package may build more of the package's dependency graph than just the package itself, and there will need to be some place to store it.

fly volumes create nix_store -a tklk-nixbuilder -s 50 -r ams -y

I'm setting 50gb, but since you are billed for space consumed regardless if the machine is powered on or not, you may wish to make this volume smaller based on your needs. You'll also need to set the path that the volume is mounted with in your fly.toml configuration file.

[mounts]
  source="nix_store"
  destination="/data/nix"

Interesting note about this configuration, even though nix stores most things in /nix, we are mounting to /data/nix initially, as otherwise if we mount it directly to /nix, it will be empty, and the machine won't have access to any binaries. So we will mount to a different directory, copy everything over to the new volume, and then set the path to the correct location.

Some internal Fly.io processes look for the /bin/sleep binary, however in NixOS, due to it being entirely reproduceable, it stores binaries with a specific path name with a cryptographic hash of the package’s build dependency graph. This means that the /bin/sleep binary is not available, so we'll either need to create a symlink to it, or we can set a path to the binary in the configuration.

[experimental]
  cmd = ['nix-shell', '-p', 'coreutils', '--run', 'sleep inf']

Your machine is now ready to be powered on. You can do this by running:

fly deploy -a tklk-nixbuilder

Your terminal will show you progress of the machine starting up, and in a short period of time it will be all ready, and you can now SSH into it to finish some final setup steps.

fly ssh console -a tklk-nixbuilder

First, copy the contents of the nix store into the mounted volume.

cp -a /nix/. /data/nix/

Back on your host machine, you can change the destination path of the mounted volume in your fly.toml configuration file to /nix and restart the machine with fly deploy -a tklk-nixbuilder

Your remote machine is now all ready to handle building of your remote nixpkgs.

You can now use the fly ssh issue command to generate an SSH key that can be used to SSH into your machine.

fly ssh issue -o personal -u root /home/tklk/.ssh/fly_key

For sake of convenience you can add some minor configuration to your ~/.ssh/config file to reduce CLI options when letting your nix-builder connect to the remote machine.

Host nixbuilder.fly
	StrictHostKeyChecking no
	UserKnownHostsFile=/dev/null
	Port 2200
	User root
	HostName localhost
	IdentityFile ~/.ssh/fly_key

Finally, you'll need to setup a proxy to forward port 2200 on your local machine to port 22 on your remote builder. This is to avoid having to configure a wireguard VPN connection to fly.io, which is a bit more involved than just adding a port forward. The & in the command below, forces the proxy into the background so you don't need to have a terminal dedicated to just the proxy command. Note: if you close the terminal then this command will quit and you'll need to start it again if you want to connect back.

fly proxy 2200:22 &

That's it! Your remote builder, and local machine are all setup for building.

You can test your remote builder by adding --store ssh-ng://nixbuilder.fly to any nix build command.

Example:

nix build nixpkgs#mercurial --store ssh-ng://nixbuilder.fly --show-trace

Some caveats are that the remote builder will be an x86-64 architecture, and if your local machine is aarch64 (aka Arm), you'll need to pass --system x86_64-linux to nix-build to ensure that the remote builder will build the package. Sadly this means that the package will be built for x86-64, and not aarch64, so you'll won't be able to copy it to your machine. As the editors note says, you are better off using something like nixbuild.net, as they support aarch64 builders (though not macOS ones as of the time of writing).

Continued reading:

Build your own CDN - Part 1: Synch TLS certificates across all your PoPs with Caddy

Tue, 31 Oct 2023 00:00:00 +0000

As a part of building a Content Delivery Network (CDN) for Gitea Pages, I'm documenting my process to share my experience with others. The reason I'm building one from scratch, rather than using a pre-built solution like Amazon CloudFront, is the potential need for many unique TLS certificates. From past experience, I've learned that there are limits to how many certificates you can have on a single CloudFront distribution. This is the first post in a series that will document the process of building a CDN from scratch.

The initial task I'll tackle is synchronizing TLS certificates across all the Points of Presence (PoPs) in the CDN. If you're unfamiliar with what PoPs are, they are geographically distributed servers located near where end users are to improve content delivery performance. I've chosen to use Caddy for this purpose. Caddy is a web server that has built-in support for Let's Encrypt and can automatically obtain and renew TLS certificates.

While there are alternative approaches using other web servers like nginx, where a central server obtains the TLS certificates and then distributes them to the other servers, Caddy offers a more decentralized approach. I can use a plugin I wrote called certmagic-s3, that lets each instance of Caddy share the TLS certificates via an S3 bucket. This approach has the advantage that any of the PoPs can obtain and renew the TLS certificates, and the other PoPs will automatically receive the updated certificates.

The most challenging part of this approach is to ensure that the plugin is properly installed in Caddy. You could use Caddy's xcaddy build tool, but Caddy also offers a build service where you can download binaries that have already been compiled. Using that build service, select certmagic-s3 as a plugin to include, download it for your platform of choice, and you'll have a Caddy binary with the plugin already installed.

Now, using that binary, you can create a Caddyfile to configure Caddy to use the plugin and obtain TLS certificates from Let's Encrypt. Here's an example Caddyfile:

{
        email webmaster@example.com  # The email associated with your Let's Encrypt account
        storage s3 {  # Configuring S3 as the storage backend
            host minio.example.com  # Your S3-compatible storage host
            bucket certmagic-s3  # Bucket where certificates will be stored
            access_key ABC123  # Your S3 access key
            secret_key XYZ789  # Your S3 secret key
            prefix "byoc"  # Optional path prefix within the bucket
        }
}
site.example.com {  # Domain to serve
    tls {
        on_demand  # Obtain TLS certificates on first HTTP request instead of on start
        # Note: you should also configure an `ask` endpoint in the global `on_demand_tls`
        # block to prevent abuse of on-demand certificate issuance. See Part 3 of this
        # series for details on setting up a validation service.
    }
    respond "hello world"  # Sample response
}

With the required information filled out from the configuration above, you can start Caddy and it will obtain the TLS certificates from Let's Encrypt and store them in the S3 bucket. If you then start another instance of Caddy with the same configuration, it will automatically obtain the TLS certificates from the S3 bucket and serve the site.

In upcoming posts I will describe how I setup nomad to distribute Caddy config to all the PoPs, and how I am using Caddy to serve custom dynamic domains for Gitea Pages. Stay tuned for more details on building out the CDN.

Using Nix with Gitea Actions

Sun, 03 Sep 2023 00:00:00 +0000

Carl Sagan once said, "If you wish to make an apple pie from scratch, you must first invent the universe." In the world of software, creating a reproducible build environment is the universe you need to invent. This post will walk you through using Nix in tandem with Gitea Actions to make that universe a reality for your projects.

I am an enthusiastic user of Nix and am a maintainer of several packages. I appreciate the reproducibility of the binaries it offers across different systems and its rapid update cycle.

Gitea Actions is a CI/CD solution that can run your build and deployment tasks. Using Nix within Gitea Actions is as straightforward as adding a few lines to your workflow file. Here's how:

name: nix

on:
  push:

jobs:
  lint-backend:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install dependencies for Nix setup action
        run: |
          apt update -y
          apt install sudo -y          
      - uses: cachix/install-nix-action@v27
        with:
          nix_path: nixpkgs=channel:nixos-unstable
      - name: Test running command with Nix
        run: nix-shell -p hello --run "hello"

This workflow will install Nix and then execute the hello command. Note that we need to install sudo, as it is a prerequisite for the cachix/install-nix-action and is not present in the default Gitea Actions runner image. If you're using a custom runner that already has sudo installed, feel free to skip that step.

Regarding the Nix package channel, I prefer to live on the bleeding edge with nixpkgs=channel:nixos-unstable. However, you're free to pin to a more stable channel if you wish. The cachix/install-nix-action Action does not have a channel configured by default, so you must specify one.

If you haven't explored Nix yet, I highly recommend you do so. It's a powerful tool for creating consistent and reproducible build environments.

Watchtower... but for Kubernetes!?!

Thu, 18 May 2023 00:00:00 +0000

Watchtower is an excellent tool for keeping your containers up to date. It's a process that runs on a schedule and checks for new versions of your containers, and if it finds one, it pulls the new image and recreates the container with the latest image. It's built for Docker, and it works great for Docker. But what about Kubernetes?

Keel, a Kubernetes operator, also achieves what Watchtower can do but can automate Helm, DaemonSet, StatefulSet & Deployment updates. It also has a friendly UI to see the status of the updates it is managing.

Installing Keel#

The first step to utilizing Keel is installing it in your Kubernetes cluster. You can use kubectl apply for this purpose:

export KEEL_NAMESPACE=keel
export KEEL_ADMIN_USER=keel
export KEEL_ADMIN_PASS=keel
kubectl apply -f https://sunstone.dev/keel?namespace=$KEEL_NAMESPACE\&username=$KEEL_ADMIN_USER\&password=$KEEL_ADMIN_PASS\&tag=latest

Configuring your Deployments for Keel#

Once you have installed Keel, you'll need to configure your deployments to use it. This is as simple as adding a few labels to your Kubernetes deployment specifications. Keel uses SemVer (Semantic Versioning), and its policies can be all, major, minor, or patch. For example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    keel.sh/policy: major
...

The above configuration means Keel will update the deployment whenever there's a new major version of the container image.

Handling private images and rate limits#

If you're using private images or Docker Hub with its strict rate limit, you'll need to configure Keel to authenticate with your registry. Keel also supports secrets for pulling images. Keel will use existing secrets that Kubernetes uses to pull the image so no additional configuration required.

Keel UI#

One of the unique features of Keel is its UI which allows you to see at a glance the status of your deployments and any updates it's managing. You can access it via a Kubernetes ingress or use kubectl port-forward:

kubectl -n keel port-forward service/keel 9300

Wrapping Up#

Keel is a powerful tool that brings the simplicity and automation of Watchtower to the Kubernetes ecosystem. Whether you have simple Deployments, use Helm, or have more complex DaemonSets or StatefulSets, Keel has you covered.

Remember, automating your image updates saves you time and ensures that you're running the latest and potentially more secure version of your containers. As always, it's essential to have robust rollback strategies and test pipelines in place, especially when using automatic updates.

Publishing a static site to Bunny.net using Gitea Actions

Tue, 18 Apr 2023 00:00:00 +0000

Bunny.net (formerly BunnyCDN) is a low-cost, high-performance CDN provider that can be used to host static sites. This post will walk through the steps to hosting a static site using it.

Creating a Storage Zone#

The first step is to create a storage zone. The storage zone is where the static site will be stored. To do this, log into the Bunny.net dashboard, click the Storage Zones tab, and follow the steps to create a new storage zone. The storage zone can be named anything, but it's best to name something that will allow you to identify it later and associate it with your site quickly. You can select the regions you would like your content replicated to, and the more regions you have, the faster your site will be in those regions. Be careful; the more regions you select, the more it will cost you.

Now that you have a storage zone, please navigate to the FTP credentials page, and have them ready for later.

Creating a Pull Zone#

The next step is to create a pull zone. The pull zone is what will be used to serve the static site. To do this, log into the Bunny.net dashboard, click the Pull Zones tab, and follow the steps to create a new pull zone. As with the storage zone, you should name it something memorable. You can also select the regions from which the CDN serves your data. The pull zone must be configured to use the storage zone you created earlier as the "origin". You can also enable a custom domain for your site. If you do, you will need to add a CNAME record to your DNS provider that points to the Bunny.net pull zone and add it to the pull zone in the Bunny.net dashboard.

Uploading the Site#

Now that you have a storage zone and a pull zone, you can upload your site to the storage zone. You can do this using the FTP credentials you created earlier. Here is an example of how to do this using Gitea Actions, but you can use any CI system you like.

# .gitea/workflows/hugo-build.yml
name: Build and Deploy to BunnyCDN

on:
  push:
    branches:
      - main

jobs:
  bunnycdn:
    name: bunnycdn-publish
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          submodules: true  # Fetch Hugo themes (true OR recursive)
          fetch-depth: 0    # Fetch all history for .GitInfo and .Lastmod

      - name: Setup Hugo
        uses: peaceiris/actions-hugo@v2
        with:
          hugo-version: '0.126.1'
          extended: true

      - name: Build
        run: hugo --minify

      - name: Deploy to BunnyCDN
        run: |
          apt update -y
          apt install -y lftp
          lftp -e "
            set ftp:ssl-allow true;
            set ftp:ssl-protect-data true;
            set ssl:verify-certificate no;
            open ${{ secrets.BUNNYCDN_FTP_HOST }};
            user ${{ secrets.BUNNYCDN_FTP_USER }} ${{ secrets.BUNNYCDN_FTP_PASSWORD }};
            mirror -R --delete -v public/ .;
            bye;
          "
      - name: purge bunny cache
        run: |
          curl --request POST \
            --url https://api.bunny.net/pullzone/${{ secrets.BUNNYCDN_ZONE_ID }}/purgeCache \
            --header 'AccessKey: ${{ secrets.BUNNYCDN_API_KEY }}' \
            --header 'content-type: application/json'

You'll need to set the BUNNYCDN_FTP_HOST and BUNNYCDN_FTP_USER secrets, they are the FTP host and user from the storage zone's FTP credentials page. The BUNNYCDN_FTP_PASSWORD is the password from the same page. The public/ directory is the directory that Hugo builds the site into. The / is the root of the storage zone. The -R flag tells lftp to mirror the directory recursively, and the -v flag tells lftp to be verbose. Finally, you'll also need to set the BUNNYCDN_ZONE_ID and the BUNNYCDN_API_KEY secrets with details you can get from the bunny.net dashboard, so that the last step in the workflow can clear the cache, and serve up the new blog posts.

Conclusion#

That's it! You should now have a static site hosted on Bunny.net. You can now use the pull zone's URL to access your site. If you enabled a custom domain, you can use that instead. You can also further configure the pull zone to enable caching, compression, and other features such as auto-https.

Notes: Bunny.net is rebuilding its dashboard, so the steps in this post may not match the current dashboard, although the process should remain largely the same.

Secure SSH Access with SSH Certificates Managed by HashiCorp's Vault

Sun, 16 Apr 2023 00:00:00 +0000

SSH certificates are an effective way to secure SSH server access. They can restrict users and the commands they can run, making them especially valuable for managing access to multiple servers. By using SSH certificates, server fingerprint validation becomes unnecessary since the certificates are signed by a Certificate Authority (CA) with the CA's public key installed on the server. Vault is an excellent tool for managing SSH certificates, offering functionalities like issuing and revoking certificates, managing SSH keys, and providing audit logs.

Warning: This post describes a non-production setup of Vault. As such, it is not hardened with appropriate security measures, and it is not recommended to use this setup in production. You should use this for learning purposes on SSH CAs and Vault.

Install Vault#

To quickly set up a development Vault server, use the official Docker image with the following command:

docker run --cap-add=IPC_LOCK -e 'VAULT_DEV_ROOT_TOKEN_ID=dev-token' -e 'VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200' -p 8200:8200 hashicorp/vault server -dev

Configure SSH Certificate Authority#

With Vault installed and running, configure it to issue SSH certificates using the SSH secrets engine.

First, set the Vault address and token for the dev server:

export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='dev-token'

Then, run the following command to enable the SSH secrets engine

vault secrets enable ssh

Next, you will need to configure the SSH CA and have it generate signing keys on your behalf to ensure that the private key stays within Vault itself.

vault write ssh/config/ca generate_signing_key=true

You can get the generated SSH public key for use later, with:

vault read -field=public_key ssh/config/ca > ssh_ca_key.pub

After enabling the SSH CA in Vault, you'll need to create a role called ops-team to issue SSH certificates. This role allows any user with access to request an SSH certificate. The example below grants broad permissions, including any option for allowed_users and port forwarding. Be sure to restrict these permissions based on your use case.

vault write ssh/roles/ops-team \
    key_type=ca \
    ttl=2h \
    max_ttl=24h \
    allow_user_certificates=true \
    allowed_users="*" \
    default_extensions='{"permit-pty":"","permit-port-forwarding":"","permit-agent-forwarding":""}'

Configure the remote server to accept the SSH certificates issued by Vault:

Copy the CA public key to the remote server:

scp ssh_ca_key.pub <username>@<target_server_ip>:/tmp/ssh_ca_key.pub

Add the public key to the OpenSSH configuration and restart the OpenSSH daemon:

echo "TrustedUserCAKeys /etc/ssh/user_ca.pub" | sudo tee -a /etc/ssh/sshd_config
sudo cp /tmp/ssh_ca_key.pub /etc/ssh/user_ca.pub
sudo systemctl restart sshd

Requesting SSH Certificates#

To request an SSH certificate from Vault and use it to SSH into the remote server, follow these steps:

Use the ops-team role to request the certificate and pass your local SSH key id_rsa.pub. Also, specify the username to use when connecting to the remote server:

vault write -field=signed_key ssh/sign/ops-team \
    public_key=@$HOME/.ssh/id_rsa.pub \
    valid_principals="<username>" > signed_id_rsa-cert.pub

Use the signed_id_rsa-cert.pub file to SSH into the remote server:

ssh -i signed_id_rsa-cert.pub -i $HOME/.ssh/id_rsa <username>@<target_server_ip>

Requesting a signed certificate manually each time can be tedious. To simplify this process, create a script called vault-ssh.sh and make it executable with chmod +x vault-ssh.sh.

#!/bin/bash

# Configuration
export VAULT_ADDR="http://<vault_server_ip>:8200"
VAULT_ROLE="ops-team"
USERNAME="<username>"
PUBLIC_KEY_PATH="$HOME/.ssh/id_rsa.pub"
CERT_PATH="$HOME/.ssh/id_rsa-cert.pub"
CONFIG_FILE="/path/to/vault-creds.conf"

# Read the Vault token from the configuration file
if [ -f "$CONFIG_FILE" ]; then
  source "$CONFIG_FILE"
else
  echo "Error: Vault configuration file not found"
  exit 1
fi

# Check if the VAULT_TOKEN variable is set
if [ -z "$VAULT_TOKEN" ]; then
  echo "Error: VAULT_TOKEN is not set in the configuration file"
  exit 1
fi
export VAULT_TOKEN

# Generate a new SSH certificate
vault write -field=signed_key ssh/sign/$VAULT_ROLE \
  public_key=@$PUBLIC_KEY_PATH \
  valid_principals="$USERNAME" > $CERT_PATH

This script requires a vault-creds.conf file containing the Vault token:

VAULT_TOKEN=<vault_token>

To integrate the certificate generation process with your SSH config, use the ProxyCommand configuration option, which allows you to run a custom command (like the script) as a "proxy" for the actual SSH connection.

Add the following to your SSH config:

Host *
  IdentityFile ~/.ssh/id_rsa
  CertificateFile ~/.ssh/id_rsa-cert.pub
  ProxyCommand bash -c '/path/to/vault-ssh.sh && nc %h %p'

Keep in mind that this approach generates a new SSH certificate for every connection. Depending on the frequency of your connections and the TTL of your certificates, you might want to modify the vault-ssh.sh script to check the current certificate's validity and generate a new one only if necessary.

Conclusion#

This post covered configuring Vault to issue SSH certificates, setting up a remote server to accept these certificates, and streamlining the process of requesting SSH certificates. Use the knowledge from this post to enhance your environment's security. Remember that this post describes a non-production setup of Vault and should be used for learning purposes only.

Credits: The above post was written from knowledge and experience of using Vault. The instructions for docker configuration of vault are from the official Vault docker documentation.

Playing around with Gitea Actions on Fly.io

Mon, 03 Apr 2023 00:00:00 +0000

Fly.io is a "serverless" hosting platform usually used to host web services. It can be used for more than just web services; it can also be used to run long-running tasks. I wanted to try a new way to run the Gitea Actions runner, and Fly.io would be interesting way to try out.

Treat this as a proof of concept, I'm unsure if this is a good idea, but it's fun to try out.

To simplify things, I will run the runner in "host" mode, meaning that each job won't be containerized but will run directly on the host. This was a decision made before getting started to limit the amount of debugging sorting out Docker in Docker.

To get started I created a new Fly.io app with the following configuration:

# fly.toml
app = "actions-on-fly"
primary_region = "ams"

[[mounts]]
  destination = "/data"
  source = "data"

I mounted a persistent volume to /data so that the runner can be registered and persist the registration token across restarts.

Since there are no prebuilt Docker images (as of the time of publishing) I created one and installed the runner in it. The Dockerfile is as follows:

# Dockerfile
FROM ghcr.io/catthehacker/ubuntu:act-latest
# the FROM image is based on ubuntu and has appropriate tools installed to run Gitea Actions

# install act_runner
RUN curl https://dl.gitea.com/act_runner/nightly/act_runner-nightly-linux-amd64 > /usr/local/bin/act_runner && \
    chmod +x /usr/local/bin/act_runner

# add start script
ADD start.sh /start.sh
RUN chmod +x /start.sh && mkdir -p /data
ENTRYPOINT ["/start.sh"]

When running the container, the startup logic will check if the runner is already registered, and if not it will register it. The registration token is passed in as an environment variable. The runner will then be started.

#!/bin/bash
# start.sh

# $ACTIONS_REGISTER_TOKEN is the registration token for the runner that is given by the Gitea runner settings page.

# set /data as the working dir
cd /data

# check if runner is already registered, and if not register it
if [ ! -f .runner ]; then
  # register runner on gitea.com, and set label as fly-runner so it runs as "host" mode
  act_runner register --no-interactive --instance "https://gitea.com" --labels "fly-runner" --token $ACTIONS_REGISTER_TOKEN
fi

# start runner
act_runner daemon

It really was a handful of lines to get a runner up and running. The runner is now running on Fly.io and can be used to run Gitea Actions. The only issue I ran into when setting this up, was that Fly.io will terminate apps if they run out of memory. This is a problem because what I was testing used a lot of memory. I ended up increasing the memory limit. Maybe a different hosting would handle OOMs differently, but I was pretty satisfied with the result. The blog you are reading right now is built using this runner.