TechKnowLogick

Vanity, thy name is Go

Tue, 10 Mar 2026 00:00:00 +0000

Go vanity imports let you put a custom domain in front of your import paths. Instead of go get gitea.com/user/repo/pkg (or, in my case, go get src.tklk.dev/tklk/x/pkg), the command becomes go get tklk.dev/pkg. The shorter path is decoupled from where the code is hosted.

After moving everything into a monorepo, this became trivial to set up. One repo means a single vanity domain covers all packages under it. On Bunny.net, the setup uses a custom 404 page stored as a static HTML file. Bunny serves it directly, so there is no server or edge compute to manage.

Go vanity import lookup#

When you run go get tklk.dev/somepkg, the Go toolchain makes an HTTP request to https://tklk.dev/somepkg?go-get=1 and parses the HTML response. It looks for a <meta name="go-import"> tag in the <head>, which tells it where the actual source code lives.

The meta tag has three space-separated values in its content attribute: the import prefix, the VCS type, and the repository URL.

<meta name="go-import" content="tklk.dev git https://src.tklk.dev/tklk/x">

For a monorepo, all subpaths under the import prefix resolve to the same repo root. tklk.dev/somepkg and tklk.dev/otherpkg both point at https://src.tklk.dev/tklk/x. The Go toolchain handles the rest, mapping the subpath to a directory within the repository.

A custom 404 page#

The Go toolchain mostly doesn't care about the HTTP status code. It fetches the URL and parses the HTML looking for the meta tag. A 200 and a 404 work equally well, as long as the response body contains what it needs.

A redirect to Gitea would've been simpler, but the Go toolchain reads the meta tag from the final response after following redirects. Gitea serves its own go-import meta tag with the repo path as the import prefix, so a redirect would make the toolchain see the Gitea hostname instead of the vanity domain. The 404 page sidesteps this because the response comes from the vanity domain itself.

Bunny.net lets you configure a custom error page for a pull zone. Set that 404 page to an HTML file containing the go-import meta tag, and every path under tklk.dev/anything that doesn't match a real file returns the custom error page. The Go toolchain reads the meta tag and resolves the import.

The full 404.html:

<!DOCTYPE html>
<html>
<head>
  <meta name="go-import" content="tklk.dev git https://src.tklk.dev/tklk/x">
</head>
<body>
  <p>Not found. See <a href="https://src.tklk.dev/tklk/x">source</a>.</p>
</body>
</html>

That's the whole file, and the meta tag is the only line Go cares about.

My pull zone also serves a few real files, like my Docker registry at /v2/. None of them conflict with package names, so the custom 404 only fires for paths that don't match anything real.

Setting it up#

If you're already using Bunny.net for static hosting, this is a small addition. I covered the basics of creating storage zones and pull zones in a previous post.

Create a storage zone and pull zone for the vanity domain (or reuse an existing one).
Upload the 404.html to the storage zone. Bunny expects custom error pages at bunnycdn_errors/404.html. You can use another path by changing the storage zone settings.
In the pull zone settings, set the storage zone as the origin.
Add a DNS CNAME record pointing tklk.dev to the pull zone hostname.

Testing it#

A quick curl confirms it's working:

curl -s 'https://tklk.dev/somepkg?go-get=1'

You should see the HTML with the meta tag come back, even though the response status is 404. From there, go get tklk.dev/somepkg resolves to the Gitea repo and pulls the code down.

The monorepo is what makes the whole setup this simple, and it costs nothing beyond what the CDN already costs. A domain fronting multiple independent repos would need a different meta tag for each import path, plus edge compute or a server to route requests.

I have a separate project, go-vanity-url, that handles that case. It generates static HTML files from a TOML config, one per package, with the correct go-import meta tag. You can host the output on any static file server or CDN. Here, one meta tag covers everything, so a static 404 page is enough.

Monorepo! Monorepo! Monorepo!

Wed, 04 Mar 2026 00:00:00 +0000

What's it called? Monorepo! That's right, monorepo!

I've been maintaining many small Go repositories for personal projects, libraries, and a bunch of experiments. The overhead that comes with all of that adds up. After recently switching to a monorepo, I'm a convert: so much toil immediately disappeared.

The inspiration#

Xe Iaso maintains a public monorepo, x, that contains many Go projects, experiments, and libraries. Sometimes full projects, such as Anubis, have even been spun out of it. The repo uses idiomatic Go project layouts with a go.mod at the root, plus repo-wide policies for things like linting, CI, and other similar items.

The problem#

When I have a new idea, I usually make a scratch folder for it, because setting up CI, linting, and dependency management for a one-off experiment feels like more work than it's worth. If the experiment turns out to be interesting and I want to keep it around, I make a repo for it and copy the code over. I'd rather not sign up to maintain something that might be a dead end.

The repos that did get created were inconsistent, and a "breaking" change in a shared library could mean updating every repo that imported it.

Dependency management was the worst of it. I use Renovate to keep things updated, and it does its job well. With several repos sharing dependencies, "doing its job" means a steady stream of PRs across all of them, even for repos I barely care about. Automatic merging handles most of the updates, but the emails and notifications still come in, and sometimes the tests fail. Then I have to decide whether to context switch and fix it or let it rot until I next need that repo and notice the breakage. It's a minor annoyance, but that still adds up as the repo count grows.

One repo to rule them all#

After looking at how Xe structured their x repo, I moved everything into a single repository. Most of those annoyances just went away.

The shared CI and build setup is available right out of the gate for any new project. A standard structure means I can copy an existing project to get started. The barrier to trying a new idea is much lower, and anything I keep ends up with the same coding style and structure.

I can also have shared utilities across multiple programs, and just have to update them once without needing to bump deps for a bunch of repos. I can also have shared CI and linting rules, so when I want to change a rule, I change it once and it applies everywhere.

Because the infrastructure is already there, more of my experiments end up in source control instead of dying in a scratch folder. That makes them easier to find if I want to revisit one later.

The freedom of building for one#

A personal monorepo with no expectation of being used by others gives me the freedom to make choices I wouldn't make for a project that has to take into account the needs of others and how different people may approach using the tool.

Those decisions might be wrong for an open source library or a shared codebase at work, where separate repos provide clear versioning and ownership boundaries. For my personal projects, that is too much overhead and it creates a barrier from actually building things.

Give it a try#

If you maintain a pile of small repos, and spend more time on repo maintenance than on the code itself, you might want to look into monorepos. Xe's monorepo is a good reference for how to structure one.

A serverless Docker registry with Bunny.net

Tue, 03 Mar 2026 00:00:00 +0000

I host all of my Docker images using my personal Gitea instance. Rather than expose my home network on the internet, I wanted to figure out a way to distribute them without having to worry about managing yet other server. I already use Bunny.net for some things, and thought I could try to figure out how to make that work.

Bunny.net has edge compute similar to AWS Lambda@Edge, but it also comes with a per-request cost. But why use those when edge-rules exist, so I wanted to see whether they were enough. The OCI distribution API turns out to be mostly static file lookups (at least for public images since you don't need to consider authentication). A CDN that can rewrite URLs and set response headers can pretend to be a registry.

Registry endpoints#

The OCI distribution spec boils down to a few GET endpoints:

GET /v2/ returns 200 OK as an auth check (conventionally with a body of {}, though the spec doesn't mandate a specific body). If you return 200, clients will assume auth isn't required.
GET /v2/<name>/manifests/<reference> returns a manifest (by tag or digest)
GET /v2/<name>/blobs/<digest> returns layers and configs

Manifests need a little extra handling. Edge compute could distinguish an index from a manifest and set headers accordingly. With plain edge rules, I instead store platform-specific manifests and multi-arch indexes in separate directories, so the CDN can set Content-Type based on the path alone. If a client requests a manifest by digest, the CDN looks for it in the manifest directory first. If it isn't there, it must be an index, so the CDN issues a redirect to the index path. The client then gets the correct Content-Type without the CDN inspecting file contents.

The responses also need specific headers: the correct Content-Type and Content-Length for each artifact, and Docker-Content-Digest on manifests and blobs. I also set Docker-Distribution-API-Version: registry/2.0 on all /v2/ responses. This is a legacy Docker header that the OCI spec considers optional, but some older clients still expect it. If you serve the right files with the right headers at these paths, Docker doesn't know the difference.

Storage layout#

The file structure in the Bunny storage zone mirrors the API paths, with some extras to help the edge rules dispatch correctly:

/v2/
  _root                              # static {} for the base endpoint
  _blobs/sha256:<hex>                # shared blob store (deduplicated)
  <image>/
    manifests/
      sha256:<hex>                   # platform-specific manifests
      tags/<tag>                     # image index JSON (multi-arch)
    indexes/sha256:<hex>             # image indexes (by digest)

Blobs live in a shared _blobs/ directory, deduplicated across images.

The edge rules#

Bunny.net's edge rules are enough to make the CDN look like a Docker registry. They handle path rewrites, response headers, and the 404-to-redirect trick for manifest vs index dispatch.

Base endpoint#

Set the Docker-Distribution-API-Version: registry/2.0 header on all /v2/* requests.
Rewrite /v2/ (exact match) to serve the _root file, which contains {}.
Set Content-Type: application/json on that base response.

Tag resolution#

When a client requests /v2/<name>/manifests/<tag> (where the tag is a human-readable name like latest, not a sha256 digest), rule 4 rewrites the path to /v2/<name>/manifests/tags/<tag>. This uses Bunny's triggerMatchAll to match /v2/*/manifests/* while excluding paths that already contain sha256: or tags/.

Rule 5 sets Content-Type: application/vnd.oci.image.index.v1+json on responses from /v2/*/manifests/tags/*. In my setup, tag lookups always return an image index (the multi-arch manifest), since I always build multi-arch with buildx. If you only built single-platform images, a tag could point directly to a platform manifest instead.

Blob requests#

Rule 6 rewrites blob requests from /v2/<name>/blobs/sha256:* to /v2/_blobs/sha256:*, pointing at the shared store. Bunny's path segment variables (%{Path.3}) extract the sha256:<hex> segment to build the rewritten path.

Manifest vs index dispatch#

When a client requests a manifest by digest (/v2/<name>/manifests/sha256:*), the CDN looks in /manifests/ first. If the digest belongs to a platform manifest (the config and layers for a single architecture, like linux/amd64), the file exists there and the CDN serves it directly. Rule 8 sets Content-Type: application/vnd.oci.image.manifest.v1+json on these paths.

If the digest refers to an image index (the document that lists which platform manifest to use for each architecture), it won't exist in /manifests/. It is stored only in /indexes/, so the request 404s. Rule 7 catches that 404 and issues a 302 redirect to /v2/<name>/indexes/sha256:*. This has to be a real redirect, not a URL override, so the client makes a fresh request. That way the response header rules evaluate against the /indexes/ path. Rule 9 then sets Content-Type: application/vnd.oci.image.index.v1+json correctly.

Digest headers#

Rules 10 and 11 extract the sha256 digest from the URL path (using the %{Path.3} path segment variable mentioned above) and set it as the Docker-Content-Digest header. One covers manifest paths, the other covers index paths after redirect. Strictly speaking, the OCI spec also requires Docker-Content-Digest on blob responses, but in practice the clients I've tested (docker, crane, skopeo) don't depend on it for blobs, so I haven't added a rule for that.

Publishing images#

Publishing goes through a script because the files have to land in the exact directory structure above. I build images with docker buildx in CI, targeting multiple architectures and exporting in OCI format. The script transforms the OCI layout into the storage zone structure, then uploads everything using the Bunny.net storage zone API. After the upload, it purges the tag manifest URL so docker pull immediately sees the new version.

publish-oci-image <image-name> <tag> <oci-layout-dir-or-tar>

  extract OCI input into a staging directory if it's a tar file

  read index.json to find the top-level index digest
  copy the index blob to:
    /v2/<name>/manifests/tags/<tag>   (tag lookup)
    /v2/<name>/indexes/<digest>       (digest lookup)

  for each platform manifest listed in the index:
    copy the manifest blob to /v2/<name>/manifests/<digest>
    for each blob (config + layers) referenced by that manifest:
      copy to /v2/_blobs/<digest> if not already there

  upload the directory to the Bunny storage zone as it matches the structure from above
  purge the CDN cache for /v2/<name>/manifests/<tag>

The staging directory mirrors the final storage layout exactly, so the upload is a straightforward sync. Even if two platform manifests reference the same layer, the script only copies it once. The cache purge at the end targets just the tag URL, because that's the only path where staleness matters (blob digests are content-addressed, so they never change).

Garbage collection#

With no registry server managing state, storage can balloon quickly, so I have a separate GC command for cleanup. It walks each image's tags, resolves every referenced blob, manifest, and index, then checks for unreferenced files in _blobs/. If it finds any, it deletes them using the Bunny API. Walking the whole graph is a bit slow, but that is fine for my use case. I publish a handful of images, and each one only ever has the latest tag.

Pulling images#

docker pull tklk.dev/abc:latest works. I've verified it with docker, crane, and skopeo. Each one sees a real registry because the headers and paths are correct.

Limitations#

There's no docker push. Images go through the publish script, which means CI is the only path to publishing. Images are public only, so this approach only works for content you're happy to expose without authentication. I also haven't implemented tag listing (/v2/<name>/tags/list) or the Docker-specific catalogue endpoint (/v2/_catalog, which isn't part of the OCI distribution spec). I know what images I have because I put them there.

If you need private images, a push API, tag listing, or fine-grained access control, you probably want something like Gitea. I use this only for public personal images that already publish through CI. It runs on a single pull zone with no server behind it.

Tricking GitHub Copilot into reviewing PRs in Gitea

Mon, 02 Mar 2026 00:00:00 +0000

Xe Iaso wrote about building a review bot that uses a self-hosted LLM to review GitHub pull requests. When I learned that some "AI" CLIs have SDKs, I wondered whether I could build something similar while leaving the loop and execution environment to a billion-dollar corporation.

Disclaimer: I chose Copilot because it was free, and I liked the novelty of using a GitHub thing somewhere other than GitHub. These companies also don't need more money for a silly experiment.

Contrary to the title, I didn't actually "trick" Copilot. These AI coding CLIs have SDKs that let you interact with them programmatically, kinda like a RESTful API. The API is as non-deterministic as the CLI, and the agent can still run its normal loop and call tools. Copilot was convenient here, but this pattern is common across these kinds of tools.

I should also note that I have reservations about AI tooling in general (a topic for another post that's been sitting in my drafts). This post is about a pattern I found interesting, not an endorsement of any particular product, and especially not an endorsement/condoning of the practices of the companies behind them.

The approach#

Xe's setup triggers via a GitHub Actions workflow when someone comments /reviewbot on a PR. It uses an OpenAI-compatible API backed by a self-hosted model running on a DGX Spark. Its agentic loop can execute Python to analyze the codebase and submit the review. The bot also clones the repo so the model has filesystem access through Python execution.

Instead of an Actions workflow, mine starts from a webhook listener that watches for a specific pattern in PR comments and fires off a review when it matches. A Go process fetches the PR's diff and existing comments, then spins up the Copilot CLI with a prompt built from that context and a set of custom tools. The agent explores the codebase and calls the tools as needed, and when it's done, the review summary gets posted back to the PR.

The tools#

The tools translate the agent's feedback into the specific API calls Gitea expects. The agent has no idea it's talking to Gitea.

post_inline_comment posts a comment on a specific file and line. Its severity is either blocker (must-fix before merge) or suggestion (recommended improvement). The PR author can triage quickly: address the blockers, consider the suggestions.

note_low_confidence records an observation the agent isn't confident about.

submit_review posts the final summary as a top-level PR comment. The agent calls it once at the end of the review.

The note_low_confidence tool came from Angie Jones' post on teaching Copilot to think like a maintainer. She recommends a confidence threshold above 80%, so the reviewer only comments when it's fairly sure something is wrong. Without that, AI reviewers dump every observation as an equally-weighted comment, and people learn to ignore the bot fast.

Note: While we may describe these machines as "thinking", it is important to recognize our anthropomorphization of what is really statistical analysis being done by the model.

I agree with the threshold idea, but my confidence in these models isn't high enough to trust their weighting. If I were building a product or running this somewhere that noisy bot comments would overload developers, I'd probably have the bot stay silent below the threshold. For an experiment, though, I wanted to see everything the model had to say and judge it myself.

Low-confidence observations go into a collapsible section at the bottom of the main review. They stay out of the way while remaining available to anyone curious. This way, I get to be my true Hannah Montana self and have the best of both worlds.

The system prompt#

Angie's post was full of great ideas, like telling the reviewer to skip linting, since CI already catches those errors and does a better job of it.

The prompt also explains the confidence model (high confidence gets an inline comment, low confidence gets a note_low_confidence call) and includes the expected output format for the final summary.

PR context goes in as a structured template: title, author, branches, description, the diff, existing comments (filtered to exclude bot noise). If the diff is too large, it's truncated with a note telling the agent to use git diff on specific files. The agent has the full cloned repo, so it can always look at more than what's in the prompt.

The feedback loop#

Once the agent is done, the review summary comment gets updated. Low-confidence notes are appended in a collapsible <details> section, and usage metadata (token counts, model used, API calls, cost if available) goes in another collapsible section at the bottom.

I include the usage metadata so that anyone reading a bot-generated review can see what produced it and roughly what it cost. A line like "claude-sonnet-4.5, 12 API calls, 45k input tokens" makes the resource use concrete, and hopefully nudges future contributions to stay focused. If a contribution is too large, it can overflow the context window and the review gets less useful.

Beyond Copilot#

Very little of this is Copilot-specific because the SDK, model, and hosting can all change. The custom tools adapt the review loop to Gitea.

Code is left as an exercise to the reader. Although, if you are a VC, I am happy to send you my routing number in exchange for $10 Billion for this unicorn.

DIY Multi-Hop Boundary Sessions without HCP

Wed, 16 Apr 2025 00:00:00 +0000

I treat my homelab as if it has the same security requirements as a production environment because for me it is a production environment. I normally use Tailscale for protected access and avoid exposing services to the internet. Some remote networks block Tailscale or WireGuard connections, so I also need a break-glass option.

As with many of my other posts, you can see I use HashiCorp's suite of tools. Boundary provides identity-based access to systems in my homelab without SSH tunnels or exposing each system to the internet. It is also a single binary that runs without administrative access, which helps when I cannot install VPN software on a machine.

The problem#

My homelab budget does not stretch to the enterprise version of Boundary or HCP (HashiCorp Cloud Platform). The community version does not support the ingress and egress workers required for multi-hop sessions.

I want multi-hop sessions so I can keep at least some VLAN isolation without letting the server in the DMZ connect directly to every server in my homelab.

The topology I wanted:

graph TB
    accTitle: DIY Boundary multi-hop network topology
    accDescr: Diagram of three network zones. A client laptop in the Internet zone connects to a Boundary controller hosted with a DMZ worker and FRP server in the DMZ. An internal worker in the Internal VLAN registers outbound with the controller, holds a tunnel to the FRP server, and reaches internal resources, so the DMZ never connects directly into the internal VLAN.
    subgraph Internet
        client[Client/Laptop]
    end
    
    subgraph DMZ
        controller[Boundary Controller]
        dmz_worker[DMZ Worker]
        frps[FRP Server]
    end
    
    subgraph Internal_VLAN[Internal VLAN]
        internal_worker[Internal Worker]
        internal_resources[Internal Resources]
    end
    
    internal_worker -- 1. Register --> controller
    client -- 2. Connect --> controller
    client -- 3. Session --> dmz_worker
    internal_worker <--> frps
    dmz_worker -- 4. Connect --> frps
    internal_worker -- 5. Access --> internal_resources

My DIY solution#

My workaround depends on Boundary's connection flow:

The client (my laptop) connects to the controller
The controller provides an authorization token
The client connects to the worker port using this token
The worker establishes the session with the target

I run an FRP (Fast Reverse Proxy) server alongside the DMZ Boundary controller and worker. The internal workers connect outward to that server, so I do not need firewall rules that permit inbound connections to the internal workers. After registration, each internal worker gets a Boundary target pointing at a unique frps port on the Boundary server's loopback interface.

The resulting session path:

sequenceDiagram
    accTitle: DIY Boundary session establishment sequence
    accDescr: Sequence of nine steps showing how a session is established. The client authenticates with the Boundary controller and receives a token, connects to the internal worker target through the DMZ worker, which forwards over the FRP tunnel the internal worker already holds. A second token then lets the client connect through that established tunnel to the internal resource target.
    participant Client
    participant Controller as Boundary Controller
    participant DMZ as DMZ Worker
    participant FRP as FRP Server
    participant Internal as Internal Worker
    participant Target as Internal Resource
    
    Client->>Controller: 1. Authenticate & request session
    Controller->>Client: 2. Return session token
    Client->>DMZ: 3. Connect to internal worker target
    DMZ->>FRP: 4. Forward to internal worker via FRP tunnel
    FRP->>Internal: 5. Connection established to internal worker
    Client->>Controller: 6. Request session for internal resource
    Controller->>Client: 7. Return session token
    Client->>Internal: 8. Connect to internal resource via established tunnel
    Internal->>Target: 9. Connect to target resource

Setting it up#

Following the standard Boundary installation process, expose the DMZ controller API, usually port 9200, and the worker, usually port 9202, to the internet. The endpoint could be boundary.tklk.dev, with Caddy providing TLS for API access. Internal workers also need to reach the cluster worker coordinator on port 9201, but that port does not need public exposure. They can reach it through the same path used for the FRP server.

Installing FRP server#

Install FRP on the DMZ server:

Download the appropriate FRP release from GitHub
Extract the archive and locate the frps binary
Create a configuration file as shown below
Set up a systemd service (optional) to ensure it runs automatically

This is the frps configuration I use:

bindPort = 7000
auth.method = "token"
auth.token = "your_secure_token"

# meaning the internal workers can only use this range for listening on
allowPorts = [ {start = 9500, end = 9999}  ]

Internal worker setup#

After the FRP server is running, configure the internal workers:

Install and configure a standard Boundary worker

# minimal worker config
# you may wish to add more configuration options, such as using a KMS
listener "tcp" {
  # this is the listener that frpc will connect to over the loop back
  address = "127.0.0.1:9202"
  purpose = "proxy"
}

worker {
  name = "internal-worker-1"
  # this is the address that the boundary controller will announce to the client
  # so this is what the target session should listen to on your local laptop/client
  public_addr = "127.0.0.1:9500"
  description = "Worker in the internal VLAN"
  controllers = ["boundary.tklk.dev:9201"]
}

Install FRP client (frpc)
Configure frpc to establish a reverse tunnel back to the primary server on a loopback interface

This is the frpc configuration I use:

serverAddr = "boundary.tklk.dev"
serverPort = 7000
auth.method = "token"
auth.token = "your_secure_token"

[[proxies]]
name = "boundary_internal_worker"
type = "tcp"
localIP = "127.0.0.1"
# 9202 is the Boundary worker's default port, but since you may have multiple internal workers, you might want to pick
# a unique port for each worker (which you'll also need to set in the worker config)
# It is strongly recommended that you don't use port 9202 as the DMZ worker will already likely be using this port, causing the frps connection to fail
localPort = 9202
remotePort = 9500

Registering the internal worker as a target#

Register each internal worker as a Boundary target pointing to the FRP server's loopback interface and the port selected in the frpc configuration. You can do this with Terraform, the Boundary CLI, or the Boundary UI.

At minimum, the target needs 127.0.0.1 as its host, the port from the frpc configuration, and a default client port that matches the worker port.

An egress worker filter can force the DMZ worker to connect to the frps port. Because the target uses localhost, Boundary may otherwise assign the route to a different worker that cannot reach it.

Using the setup#

Authenticate with Boundary, then use boundary connect to establish a connection to the internal worker target.

# Authenticate with Boundary (if not already authenticated)
boundary authenticate

# List available targets
boundary targets list -recursive

Accessing an internal resource takes two connections:

Establish a connection to the internal worker:
```
boundary connect -target-id=ttcp_1234567890  # Your internal worker target ID
```
If nothing else on your local machine is listening on the worker port as defined above, you'll open a connection to the internal worker. In your session list, this connection may be marked as "pending" since no traffic is being sent yet.

Once connected, you can access targets in the internal VLAN:

boundary connect -target-id=ttcp_0987654321  # Internal resource target ID

This avoids HCP or enterprise license fees, but it adds FRP management and a second connection step.

Limitations and considerations#

You need to manage FRP alongside Boundary
The connection process requires an extra step compared with the enterprise solution
Nested connections can add a small amount of latency
Protect the FRP token to prevent unauthorized access

Single-hop sessions are likely enough for my needs, with multi-hop being a tad over the top. Building this still gave me a better understanding of Boundary's connection flow and another way to debug its networking.

Boundary can also be integrated with Vault to broker SSH credentials. Vault can be used as an SSH certificate authority, and so perhaps in a future post I might connect the two systems.

SSH into your Gitea Actions for debugging

Thu, 10 Apr 2025 00:00:00 +0000

While Gitea Actions allows for local debugging of workflows, sometimes you need to debug directly on the runner. This is especially challenging since the workflows only exist for the duration of the run. Even if you have access to the server and can docker exec into the container, it may terminate while you are debugging.

Since Gitea Actions are compatible with GitHub Actions, you can use action-tmate to SSH directly into the runner environment and debug your workflow in real time.

Adding action-tmate to your workflow#

First, edit your Gitea Actions workflow YAML to add the action-tmate step.

name: CI
on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      # Add this step wherever you need debugging access
      - name: Debug with SSH
        uses: mxschmitt/action-tmate@v3

When this step runs, it will pause the workflow and output SSH connection details in the logs. The workflow will wait until you've connected, finished debugging, and exited the session. However, if you place this step after a failing one, the workflow may stop before reaching it.

Connecting to your runner#

Once the action runs, check your workflow logs. You'll see connection details like:

WebURL: https://tmate.io/t/...
SSH: ssh abc123...@...tmate.io

You can connect using either SSH or the web interface. The web interface is especially useful if your network's firewall blocks SSH connections.

Once connected, you'll have full shell access to the runner environment, allowing you to inspect files, run commands, and verify network connectivity.

When you're done debugging, exit the SSH session. The workflow will then continue with the next step after action-tmate.

Security: Anyone with the connection details can access the session, so don't leave it idle longer than needed. The example connection details use tmate.io, which adds an external service to the debugging path.

Targeted debugging approaches#

You probably don't want to SSH into every workflow run. Here are some more practical approaches:

Only on failure#

- name: Debug with SSH
  if: ${{ failure() }}
  uses: mxschmitt/action-tmate@v3

This only activates the SSH session if a step in the workflow failed.

Manual trigger#

With the introduction of workflow_dispatch in Gitea 1.24, you can manually trigger workflows with an option to enable SSH debugging:

on:
  workflow_dispatch:
    inputs:
      debug_enabled:
        type: boolean
        description: 'Enable SSH debugging'
        required: false
        default: false

# Then in your job:
steps:
  - name: Debug with SSH
    if: ${{ github.event_name == 'workflow_dispatch' && inputs.debug_enabled }}
    uses: mxschmitt/action-tmate@v3

Direct access is useful when logs are not enough. For other session controls, see the additional configuration options in action-tmate.

Manage an entire fleet of macOS VMs with HashiCorp Nomad

Wed, 09 Apr 2025 00:00:00 +0000

To learn how Nomad driver plugins work, I built a driver plugin for Tart. It lets Nomad manage Tart macOS VMs in much the same way that it manages Docker containers.

Disclaimer: This driver is a proof of concept and is not ready for production use. For a production-ready option built by Tart's developers, consider Orchard.

Tart#

Tart is a VM manager for macOS VMs on Apple Silicon. It integrates with OCI-compatible container registries, so VM images can be stored and distributed like Docker container images.

The driver experiment#

One of the ways I learn a new technology is by building something with it. Nomad already has drivers for workloads such as Docker, so I chose Tart while learning the plugin system. Tart manages VMs on one machine rather than orchestrating them across several, which made it a useful fit for a driver. Tart also has its own orchestration manager, Orchard, but Nomad can manage other workloads alongside the VMs. Potential uses include CI/CD workflows, ephemeral development environments, and testing infrastructure.

Sample Nomad job#

This job starts a macOS VM:

job "tart-demo" {
  datacenters = ["dc1"]
  type        = "service"

  group "demo" {
    task "vm" {
      driver = "tart"

      config {
        image     = "ghcr.io/cirruslabs/macos-sequoia-base:latest"
        cpu       = 4
        memory    = 8192
        disk_size = 50
        command   = "echo 'Hello from Tart VM'"
      }
    }
  }
}

Trouble with Azure Functions and managed identities

Mon, 31 Mar 2025 00:00:00 +0000

While automating TLS certificates for Azure Application Gateways, I ran into an issue with Azure Functions and managed identities. The tool I was using could not fetch a managed identity token. After reviewing its code and the Azure documentation, I found that Azure Functions exposes environment variables for endpoint information that differs slightly from the standard endpoint.

Editor's Note: Please don't take any of this in a negative light. The functionality that would normally be provided by an SDK couldn't be used, and the implementation is uncommon enough that it's understandable a tool like acme.sh wouldn't cover this edge case. It is a great tool, and I'm very happy with it.

I firmly believe that all TLS certificates should be automated. The goal of this project is to request certificates from Let's Encrypt or ZeroSSL and store them in Azure Key Vault for Azure Application Gateway to use. The first problem I hit was managed identity authentication.

The setup#

My plan was to run acme.sh in an Azure Function and use Azure's managed identity to authenticate with Azure Key Vault. The function could then store certificates directly in Key Vault for Application Gateway.

Running the Azure Function on a schedule would renew certificates before expiration without manual work.

The problem#

Installing acme.sh in the Azure Function was quick, but DNS validation with a managed identity could not fetch a managed identity token.

The error showed that acme.sh could not connect to the managed identity token service. In most Azure environments, services access that token service through a link-local address, typically 169.254.169.254. This endpoint is well documented and widely used, so acme.sh would normally use the correct endpoint.

After connecting to the Azure Function over SSH, I confirmed that it could not reach the metadata endpoint. The Python SDK documentation pointed me to the IDENTITY_ENDPOINT environment variable for the correct endpoint.

The fix#

With the correct endpoint and the additional authentication header, I fetched the managed identity token with curl. I then searched acme.sh's code to see how it fetched the token. It made the same hardcoded-endpoint assumption I had, so I put together a small change to fix it.

In the spirit of open-source, and to thank the authors for their work on acme.sh, I was able to send that change as a PR to fix the issue so that no one else would have to go through the same debugging process.

Next steps#

I plan to cover the certificate automation for Azure Application Gateway in a later post.

SSH Multi-Factor Authentication with SoloKey V2

Sun, 30 Mar 2025 00:00:00 +0000

In a bout of attempting to improve my operational security, I decided to review my current practices and see which areas I could improve. While I already use SSH keys for auth, having a file that is essentially a password to all my servers is a bit of a security risk if it ever is compromised. I already use my SoloKey for 2FA on various websites, so why not for SSH too? This way, even if someone gets my private key file, they can't access my servers without physically having my SoloKey. Still a risk (see the XKCD comic about a $5 wrench), but less so.

Editor's Note: While MFA by using a physical key is a great improvement, you really should look into using SSH Certificates as they are time scoped, and depending on how you have them set up, have automatic rotation. Among other benefits, this is a great way to ensure that even if a key is unknowingly compromised, it can't be used for long.

Also, this same process could be used for other physical keys like YubiKeys, but I have several SoloKeys V2 (cheaper and open-source) so that is what I used.

OpenSSH support on macOS#

Creating the key starts with ssh-keygen -t ed25519-sk and touching the SoloKey when prompted. On the macOS devices I tested, the bundled SSH version did not support this key type, so I installed a newer version of OpenSSH:

brew install openssh libfido2

After installation, you can check that you have the right version:

ssh -V

You should see OpenSSH_8.2 or later, which supports the FIDO2 keys. If you don't, then you may need to update your $PATH environment variable to point to the Homebrew version of SSH.

Creating a key tied to your SoloKey#

With the newer OpenSSH installed, create the key:

ssh-keygen -t ed25519-sk -f ~/.ssh/id_ed25519_sk

You'll be asked for a passphrase (recommended), and be prompted to touch your SoloKey to confirm. Once done, you'll have two new files:

~/.ssh/id_ed25519_sk - your private key
~/.ssh/id_ed25519_sk.pub - your public key

Making it convenient#

For convenience, I added the key to my SSH agent:

ssh-add ~/.ssh/id_ed25519_sk

If you don't use the SSH agent, you could instead add the key to your SSH configuration file

vi ~/.ssh/config

# Use the security key for all connections
Host *
    IdentityFile ~/.ssh/id_ed25519_sk

Setting up the server side#

Similarly to how you would add a regular SSH key, you can add the public key to your server:

cat ~/.ssh/id_ed25519_sk.pub | pbcopy
ssh user@remote-server "vi ~/.ssh/authorized_keys" # Paste the public key and save

Just like with your local machine, you'll need to make sure the remote server also supports the new key type. You can check this by running:

ssh user@remote-server "ssh -V"

The moment of truth#

Now try connecting:

ssh user@remote-server

If everything's set up correctly, you'll be prompted to touch your SoloKey to authenticate.

The older OpenSSH version bundled with macOS was the only setup problem I hit. The same key can also authenticate to other services, including Gitea.

If you are on a Mac, you could also look into Secretive, which stores SSH keys in the Secure Enclave and integrates with Touch ID.

Gitea Actions: Forget about YAML, let's use Nix

Sun, 04 Aug 2024 00:00:00 +0000

I maintain many repositories using Gitea Actions, and I've run into pipeline failures over small issues. Whitespace differences in YAML files are the most common cause, leading to unexpected parsing or complete failures. Recently, I helped debug a copied workflow where some of the indentation used tabs and the rest used spaces. That was enough to make the pipeline fail.

Update: After writing this post, I discovered a similar approach using Dhall. If you're interested in this concept, check out github.com/regadas/github-actions-dhall, which has already implemented much of what I discuss here with Dhall instead of Nix.

The problem with YAML#

YAML is extremely sensitive to whitespace, which can lead to subtle issues that are hard to debug when tabs and spaces look alike.

The solution: Nix#

In my case, as is the case with all my problems, the solution was to use Nix. I had been using Terranix to manage some infrastructure. Terranix translates Nix to JSON for Terraform, and I realized a similar approach could work for Gitea Actions. JSON and YAML both represent data as key-value pairs, so I can convert Nix to JSON and then to YAML.

Editor's note: The Nix below is an untested rough draft of the approach. If you try it, I'd be interested in any feedback so I can update this post.

Potential Nix benefits#

Fewer whitespace-related errors in handwritten workflows
Nix features such as conditionals, loops, and functions
Code reuse across workflows
Workflows split into smaller components

Implementing Nix-based workflows#

Step 1: Create a basic step function#

A step is the most common workflow component, so I'll start with a function that creates one.

# steps.nix
let
  mkStep = { name ? null, uses ? null, run ? null, with' ? null }:
    let
      step = {
        name = if name != null then name else null;
        uses = if uses != null then uses else null;
        run = if run != null then run else null;
        with = if with' != null then with' else null;
      };
    in
      builtins.filterAttrs (a: v: v != null) step;
in
  mkStep

This mkStep function creates a step object and removes null fields from the YAML output.

Note: with is a reserved keyword in Nix, so I used with' instead.

Step 2: Create a checkout function#

Checkout is another common workflow step. This function builds it with mkStep.

# steps.nix
# ... keep the previous code and add the new `mkCheckout` function
  mkCheckout = { name, uses ? "actions/checkout@v4", with' }:
    mkStep {
      name = name;
      uses = uses;
      with' = with';
    };

# Export both functions
{
    mkStep = mkStep;
    mkCheckout = mkCheckout;
}

The same pattern could cover other common actions. Generating functions from action.yaml files is another possibility.

Step 3: Create a workflow#

The functions above can create a complete workflow.

# workflow.nix
let
  steps = import ./steps.nix;

  mySteps = [
    (steps.mkCheckout { with' = { fetch-depth = 0; }; })
    (steps.mkStep { run = "echo 'Hello, World!'" })
    # More steps could be added here
  ];

in
  {
    name = "Example Workflow";
    on = "push";
    jobs = {
      build = {
        "runs-on" = "ubuntu-latest";
        steps = mySteps;
      };
    };
  }

Generate the Gitea Actions YAML workflow#

Convert the Nix workflow to YAML with:

nix-instantiate --eval --strict --json ./workflow.nix | yq eval -P - > example.yaml

This requires yq and writes the generated workflow to example.yaml.

Possible next steps#

This may be most useful where projects have similar workflows. The Gitea project has one workflow for nightly release artifacts and another for tagged release artifacts. Arguments and conditionals could move their shared definitions into one place instead of maintaining two similar files.

Nix flakes are another possible way to define inputs for an Action step and pass them to the workflow.

This approach replaces handwritten YAML with Nix, but it adds Nix and yq to the workflow toolchain and still generates YAML as an intermediate file. Whether that tradeoff is worthwhile depends on how much workflow code can be shared.

Streamlining My GCP SSH Workflow: Using SSH Configuration Tricks

Thu, 18 Jul 2024 00:00:00 +0000

I often recreate virtual machines (VMs) in Google Cloud Platform (GCP), which means their connection information changes frequently. That leaves stale entries in known_hosts and creates more DNS work as the number of servers grows. Using GCP's Identity-Aware Proxy (IAP) also requires the gcloud CLI. So instead of dealing with this complexity each time, I used my past exprience with SSH configuration customization to handle those moving parts.

Editor's Note: At the time I made heavy use of GCP, however now I am using other hyperscalers. This post is kept up for historical purposes.

The challenge#

There were several issues I needed to solve:

Frequent VM recreation leading to connection information changes
Managing stale known_hosts entries
DNS management for numerous servers
The necessity of using IAP for secure access
The complexity of the gcloud compute ssh command

Requirements#

The setup needs:

The gcloud CLI installed locally
nc (netcat) installed on the remote servers to proxy the connection from IAP to the VM

gcloud compute ssh wraps ssh to handle authentication and IP resolution. It becomes unwieldy with other SSH-based tools such as Ansible and rsync because each tool needs the command passed through to it. I had previously used Vault to make SSH run a command for each connection, so I tried the same approach here. A Stack Overflow answer gave me a starting point that I adapted for this setup.

I use GCP's Identity-Aware Proxy to tunnel connections without exposing SSH to the public or assigning public IPs to each VM. The gcloud CLI provides this through the --tunnel-through-iap flag.

SSH configuration#

I added this block to my SSH configuration:

Host *.gcloud
    ProxyCommand bash -c 'IFS=. read -r server zone project _ <<< "${1}"; gcloud compute --project "$project" ssh --zone "$zone" --ssh-key-file ~/.ssh/gcp_key --tunnel-through-iap "${2}@$server" --command="nc 0.0.0.0 22"' _ %h %r
    IdentityFile ~/.ssh/gcp_key
    # GCP handles the known hosts, and hostkey checking separately, so we can ignore them here
    StrictHostKeyChecking no
    UserKnownHostsFile=/dev/null

This configuration takes a hostname I provide (e.g., server1.us-west1-b.project1-id.gcloud) and converts it into variables that are then passed to the gcloud CLI.

I also wanted to forward ports from inside GCP's network to my local machine. For a particular host, define a more specific Host block and add its options there. SSH will also apply the settings from the wildcard block.

Host server1.us-west1-b.project1-id.gcloud
    LocalForward 3306 10.20.30.40:3306

# Host *.gcloud...

Troubleshooting#

If this setup does not connect, make sure you've added the IAP IP addresses to your GCP network ingress rules. You can find the list in the GCP documentation. You may also need to sign in to the gcloud CLI.

This setup has saved me a lot of time. I could manage DNS entries when creating servers with infrastructure as code, but that would add another piece of configuration that can fail. This approach also keeps the VMs' SSH ports off the public internet.

Use Drone plugins in Gitea Actions

Sat, 29 Jun 2024 00:00:00 +0000

Drone.IO runs each pipeline step in a Docker container. Drone plugins are containers with a predefined task and settings supplied by the pipeline. Because they are published as Docker images, the same plugins can also run in Gitea Actions.

One plugin I used was drone-scp, which copies files from the build pipeline to a remote server. My Drone step looked like this:

- name: scp files
  image: appleboy/drone-scp
  settings:
    host: example1.com
    username: ubuntu
    password:
      from_secret: ssh_password
    target: /home/deploy/web
    source:
      - release/*.tar.gz

The step calls the predefined binary in the drone-scp image, which copies files from the release directory to the target directory on the remote server. The full script does not need to live in the pipeline.

Because drone-scp is published as a Docker image, Gitea Actions can run it too. The quick approach works for most plugins. The more involved approach wraps the plugin as a full Gitea Action.

The simple approach: use the Docker image directly#

The easiest way to use a Drone plugin in Gitea Actions is to reference the Docker image directly with uses: docker:// and pass the plugin's expected PLUGIN_ environment variables via the step's env: block:

name: copy files
on: [push]
jobs:
  build:
    name: Build
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: docker://appleboy/drone-scp
      env:
        PLUGIN_HOST: ${{ secrets.HOST }}
        PLUGIN_USERNAME: ubuntu
        PLUGIN_PASSWORD: ${{ secrets.SSH_PASSWORD }}
        PLUGIN_TARGET: /home/deploy/web
        PLUGIN_SOURCE: release/*.tar.gz

This works because Drone plugins read their configuration from PLUGIN_-prefixed environment variables (derived from the settings: block in Drone), and step-level env: variables in Gitea Actions are passed directly into Docker containers. No repackaging is needed.

The full approach: wrap the plugin as a Gitea Action#

If you want to publish a reusable Action with defined inputs (so consumers use with: instead of env:), you can wrap the Drone plugin in a Dockerfile, entrypoint script, and action.yml. The Dockerfile should look something like this:

FROM appleboy/drone-scp

COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

ENTRYPOINT ["/entrypoint.sh"]

The entrypoint passes the action's input arguments to the drone-scp binary:

#!/bin/sh

set -eu

sh -c "/bin/drone-scp $*"

Note: When using the with: block in Gitea Actions (like GitHub Actions), inputs are passed to Docker containers as environment variables prefixed with INPUT_ (e.g. INPUT_HOST). Drone plugins natively expect the PLUGIN_ prefix (e.g. PLUGIN_HOST). Many popular Drone plugin authors (including Appleboy) have updated their binaries to accept both prefixes. If a plugin does not recognise INPUT_ variables, you may need to translate them in the entrypoint script by mapping each INPUT_* variable to its corresponding PLUGIN_* variable.

The action.yml file tells Gitea Actions which inputs to use.

name: 'drone-scp Gitea Action'
# This tells the Gitea Action runner what type of Action this is
# in this case, it's "docker", and the image to use is the Dockerfile
runs:
  using: 'docker'
  image: 'Dockerfile'
inputs:
  host:
    description: 'scp remote host'
# ... define all the remaining inputs that exist for the Drone Plugin

You can then reference this action in a Gitea Actions workflow the same way you used the Drone.IO plugin.

For example:

name: copy files
on: [push]
jobs:
  build:
    name: Build
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: https://gitea.com/techknowlogick/scp-gitea-action@main
      with:
        host: ${{ secrets.HOST }}
...

Appleboy, the author of the SCP Drone plugin, has already done this. The resulting scp-action can be used directly.

Plugins that accept INPUT_ variables can use this wrapper directly. Plugins that only accept PLUGIN_ variables need the mapping described above, but the plugin itself does not need to be rewritten. This keeps the existing implementation, including plugins written in languages the runner does not support natively.

Experimenting with Proxmox on Apple Silicon: Turning Your M1 MacBook Air into a Virtual Datacentre

Fri, 28 Jun 2024 00:00:00 +0000

In an effort to create an easy-to-manage VM environment that I can use on the go, even with spotty internet, I decided to try running Proxmox on my Apple Silicon MacBook Air. Parallels and VMware Fusion were options, but I chose Proxmox to support open-source software. For production-level workloads, I would purchase a Proxmox license and use a dedicated server. Proxmox doesn't distribute arm64/aarch64 packages, so this experiment depended on an unofficial port.

Editor's Note: These instructions are now out of date, as there are newer versions of Proxmox, and they may support Arm too. This post is kept up for historical reference.

Getting started#

Proxmox is based on Debian but does not come packaged for arm64. A Debian installation can be converted to Proxmox, so I tried the same route used for x86_64.

UTM provides an interface for Apple's Hypervisor and QEMU. I could have used QEMU directly, but I chose UTM and created a Debian VM from one of the premade images in its gallery. Its presets supplied the VM configuration.

Converting Debian to Proxmox#

With Debian running, I needed Proxmox packages for Arm. Since Proxmox doesn't support Arm or distribute their packages for Arm I would have to build those myself. Luckily, some other kind soul already did this and published them.

Warning: I'm installing packages from an unofficial source. My risk level is fine with this, but if you are considering doing this, I strongly recommend you evaluate the choice carefully. If you are not comfortable, you can build the packages from source.

Before installing the packages, I made sure the VM's hostname resolved through /etc/hosts. The VM already used debian, so I added 192.168.64.11 debian to the end of the hosts file. This allowed both the VM and the Proxmox installer to resolve the hostname.

Update: These packages depended on OpenSSL 1.1, which is now removed from Debian, and so you'll need to build from source and link the packages to OpenSSL 3+

These are the commands I used for the original package installation:

# Add the 3rd party repository
echo "deb https://global.mirrors.apqa.cn/proxmox/debian/pve bullseye port">/etc/apt/sources.list.d/pveport.list

# Download/Trust the key that signs the 3rd party packages
curl https://global.mirrors.apqa.cn/proxmox/debian/pveport.gpg -o /etc/apt/trusted.gpg.d/pveport.gpg 

# pull the latest updates from the repositories, and update the system
apt update && apt full-upgrade

# install ifupdown2, which is required for Proxmox
apt install ifupdown2

# and finally, install Proxmox
apt install proxmox-ve postfix open-iscsi

After running these commands, I rebooted the VM and opened the Proxmox admin interface at https://YOUR_IP_HERE:8006.

Running Proxmox#

The resulting installation used the standard Proxmox interface. Any container or VM still needed an arm64/aarch64 image.

Alternatives considered#

Other options included Parallels, VMware Fusion, or Asahi Linux with VMs running directly on the host. A separate device would avoid running Proxmox inside another hypervisor, but I did not want to carry one when I was away from home. I preferred keeping the environment on my laptop without disrupting my day-to-day desktop.

Results#

On the M1 MacBook Air, I was able to run several VMs and containers without a noticeable performance hit. The experience was better than I expected from an x86_64 system with similar specifications, but I was not running a serious workload.

Terraform could've automated the environment setup, and the Proxmox API could script operations. I plan to keep experimenting with both.

Disclaimer#

Editor's Note: This setup is not recommended for production use. The arm64/aarch64 packages are unofficial and not provided by Proxmox. An M1 MacBook Air is not intended to be a server.

Bye-bye Docker Hub, time to self-host all of my Docker images

Wed, 26 Jun 2024 00:00:00 +0000

Gitea includes a built-in Docker registry, so I can host Docker and other OCI-compliant images without relying on Docker Hub or dealing with its rate limits.

The MVP ("most valuable player") of this whole thing is a tool called skopeo. It is a command-line tool for working with container images and registries, and it does not require root access or a running Docker daemon.

I used skopeo and a small Bash script to migrate some of my Docker images from Docker Hub to Gitea. Gitea enables its package registry by default, so this only covers moving the images. The registry operations are not specific to Gitea, although the script uses the Docker Hub API to discover source repositories.

Skopeo cannot fetch the full list of images in a namespace, so the script uses the Docker Hub API to get that list before passing each image to skopeo. It uses skopeo sync rather than skopeo copy because sync copies all tags without another API call to list them.

#!/bin/bash

## Variables to change
# TODO: these shouldn't be hardcoded, but rather passed in as arguments or via env vars
#       but that's a future problem

# Docker Hub credentials
HUB_NAMESPACE="<namespace>" # the user/org namespace on Docker Hub that you want to bring over
HUB_USERNAME="<username>"
HUB_PASSWORD="<password>"

GITEA_DOMAIN="<domain>" # e.g. gitea.example.com
GITEA_PACKAGE_NAMESPACE="<namespace>" # the user/org that you wish to store the images under
GITEA_USERNAME="<username>" # user that has access to the package registry namespace
GITEA_TOKEN="<password>" # token needs the packages:write permission

# Function to check if a command exists
command_exists() {
    command -v "$1" >/dev/null 2>&1
}

# Check if jq and skopeo are installed
if ! command_exists jq || ! command_exists skopeo; then
    echo "either jq or skopeo is missing, please make sure they are both installed" >&2
    exit 1
fi

# login into Docker Hub and fetch an API token
# its possible to do this without the token, but you'll likely run into ratelimits, and it would also only provide public images
TOKEN=$(curl -s -H "Content-Type: application/json" -X POST -d '{"username": "'${HUB_USERNAME}'", "password": "'${HUB_PASSWORD}'"}' https://hub.docker.com/v2/users/login/ | jq -r .token)

# Get list 100 of images that the namespace has, if you have more than that then pagination will need to be added
REPOS=$(curl -s -H "Authorization: JWT ${TOKEN}" "https://hub.docker.com/v2/repositories/${HUB_NAMESPACE}/?page_size=100" | jq -r '.results[].name')
# TODO: The fetching of images above is Docker Hub API specific, and so if your source is non-Docker Hub this will need to be adjusted

# Loop through images and pass them to skopeo
for repo in $REPOS; do
    echo "Syncing ${HUB_NAMESPACE}/${repo} to ${GITEA_DOMAIN}/${GITEA_PACKAGE_NAMESPACE}/${repo}"
    skopeo sync \
        --src docker --src-creds ${HUB_USERNAME}:${HUB_PASSWORD} \
        --dest docker --dest-creds ${GITEA_USERNAME}:${GITEA_TOKEN} \
        "docker.io/${HUB_NAMESPACE}/${repo}" \
        "${GITEA_DOMAIN}/${GITEA_PACKAGE_NAMESPACE}"
done

echo "Finished moving images!"

Changing HUB_NAMESPACE lets the same script copy another Docker Hub namespace, such as Bitnami.

The script worked for my namespace, which had fewer than 100 images. It still needs argument or environment-variable handling, error handling, and Docker Hub API pagination for larger namespaces. If you modify it, I'd love to hear about it.

If you want to read more about the skopeo sync options, you can check out the skopeo sync documentation.

Disclaimer: I am a maintainer of Gitea. Skopeo can use other registries, so the migration does not require Gitea as the destination.

Let WireGuard clients use your BGP-advertised IP blocks

Sun, 16 Jun 2024 00:00:00 +0000

This is an addendum to my previous post on giving yourself an IPv6 address using WireGuard. If you have an IPv4 range from before the "great IPv4 exhaustion", you can assign addresses from that range to WireGuard clients.

Several cloud providers, including OVH, Vultr, and Equinix Metal, allow customers to advertise their own IP ranges. OVH is the example here because its bandwidth costs are predictable and it does not require a separate BGP connection, which makes setup less work.

Note: If you followed part 1 and added a MASQUERADE rule for your network interface, you will need to remove it when assigning real public IPs from your own range. The MASQUERADE rule rewrites the source IP of outgoing packets to the server's address, which defeats the purpose of giving clients their own public IP. Remove it with iptables -t nat -D POSTROUTING -o <YOUR_NETWORK_INTERFACE> -j MASQUERADE (and the equivalent ip6tables rule if applicable).

With the WireGuard connections from the previous post already set up, assign each client an address from your range. This example uses 8.8.8.0/24, the TEST-NET-1 block reserved for documentation.

# server adjustments to wg0.conf
[Peer]
# change each peer to have a specific "allowed IP" from your block of addresses
AllowedIPs = 8.8.8.1/32  # Replace this documentation address with one from your advertised range.
#...

On the client, make the Interface address match the address defined for that peer on the server:

# client adjustments to wg0.conf
[Interface]
Address = 8.8.8.1/32 # must match the "AllowedIPs" configuration on the server for this peer
# ...

The WireGuard configuration only assigns an address to a peer. Your server must also be able to receive and send traffic for the range. How that is configured depends on whether you maintain a BGP connection with the provider or use a setup such as OVH's, where no separate BGP connection is required.

Editor's Note: If you don't have your own IPv4 range, you can do this with IPv6, which is much easier to get. If you are an EU citizen, I can assist with the process. Send an email to hello at 'this domain without the text part'.

Giving yourself IPv6 with WireGuard

Sat, 15 Jun 2024 00:00:00 +0000

After many years of asking, my Internet Service Provider still doesn't support IPv6. The same goes for many other ISPs. IPv4 exhaustion has pushed some ISPs toward "Carrier-Grade NAT", where many customers share one address. Dedicated IPv4 addresses are also getting more expensive for small websites, and cloud providers have started charging for them. The obvious solution is for ISPs to support IPv6, but that takes work and many customers do not know there is a problem. Keeping the status quo is easier.

I work around this by renting a server in a datacentre with an IPv6 address and tunnelling my traffic through it with WireGuard. This is moderately involved and costs money, but it also gives websites another source of IPv6 traffic. Perhaps that will help show that IPv6 is worth implementing.

You can do this with any server that has an IPv6 address. I'll use one from OVH because it does not charge for bandwidth and has a datacentre near me, which limits the latency added by the tunnel. DigitalOcean, Vultr, Linode, and other providers could also work, but check their bandwidth costs.

I installed WireGuard on my laptop and server. On Debian or Ubuntu, run apt install wireguard on both machines.

Next, I created a public and private key on both machines with the WireGuard tools:

mkdir ~/.wg
cd ~/.wg
wg genkey | tee privatekey | wg pubkey > publickey

This creates a private key and public key in the ~/.wg directory. Next, configure the tunnel on the client and server.

# server config
# /etc/wireguard/wg0.conf
[Interface]
Address = 10.0.0.1/24  # Private IPv4 subnet for the VPN
Address = fd42:42:42::1/64  # Private IPv6 subnet for the VPN
PrivateKey = <private_key_from_server>
ListenPort = 51820

[Peer] # your laptop
PublicKey = <public_key_from_laptop>
AllowedIPs = 10.0.0.2/32, fd42:42:42::2/128

# ... you can add many other peers to this file, just make sure to increment their allowed IPs

# client config
# /etc/wireguard/wg0.conf
[Interface]
PrivateKey = <private_key_from_laptop>
Address = 10.0.0.2/24, fd42:42:42::2/64 # addresses that matches "AllowedIPs" from server config
DNS = 1.1.1.1, 2606:4700:4700::1111 # your choice of DNS server, these are cloudflare's but you could choose your own

[Peer]
PublicKey = <public_key_from_server>
AllowedIPs = 0.0.0.0/0, ::/0 # this says to send all traffic over the tunnel
Endpoint = <server_public_ip>:51820

Now, make sure the server can route IPv6 traffic. This requires iptables rules and system settings.

# as root run the following commands
# the next three commands will let your system packet forward for IPv4 and IPv6
echo 'net.ipv4.ip_forward = 1' | tee -a /etc/sysctl.conf
echo 'net.ipv6.conf.all.forwarding = 1' | tee -a /etc/sysctl.conf
sysctl -p # this will immediately apply the configuration additions

# now we need to let iptables to route the traffic appropriately
# on my machine, my network interface is eno1, but it may be eth0 for you or something else, you can find this using: ip addr
iptables -A FORWARD -i wg0 -j ACCEPT
iptables -A FORWARD -o wg0 -j ACCEPT
iptables -t nat -A POSTROUTING -o <YOUR_NETWORK_INTERFACE> -j MASQUERADE
ip6tables -A FORWARD -i wg0 -j ACCEPT
ip6tables -A FORWARD -o wg0 -j ACCEPT
ip6tables -t nat -A POSTROUTING -o <YOUR_NETWORK_INTERFACE> -j MASQUERADE

# ensure that your server has the wireguard ports open
iptables -A INPUT -p udp --dport 51820 -j ACCEPT
ip6tables -A INPUT -p udp --dport 51820 -j ACCEPT

# don't allow outside traffic to connect back to your machine that you didn't initiate
# this is to prevent your machine becoming world accessible
# if you plan on running a public service, such as a webserver on your laptop, you may need
# to adjust these
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -j DROP
ip6tables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
ip6tables -A INPUT -j DROP

Warning: The INPUT DROP rules above will drop all traffic that is not already established or explicitly allowed. If you are connected to this server over SSH, make sure you add a rule to allow SSH traffic before adding the DROP rules (e.g. iptables -A INPUT -p tcp --dport 22 -j ACCEPT and the equivalent ip6tables rule). Otherwise, you will lock yourself out.

Note: These iptables rules are not persistent across reboots. To make them survive a reboot, you can either use the iptables-persistent package (apt install iptables-persistent and then netfilter-persistent save), or use WireGuard's PostUp and PostDown directives in wg0.conf to apply and remove the rules when the tunnel starts and stops.

Both machines are now ready. Run wg-quick up wg0 on the server and then on the laptop. Run wg show on both machines to check the tunnel.

Editor's Note: Tailscale exit nodes are another way to route the traffic without managing the WireGuard configuration directly.

Instead of Docker-in-Docker, what about Docker-on-Fly?

Fri, 07 Jun 2024 00:00:00 +0000

In my ever-ending quest to do things ~~that are probably not the best idea~~ that aren't a part of a usual use-case, I wanted to see if I could run a docker daemon on Fly.io.

Fly.io uses Firecracker micro-VMs to run services. It "transmogrifies" Docker images for those micro-VMs. This avoids running the Docker daemon inside another Docker container.

After creating a whole Docker image with the Docker daemon and getting it working on Fly, I found that Fly had already made one. Their version is much more succinct than mine, so this post uses it instead. I suspect mine will never see the light of day.

# first clone the repo
git clone https://github.com/fly-apps/docker-daemon.git docker-on-fly
# then enter the directory
cd docker-on-fly
# --build-only because we need to create a volume
fly launch --build-only
# creating a volume with 50gb of storage
fly volumes create data --size 50 --region ams
# run the service
fly deploy
# ssh into the machine (this example runs a container)
fly ssh console -C "docker run --rm hello-world"

You can now do some interesting things on there such as running entire services with docker compose. Instead of translating a docker-compose.yml file into fly.toml, the service can use the existing Compose file.

Using Compose avoids translating the service definition, but it replaces that work with a Docker daemon, a persistent volume, and a network path to maintain. A next step would be a WireGuard tunnel that lets the local Docker client connect to the daemon.

Adding SFTP abilities to Namespace.so's ephemeral instances

Thu, 06 Jun 2024 00:00:00 +0000

Update (February 2026): Namespace has since added native file upload support to their CLI. You can now use nsc instance upload to copy files directly to an instance, making the shell scripts in this post unnecessary. This post is kept for historical purposes.

Namespace.so provides ephemeral Linux and macOS instances, primarily for CI/CD workflows. Like Fly.io, it lets me avoid managing servers.

I use it for some of my CI/CD workflows and wanted to try it for other tasks. One idea is an on-demand nixpkgs builder, similar to my Fly.io setup. With Fly.io I manage the instances myself, while Namespace terminates them after a set period. That reduces the monthly hosting spend.

At the time I wrote this, Namespace did not support uploading files directly to instances. I wanted something I could use immediately, which turned into a small side project.

I asked in the Namespace Discord and was pointed to croc, which others had used. I wanted an option that did not require another tool on the remote server as well as locally. There are plenty of other ways to move the files, but I like making little Bash tools for myself, and this was a good excuse to play around.

Unlike Fly.io, Namespace does not provide direct SSH access, so I needed a way to send files through its existing command interface. The files might be configuration, binaries, or other inputs for whatever I was running.

The nsc client supports pseudo-shell sessions and arbitrary commands. My first pass copied a small file with nsc ssh $machine_id 'echo "hi" > /root/test.txt', but that does not work well for binary files or large text files. Terminal length limits also meant I had to split files into chunks and reassemble them on the remote machine. Each chunk establishes a new connection, so the transfer is slower than a direct copy. Parallelizing those connections can reduce the transfer time, but that was outside the initial scope.

#!/bin/bash

# help text
usage() {
    echo "Usage: $0 -l <local_file> -r <remote_file> -m <machine_id> [-c <chunk_size>]"
    echo ""
    echo "This Script uses the Namespace Client 'nsc' to copy files to your instance"
    echo "Note: You'll need to ensure you've logged in with 'nsc login' first"
    echo ""
    echo "Options:"
    echo "  -l <local_file>   Path to the local file to be transferred"
    echo "  -r <remote_file>  Path to the remote file to be created"
    echo "  -m <machine_id>   Machine ID for the nsc ssh command"
    echo "  -c <chunk_size>   Size of the chunks for splitting the base64 encoded file (default: 1k)"
    echo "  -h, --help        Show this help message and exit"
    exit 1
}

chunk_size="1k" # default chunk size

# parse args
while getopts ":l:r:m:c:h" opt; do
    case ${opt} in
        l )
            local_file=$OPTARG
            ;;
        r )
            remote_file=$OPTARG
            ;;
        m )
            machine_id=$OPTARG
            ;;
        c )
            chunk_size=$OPTARG
            ;;
        h )
            usage
            ;;
        \? )
            usage
            ;;
    esac
done

# check for (full) help flag
for arg in "$@"; do
    if [ "$arg" == "--help" ]; then
        usage
    fi
done

# make sure all args are set
if [ -z "${local_file}" ] || [ -z "${remote_file}" ] || [ -z "${machine_id}" ]; then
    usage
fi

#  base64 encode file and split into chunks
cat $local_file | base64 > /tmp/local_file.b64
split -l $chunk_size /tmp/local_file.b64 /tmp/chunk_

# init remote file
nsc ssh $machine_id "echo -n '' > /tmp/remote_file.b64"

# FIXME: transfer chunks individually, then reassemble them on server (this would allow for parallelization)
# loop over chunks and send each one
for chunk in /tmp/chunk_*; do
    chunk_content=$(cat $chunk)
    nsc ssh $machine_id "echo -n '$chunk_content' >> /tmp/remote_file.b64"
done

# decode base64 file
nsc ssh $machine_id "base64 -d /tmp/remote_file.b64 > $remote_file"

# clean up local temp files
rm /tmp/local_file.b64 /tmp/chunk_*

echo "File transferred successfully."

Run it like this:

# ensure you are logged into namespace.so
nsc login

# create a new ephemeral instance (4 cores, 8gb ram)
machine_id=$(nsc create --machine_type 4x8 --bare --output json | jq -r .cluster_id)

# copy file up to server (assuming you have chmod +x the script already)
./transfer_file.sh -l /home/tklk/Photos/nyan_cat.gif -r /root/nyan.gif -m $machine_id

Another option is to install Tailscale in the ephemeral instance and use tailscale file cp. I left that for another day.

Update:

After writing this post, I added parallelization with GNU parallel to send the chunks to the remote machine. This reduced the transfer time. You need parallel installed on the local machine because several common operating systems do not include it by default. Replace the transfer section of the script above with this version:

nsc ssh $machine_id "mkdir -p /tmp/chunks && echo -n '' > /tmp/remote_file.b64"
send_chunk() {
    chunk=$1
    chunk_name=$(basename $chunk)
    chunk_content=$(cat $chunk)
    nsc ssh $machine_id "echo -n '$chunk_content' > /tmp/chunks/$chunk_name"
}
export -f send_chunk
export machine_id

# use gnu parallel to transfer files
# FIXME: accept -j as an argument to be able to adjust the hardcoded number
#        left as an exercise for the reader
find /tmp/chunk_* | parallel -j 4 send_chunk

# reassemble chunks
nsc ssh $machine_id "cat /tmp/chunks/* > /tmp/remote_file.b64 && rm -r /tmp/chunks"

# based64 decode and write to destination path
nsc ssh $machine_id "base64 -d /tmp/remote_file.b64 > $remote_file && rm /tmp/remote_file.b64"

Build your own CDN - Part 3: Allowing bring-your-own domains with automatic TLS certificates

Wed, 05 Jun 2024 00:00:00 +0000

Part 1 synchronized TLS certificates across the PoPs, and Part 2 distributed Caddy configuration with Nomad. Caddy still needs to request certificates only for names approved by Gitea Pages. Customer domains are not known in advance, so this requires a validation service. The same pattern applies to any SaaS that lets customers bring their own domain.

Caddy's on-demand TLS configuration accepts an ask endpoint. Before requesting a certificate, Caddy sends a GET request with the domain name to that endpoint. A 2xx response approves the request, while any other status code denies it.

The check protects the Let's Encrypt quota from requests for domains outside the application's control. It also prevents someone from pointing an unapproved domain at the infrastructure and receiving a valid TLS certificate.

The Nomad template from Part 2 can add a customer's domain to the Caddy configuration. Adding the name is not enough to establish that the customer controls it. The validation service must make that decision before approving Caddy's request.

The Caddy side of the configuration is small:

{
    on_demand_tls {
        ask https://validation.service.tld/endpoint
    }
}

example.com {
    tls {
        on_demand
    }
    respond "Hello World!"
}

For this configuration, Caddy calls https://validation.service.tld/endpoint with a query string such as ?domain=example.com. The initial request waits while Caddy checks the domain and obtains a certificate, so the validation endpoint should respond quickly.

The service can be a small Go application that reads a file of approved domains or a database lookup inside a larger application. If approval depends on external checks, such as confirming that the domain points to the load balancer, those checks can run asynchronously and cache the result in the application's database instead of running for every Caddy request.

The validation handler depends on where the application stores customer domains and which checks it performs, so there is no single implementation to show here. It can be a separate HTTP service or part of an application that already handles customer and billing data. Caddy only depends on the HTTP response, not the implementation language. An endpoint that returns 200 for every name would remove the protection that the ask check is meant to provide.

Build your own CDN - Part 2: Using Nomad to manage Caddy

Fri, 17 May 2024 00:00:00 +0000

Part 1 covered provisioning TLS certificates and keeping them synchronized across Caddy instances. The next problem I faced was creating Caddy configuration from the services running at each Point of Presence (PoP) and distributing it to the edge nodes. I used HashiCorp Nomad for that.

Like Kubernetes, Nomad schedules containers and non-containerized applications across servers from a shared configuration.

Nomad manages the servers across the CDN's datacentres. The job below runs Caddy on each edge node. Later templates turn Nomad service metadata into Caddy reverse proxy configuration.

The Caddy job specification starts with the edge nodes:

job "caddy" {
  # with a CDN, you'll want datacentres close to the people browsing your data
  # you can define as many PoPs here as you wish
  datacenters = ["dc1", "dc2", "dc3"]
  constraint {
    # this constraint will ensure that the job will only be run on nodes that have
    # a certain hostname, in this case only those that are loadbalancers
    attribute = "${attr.unique.hostname}"
    value     = "edge-lb-.+"
    operator  = "regexp"
  }
  group "loadbalancer" {
    # by default count is 1, meaning only one container will be created
    # however you can use nomad's autoscaler to dynamically change this
    # otherwise if you know the number of nodes in advance, and they will
    # remain fixed, you can hardcode as you wish
    # count = 1
    constraint {
      # ensure that a Caddy servers in the group are scheduled on separate physical hosts
      operator  = "distinct_hosts"
    }
    task "server" {
      driver = "docker"
      config {
        # we'll use the official caddy image, but if you want to use the S3 cert sharing from
        # the previous post, you'll need to build your own image with the caddy plugin included
        image = "caddy:2"
        # to allow Caddy to bind directly to any port for the host, instead of using docker port forwarding
        # you can give caddy access to the host namespace
        network_mode = "host"
      }
    }
  }
}

The job creates a Caddy instance on every node whose hostname matches edge-lb-.+, which supports multiple load balancers in each datacentre. At this point Caddy still serves its default landing page. It does not proxy application requests or request TLS certificates.

Serving applications and TLS certificates requires a Caddyfile template in the Nomad job.

Nomad renders template stanzas into files mounted in the container:

...
task "server" {
config {
    ...
    # mount in the generated caddy configuration as a ready only volume
    # the path used, is the path to the generated files from the below template stanzas
    mount {
        type     = "bind"
        source   = "..${NOMAD_ALLOC_DIR}/../server/caddy"
        target   = "/etc/caddy"
        readonly = true
    }
}

template {
        data = <<EOH
# caddy configuration goes here
# this is hardcoded configuration that responds with the hostname of the node that is handling the response
:80 {
  respond "Hello World from {{ env "node.unique.id" }}!"
}
        EOH
        # where to write out the configuration
        destination = "caddy/Caddyfile"
    }
}

Nomad templates can use variables and service data. The next template generates Caddy configuration from endpoints for services managed by Nomad:

... Caddyfile template

{{range service "nomad-example-app"}}
{{index .ServiceMeta "domain"}} {
  tls {
    on_demand
  }
  header {
    X-Balance "{{ env "node.unique.id" }}"
  }
  reverse_proxy {{.NodeAddress}}:{{.Port}}
}
{{end}}

The template reads each nomad-example-app service, uses its domain metadata for the Caddy site address, and proxies traffic to that service's node address and port. These connections use the network attached to the Docker bridge. If the published IPs are public, firewall rules need to prevent requests from bypassing Caddy.

The application job also needs a port for the container. In this example, Nomad maps container port 80 to a random high port on the host. Its service metadata includes the domain that Nomad uses for the generated Caddy block.

group "nomad-example-app" {

    network {
      port "http"  { to = 80 }
      mode = "bridge"
    }
    service {
      name = "nomad-example-app"
      port = "http"
      meta {
        domain = "nomad-app.example.com"
      }
    }
    task "server" {
      driver = "docker"

      config {
        image = "traefik/whoami:latest"
        ports = ["http"]
...

Nomad updates the Caddy configuration as services appear or disappear.

By default, each configuration change restarts the Caddy container. A full restart can be undesirable for a production load balancer. Setting change_mode = "signal" and change_signal = "SIGUSR1" on the template stanza sends a signal instead. Caddy handles SIGUSR1 with a graceful configuration reload, which avoids the downtime of a full container restart.

Possible extensions include health checks, services created through GitOps or the Nomad API, and Consul Connect for mTLS between Caddy and the application.

For Gitea Pages, I plan to run one container for each static site in every region and create a Nomad service for each site. Each Caddy instance would proxy to the site container in its own region, while the sites remain loosely coupled. When a new static-site container is published, Nomad would update the services globally.

Part 3 covers Caddy's on-demand certificates for customer-provided domains and the validation endpoint that controls those requests.

Using Fly.io as a remote builder for nixpkgs

Mon, 13 May 2024 00:00:00 +0000

A service I enjoy using, and write about often, is fly.io, as it lets me have compute on demand, without having to worry about provisioning and managing an entire server.

Editor's Note: This was a fun experiment, but you are likely better off using a managed service such as nixbuild.net for anything even resembling a production use case.

One of my latest use cases for it is offloading building various nixpkgs from my low-powered laptop to a remote Fly.io builder. I can give the builder more resources for the period when I need them.

Start by creating the builder machine. The default has 256 MB of RAM, while this example requests 2 GB and four CPUs. Adjust those values for the package you are building. Fly machines can be created from container images, so this setup uses the same nixos/nix image as Docker.

fly launch --name tklk-nixbuilder --region ams -o personal --build-only --image nixos/nix --no-public-ips  --vm-cpus 4 --vm-memory 2048

The --build-only flag keeps Fly from starting the machine before the remaining configuration is in place.

The builder needs additional storage because building a package can pull in more of its dependency graph. Create a volume for the Nix store:

fly volumes create nix_store -a tklk-nixbuilder -s 50 -r ams -y

This example uses a 50 GB volume. Fly bills for the storage regardless of whether the machine is powered on, so choose the size based on the expected store. Set the mount path in fly.toml:

[mounts]
  source="nix_store"
  destination="/data/nix"

Do not mount the empty volume at /nix yet. It would hide the existing Nix store, leaving the machine without the binaries it needs. Mount it at /data/nix first, copy the existing store, and then change the destination to /nix.

Some internal Fly.io processes look for the /bin/sleep binary. NixOS stores binaries under hashed /nix/store paths instead of /bin, so that path does not exist. Either create a symlink or set the command in the configuration. This example sets the command explicitly:

[experimental]
  cmd = ['nix-shell', '-p', 'coreutils', '--run', 'sleep inf']

Deploy the machine after the configuration is in place:

fly deploy -a tklk-nixbuilder

After the machine starts, SSH into it to finish the setup:

fly ssh console -a tklk-nixbuilder

First, copy the contents of the nix store into the mounted volume.

cp -a /nix/. /data/nix/

Back on your host machine, change the destination path of the mounted volume in fly.toml to /nix. Then restart the machine with fly deploy -a tklk-nixbuilder.

Generate an SSH key for the remote connection with fly ssh issue:

fly ssh issue -o personal -u root /home/tklk/.ssh/fly_key

Add the following entry to ~/.ssh/config to avoid repeating the connection options:

Host nixbuilder.fly
	StrictHostKeyChecking no
	UserKnownHostsFile=/dev/null
	Port 2200
	User root
	HostName localhost
	IdentityFile ~/.ssh/fly_key

Editor's Note: This SSH entry disables host key checking for nixbuilder.fly. That avoids prompts for the local proxy endpoint, but it also removes SSH host verification for this entry.

Start a local proxy from port 2200 to port 22 on the remote builder. This avoids configuring a WireGuard VPN connection to Fly.io, which is more involved than this port forward. The trailing & runs the proxy in the background. If the terminal closes, the proxy stops and must be restarted before reconnecting.

fly proxy 2200:22 &

With the proxy running, test the remote builder by adding --store ssh-ng://nixbuilder.fly to a nix build command.

Example:

nix build nixpkgs#mercurial --store ssh-ng://nixbuilder.fly --show-trace

The Fly builder uses x86-64 so if your local machine is aarch64, pass --system x86_64-linux to nix-build so the remote builder can build the package. You can copy the x86-64 output back, but it will not run natively on the Arm machine. For aarch64 builds, nixbuild.net supports aarch64 builders, though not macOS builders as of this writing.

Build your own CDN - Part 1: Sync TLS certificates across your PoPs with Caddy

Tue, 31 Oct 2023 00:00:00 +0000

Gitea Pages will need many unique TLS certificates, and from past experience I know that a single Amazon CloudFront distribution limits how many certificates it can hold. I am building a Content Delivery Network (CDN) from scratch to be able to handle that constraint.

The first task is keeping TLS certificates synchronized across the CDN's Points of Presence (PoPs), because the servers spread across the globe close to be able to serve the end users in a locatin close to them. I chose Caddy because it has built-in Let's Encrypt/ACME support and can obtain and renew TLS certificates automatically.

Caddy lets me use a plugin I wrote, certmagic-s3, so each instance can share certificates through an S3 bucket. Any PoP can obtain or renew a certificate, and the other PoPs can retrieve the update from the bucket.

This removes the central certificate distributor, but it makes the S3 bucket and custom Caddy binary part of the setup. Each PoP needs credentials for the bucket, and each Caddy upgrade needs another binary that includes the plugin. You can build Caddy with xcaddy, or use Caddy's build service to download a compiled binary. In the build service, select certmagic-s3, choose a platform, and download the resulting binary.

That binary can load the plugin from a Caddyfile and obtain TLS certificates from Let's Encrypt:

{
        email webmaster@example.com  # The email associated with your Let's Encrypt account
        storage s3 {  # Configuring S3 as the storage backend
            host minio.example.com  # Your S3-compatible storage host
            bucket certmagic-s3  # Bucket where certificates will be stored
            access_key ABC123  # Your S3 access key
            secret_key XYZ789  # Your S3 secret key
            prefix "byoc"  # Optional path prefix within the bucket
        }
}
site.example.com {  # Domain to serve
    tls {
        on_demand  # Obtain TLS certificates on first HTTP request instead of on start
        # Note: you should also configure an `ask` endpoint in the global `on_demand_tls`
        # block to prevent abuse of on-demand certificate issuance. See Part 3 of this
        # series for details on setting up a validation service.
    }
    respond "hello world"  # Sample response
}

After configuring and starting Caddy, the first request for the site will cause Caddy to obtain the certificate from Let's Encrypt and store it in the S3 bucket. A second Caddy instance with the same storage configuration can retrieve the certificate from the bucket and serve the site.

Part 2 covers distributing this Caddy configuration across the PoPs with Nomad. Because this configuration uses on_demand, the deployment also needs an ask endpoint so unapproved domains cannot consume certificate issuance. Part 3 covers that validation step.

Using Nix with Gitea Actions

Sun, 03 Sep 2023 00:00:00 +0000

Carl Sagan once said, "If you wish to make an apple pie from scratch, you must first invent the universe." A reproducible build does not need to go quite that far. Nix can provide the build environment inside Gitea Actions with a small addition to the workflow.

I use Nix and maintain several packages. I appreciate the reproducible build environments it provides and how quickly packages are updated.

The workflow below installs Nix in a Gitea Actions job and runs hello from nix-shell:

name: nix

on:
  push:

jobs:
  lint-backend:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install dependencies for Nix setup action
        run: |
          apt update -y
          apt install sudo -y          
      - uses: cachix/install-nix-action@v27
        with:
          nix_path: nixpkgs=channel:nixos-unstable
      - name: Test running command with Nix
        run: nix-shell -p hello --run "hello"

The default Gitea Actions runner image does not include sudo, which cachix/install-nix-action requires, so the workflow installs it first. A custom runner that already includes sudo can skip that step.

I prefer to live on the bleeding edge with nixpkgs=channel:nixos-unstable, but you can pin a more stable channel instead. The cachix/install-nix-action Action does not configure a channel by default, so the workflow must specify one.

Installing Nix adds setup time and requires sudo on this runner image. In return, the job can use Nix package definitions instead of reproducing the environment with runner-specific installation commands.

Editor's Note: Using DetSys' Nix installer action can speed up the install of nix in your workflows, and also reduce the customization of the runner needed.

Watchtower... but for Kubernetes!?!

Thu, 18 May 2023 00:00:00 +0000

Watchtower runs on a schedule and checks for newer Docker container images. When it finds one, it pulls the image and recreates the container. Keel applies the same approach to Kubernetes workloads, including Helm releases, DaemonSets, StatefulSets, and Deployments. It also provides a UI for viewing the updates it manages.

Installing Keel#

Install Keel in the cluster with kubectl apply:

export KEEL_NAMESPACE=keel
export KEEL_ADMIN_USER=keel
export KEEL_ADMIN_PASS=keel
kubectl apply -f https://sunstone.dev/keel?namespace=$KEEL_NAMESPACE\&username=$KEEL_ADMIN_USER\&password=$KEEL_ADMIN_PASS\&tag=latest

Configuring deployments for Keel#

Add Keel labels to the Kubernetes Deployment specification. Keel uses SemVer (Semantic Versioning), and its policies can be all, major, minor, or patch:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    keel.sh/policy: major
...

The major policy tells Keel to update the Deployment when a new major version of the container image is available.

Handling private images and rate limits#

If you use private images or Docker Hub with its rate limit, Keel needs registry credentials. Keel can use the image pull secrets already configured in Kubernetes, so no additional secret is required.

Keel UI#

Keel's UI shows the status of deployments and the updates it manages. Expose it through a Kubernetes ingress or use kubectl port-forward:

kubectl -n keel port-forward service/keel 9300

Update tradeoffs#

Automatic image updates save time and keep containers on newer images, which may include security fixes. They still require a rollback strategy and test pipeline.

Publishing a static site to Bunny.net using Gitea Actions

Tue, 18 Apr 2023 00:00:00 +0000

Bunny.net (formerly BunnyCDN) can host a static site by serving files from a storage zone through a pull zone. The workflow below uses Gitea Actions to build a Hugo site, upload it over FTP, and purge the CDN cache after each deployment.

Creating a storage zone#

Create a storage zone in the Bunny.net dashboard and give it a name that you can associate with the site. Select the regions where Bunny.net should replicate the content. More regions can make the site faster in those locations. They also cost more.

Open the storage zone's FTP credentials page and keep the host, username, and password for the workflow.

Creating a pull zone#

Create a pull zone in the Bunny.net dashboard, give it a recognizable name, and set the storage zone as its "origin". Select the regions from which the CDN should serve the site. For a custom domain, add it to the pull zone and create a CNAME record at your DNS provider that points to the Bunny.net pull zone.

Uploading the site#

This Gitea Actions workflow builds the Hugo site, mirrors the generated public/ directory to the storage zone with its FTP credentials, and purges the pull zone cache. It can be adapted to another CI system.

# .gitea/workflows/hugo-build.yml
name: Build and Deploy to BunnyCDN

on:
  push:
    branches:
      - main

jobs:
  bunnycdn:
    name: bunnycdn-publish
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          submodules: true  # Fetch Hugo themes (true OR recursive)
          fetch-depth: 0    # Fetch all history for .GitInfo and .Lastmod

      - name: Setup Hugo
        uses: peaceiris/actions-hugo@v2
        with:
          hugo-version: '0.126.1'
          extended: true

      - name: Build
        run: hugo --minify

      - name: Deploy to BunnyCDN
        run: |
          apt update -y
          apt install -y lftp
          lftp -e "
            set ftp:ssl-allow true;
            set ftp:ssl-protect-data true;
            set ssl:verify-certificate no;
            open ${{ secrets.BUNNYCDN_FTP_HOST }};
            user ${{ secrets.BUNNYCDN_FTP_USER }} ${{ secrets.BUNNYCDN_FTP_PASSWORD }};
            mirror -R --delete -v public/ .;
            bye;
          "
      - name: purge bunny cache
        run: |
          curl --request POST \
            --url https://api.bunny.net/pullzone/${{ secrets.BUNNYCDN_ZONE_ID }}/purgeCache \
            --header 'AccessKey: ${{ secrets.BUNNYCDN_API_KEY }}' \
            --header 'content-type: application/json'

Set the BUNNYCDN_FTP_HOST, BUNNYCDN_FTP_USER, and BUNNYCDN_FTP_PASSWORD secrets from the storage zone's FTP credentials page. Hugo builds the site into public/, which lftp recursively mirrors to the root of the storage zone. The --delete option removes remote files that are no longer in public/, and -v prints verbose transfer output.

Set BUNNYCDN_ZONE_ID and BUNNYCDN_API_KEY secrets in your Action's settings found on the Bunny.net dashboard. The final workflow step uses them to purge the pull zone cache after the upload.

The pull zone URL serves the uploaded site. If you configured a custom domain, it serves the same content. Caching, compression, and auto-HTTPS can be configured on the pull zone.

Note: Bunny.net is rebuilding its dashboard, so these steps may not match the current dashboard. The underlying process should remain largely the same.

Secure SSH Access with SSH Certificates Managed by HashiCorp's Vault

Sun, 16 Apr 2023 00:00:00 +0000

SSH user certificates can restrict which users may connect and which commands they may run. In the setup below, the server trusts a Certificate Authority (CA) public key and accepts user certificates signed by that CA. This does not replace host key verification on the client. Avoiding per-server host fingerprints requires a separate host certificate setup and client-side trust in the host CA. Vault can issue and revoke certificates, manage SSH keys, and provide audit logs. Vault can also manage host certificates to also prevent that TOFU issue.

Warning: This post describes a non-production Vault setup. It is not hardened and should only be used to learn about SSH CAs and Vault.

Install Vault#

To quickly set up a development Vault server, use the official Docker image with the following command:

docker run --cap-add=IPC_LOCK -e 'VAULT_DEV_ROOT_TOKEN_ID=dev-token' -e 'VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200' -p 8200:8200 hashicorp/vault server -dev

Configure the SSH certificate authority#

With Vault installed and running, configure it to issue SSH certificates using the SSH secrets engine.

First, set the Vault address and token for the dev server:

export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='dev-token'

Then, run the following command to enable the SSH secrets engine

vault secrets enable ssh

Next, you will need to configure the SSH CA and have it generate signing keys on your behalf to ensure that the private key stays within Vault itself.

vault write ssh/config/ca generate_signing_key=true

You can get the generated SSH public key for use later, with:

vault read -field=public_key ssh/config/ca > ssh_ca_key.pub

After enabling the SSH CA in Vault, you'll need to create a role called ops-team to issue SSH certificates. This role allows any user with access to request an SSH certificate. The example below grants broad permissions, including any option for allowed_users and port forwarding. Be sure to restrict these permissions based on your use case.

vault write ssh/roles/ops-team \
    key_type=ca \
    ttl=2h \
    max_ttl=24h \
    allow_user_certificates=true \
    allowed_users="*" \
    default_extensions='{"permit-pty":"","permit-port-forwarding":"","permit-agent-forwarding":""}'

Configure the remote server to accept the SSH certificates issued by Vault:

Copy the CA public key to the remote server:

scp ssh_ca_key.pub <username>@<target_server_ip>:/tmp/ssh_ca_key.pub

Add the public key to the OpenSSH configuration and restart the OpenSSH daemon:

echo "TrustedUserCAKeys /etc/ssh/user_ca.pub" | sudo tee -a /etc/ssh/sshd_config
sudo cp /tmp/ssh_ca_key.pub /etc/ssh/user_ca.pub
sudo systemctl restart sshd

Request SSH certificates#

To request an SSH certificate from Vault and use it to SSH into the remote server, follow these steps:

Use the ops-team role to request the certificate and pass your local SSH key id_rsa.pub. Also, specify the username to use when connecting to the remote server:

vault write -field=signed_key ssh/sign/ops-team \
    public_key=@$HOME/.ssh/id_rsa.pub \
    valid_principals="<username>" > signed_id_rsa-cert.pub

Use the signed_id_rsa-cert.pub file to SSH into the remote server:

ssh -i signed_id_rsa-cert.pub -i $HOME/.ssh/id_rsa <username>@<target_server_ip>

Requesting a signed certificate manually each time can be tedious. To simplify this process, create a script called vault-ssh.sh and make it executable with chmod +x vault-ssh.sh.

#!/bin/bash

# Configuration
export VAULT_ADDR="http://<vault_server_ip>:8200"
VAULT_ROLE="ops-team"
USERNAME="<username>"
PUBLIC_KEY_PATH="$HOME/.ssh/id_rsa.pub"
CERT_PATH="$HOME/.ssh/id_rsa-cert.pub"
CONFIG_FILE="/path/to/vault-creds.conf"

# Read the Vault token from the configuration file
if [ -f "$CONFIG_FILE" ]; then
  source "$CONFIG_FILE"
else
  echo "Error: Vault configuration file not found"
  exit 1
fi

# Check if the VAULT_TOKEN variable is set
if [ -z "$VAULT_TOKEN" ]; then
  echo "Error: VAULT_TOKEN is not set in the configuration file"
  exit 1
fi
export VAULT_TOKEN

# Generate a new SSH certificate
vault write -field=signed_key ssh/sign/$VAULT_ROLE \
  public_key=@$PUBLIC_KEY_PATH \
  valid_principals="$USERNAME" > $CERT_PATH

This script requires a vault-creds.conf file containing the Vault token:

VAULT_TOKEN=<vault_token>

To integrate the certificate generation process with your SSH config, use the ProxyCommand configuration option, which allows you to run a custom command (like the script) as a "proxy" for the actual SSH connection.

Add the following to your SSH config:

Host *
  IdentityFile ~/.ssh/id_rsa
  CertificateFile ~/.ssh/id_rsa-cert.pub
  ProxyCommand bash -c '/path/to/vault-ssh.sh && nc %h %p'

Keep in mind that this approach generates a new SSH certificate for every connection. Depending on the frequency of your connections and the TTL of your certificates, you might want to modify the vault-ssh.sh script to check the current certificate's validity and generate a new one only if necessary.

Credits: This post draws on my experience using Vault and the official Vault Docker documentation for the development server configuration.

Playing around with Gitea Actions on Fly.io

Mon, 03 Apr 2023 00:00:00 +0000

Fly.io is a "serverless" hosting platform usually used for web services, but it can also run long-running tasks. I wanted to try running the Gitea Actions runner there.

Treat this as a proof of concept. There are better ways to do this, but it still was fun to try.

To keep the experiment simple, I ran the runner in "host" mode. Jobs run directly on the host rather than in containers, which kept Docker-in-Docker debugging out of the initial setup.

To get started, I created a new Fly.io app with the following configuration:

# fly.toml
app = "actions-on-fly"
primary_region = "ams"

[[mounts]]
  destination = "/data"
  source = "data"

I mounted a persistent volume to /data so the runner registration persists across restarts.

Since there were no prebuilt Docker images as of publication, I created one and installed the runner in it. The Dockerfile is as follows:

# Dockerfile
FROM ghcr.io/catthehacker/ubuntu:act-latest
# the FROM image is based on ubuntu and has appropriate tools installed to run Gitea Actions

# install act_runner
RUN curl https://dl.gitea.com/act_runner/nightly/act_runner-nightly-linux-amd64 > /usr/local/bin/act_runner && \
    chmod +x /usr/local/bin/act_runner

# add start script
ADD start.sh /start.sh
RUN chmod +x /start.sh && mkdir -p /data
ENTRYPOINT ["/start.sh"]

At startup, the container checks whether the runner is already registered and registers it if needed. The registration token comes from an environment variable, and then the runner starts.

#!/bin/bash
# start.sh

# $ACTIONS_REGISTER_TOKEN is the registration token for the runner that is given by the Gitea runner settings page.

# set /data as the working dir
cd /data

# check if runner is already registered, and if not register it
if [ ! -f .runner ]; then
  # register runner on gitea.com, and set label as fly-runner so it runs as "host" mode
  act_runner register --no-interactive --instance "https://gitea.com" --labels "fly-runner" --token $ACTIONS_REGISTER_TOKEN
fi

# start runner
act_runner daemon

It took only a handful of lines to get the runner working on Fly.io. The issue I ran into was memory: Fly.io terminates apps when they run out of memory. The workload I was testing needed more memory, so I increased the limit. Memory-heavy jobs may therefore need a larger machine than the default configuration.

This remains a proof of concept rather than a recommendation. The blog you are reading right now is built using this runner.

Editor's Note: This blog is no longer built using this approach, but still uses Gitea Actions to build and publish. The post is kept as historical reference.