Build your own CDN - Part 1: Sync TLS certificates across your PoPs with Caddy

Gitea Pages will need many unique TLS certificates, and from past experience I know that a single Amazon CloudFront distribution limits how many certificates it can hold. I am building a Content Delivery Network (CDN) from scratch to be able to handle that constraint.

The first task is keeping TLS certificates synchronized across the CDN's Points of Presence (PoPs), because the servers spread across the globe close to be able to serve the end users in a locatin close to them. I chose Caddy because it has built-in Let's Encrypt/ACME support and can obtain and renew TLS certificates automatically.

Caddy lets me use a plugin I wrote, certmagic-s3, so each instance can share certificates through an S3 bucket. Any PoP can obtain or renew a certificate, and the other PoPs can retrieve the update from the bucket.

This removes the central certificate distributor, but it makes the S3 bucket and custom Caddy binary part of the setup. Each PoP needs credentials for the bucket, and each Caddy upgrade needs another binary that includes the plugin. You can build Caddy with xcaddy, or use Caddy's build service to download a compiled binary. In the build service, select certmagic-s3, choose a platform, and download the resulting binary.

That binary can load the plugin from a Caddyfile and obtain TLS certificates from Let's Encrypt:

{
        email webmaster@example.com  # The email associated with your Let's Encrypt account
        storage s3 {  # Configuring S3 as the storage backend
            host minio.example.com  # Your S3-compatible storage host
            bucket certmagic-s3  # Bucket where certificates will be stored
            access_key ABC123  # Your S3 access key
            secret_key XYZ789  # Your S3 secret key
            prefix "byoc"  # Optional path prefix within the bucket
        }
}
site.example.com {  # Domain to serve
    tls {
        on_demand  # Obtain TLS certificates on first HTTP request instead of on start
        # Note: you should also configure an `ask` endpoint in the global `on_demand_tls`
        # block to prevent abuse of on-demand certificate issuance. See Part 3 of this
        # series for details on setting up a validation service.
    }
    respond "hello world"  # Sample response
}

After configuring and starting Caddy, the first request for the site will cause Caddy to obtain the certificate from Let's Encrypt and store it in the S3 bucket. A second Caddy instance with the same storage configuration can retrieve the certificate from the bucket and serve the site.

Part 2 covers distributing this Caddy configuration across the PoPs with Nomad. Because this configuration uses on_demand, the deployment also needs an ask endpoint so unapproved domains cannot consume certificate issuance. Part 3 covers that validation step.