Build your own CDN - Part 1: Synch TLS certificates across all your PoPs with Caddy

As a part of building a Content Delivery Network (CDN) for Gitea Pages, I’m documenting my process to share my experience with others. The reason I’m building one from scratch, rather than using a pre-built solution like Amazon CloudFront, is the potential need for many unique TLS certificates. From past experience, I’ve learned that there are limits to how many certificates you can have on a single CloudFront distribution. This is the first post in a series that will document the process of building a CDN from scratch.

The initial task I’ll tackle is synchronizing TLS certificates across all the Points of Presence (PoPs) in the CDN. If you’re unfamiliar with what PoPs are, they are geographically distributed servers located near where end users are to improve content delivery performance. I’ve chosen to use Caddy for this purpose. Caddy is a web server that has built-in support for Let’s Encrypt and can automatically obtain and renew TLS certificates.

While there are alternative approaches using other web servers like nginx, where a central server obtains the TLS certificates and then distributes them to the other servers, Caddy offers a more decentralized approach. I can use a plugin I wrote called certmagic-s3, that lets each instance of Caddy share the TLS certificates via an S3 bucket. This approach has the advantage that any of the PoPs can obtain and renew the TLS certificates, and the other PoPs will automatically receive the updated certificates.

The most challenging part of this approach is to ensure that the plugin is properly installed in Caddy. You could use Caddy’s xcaddy build tool, but Caddy also offers a build service where you can download binaries that have already been compiled. Using that build service, select certmagic-s3 as a plugin to include, download it for your platform of choice, and you’ll have a Caddy binary with the plugin already installed.

Now, using that binary, you can create a Caddyfile to configure Caddy to use the plugin and obtain TLS certificates from Let’s Encrypt. Here’s an example Caddyfile:

{
        email webmaster@example.com  // The email associated with your Let's Encrypt account
        storage s3 {  // Configuring S3 as the storage backend
            host minio.example.com  // Your S3-compatible storage host
            bucket certmagic-s3  // Bucket where certificates will be stored
            access_key ABC123  // Your S3 access key
            secret_key XYZ789  // Your S3 secret key
            prefix "byoc"  // Optional path prefix within the bucket
        }
}
site.example.com {  // Domain to serve
    tls {
        on_demand  // Obtain TLS certificates on first HTTP request instead of on start
    }
    respond "hello world"  // Sample response
}

With the required information filled out from the configuration above, you can start Caddy and it will obtain the TLS certificates from Let’s Encrypt and store them in the S3 bucket. If you then start another instance of Caddy with the same configuration, it will automatically obtain the TLS certificates from the S3 bucket and serve the site.

In upcoming posts I will describe how I setup nomad to distribute Caddy config to all the PoPs, and how I am using Caddy to serve custom dynamic domains for Gitea Pages. Stay tuned for more details on building out the CDN.