Build your own CDN - Part 2: Using Nomad to manage Caddy

Part 1 covered provisioning TLS certificates and keeping them synchronized across Caddy instances. The next problem I faced was creating Caddy configuration from the services running at each Point of Presence (PoP) and distributing it to the edge nodes. I used HashiCorp Nomad for that.

Like Kubernetes, Nomad schedules containers and non-containerized applications across servers from a shared configuration.

Nomad manages the servers across the CDN's datacentres. The job below runs Caddy on each edge node. Later templates turn Nomad service metadata into Caddy reverse proxy configuration.

The Caddy job specification starts with the edge nodes:

job "caddy" {
  # with a CDN, you'll want datacentres close to the people browsing your data
  # you can define as many PoPs here as you wish
  datacenters = ["dc1", "dc2", "dc3"]
  constraint {
    # this constraint will ensure that the job will only be run on nodes that have
    # a certain hostname, in this case only those that are loadbalancers
    attribute = "${attr.unique.hostname}"
    value     = "edge-lb-.+"
    operator  = "regexp"
  }
  group "loadbalancer" {
    # by default count is 1, meaning only one container will be created
    # however you can use nomad's autoscaler to dynamically change this
    # otherwise if you know the number of nodes in advance, and they will
    # remain fixed, you can hardcode as you wish
    # count = 1
    constraint {
      # ensure that a Caddy servers in the group are scheduled on separate physical hosts
      operator  = "distinct_hosts"
    }
    task "server" {
      driver = "docker"
      config {
        # we'll use the official caddy image, but if you want to use the S3 cert sharing from
        # the previous post, you'll need to build your own image with the caddy plugin included
        image = "caddy:2"
        # to allow Caddy to bind directly to any port for the host, instead of using docker port forwarding
        # you can give caddy access to the host namespace
        network_mode = "host"
      }
    }
  }
}

The job creates a Caddy instance on every node whose hostname matches edge-lb-.+, which supports multiple load balancers in each datacentre. At this point Caddy still serves its default landing page. It does not proxy application requests or request TLS certificates.

Serving applications and TLS certificates requires a Caddyfile template in the Nomad job.

Nomad renders template stanzas into files mounted in the container:

...
task "server" {
config {
    ...
    # mount in the generated caddy configuration as a ready only volume
    # the path used, is the path to the generated files from the below template stanzas
    mount {
        type     = "bind"
        source   = "..${NOMAD_ALLOC_DIR}/../server/caddy"
        target   = "/etc/caddy"
        readonly = true
    }
}

template {
        data = <<EOH
# caddy configuration goes here
# this is hardcoded configuration that responds with the hostname of the node that is handling the response
:80 {
  respond "Hello World from {{ env "node.unique.id" }}!"
}
        EOH
        # where to write out the configuration
        destination = "caddy/Caddyfile"
    }
}

Nomad templates can use variables and service data. The next template generates Caddy configuration from endpoints for services managed by Nomad:

... Caddyfile template

{{range service "nomad-example-app"}}
{{index .ServiceMeta "domain"}} {
  tls {
    on_demand
  }
  header {
    X-Balance "{{ env "node.unique.id" }}"
  }
  reverse_proxy {{.NodeAddress}}:{{.Port}}
}
{{end}}

The template reads each nomad-example-app service, uses its domain metadata for the Caddy site address, and proxies traffic to that service's node address and port. These connections use the network attached to the Docker bridge. If the published IPs are public, firewall rules need to prevent requests from bypassing Caddy.

The application job also needs a port for the container. In this example, Nomad maps container port 80 to a random high port on the host. Its service metadata includes the domain that Nomad uses for the generated Caddy block.

group "nomad-example-app" {

    network {
      port "http"  { to = 80 }
      mode = "bridge"
    }
    service {
      name = "nomad-example-app"
      port = "http"
      meta {
        domain = "nomad-app.example.com"
      }
    }
    task "server" {
      driver = "docker"

      config {
        image = "traefik/whoami:latest"
        ports = ["http"]
...

Nomad updates the Caddy configuration as services appear or disappear.

By default, each configuration change restarts the Caddy container. A full restart can be undesirable for a production load balancer. Setting change_mode = "signal" and change_signal = "SIGUSR1" on the template stanza sends a signal instead. Caddy handles SIGUSR1 with a graceful configuration reload, which avoids the downtime of a full container restart.

Possible extensions include health checks, services created through GitOps or the Nomad API, and Consul Connect for mTLS between Caddy and the application.

For Gitea Pages, I plan to run one container for each static site in every region and create a Nomad service for each site. Each Caddy instance would proxy to the site container in its own region, while the sites remain loosely coupled. When a new static-site container is published, Nomad would update the services globally.

Part 3 covers Caddy's on-demand certificates for customer-provided domains and the validation endpoint that controls those requests.