Caddy

Introduction to Caddy

Caddy is a simple yet powerful open-source web server with automatic HTTPS. What makes Caddy so powerful is its plug-in ecosystem. You start with a simple Caddy server and, based on your needs, customise it by adding plugins. Out of the tools I have tried/considered (Nginx or Traefik), Caddy is the simplest option.

In the infra-bootstrap-tools setup, Caddy serves as a crucial reverse proxy and authentication gateway for applications deployed within the Docker Swarm cluster. It simplifies exposing services to the internet, handles SSL/TLS certificate management, and enforces access control.

Deployment Process

Caddy is deployed as a Docker Swarm service using the docker_swarm_app_caddy Ansible role. This role automates the entire setup process.

Key actions performed by the Ansible role include:

  • Copying Essential Files:
    • Caddyfile: The main configuration file for Caddy, defining how requests are handled, reverse proxy rules, and authentication policies.
    • Dockerfile: Used to build a custom Caddy image that includes necessary plugins.
    • caddy-stack.yml: A Docker Compose file that defines the Caddy service, its network, and dependencies.
  • Managing Caddyfile as a Docker Swarm Config: The Caddyfile is managed as a Docker Swarm configuration object. This allows for dynamic updates to the Caddy configuration without restarting the service. The utils_rotate_docker_configs Ansible role is utilized for this purpose, ensuring versioned and controlled rollout of configuration changes.
  • Managing Sensitive Data as Docker Swarm Secrets: Sensitive information required by Caddy and its plugins is securely managed as Docker Swarm secrets. This includes:
    • GitHub OAuth credentials (CADDY_GITHUB_CLIENT_ID, CADDY_GITHUB_CLIENT_SECRET) for the authentication portal.
    • DigitalOcean API token (CADDY_DIGITALOCEAN_API_TOKEN) for DNS challenges (if using DigitalOcean for DNS).
    • JWT signing key (CADDY_JWT_SHARED_KEY) for securing authentication tokens. The utils_rotate_docker_secrets Ansible role handles the creation and rotation of these secrets.
  • Deploying the Caddy Stack: The Caddy service itself is deployed as a Docker stack using the docker_stack Ansible module, based on the caddy-stack.yml file.

Custom Caddy Image

A custom Caddy image is built and used to ensure all necessary functionalities are available. The image is ghcr.io/xnok/infra-bootstrap-tools-caddy:main and includes the following key plugins:

  • github.com/lucaslorentz/caddy-docker-proxy/v2: This plugin enables Caddy to dynamically discover and configure reverse proxying for Docker Swarm services based on labels applied to those services.
  • github.com/greenpau/caddy-security: This plugin provides robust authentication and authorization capabilities, including support for OAuth/OIDC providers like GitHub, JWT, and more.
  • github.com/mholt/caddy-dynamicdns: This plugin allows Caddy to automatically update DNS records, which is useful for dynamic IP environments or for managing DNS records for services it exposes.

Securing Applications with Caddy and Docker Labels

The core concept for exposing and securing applications with Caddy in this environment revolves around Docker Swarm service labels. The caddy-docker-proxy plugin continuously monitors Docker Swarm for services with specific labels and automatically configures Caddy to act as a reverse proxy for them.

To expose a service through Caddy, you define labels on your other Docker Swarm services (not on Caddy itself). These labels instruct Caddy on how to route traffic to your application.

Here are the two most common labels:

  • caddy: example.com: This label defines which domain or domain your application will be accessible from.
  • caddy.reverse_proxy: "{{ upstreams 80 }}": This label indicates to Caddy that we want to serve the port 80 of our application via reverse proxy, meaning Caddy acts as an intermediary for client requests.

Automatic HTTPS: A significant advantage of this setup is that any service exposed through Caddy using these labels will automatically benefit from HTTPS. Caddy handles the entire lifecycle of SSL/TLS certificates, including issuance and renewal, typically using Let’s Encrypt.

Your application would be publicly exposed with those two labels, which is perfectly fine for a website. Still, in my case, I am planning to deploy automation tools or experiments that I don’t want anyone to access. Let’s see how we can simply secure our application with social login, for instance.

Authentication and Authorization

The caddy-security plugin provides the features described in this section. The first thing I would like to point out is that the Authentication portal is available, typically at auth.YOUR_DOMAIN (defined in the main Caddyfile). This portal allows for a landing page for authentication and links to various applications. Important to note that links in the portal are defined in the main Caddyfile and are not dynamically populated from the Docker labels, which is very sad. So far, I haven’t found a solution to that problem.

On top of the authentication portal Caddy Security offers to very important features:

  • Identity Provider: In my case, I use GitHub OAuth, which is configured as the primary identity provider. Users will be redirected to GitHub to log in.
  • Authorisation Policy: Access to applications secured by Caddy is typically granted based on attributes provided by the Identity provider; in this example, the policies are based on being a member of a specific GitHub organisation. The policy is defined within the Caddyfile and enforced such that users must be authenticated before Caddy handles any request. Instead, unauthenticated users are redirected to GitHub.

Once policies are defined, the main Caddyfile can be used to restrict access to the application using Docker labels. For instance, to apply the admin policy to a container, you can apply the following label, assuming admins_policy is one of the predefined policies.

  whoami:
    # A container that exposes an API to show its IP address
    image: containous/whoami
    networks:
      - caddy
    deploy:
      labels:
        caddy: whoami.$DOMAIN
        caddy.reverse_proxy: "{{ upstreams 80 }}"
        caddy.authorize: with admins_policy  # The policy to apply for accessing this container

Sometimes you need to only secure part of your application, let’s assume here that example.com/test is publicly accessible while the rest is not, then you can define multiple routes on the application and apply the policy to only the route you wish to secure and require authentication.

    deploy:
      labels:
        caddy: ${SUBDOMAIN}.${DOMAIN_NAME}
        # Public route
        caddy.0_route: /test
        caddy.0_route.reverse_proxy: "{{ upstreams 5678 }}"
        # Secured Routes
        caddy.1_route: "*"
        caddy.1_route.reverse_proxy: "{{ upstreams 5678 }}"
        caddy.1_route.authorize: with admins_policy

Requirements

For the docker_swarm_app_caddy Ansible role to deploy and configure Caddy successfully, the following prerequisites must be met:

  • DigitalOcean API Token: A DigitalOcean API token with DNS read and write permissions for your domain. This is required if Caddy is managing DNS records via the caddy-dynamicdns plugin with the DigitalOcean provider.
  • Configured GitHub OAuth Application:
    • Homepage URL: https://auth.YOUR_DOMAIN (e.g., https://auth.example.com)
    • Authorization callback URL: https://auth.YOUR_DOMAIN/oauth2/github/callback (e.g., https://auth.example.com/oauth2/github/callback)
  • Environment Variables on Ansible Control Node: The following environment variables must be set on the machine where you run Ansible:
    • CADDY_GITHUB_CLIENT_ID: The Client ID of your GitHub OAuth application.
    • CADDY_GITHUB_CLIENT_SECRET: The Client Secret of your GitHub OAuth application.
    • CADDY_JWT_SHARED_KEY: A strong, randomly generated secret key used for signing JWTs by caddy-security. You can generate one with openssl rand -hex 32.
    • CADDY_DIGITALOCEAN_API_TOKEN: Your DigitalOcean API token (if applicable).