AI & Harness

nginx: cannot load certificate, before certbot has issued it

Write the HTTPS block for a new site before certbot has run, and nginx refuses the whole config:

nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/app.example.net/fullchain.pem": BIO_new_file() failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/etc/letsencrypt/live/app.example.net/fullchain.pem, r) error:10000080:BIO routines::no such file)
nginx: configuration file /etc/nginx/nginx.conf test failed

That’s nginx -t in the nginx:1.27-alpine image with one server block pointing at a certificate that doesn’t exist. When one nginx fronts several sites, as it did in my home lab, a failed test means no reload for any of them. And certbot’s webroot challenge needs nginx already serving /.well-known/acme-challenge/ over plain HTTP on that domain, so the site has to exist before its certificate does.

HTTPS firstnginx -t failsno reloadany sitecannot load certificateHTTP + ACMEcertbotHTTPS blockwebroot challenge over plain HTTP

My home lab’s new-service skill had Claude add a site in an order that avoids both problems. It wrote the nginx config in two phases from one template: an HTTP-only block with the ACME location first, then the HTTPS block once the certificate existed.

The order

  1. DNS resolves to your public IP (dig +short). nginx and certbot wait until it does.
  2. The container is up, on the same Docker network as nginx.
  3. The HTTP-only block, then nginx -t and a reload, never a container restart.
  4. certbot in webroot mode, on the same network. Every Problem has the week a one-shot certbot container spent failing DNS on Docker’s default bridge.
  5. The HTTPS block, test, reload, and curl -sI https://<domain>/ from outside.
  6. Docs updated, with the diff shown first.

The HTTPS template also carries a warning: don’t set a Content-Security-Policy in nginx if the app sets its own (Helmet does). On that server, duplicate CSP headers had broken an app.

Save it as .claude/skills/new-service/SKILL.md with the template as templates/nginx-site.conf. My version also covered DuckDNS, pinned container IPs, firewall rules and a BuildKit workaround; this one keeps the order and leaves those out. The template loses its log paths, two proxy settings, and the include and ssl_dhparam lines for certbot SSL files my server already had, which a fresh webroot setup may not. And the skill’s own reason for the order was that certbot can’t reach the challenge; the nginx -t failure above is the one reproduced for this note. Fill in the network name, paths and the certbot email for your server.

Markdown
name
new-service
description
Use whenever the user wants to put a new site or Docker service behind nginx with a Let's Encrypt certificate: "add a new site", "deploy <app> behind nginx", "set up <domain>", "new subdomain", "spin up a container behind nginx".

new-service: onboard a site behind nginx + certbot

The order matters. DNS first, then an HTTP-only nginx block for the ACME challenge, then certbot, then the HTTPS block, then verify, then docs. An HTTPS block that points at a certificate that doesn't exist yet fails nginx -t ("cannot load certificate"), and on a shared nginx that blocks the reload for every site.

Fill these in for your server

What Yours
Docker network nginx and the apps share e.g. web
nginx config dir e.g. ~/docker/nginx/conf.d/
Compose file that runs nginx e.g. ~/docker/docker-compose.yml
certbot dirs e.g. ~/docker/certbot/conf, ~/docker/certbot/www
Where services are documented e.g. docs/services.md

Step 1: Gather intent

Ask only for what's missing:

  • Service name: kebab-case; used for the compose project, container and config filename
  • Domain: full hostname, or a path under an existing domain
  • Type: Docker container, or static files served by nginx
  • Upstream: container name + internal port, or the static path Optional: WebSocket, a body size above nginx's 1m default, special headers.

Step 2: DNS

A new hostname must resolve to the server's public IP: dig +short <domain>. A path under an existing domain needs no DNS change. Don't go past Step 3 until DNS resolves.

Step 3: The container (if Docker)

Its own compose project unless the user says otherwise. On the shared network (external: true), container_name equal to the service name, restart: unless-stopped, env vars from a sibling .env. docker compose up -d.

Step 4: HTTP-only nginx block

Use Phase 1 of templates/nginx-site.conf as <service>.conf. Then:

docker compose -f <compose file> exec -T nginx nginx -t \
  && docker compose -f <compose file> exec -T nginx nginx -s reload

If nginx -t fails, fix the config and re-test. Never restart the nginx container.

Step 5: certbot

docker run --rm --network <network> \
  -v <certbot conf>:/etc/letsencrypt \
  -v <certbot www>:/var/www/certbot \
  certbot/certbot certonly --webroot -w /var/www/certbot \
  -d <domain> --email <email> --agree-tos --no-eff-email --non-interactive

It must run on the shared network: one-shot containers on Docker's default bridge may not resolve DNS or reach nginx. Webroot can't validate wildcards; those need a DNS challenge.

Symptom Cause Check
Failed to connect ... for HTTP-01 DNS not propagated or port 80 blocked dig, then curl -sI http://<domain>/.well-known/acme-challenge/test
unauthorized: Invalid response ACME location missing or wrong root the location /.well-known/acme-challenge/ block
connection refused nginx not on the shared network docker network inspect <network>

Step 6: HTTPS block

Uncomment Phase 2 with the issued certificate paths, test and reload as in Step 4, then verify from outside: curl -sI https://<domain>/ should give the expected status and a valid certificate.

Step 7: Docs

Add the service to the docs. Show the diff to the user before writing.

Step 8: Validate

  • dig +short <domain> returns the public IP
  • curl -sI https://<domain>/ returns the expected status
  • docker compose ps shows the service running
  • <certbot conf>/live/<domain>/fullchain.pem exists and is fresh
  • nginx's access log shows the test request
  • Docs updated

Output

  1. Every file created or changed
  2. The exact commands run, so the user can replay them
  3. The Step 8 results
  4. Anything deferred
---
name: new-service
description: >
  Use whenever the user wants to put a new site or Docker service behind nginx with a Let's
  Encrypt certificate: "add a new site", "deploy <app> behind nginx", "set up <domain>",
  "new subdomain", "spin up a container behind nginx".
---

# new-service: onboard a site behind nginx + certbot

The order matters. DNS first, then an HTTP-only nginx block for the ACME challenge, then
certbot, then the HTTPS block, then verify, then docs. An HTTPS block that points at a
certificate that doesn't exist yet fails `nginx -t` ("cannot load certificate"), and on a
shared nginx that blocks the reload for every site.

## Fill these in for your server

| What | Yours |
|---|---|
| Docker network nginx and the apps share | e.g. `web` |
| nginx config dir | e.g. `~/docker/nginx/conf.d/` |
| Compose file that runs nginx | e.g. `~/docker/docker-compose.yml` |
| certbot dirs | e.g. `~/docker/certbot/conf`, `~/docker/certbot/www` |
| Where services are documented | e.g. `docs/services.md` |

## Step 1: Gather intent

Ask only for what's missing:
- **Service name**: kebab-case; used for the compose project, container and config filename
- **Domain**: full hostname, or a path under an existing domain
- **Type**: Docker container, or static files served by nginx
- **Upstream**: container name + internal port, or the static path
Optional: WebSocket, a body size above nginx's 1m default, special headers.

## Step 2: DNS

A new hostname must resolve to the server's public IP: `dig +short <domain>`. A path under
an existing domain needs no DNS change. Don't go past Step 3 until DNS resolves.

## Step 3: The container (if Docker)

Its own compose project unless the user says otherwise. On the shared network
(`external: true`), `container_name` equal to the service name, `restart: unless-stopped`,
env vars from a sibling `.env`. `docker compose up -d`.

## Step 4: HTTP-only nginx block

Use Phase 1 of `templates/nginx-site.conf` as `<service>.conf`. Then:

```bash
docker compose -f <compose file> exec -T nginx nginx -t \
  && docker compose -f <compose file> exec -T nginx nginx -s reload
```

If `nginx -t` fails, fix the config and re-test. Never restart the nginx container.

## Step 5: certbot

```bash
docker run --rm --network <network> \
  -v <certbot conf>:/etc/letsencrypt \
  -v <certbot www>:/var/www/certbot \
  certbot/certbot certonly --webroot -w /var/www/certbot \
  -d <domain> --email <email> --agree-tos --no-eff-email --non-interactive
```

It must run on the shared network: one-shot containers on Docker's default bridge may not
resolve DNS or reach nginx. Webroot can't validate wildcards; those need a DNS challenge.

| Symptom | Cause | Check |
|---|---|---|
| `Failed to connect ... for HTTP-01` | DNS not propagated or port 80 blocked | `dig`, then `curl -sI http://<domain>/.well-known/acme-challenge/test` |
| `unauthorized: Invalid response` | ACME location missing or wrong root | the `location /.well-known/acme-challenge/` block |
| `connection refused` | nginx not on the shared network | `docker network inspect <network>` |

## Step 6: HTTPS block

Uncomment Phase 2 with the issued certificate paths, test and reload as in Step 4, then
verify from outside: `curl -sI https://<domain>/` should give the expected status and a valid
certificate.

## Step 7: Docs

Add the service to the docs. Show the diff to the user before writing.

## Step 8: Validate

- [ ] `dig +short <domain>` returns the public IP
- [ ] `curl -sI https://<domain>/` returns the expected status
- [ ] `docker compose ps` shows the service running
- [ ] `<certbot conf>/live/<domain>/fullchain.pem` exists and is fresh
- [ ] nginx's access log shows the test request
- [ ] Docs updated

## Output

1. Every file created or changed
2. The exact commands run, so the user can replay them
3. The Step 8 results
4. Anything deferred

templates/nginx-site.conf:

# Phase 1: HTTP-only. Live first, so certbot can complete the ACME challenge.
server {
    listen 80;
    listen [::]:80;
    server_name [DOMAIN];

    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

# Phase 2: HTTPS. Uncomment only after certbot has issued the certificate.
# server {
#     listen 443 ssl;
#     listen [::]:443 ssl;
#     http2 on;
#     server_name [DOMAIN];
#
#     ssl_certificate     /etc/letsencrypt/live/[DOMAIN]/fullchain.pem;
#     ssl_certificate_key /etc/letsencrypt/live/[DOMAIN]/privkey.pem;
#
#     # Don't set a CSP here if the app sets its own (Helmet does):
#     # duplicate CSP headers break pages.
#     add_header X-Content-Type-Options nosniff always;
#     add_header X-Frame-Options SAMEORIGIN always;
#     add_header Referrer-Policy strict-origin-when-cross-origin always;
#
#     client_max_body_size 1m;
#
#     location / {
#         proxy_pass http://[UPSTREAM_HOST]:[UPSTREAM_PORT];
#         proxy_http_version 1.1;
#         proxy_set_header Host              $host;
#         proxy_set_header X-Real-IP         $remote_addr;
#         proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
#         proxy_set_header X-Forwarded-Proto $scheme;
#         # WebSocket upgrade: a no-op if the upstream doesn't use it
#         proxy_set_header Upgrade           $http_upgrade;
#         proxy_set_header Connection        "upgrade";
#         proxy_read_timeout 60s;
#     }
# }

see also: Every Problem, The Setup, Incident Triage Skill