# Home Lab — Production Linux Infrastructure

Two laptops from 2010 and 2012, a solar supply that cut out every day, and
a filtered national network. That was the hardware budget. For seven months, February to
August 2026, it ran Pi-hole, Samba, Gitea, an rsyslog collector and a handful of
Dockerized web services behind Nginx with automated SSL.

Then I shut it down on purpose, and wrote up what happened.

## What was actually on it

Two boxes, split by exposure. **n-server** held the things the house depended on —
DNS filtering, file shares, the git remote. **x-server** held the things the internet
could reach: the Dockerized sites, the reverse proxy, the certificates.

The part I'd point at first isn't a service, it's the alerting. I wrote a Python watcher
running under systemd that tailed the Nginx access logs and pushed notifications over
ntfy when something got through — not when something was *attempted*. That distinction
is the whole design, and it took a while to arrive at, because the first version alerted
on attacks and the channel was muted within a week. The internet attacks a public IP
constantly; the only interesting question is whether an attack worked.

On top of that sat a small ChatOps tool: ban an IP, restart a container, get a status
summary, from a phone, without a shell.

## The constraints did the designing

Power was the loudest one. A box that loses power every few days can't be treated as
reliable, so everything had to come back on its own — systemd units with the right
`Restart=` and `After=`, Docker restart policies that survive an unclean shutdown, and
no manual step anywhere in the boot path. That constraint turned out to be a good
teacher. Infrastructure that assumes it will be killed at a random moment is better
infrastructure.

The security work was a full audit rather than a checklist: Docker's `FORWARD` chain
rules (Docker will happily punch through a firewall you thought you'd configured), AIDE
for file integrity, secrets pulled out of the compose files and centralized, and fail2ban
in front of SSH.

## Why it ended

Not because it broke. Because the services that actually needed uptime had moved to the
edge, and what was left didn't justify two machines burning solar power. The portfolio
went to Cloudflare Workers in August 2026, and once it had, the honest answer to "what is
x-server for" became "nothing."

So I decommissioned it deliberately — verified 133 GB of data before deleting any of it,
which is its own story, and the only phase of a home lab with a step you cannot undo.
Both laptops still run: one as file storage, one as Pi-hole.

## The write-up

The interesting material is the failures, and there are eight posts of them:
the bugs that never threw an error, the alerting rebuild, the migration plan that turned
out to answer the wrong question, the false alarm during decommissioning, and a full
list of all 68 problems from seven months — the reference version, not the curated one.

[Read the series →](/articles/#home-lab)

The runbooks, operational scripts and the agent setup stayed in the repo; the writing
lives here.
