Projects

Home Lab — Production Linux Infrastructure

Repository · Eight posts on how it ran and why it ended

Two laptops from 2010 and 2012, a solar supply that cut out every day, and a filtered national network. That was the hardware budget. For seven months, February to August 2026, it ran Pi-hole, Samba, Gitea, an rsyslog collector and a handful of Dockerized web services behind Nginx with automated SSL.

Then I shut it down on purpose, and wrote up what happened.

What was actually on it

Two boxes, split by exposure. n-server held the things the house depended on — DNS filtering, file shares, the git remote. x-server held the things the internet could reach: the Dockerized sites, the reverse proxy, the certificates.

The part I’d point at first isn’t a service, it’s the alerting. I wrote a Python watcher running under systemd that tailed the Nginx access logs and pushed notifications over ntfy when something got through — not when something was attempted. That distinction is the whole design, and it took a while to arrive at, because the first version alerted on attacks and the channel was muted within a week. The internet attacks a public IP constantly; the only interesting question is whether an attack worked.

On top of that sat a small ChatOps tool: ban an IP, restart a container, get a status summary, from a phone, without a shell.

The constraints did the designing

Power was the loudest one. A box that loses power every few days can’t be treated as reliable, so everything had to come back on its own — systemd units with the right Restart= and After=, Docker restart policies that survive an unclean shutdown, and no manual step anywhere in the boot path. That constraint turned out to be a good teacher. Infrastructure that assumes it will be killed at a random moment is better infrastructure.

The security work was a full audit rather than a checklist: Docker’s FORWARD chain rules (Docker will happily punch through a firewall you thought you’d configured), AIDE for file integrity, secrets pulled out of the compose files and centralized, and fail2ban in front of SSH.

solarsupplyx-servernginx · sitesn-serverdns, files, gitinternetwatcherntfy → phonealerts only when an attack gets through

Why it ended

Not because it broke. Because the services that actually needed uptime had moved to the edge, and what was left didn’t justify two machines burning solar power. The portfolio went to Cloudflare Workers in August 2026, and once it had, the honest answer to “what is x-server for” became “nothing.”

So I decommissioned it deliberately — verified 133 GB of data before deleting any of it, which is its own story, and the only phase of a home lab with a step you cannot undo. Both laptops still run: one as file storage, one as Pi-hole.

The write-up

The interesting material is the failures, and there are eight posts of them: the bugs that never threw an error, the alerting rebuild, the migration plan that turned out to answer the wrong question, the false alarm during decommissioning, and a full list of all 68 problems from seven months — the reference version, not the curated one.

Read the series →

The runbooks, operational scripts and the agent setup stayed in the repo; the writing lives here.