Home Lab

How to shut down a home lab

There are thousands of guides for building one of these. I couldn’t find a single one for ending it, which is odd, because decommissioning is the only phase with a step you can’t undo.

Break nginx during the build and you fix nginx. Delete 133 GB and discover the copy was incomplete and that’s the end of it.


Knowing it’s over

No dramatic failure. Just a slow accumulation of things that didn’t need to be there.

The site had moved to the edge and was obviously better off. The VPN hadn’t been used in months. Then I checked two things properly for the first time.

The task app — a Laravel thing I’d built and deployed and felt good about — had 2 users and 14 tasks, and its last real browser session was six weeks earlier. The traffic I’d been vaguely aware of was scanners sweeping for Laravel’s admin path. Requests to /dashboard redirecting to /login, from addresses in Bulgaria and Germany, where I have no users.

And Pi-hole, which I thought the whole house depended on, had been queried by exactly one device. The router had been handing out itself as the resolver the entire time. 143 queries in 24 hours, against a gravity list of 621,000 domains.

Both took five minutes to check. I’d assumed the opposite for months, and the assumption was flattering in both cases, which is probably not a coincidence.


Order

Everything is reversible except deleting data. So: verify, then archive, then wipe. And don’t overlap them — don’t start wiping one thing while still verifying another, because that’s how you delete the wrong copy while distracted.


The verification mistake

This is the part worth the post, because my method was wrong in a way that looked completely reasonable.

133 GB on the server’s spinning disk, already copied to an external drive months earlier. To confirm before wiping, I compared the two trees directory by directory — size and file count per folder.

Folder Server Copy
Design 2.9 G / 7,845 3.5 G / 7,992 fine
Documents 4.1 G / 4,889 34 G / 11,260 fine
Music 35 G / 5,125 35 G / 5,143 fine
IT 90 G / 4,071 90 G / 1,947 ⚠️
Temp 2.4 G / 1 missing ⚠️

2,124 files missing. Same total size in IT, half the file count. Three subfolders apparently gone entirely. It looked exactly like a copy that had silently truncated partway through.

Every single file was already there.

Months earlier I’d reorganized the drive and moved a whole subtree out of IT/ and into Documents/. Which is why IT had fewer files on the copy. The missing 2.4 GB video was in the recycle bin, deleted deliberately.

A path-level comparison cannot see a move. When a file goes from A/x to B/x, a per-directory diff reports it missing from A and surplus in B — two separate observations in two separate rows that never visibly cancel out. Scan a reorganized tree and you get a long list of missing files and a vague sense that some folders got bigger, and nothing connects them for you.

What actually works is comparing content manifests instead of paths:

find /source -type f -printf '%f|%s\n' | sort -u > source.txt
find /copy   -type f -printf '%f|%s\n' | sort -u > copy.txt
comm -23 source.txt copy.txt

Path-independent, so a move is invisible to it. Which is the point. That resolved 2,124 missing files to zero in one pass.

serverA/xcopyB/xsame fileper-directory diffcontent manifest2,124 missing0 missing

The worse mistake was that I started copying the “missing” files back before I’d finished diagnosing. So then I had duplicates on the drive, which had to be verified byte-identical with diff -rq and removed — cleanup created entirely by acting on a bad finding. Reaching the wrong conclusion cost me nothing. Acting on it cost an hour.

When a verification tells you you’ve lost data, check the verification first.


What to keep

I swept the actual disks rather than trusting my own documentation, and found things nobody remembered.

A Docker volume for a Telegram bot whose container had been deleted months earlier, still holding tasks.db and notes.db. Not in any service list, because the service didn’t exist. A retired site’s files sitting in ~/docker/website.retired-20260523/, untouched for a quarter. The nginx access logs, which are the only copy of the entire security record. The ufw before.rules file, which — because Docker was configured not to manage iptables — was the only place on the machine that described how the network actually worked. And a 76 MB dataset that couldn’t be regenerated because it was a point-in-time pull.

Also a directory called backups sitting on the same disk as the thing it was backing up. Both copies, one failure domain. Found it during the sweep, which is a lucky place to find it.

Your service list describes what you remembered to write down. The volumes and the orphaned directories and the /mnt subfolders describe what’s actually there. Those are different sets.

What I skipped on purpose: expiring certificates, rebuildable images, Pi-hole’s gravity database (100 MB+ and regenerates from the adlists), the AIDE baseline, and SSH host keys, which should never be reused anyway.

The archive came out at about 100 MB from both machines. Checksummed where checksums already existed. The most important file in it is a README explaining what everything else is and why it’s there, because an archive you can’t navigate in two years is the same as no archive.


The stuff not on the machines

The parts I nearly forgot, because they don’t live on either server.

Router port forwards — remove them. Leaving forwards pointed at a freed address means whatever gets that address next by DHCP inherits them. The certbot cron, disabled before it starts failing weekly. DNS repointed on the one device that was still using Pi-hole. And a GitHub Actions runner registered against someone else’s repository, which would otherwise sit permanently offline in their settings.

Verify the port forwards from outside your network. A request from inside can hairpin through the router and succeed against a forward that’s already gone, which will tell you the opposite of the truth. I confirmed mine by fetching a unique path from an external host and grepping the access log for it. It never arrived, while internal health checks kept logging normally every ten minutes. That combination is proof.

from outsidefrom insideexternal hostrouteraccessloghealth checksnever arrivedkept logging

What the documentation becomes

My config repo now describes infrastructure that doesn’t exist. That’s fine — it’s a record.

But the deploy pipeline in it is dead, the “push to main runs sudo on both servers” trust model describes a setup that no longer exists, and every skill and command assumes live hosts. Both laptops still run, one as file storage and one as Pi-hole, but nothing deploys to them anymore. So the top of that file now says exactly that, before anything else, splitting what’s historical from what still binds. Not because I’ll forget. Because the version of me who opens it in a year, in a hurry, won’t read past the first screen.


The ending

I didn’t shut it down because it failed. I shut it down because I finally stopped answering two questions as though they were one.

Do I enjoy running this — yes, genuinely, and it taught me more about reverse proxies, certificates, container networking and firewalls than any amount of reading would have.

Should this be running here — no. Not on a 2010 laptop on a solar system that cuts out every day in a country where half the internet is blocked, serving a site with real visitors.

Both of those were true the entire time. Building the lab was right and ending it was right. The only mistake was the months I spent letting the first answer settle the second one.