# Everything reported clean

This repo is the public version of a private one. The lab's real documentation carries addresses,
hostnames, account names and paths that have no business being published, so porting it meant
substituting all of that — and substitution is not a job to do by eye across two thousand lines.

So I wrote a checker. A list of patterns, run over every file, non-zero exit on a match, wired to
`pre-commit`.

It passed. It passed every single time I ran it.

It was also wrong in five separate ways, and every one of them was reporting clean the entire time
it was wrong. Which makes this the same post as [post 2](/articles/nothing-crashed), except the thing
quietly doing nothing was the thing whose only job was to catch things quietly doing nothing.

Four of those I found before publishing. The fifth I found afterwards, and by then it had already
let something through.

---

## `-I` is not ignore-case

```bash
grep -rInE "$pattern" .
```

[`-I`](https://man7.org/linux/man-pages/man1/grep.1.html) means *ignore binary files*. Ignore-case is `-i`, lowercase. I had been reading that flag as
case-insensitivity for years, in a script where every pattern was written in lowercase.

So the check for family-relationship words — the ones that turn a technical writeup into something
about my household — passed for weeks with a capitalised one sitting in a document. It matched the
lowercase form. The document had a capital letter.

One capital letter, and a check that was structurally incapable of ever firing.

## A pattern grep rejected outright

The public-IP check is the one that actually matters. A residential address in a document is the
single value that points at a house, and everything else in the repo is comparatively harmless.

I'd written it with a negative lookahead to skip the private ranges:

```
\b(?!10\.|192\.168\.)([0-9]{1,3}\.){3}[0-9]{1,3}\b
```

`grep -E` has no lookarounds. It doesn't ignore them — it errors. The error went to stderr, which
the script was discarding, and an empty result set looks precisely like nothing matched.

The most important check in the file had been erroring on every run since the day I wrote it, and
printing `clean` each time.

The fix was to stop being clever: match every address, then filter the private ranges out with a
second `grep -v`. Uglier, and it works.

The general version is worth more than the specific one. **A check that can't tell "no matches" from
"didn't run" isn't a check.** Both look like success from the outside, and one of them is a lie.

## The escaped form

This is the one I would never have found by reading the code.

The port scripts are `sed` scripts. A `sed` script that replaces a real value necessarily contains
that value — that's what a substitution is. So the tooling built to strip identifying values out of
the docs was carrying a complete list of them.

Which I knew, and had decided was fine, because the checker was scanning everything.

It wasn't finding them. In a `sed` script the value appears escaped:

```
s/203\.0\.113\.7/<WAN-IP>/g
```

A pattern hunting for `203\.0\.113\.7` matches literal dots. It does not match a literal backslash
followed by a dot. The two forms don't see each other, and the escaped one is the form that lives in
every substitution map you'll ever write.

So the scan specifically looking for those values walked past all of them, in the one file
guaranteed to contain all of them, and reported clean. Then I committed it.

The fix had two halves. The maps moved out of the repo entirely — the scripts now take a map by
path and hold no values themselves. And the checker strips backslashes from each file before
matching, so the escaped form and the plain form collapse into the same thing.

The lesson generalizes further than I'd like: **anything that transforms private data into public
data contains all of it by definition.** Scrubbers, substitution maps, allowlists, test fixtures,
the golden files in your test suite. It's the last place anyone looks, because it reads as tooling
rather than content.

## The pattern that wasn't there

I found the fourth one the day before publishing, and not by running the checker. By reading.

There was no email pattern. Not a broken one — none at all. It had never occurred to me to write it.

My own address was sitting in a certbot example in the agent docs and on a line in a runbook, and
in both places it reads as entirely ordinary configuration. Nothing about `--email <EMAIL>` looks
like a leak. It looks like documentation.

It was also the commit author, which no amount of scanning file contents will ever reach.

The pattern took two minutes to add. The part that took longer was accepting what it implied: "the
checker passed" had quietly become the whole of my confidence, and the checker only knows the
things I already thought of. Every category it covers is a category I'd been burned by. The one I
hadn't been burned by yet was invisible.

---

## The working tree is not the history

All four of those are content bugs. There's a structural one underneath them.

The checker only ever saw current files. A value committed and later removed is gone from the
working tree and still sitting in the history — and the history is what goes public, in full, the
moment you flip the switch. Nobody has to dig for it. `git log -p` is right there.

That's how the deliberately weak sudo password from [post 6](/articles/the-agent-knew-more) ended up
somewhere it shouldn't have been. Not through the agent, not through the mechanism I'd built so
carefully. Through a commit.

There's no clever fix for this one. If a value ever entered the history, the history is the thing
that has to change — and it's much cheaper to do that while a repo is still private and nobody has
cloned it.

And then the part I nearly missed entirely: **the commit message is public too.** Mine described
this incident in detail, including a tidy list of exactly which categories of value had leaked and
where the substitution maps now lived. A summary of what's hidden and a pointer to where it's kept,
sitting permanently in `git log`, written by me, in the commit whose whole purpose was to establish
a clean history.

Scan the messages. Read them like a stranger would.

---

## Canary tests

Every check now gets tested in both directions before I trust it: a file containing a value it must
catch, and the clean tree it must not fire on.

That's the entire methodology and it would have caught three of the first four immediately. The
case-sensitivity bug dies the first time you feed it a capitalised word. The broken lookahead dies
the first time you plant an address and watch the check print `clean` anyway. The escaped form dies
the moment you test against a real map file instead of a hand-written example.

**If I can't make a check fail on demand, I don't believe it when it passes.**

---

## The fifth, found after publishing

I wrote that section, and then I published the repo.

What I had not done — and did not notice I had not done — was run it. "Every check now gets tested
in both directions" was a rule I had written down. It was not a script that existed.

When I finally wrote the script, thirteen checks proved they could fail on demand. One did not:

```bash
check "wireguard keys" '\b[A-Za-z0-9+/]{42}[A-Za-z0-9+/=]{1}=\b'
```

A WireGuard key is 43 base64 characters and a trailing `=`. That final `\b` sits after the `=`, and
`=` is not a word character — so the boundary only holds when a word character *follows* the key.
At the end of a line there is nothing to bound against, and the end of a line is where every config
file on earth puts one.

Plant a key mid-line with a letter after it and the check fires. Write it the way WireGuard writes
it and the check is silent. It had been silent since the day I wrote it.

What it missed was a real server `PublicKey`, in the VPN section of the public server runbook, in a
block where the client private key and the WAN address had both been correctly replaced with
placeholders. It had never been in the substitution maps, so the port scripts had nothing to swap it
for and no other check had an opinion about it. One real value in a paragraph of placeholders, which
is the hardest thing there is to catch by reading — and I had read that file twice, looking for
exactly this.

It's a public key, and the endpoint beside it is scrubbed, so on its own it points at nothing. It's
still key material the scrubbing existed to remove, and it's still in the history at the initial
commit, because the repo was already public by the time I found it. Rotating the keypair is cheaper
than rewriting a published history, and the lab is off anyway.

The part worth keeping isn't the regex. It's that the section immediately above this one — the one
recommending canary tests, in a post about checks that lie — was itself an untested claim.
**Writing down the discipline is not the same as running it.** I published an argument for exactly
that point while doing the opposite one heading earlier.

---

## If you're publishing a lab writeup

Everything I'd tell someone starting this, in order of how much it cost me:

- **Write a public repo. Never flip the private one.** Derived artifact, one direction, no exceptions.
- **Keep the substitution map outside the repo.** It contains every value you're removing.
- **Scan the history, not the working tree.** They are completely different questions.
- **Read your commit messages.** Yours are longer and more candid than you remember.
- **Canary every check in both directions — and run it, don't just resolve to.** A clean report from
  an untested check is worth nothing. Mine cost me a server key between resolving and running.
- **Then read the whole thing anyway, slowly.** The email was found by reading. No pattern I owned
  was ever going to catch it, because I hadn't thought of it — and that's exactly the category that
  matters.

The through-line of this entire series turned out to be the same sentence in a different costume:
the failures that cost the most are the ones that report success. Seven months of that on two
laptops, then the tool I wrote to make sure I'd learned it did it to me one more time on the way out
the door — and then once more after the door had shut, in the section where I explained how to stop
it happening.
