A test that fails if my tool says "savings"
ZeroContext has a test whose only job is to stop the tool from claiming it saves money:
assert.ok(/fewer rule tokens in the window/.test(out));
assert.ok(/Scope, not \$/.test(out));
assert.ok(!/saving/i.test(out), "report must not claim 'savings'");
It’s there because the first version of the report called its number savings.
What the A/B said
ZeroContext serves an agent only the slice of your rules that matches the file it’s on, instead of loading the whole CLAUDE.md into every message. zerocontext analyze measures the difference. On its own repo today:
= 388 tok resident vs 1,913 tok preloaded (every message)
→ 80% fewer rule tokens in the window (1,525 tok)
That number used to be called savings. Then I ran an A/B end to end, because I didn’t trust it: fewer tokens in the window says nothing about the bill. Just-in-time came out cost-neutral to higher. A preloaded CLAUDE.md gets prompt-cached, so carrying it is close to free. The commit at the time put rules at about 7% of each turn’s context, and the extra turn that fetches a slice is full price.
What changed
The number stayed the same. Everything around it changed:
- inside
analyze,jitbecameresidentandsavingsbecamereduction - the report leads with “N% fewer rule tokens in the window” and says “Scope, not $” right under it
- the README dropped the cost row from its comparison table
- the project’s
CLAUDE.mdcalls it “a scoping number, not a session-cost claim”
And the test, so the framing can’t quietly slide back. “Savings” is the obvious word for anyone editing the report later, and an agent might not read CLAUDE.md first. It will see a red test.
The result itself is in Zerocontext.