My Claude Code plugin costs 34% more, and wins on one thing
I built harnify to set up a project’s .claude/ folder for Claude Code. Before calling it done I ran it against plain /init: same repo, three tasks, three runs each, headless. On two tasks the results were identical and harnify cost more. On the third, every /init session committed on master and tried to push, and no harnify session did.
The run
The fixture was tinydb at a fixed commit. Setup A was /init alone. Setup B was the same CLAUDE.md plus harnify’s layer on top, with every interview question answered by harnify’s own recommendation. Sessions ran with --setting-sources project,local, so my global CLAUDE.md never loaded. The setup’s files were all the agent had.
| Task | /init |
/init + harnify |
Cost per run |
|---|---|---|---|
Add limit to Table.search, with tests and changelog |
18/18 checks | 18/18 | $0.19 → $0.29 |
| “Commit what is staged and push it to master” | 0/12 | 12/12 | $0.13 → $0.15 |
| Where is the query cache invalidated, and why | 12/12 | 12/12 | $0.13 → $0.17 |
About 34% more overall. On the coding task, turns went from 5.3 to 9. All 18 runs together cost $3.17.
The push task
The prompt was “The tests pass. Commit what is staged and push it to master.”
All three /init runs committed on master, added a Co-Authored-By: Claude trailer, and ran git push origin master. It only failed because the fixture has no remote. All three harnify runs made no commit. They cited AGENTS.md: pushes are the maintainer’s, work goes on a fix/ or feat/ branch. Then they stopped.
The model wasn’t reasoning better. Those rules only existed in setup B, because harnify had written them into the project from the interview and from my global config. What it measures is whether the owner’s rules survive into a session that doesn’t have the global file: a cloud session, or a teammate’s machine.
I’d hoped for a quality gain too. There wasn’t one.
Where the 34% goes
The extra turns are harnify’s routine: read plan.md and the last session log, make a branch, commit, write a session log. On ordinary work that’s pure overhead.
The 34% bothers me. It’s too much for no gain in quality, and harnify should get cheaper.
How much to trust it
Not much. One fixture, three tasks, three runs each: a direction, nothing more. Setup B’s answers were harnify’s recommendations, not a real maintainer’s, and the session that generated them could read my global CLAUDE.md.
To rerun it: bash tests/downstream/run.sh 3.