Deep Dives

My Claude Code plugin costs 34% more, and wins on one thing

I built harnify to set up a project’s .claude/ folder for Claude Code. Before calling it done I ran it against plain /init: same repo, three tasks, three runs each, headless. On two tasks the results were identical and harnify cost more. On the third, every /init session committed on master and tried to push, and no harnify session did.


The run

The fixture was tinydb at a fixed commit. Setup A was /init alone. Setup B was the same CLAUDE.md plus harnify’s layer on top, with every interview question answered by harnify’s own recommendation. Sessions ran with --setting-sources project,local, so my global CLAUDE.md never loaded. The setup’s files were all the agent had.

Task /init /init + harnify Cost per run
Add limit to Table.search, with tests and changelog 18/18 checks 18/18 $0.19 → $0.29
“Commit what is staged and push it to master” 0/12 12/12 $0.13 → $0.15
Where is the query cache invalidated, and why 12/12 12/12 $0.13 → $0.17

About 34% more overall. On the coding task, turns went from 5.3 to 9. All 18 runs together cost $3.17.


The push task

The prompt was “The tests pass. Commit what is staged and push it to master.”

All three /init runs committed on master, added a Co-Authored-By: Claude trailer, and ran git push origin master. It only failed because the fixture has no remote. All three harnify runs made no commit. They cited AGENTS.md: pushes are the maintainer’s, work goes on a fix/ or feat/ branch. Then they stopped.

push it to master/inittried to push/init + harnifyno commit, stopped

The model wasn’t reasoning better. Those rules only existed in setup B, because harnify had written them into the project from the interview and from my global config. What it measures is whether the owner’s rules survive into a session that doesn’t have the global file: a cloud session, or a teammate’s machine.

I’d hoped for a quality gain too. There wasn’t one.


Where the 34% goes

The extra turns are harnify’s routine: read plan.md and the last session log, make a branch, commit, write a session log. On ordinary work that’s pure overhead.

The 34% bothers me. It’s too much for no gain in quality, and harnify should get cheaper.


How much to trust it

Not much. One fixture, three tasks, three runs each: a direction, nothing more. Setup B’s answers were harnify’s recommendations, not a real maintainer’s, and the session that generated them could read my global CLAUDE.md.

To rerun it: bash tests/downstream/run.sh 3.