I kept seeing the same skill in my feeds. A Claude Code plugin, 84,000 stars, that saves you tokens by making the agent talk like a caveman. No articles, no filler, just grunts.
The pitch made sense. Fewer words out, smaller bill. So I measured it, plus a token-efficient CLAUDE.md sitting near it in the charts, 5,700 stars, same promise. Each task run twice, once with the skill and once without, on real coding sessions, scored on the dollar cost instead of a token count.
The savings didn't show up. That part I half-expected. What got me was why: the number these skills optimize and the number I pay are two different numbers. Nobody's being conned. Output tokens are just the number you can watch move on a single prompt, so it's the one everyone reaches for, sellers and skeptics alike. It simply isn't the number that lands on your bill.
Quick disclosure: I build an eval harness, so poking holes in claims like this is my idea of a fun weekend. You don't need my tool for any of it. Two runs and a token counter reproduces the whole thing. The finding is what matters, not the instrument.
Three kinds of token, and only one is expensive
The first time I actually read a Claude bill instead of just paying it, I found three kinds of token, priced nothing alike.
Output, the words the model writes back, is $15 per million on Sonnet. Input, the fresh text I send in, is $3. Cache-read, the context it has already seen and re-reads every turn, is $0.30.
So output is the expensive one. Five times input, fifty times cache-read. And cache-read is nearly free, with a mountain of it.
Every token-saving skill goes after output, the model's prose. Not because that's where the money is. Because it's the number you can see on a single prompt: ask a question, count the reply with and without the skill, subtract. Big number, easy to screenshot.
That number answers a question I never ask.
What I'm really paying for
A real coding session isn't one prompt. I watched one of mine go by: read a file, run a test, read the result, think, edit, re-read. Every turn re-bills the whole context, almost all of it cache-read, the cheap giant bucket.
Output, the part these skills shrink, is a sliver of that. About 1% of the tokens in my runs. You
can check your own; caveman even ships a caveman-stats hook that prints the split. Mine always
had the same shape, a few thousand output tokens riding on a hundred times as many cached ones.
But tokens aren't dollars, and that's where it turns. Output is priced fifty times higher than cache-read, so its share of the money runs far ahead of its share of the volume. So I stopped estimating and added up the actual dollars across every run. Output came to about 20% of the bill. A fifth, not a hundredth. (Your mix will vary. A session re-reading big files over and over is nearly all cheap cache and tilts output lower; one firing fresh prompts tilts it higher. A fifth is the middle of what I measured, and it's already generous, because that number counts the code and tool calls in "output" that caveman deliberately never touches.)
Hold onto that fifth, because it sets a ceiling. Even a perfect 65% cut of a fifth of your bill is about a 13% saving, and that is the dreamy best case, before a single real-world complication. The lever is small before we even ask whether the skills pull it. One of them, it turns out, does. It still doesn't matter.
So I measured the number I actually pay
I wanted the honest version, so I scored three things on every task instead of one: the output tokens (what the skill claims to cut), the real dollar cost of the whole run, and whether the answer was still correct.
Seven tasks on Sonnet, from a one-line slugify up to a multi-file refactor with its own tests and a code review. Five runs each, both skills. A hundred and forty runs, about $10 of API-equivalent, billed $0 against my subscription. I installed each one the way you actually would (Caveman as a real Claude Code plugin, which switches itself on from the first message) and ran a Welch t-test on every result, so a "cut" has to clear the noise before I believe it.
The two skills: Caveman (~84,000 stars, README claims 65%, self-measured on 10 one-shot prompts) and a token-efficient CLAUDE.md (~5,700 stars, claims 63%, on what its own repo calls "a 5-prompt directional indicator, no variance controls").
Caveman really does cut output
Here's the part I didn't expect: it works. Installed for real, caveman comes on from the first message, and on five of my seven tasks output came down, four of them visibly — two big enough to pass a significance test (an off-by-one bugfix down 31%, an email-regex task down 28%). This is not vaporware. The agent really does drop its articles and grunt.
It just doesn't do it at 65%. Average the seven tasks and the output cut is about 6%, not 65, and it's lumpy: real cuts on some, and on two others output actually grew (the one-line slugify, the multi-file refactor). A 59-point gap between the claim and the average, and most individual tasks don't even clear the noise bar. Caveman compresses. It compresses a little, unevenly, and nowhere near the number on the tin.
The token-efficient CLAUDE.md doesn't get that far. Its output grew 29% on average, a few small trims on the short tasks and then more than a doubling on one and a +54% on the review, both dragging the mean the wrong way. Claimed 63% down; measured 29% up. A 92-point gap.
And most of it barely clears the noise. Take the one-line slugify: its output grew 54% in this run, but the five trials swung so wildly the move fails a significance test (p=.27), and most tasks are like that, real-looking deltas sitting inside the run-to-run spread. A strange property for a number sold to the percentage point.
In dollars, nothing changed
Then the part I actually care about: the bill. Here's the punchline. Caveman genuinely cut my output, and my bill did not move. Add up every dollar across the whole run (the total that hits your card, not the per-task averages) and the two arms land within one percent of each other. No saving, no significant cost. A wash.
That's the whole piece in one number. A real output cut, and the bill is flat, because output is a fifth of the cost and caveman spends its own ~1 to 1.5k input tokens per turn carrying its instructions, which quietly eats back the little it saves. The token-efficient CLAUDE.md is worse: its bill came out higher in every run I've done, 10% in this one.
Correctness held on all fourteen task-pairs, so none of this is dangerous, just not economical. The skill does exactly what it advertises, shrinking the words the model writes, and the number you pay never notices.
The authors already know this
What got me when I read the repos: both authors already say all of this.
Caveman's README has a little ASCII bar chart right at the top. Output tokens saved: 65%. Input tokens saved: 0%. It says in plain text that it "only shrinks output tokens," adds "~1 to 1.5k input tokens per turn," and can go "net-negative" on already-terse work. The token-efficient repo calls its own 63% "a directional signal, not a precise universal measurement," and notes the savings only turn positive when output volume is high enough to cover the standing input cost.
They footnote the exact thing I measured. Nobody's lying.
The problem is which number travels. The 65% is what 84,000 people star and repeat. The 0% right next to it, and the net-negative warning below, is a line most of them never scroll to. If you want the honest number to win, it has to be the headline: real dollars, on a real session.
I tried its best tool, too
To its credit, caveman knows output is the small lever, so it ships bigger ones, two of them, both
aimed at input. /caveman-compress rewrites a memory file (your CLAUDE.md, your notes) into
caveman-speak to cut input tokens "every session after," and caveman-shrink trims the tool
definitions an MCP server injects every turn. I tested the first; the second needs a live MCP
server I didn't wire up. Input is a larger slice of the bill than output, so this is the fairer
test, and I wanted to give it the best possible shot.
So I took a verbose conventions file and compressed it 68%, better than caveman's own claimed 46%, keeping every rule intact. Then I ran the same task under both versions and measured the real session.
The file got 68% smaller. The bill didn't care. Cache tokens moved 2%, total cost 14%, and neither one was statistically significant. Noise, both of them, because a 500-token memory file is a rounding error next to a 100,000-token session. Same shape as the grunting: a real cut of a small thing, and the small thing is a sliver of what you pay. One genuine point in its favor, though: the compressed file cost nothing in accuracy. The agent followed every rule either way.
The number people star isn't the number they pay
The mismatch cuts both ways, and that's the part worth stealing.
The 63–65% figures aren't fraud. They're the easy number to see: an output-token delta on a single prompt. But that's not the number you pay. You pay dollars, across three price tiers, piled up over a long session, with output a small and badly-moved slice of the total.
The trap catches debunkers too. Count output tokens, land on something smaller, declare the skill busted, and you're holding the same wrong ruler, just more skeptically. Output-only is the wrong measure whether you're selling or dunking. Measure the real cost of a real run, check the answer still works, and the percentage on the tin stops meaning much.
Where the savings actually hide
If you actually want a smaller bill — or more runway before the subscription quota wall — output compression is the wrong lever. You're squeezing the cheap 1%. The money is in what re-enters context every turn.
- MCP servers you forgot you connected. Each one injects its tool definitions into every turn whether you call it or not. Dropping the unused ones is the biggest lever nobody pulls.
- A bloated main thread. Hand work to subagents so the main conversation stays small instead of dragging a growing context through every turn.
- No pruning.
/compactand deliberate trimming cut the cache-read pile directly — the pile with all the volume.
That's where I'd look first. And I'll hold myself to my own rule: I haven't benchmarked those three the way I benchmarked Caveman, so treat them as leads, not measured claims. Subagents especially can backfire on a small task, since spawning one re-reads a chunk of context of its own.
One more cost I didn't measure: installing any of these skills rewrites your system prompt or CLAUDE.md, which can invalidate the prompt cache and force a full re-read next turn. That's a real charge, and it has nothing to do with how long the replies are. The point holds either way. The money is in the context, not in teaching the model to grunt.
A note on getting this wrong
Since the whole piece is about measuring right, I owe you this. My first cut of this benchmark was broken. I had installed caveman in a way that never actually loaded it, so for a while I was measuring a skill that wasn't even in the room. A blunt review and a careful read of caveman's own README, which spells out that it switches on through a session hook, caught the mistake. I fixed the delivery, verified the skill was actually on this time (its output turns measurably telegraphic), and re-ran everything above. The corrected numbers came out kinder to caveman than my broken ones were, and that is the point: measure right, especially when it helps the thing you're poking at.
Run it yourself
None of this is trust-me. One paired run of a single real task, scored on output tokens, real
dollars, and whether the answer still works, shows you the direction in an afternoon. I ran a
hundred and forty to be sure the direction wasn't noise. The harness is open source
(vigiles), but any two-arm setup and a token counter does
the job.
So before you install the next thing promising to cut your tokens by some percent, ask it one question: which tokens, priced how, measured on what? If the answer is "output, on one prompt," you already know what it does to a real session. Maybe it works. Maybe it really does trim what the model writes. But that's a sliver of what you pay, and your bill will barely feel it.