What a code index actually saves an agent.

The question

Reducing the token cost of long-running agents names six levers. The fourth is an external index of the codebase, so an agent stops re-searching files it has already read. That note also commits to a scoring rule: tokens saved per step against task success, with failure modes documented as carefully as the wins. This is that measurement for the fourth lever, and the failure mode turned out to be the interesting part.

The tool measured is codebase-memory-mcp, which is MIT licensed, ships as a single static binary, and publishes an evaluation across 31 repositories claiming 83% answer quality with 10x fewer tokens and 2.1x fewer tool calls against file by file exploration. It is a good tool. We use it. The question was only whether the 10x holds on our workloads.

Method

Built from source at main, Linux x86_64. Two repositories: a public Go project of 51,189 lines that nobody here had read before, and a private Python and TypeScript codebase of ours.

The baseline is grep -rn, which is what an agent actually does, and then the same plus reading the one file that defines the symbol, which is what it does next. Output is measured in bytes.

On the Go repository the symbols were chosen mechanically: every capitalised identifier of six or more characters was extracted, ranked by frequency, and sampled at fixed ranks. Nobody looked at the list before it was measured. That matters more than the sample size, because the first question anyone should ask of a result like this is whether the queries were picked to produce it.

What we found

At the tool's default limit of 10 results:

• 2.8x fewer bytes than grep on our own codebase.
• 4.0x on the Go project.

Raise the limit so both sides return the complete answer and it inverts:

• It costs more than grep on 22 of the 24 symbols.
• Median 0.63x and 0.81x. On our own codebase it did not win once.

Both numbers are real. They measure different things, and the gap between them is the finding:

The token saving is selection, not compression. The default returns a capped, ranked, symbol deduplicated shortlist. It is cheaper because it returns less, not because it encodes the same answer more efficiently.

That is not a criticism. A ranked shortlist of ten is very often exactly what an agent wants, and the per record structure it returns — qualified name, file, line range, inbound and outbound edge counts — is information grep cannot produce at any price. Deduplication does eventually win outright: one symbol matching 1,886 lines collapsed to 301 distinct symbols and beat grep by 3.4x. But the crossover is above 300 matching lines, not the 85 we first thought.

Full results, Go project

Symbols chosen by frequency rank, not by us.

Symbol	matching lines	grep	index, default	index, complete	vs grep
Prepend	1	107	317	317	0.34x
FormatDuration	7	646	966	966	0.67x
Foreground	10	772	1,487	1,487	0.52x
ChatCompletion	80	8,082	2,136	6,316	1.28x
Pattern	120	12,289	3,259	12,756	0.96x
Collector	121	10,085	2,170	13,148	0.77x
Runner	152	14,159	3,289	18,815	0.75x
CustomProviders	213	21,042	2,236	22,130	0.95x
Client	330	28,882	3,731	33,879	0.85x
Errorf	1,886	199,618	1,885	57,960	3.44x

The private codebase produced 14 more rows in the same shape, 0 wins of 14 at full fidelity, median 0.63x. Its symbol names are withheld; the ratios are not.

Three numbers we measured and did not use

The method is the claim here, so the discarded numbers matter as much as the kept one.

51x. Against a baseline that read every file containing a match. No competent agent does that, so the baseline would have been doing the winning.

20x. Against grep -C5. Same objection, smaller.

105x. The largest figure we produced, and the one that nearly became the headline. That query matched 1,886 lines; the tool returned 10, because the limit defaults to 10. grep returned all 1,886. That is truncation, not compression, and the same objection quietly eats the top of our own tables wherever a symbol is common enough to hit the cap.

And one we got wrong

At six symbols we concluded the tool answered in a near constant 1.8 to 2.0 KB and built an explanation on it. At fourteen that collapsed: the range is 974 bytes to 7,097. The narrow band was an artifact of too small a sample, and publishing a conclusion drawn from it would have been the same error this work exists to check for in other people's numbers.

What this does not measure

Two things, and they are the two that could change the conclusion.

Tool calls. The published claim is 2.1x fewer, and nothing here touches it. It is also the claim the shortlist framing would most plausibly support: fewer round trips because the ranking is good, rather than fewer tokens per call. A good shortlist that ends the search in one call is worth more than these byte counts suggest.

Answer quality. The published figure is 83%, and nothing here tests whether the ranked ten actually contained what you needed. A shortlist that is cheap and wrong is worse than grep at any price.

Two repositories and one question shape is not a benchmark, and this is not published as one.

What we take from it

For the lever it belongs to, the answer is conditional rather than a multiplier. An index earns its keep when a symbol is scattered widely enough that a ranked shortlist beats reading the matches, and when the structure it returns — the call graph, the edge counts — is the thing you actually needed. Asked for exhaustive results on a symbol that appears in a handful of places, it is more expensive than the obvious tool.

Which is a more useful thing to know than a ratio.