the paper promised 16×. my cache already had 9×.

a paper landed on my desk with a great number on it: 16–30× faster time-to-first-token on long retrieval contexts, no loss in quality. i run a self-hosted LLM stack for svadhyay, and the slowest thing in it is exactly what the paper targets — stuffing a whole textbook into the prompt so the tutor can answer from it. the instinct is obvious: build it.

i didn't. i measured first. this is that afternoon.

what the paper actually proposes

the paper is REFRAG (Meta, 2025). the standard way to do retrieval-augmented generation is to paste the retrieved passages into the prompt as text and let the model read all of it. for a long context that means thousands of tokens the decoder has to chew through before it can say a single word — that first-word delay is time-to-first-token, TTFT, and it grows with the context.

REFRAG's move: don't feed the passages as text. compress each little chunk into a single embedding with a light encoder, hand those embeddings straight to the decoder, and let a small reinforcement-learning policy decide which few chunks actually need to be expanded back to full detail. roughly 16× fewer things for the decoder to prefill, so TTFT drops hard.

it is not a flag you flip

here's the catch that decides everything. REFRAG feeds the decoder embeddings instead of tokens — which means you need a hook into the decoder's input layer, and you need to have trained a matched encoder and projection against that specific decoder (continual pre-training, then fine-tuning). you cannot do this over a hosted chat-completions API, and you cannot do it to a 400B-parameter model on a single box.

so "adopt REFRAG" doesn't mean adopt anything. it means: stand up a separate, smaller decoder, run a multi-week training recipe, fork your serving layer to accept embeddings, and then maintain a second model in the fleet. that is a multi-month research project, not a feature. a project that size deserves a number attached before you start it — not after.

phase 0: measure what you already have

so before any of that, one question: what would REFRAG have to beat? not the number in the paper — the number on my stack, today.

i wrote a small benchmark — maybe 250 lines — that hits my real qwen-3.5 endpoint the same way the tutor does: system prompt, a book's worth of text, a question, streaming on. it sweeps the context from ~1k tokens up to the server's ceiling and records TTFT at each size. the only trick that matters: it measures each size twice.

• cold — a fresh, never-seen prompt. the decoder prefills the whole thing from scratch.
• warm — the identical prompt sent again immediately, so the serving layer's prefix cache is hot.

that distinction is the whole game, and it's the one the paper's headline quietly skips.

what came back

context (tokens)	cold TTFT	warm TTFT	cache gain
1.2k	0.50s	0.43s	1.2×
4.6k	0.82s	0.61s	1.4×
9k	1.01s	0.62s	1.6×
18k	2.05s	0.60s	3.4×
37k	5.39s	1.10s	4.9×
74k	11.07s	1.20s	9.2×
90k	14.44s	1.44s	10.0×

two things fell out of this, and one of them killed the project.

the cold curve is genuinely bad, and then it hits a wall. 11 seconds to first token at 74k tokens, 14 at 90k — climbing roughly linearly. and at exactly 93,100 tokens the server stops playing: Input length exceeds the maximum allowed length. a 400-page book is 150k+ tokens, so it never earns a slow prefill — it gets rejected outright, and the whole-book path quietly falls back to retrieval. the slowest a user waits is ~14 seconds, right at the ceiling. that's the pain REFRAG is built for, and it's real.

but almost nobody pays it. the prefix cache already flattens warm TTFT to under a second and a half, all the way up the curve — 9× at 74k, 10× at the ceiling — and i get it for free, today, with no training and no serving fork. and here's the sleight of hand: the paper's 16× is measured against an uncached baseline. my real baseline on any repeat turn is warm. REFRAG wasn't racing my cold number — it was racing my warm one, and against that its margin is thin.

when it would still pay

measuring doesn't mean the answer is always no. it means you know exactly where the remaining win lives:

• the first touch. the very first question on a fresh book still eats the full cold cost. but i can kill that for free by pre-warming the cache when the user opens the book — hours of work, not months.
• memory, not latency. the one thing the cache cannot give me: REFRAG stores ~16× less key-value cache per context, so i could fit far more concurrent long-context users on the same GPU. my single-stream benchmark says nothing about that — it's the one axis where REFRAG might still earn its keep, and the next thing worth measuring.

everything else the paper promised, my serving layer already delivered.

the actual lesson

i've watched good engineers — myself included — read a paper, fall for the headline number, and start building. the number is almost always real. it's just measured against a baseline that isn't yours.

phase 0 costs an afternoon and a couple hundred lines. a research project costs a quarter. the afternoon told me the quarter would have bought a 9× speedup i already had. the benchmark isn't the boring step before the work. sometimes it is the work — the thing that tells you the most valuable move is the one you don't make.