It sounds like a mismatch on paper. As a modern accelerator, the NVIDIA B200 delivers roughly 9,000 trillion operations per second. A fast NVMe drive moves a few gigabytes per second, sequentially. Compared head-to-head on raw throughput, the GPU isn't just faster, it isn't in the same category.
And yet, for one of the most frequent LLM operations in production, reloading a conversation's context after it's been pushed out of GPU memory, the drive wins, and not narrowly. At long context lengths, storage beats the GPU by more than 50x. This piece works through why that happens, what it means for telco-scale inference architecture and where the crossover point actually sits.
This builds on an earlier independent technical analysis the author published covering the same underlying mechanics; the walkthrough below extends it with the operational framing that matters for serving infrastructure at scale.
Transformer-based language models process text as a sequence of tokens, passed through a stack of identical layers. Inside each layer, the attention mechanism computes three vectors per token: Query, Key and Value. To generate the next token, the model compares its current Query against the Keys of every token that came before it, which means those prior Key and Value vectors must remain accessible for the duration of the request.
The KV cache is just that accumulated store, every Key and Value vector for every token, across every layer, kept around so the model never has to redo the work of computing them. Skip straight to the next section if this part is already familiar territory. The operational story is what matters here.
Here's a detail that surprises people outside the inference world: during token generation, a GPU spends most of its cycles waiting, rather than computing. Producing a single token for a single request means reading through the model's entire weight set, hundreds of gigabytes, and that read dominates the timeline. In the language of the field, the chip is memory-bound rather than compute-bound.
Batching is the lever that fixes this. Model weights get read once per generation step, no matter how many requests share that step, so running 100 requests in parallel extracts roughly 100x the useful computation from the same memory pass. The ratio of computation performed to bytes read is called arithmetic intensity, and it's what determines whether a GPU is actually earning its keep or sitting mostly idle.
Where that crossover happens depends entirely on the model. A 7-billion-parameter model might saturate GPU compute at a batch of around 10 concurrent requests. DeepSeek-V3, a 671-billion-parameter mixture-of-experts model, needs roughly 177 concurrent requests on the same class of hardware to reach the same point. Same silicon, very different answer—because the model's shape, not the GPU's spec sheet, decides how much batching is required to stop wasting cycles.
Every token held in a request's context—the prompt, the conversation so far, any uploaded document—carries its own KV cache entry, and that entry has to live in GPU high-bandwidth memory for as long as the request is active. As a rough guide, 10,000 tokens works out to roughly 7,500 words, or about 30 pages of text. The relationship between context length and memory consumed is linear: double the tokens, double the HBM footprint.
The per-token cost varies by model and precision. For a model like Gemma 4 31B Dense running at BF16, each token costs around 480 KB of HBM, which means a single 128K-token request alone eats close to 63 GB, leaving almost nothing else for an H100 to work with. Dropping to FP8 precision cuts that in half, to roughly 31.5 GB per request, which is by far the cheapest way to relieve memory pressure without touching hardware.
Put the two pressures together, and the conflict becomes obvious: hitting compute saturation requires many concurrent sessions held in memory at once—177 of them, in the DeepSeek-V3 example—but each of those sessions is also consuming HBM at the rate above. At 480 KB per token, 177 sessions each carrying a 32K-token context already add up to more than 2.7 terabytes, well past what any current GPU cluster can hold in memory simultaneously. In practice, the system has no choice but to evict older caches to make room for new traffic. When an evicted session returns, its cache has to be reconstructed, and either recomputed or reloaded.
When a session's cache has been evicted, and that user sends a follow-up, the serving system has two options:
Both paths cost something. The only real question is which one costs less time. That turns out to depend heavily on context length.
Recompute cost is measured in FLOPs, the floating-point operations a GPU has to perform. For DeepSeek-V3, processing a 10,000-token conversation history requires on the order of 1.4 quadrillion FLOPs, which an 8-way B200 cluster handles in roughly 85 milliseconds.
The trouble is that this cost doesn't scale gently. With full attention, every token must be compared against every other token in the context, so the computational burden grows quadratically rather than linearly. At 10K tokens, attention itself accounts for only about 13% of total FLOPs spent on the request. Push that same request out to 128K tokens, and attention's share climbs to roughly 65%, meaning doubling the context very nearly quadruples the total work. That's the part that makes recompute dangerous at scale: it isn't a proportional tax, it's a trap that gets worse the longer a conversation runs.
Newer architectural choices, including sliding-window attention in Gemma-4, and sparse attention in DeepSeek-V3.2, push back against this by limiting how many neighboring tokens each token actually attends to, which changes the underlying FLOPs formula and softens, but doesn't eliminate, the quadratic growth.
Reload cost is a much simpler story: it's bound by sequential transfer speed. A single PCIe Gen5 NVMe drive reads at roughly 12 GB/s, and an 8-drive server aggregates that to around 96 GB/s. DeepSeek-V3's compressed KV format, called Multi-head Latent Attention (MLA), stores just 35 KB per token—a fraction of what most architectures need. At that footprint, reloading a 10,000-token cache takes about 4 milliseconds. Stretch that out to 128,000 tokens, and reload time only climbs to roughly 47 milliseconds.
There's a small fixed overhead on top of any NVMe read, on the order of microseconds to low milliseconds, but at the transfer sizes involved here, it doesn't meaningfully change the picture. The defining property of reload cost is that it scales linearly with token count. More tokens simply means more bytes to move. There are no quadratic surprises and no hidden acceleration in cost as context grows.
Define a single decision ratio: R equals recompute time divided by reload time. When R is greater than 1, storage wins and reloading is the faster path. When R is below 1, recomputing on the GPU is still cheaper.
At 128K tokens specifically: recompute lands around 2,700 milliseconds, reload around 47 milliseconds, putting R at roughly 57. The drive isn't just faster here; it's faster by more than an order of magnitude.
Picture a network operations assistant deployed at a regional edge site, used by engineers troubleshooting recurring RAN issues. A technician opens a long diagnostic session in the morning, uploading site logs, prior maintenance notes and a multi-week history of alarms, building up a context well past 100K tokens by the time they've worked through the issue. They step away for two hours to conduct a physical inspection and then come back with a follow-up question.
In that two-hour gap, under any realistic traffic load, that session's KV cache has almost certainly been evicted from GPU HBM to make room for other active sessions across the site. The system now faces exactly the recompute-versus-reload choice described above, except the stakes are concrete: recomputing a context that large could mean seconds of dead air before the assistant responds, on a connection where the technician is standing in a cabinet expecting an immediate answer. Reloading from local NVMe, by contrast, keeps that response in the tens-of-milliseconds range regardless of how long the original session ran.
This is precisely the scenario where the architectural choice in the previous section stops being theoretical. A storage tier sized and placed to hold evicted KV caches at the edge, rather than discarding them and forcing every reconnect through full recompute, is the difference between a diagnostic tool engineers trust and one they learn to work around.
DeepSeek-V3 is a compelling example of the storage argument because its compressed KV format keeps so little data per token. Most production models don't get that advantage, and the ratio shifts accordingly.
The mainstream approach, Grouped Query Attention (GQA), scales KV cache size with the number of layers, attention heads, and hidden dimensions in the model. Llama-3 70B at FP8 precision stores 163 KB per token under this scheme, meaning a single 32K-token session alone holds more than 5 GB of cache. DeepSeek-V3's Multi-head Latent Attention (MLA) takes a fundamentally different path, compressing Keys and Values into a compact latent representation instead of storing them in full, landing at just 35 KB per token, nearly five times leaner than Llama-3 70B.
That gap affects the decision ratio because a smaller per-token footprint decreases reload time while leaving recompute time untouched. The architecture doesn't change how much FLOPs a recompute costs, only how many bytes a reload has to move. A leaner KV format pushes R higher across the board. For example, a 10K-token Llama-3 70B cache runs about 1.6 GB and takes roughly 17 milliseconds to reload at 96 GB/s; the equivalent DeepSeek-V3 cache is just 351 MB and takes about 4 milliseconds, a 4x gap from architecture alone, with everything else held constant.
Quantization is the second lever, independent of architecture. FP8 halves KV cache size relative to BF16, and more aggressive schemes like INT4 compress further still. Every byte shaved off the cache speeds up reload time, which means quantization doesn't just relieve HBM pressure, it actively pushes the recompute-versus-reload decision further in storage's favor, even at shorter context lengths where the case might otherwise be marginal.
The throughput specs on a GPU's datasheet describe its theoretical ceiling, not what it's actually doing during a typical inference request. At low batch sizes, token generation uses a sliver of a B200's total capacity, and the cost of reconstructing a long conversation from scratch grows quadratically with context length, while the cost of pulling that same conversation off a drive grows linearly. Those two growth curves are why the match stops favoring the GPU past a certain context length.
When running a server stack at distributed edge or MEC sites where every millisecond of round-trip matters more than it does in a centralized data center, the practical takeaway for anyone is that local fast storage isn't just a safety net for when GPU memory runs out. Past the crossover point, it's the faster path outright, and provisioning for it deliberately is a capacity-planning decision, not an afterthought.
Where that crossover sits depends on the specific model, hardware generation and traffic pattern in play, but the underlying shape of the problem, quadratic recompute against linear reload, doesn't change. Knowing where that line falls for a given deployment is what separates a serving architecture that scales cleanly from one that quietly recomputes its way into a latency problem.
The author, Jagadish Mukku, writes on Medium. You can follow him here.
