As mobile operators push generative AI deeper into the network, from customer-facing assistants to RAN-side automation, one infrastructure question keeps resurfacing in planning meetings: what storage should sit behind our inference workloads? It is the wrong question to start with because it assumes inference is one workload. It isn't.
Every AI response a network produces, whether it's a triage suggestion from a field-service copilot, a personalized retail recommendation, or an automated RAN anomaly explanation, is actually assembled from several distinct data flows working together. Each of those flows has a different shape, a different tolerance for loss, and a different relationship with time. Treating them as one storage problem is how telco AI deployments quietly become slower, costlier, and harder to scale than they need to be.
This article breaks down what's actually happening behind the scenes of a telco-grade inference stack—five data workloads, each with its own demands—and what that means for how operators should architect storage across core, edge, and RAN. It builds on an earlier general-audience analysis the author published independently, extended here, with the telco-specific edge and RAN considerations that determine how this plays out in a network environment.
Strip away the marketing language around “AI-native networks” and what you're left with is a pipeline. A request comes in, a customer query, a sensor reading, a network event, and before a model produces an answer, several things have to happen in sequence, each touching storage differently:
These aren't five flavors of the same storage requirement. They're five different problems wearing the same trench coat. A telco architecting its AI infrastructure around a single storage class, whether that's because it's what's already deployed, or because it seemed simpler to standardize., is solving for the average of five very different needs, which means it's solving for none of them well.
Before going workload by workload, here's the comparison that the rest of this article is built around. The baseline replica counts below assume standard fault tolerance; an operator's own failure-domain requirements—particularly at distributed edge sites—may call for more.
Before a network-facing model can answer a single query, it must be fully loaded from storage into GPU memory. Depending on the model, that's anywhere from tens of gigabytes to the better part of a terabyte, read sequentially, in one pass.
This load happens at startup, but calling it non-critical would be wrong. When an inference engine crashes and restarts, it cannot process a single request until the entire model is back in GPU memory. Load time is recovery time. In a telco environment where edge sites or MEC nodes may restart GPU fleets in coordinated batches—after a software rollout, a regional failover, or an unexpected outage—every server in that fleet races to load the same model simultaneously. A single storage copy becomes a bottleneck across the whole site, and every second the model isn't loaded is a second the site isn't serving.
The right response is multiple read replicas of frequently loaded models—not for fault tolerance in the traditional sense, since the model registry upstream remains the authoritative copy and any lost replica can be re-fetched—but purely to distribute simultaneous read load across the fleet and bring recovery time down. Snapshots and backups add nothing here. The registry is the source of truth; the inference-tier copy is a performance asset, not a primary record.
Of all the workloads in an inference pipeline, KV cache has the least tolerance for a slow storage path. During the prefill phase, the model generates key-value tensors for every attention layer, stored as the KV cache. When a later request shares the same prefix, the engine reads from cache instead of recomputing, and at production volumes that difference quickly compounds into the gap between a response that feels instant and one that lags.
This has a direct architectural consequence: KV cache wants to live as close to the GPU as possible. Co-located NVMe—drives inside the same chassis as the accelerator, connected over PCIe—keeps this path fast and low-latency. Where GPU Direct Storage (GDS) is supported, data moves straight from disk into GPU memory without the extra hop through system RAM, eliminating a step that matters when prefill latency is directly visible to the end user.
For scheduled or predictable workloads, KV cache can be pre-staged from cheaper object storage down to local NVMe ahead of time, so the live serving path never waits on a network transfer. Unpredictable requests stay on fast local storage; scheduled ones are hydrated in the background.
Two replicas are generally sufficient, enough to absorb a node failure or a cache-warming surge on restart, without overbuilding for inherently disposable data. KV cache needs no backups: if an entry is lost, the engine recomputes it from the original prompt, and TTL-based expiry handles cleanup automatically.
Everything above is, in some sense, recoverable or re-derivable. Conversational history is not. Once a multi-turn exchange between a customer and an AI assistant happens, it's the actual record of what was said. There's no upstream source to regenerate it from if it's lost.
The I/O shape reflects that: small, frequent writes as each turn gets appended, and full-session reads when context is reloaded for a new turn. Some teams will access this through a plain filesystem; others will route it through a database layer fronting that storage. Both are legitimate, and the underlying storage needs to support either pattern without forcing a choice.
This is also the one workload in the stack where three replicas and proper backup and snapshot capability genuinely earn their cost—point-in-time recovery matters for compliance, for disaster recovery, and simply because customer conversation data carries real consequences if it's lost or corrupted. It's the closest thing in this entire pipeline to conventional enterprise storage, and it should be planned for accordingly rather than bundled in with the more disposable workloads around it.
Telco use cases for vector search are expanding fast, matching a customer's support query against a knowledge base without invoking a full model, surfacing the right equipment manual for a field technician, or routing a network alert to the right runbook. Underneath all of it is the same I/O pattern: large, sequential writes when embeddings are generated and indexed, followed by small, random reads at query time as the index is searched.
That lopsided pattern, heavy, batched writes on the way in, scattered single-record lookups on the way out, has more in common with how a classic search engine index behaves than with anything else in this pipeline, and it deserves its own storage planning.
Three replicas are warranted because rebuilding a vector index isn't cheap—regenerating embeddings across a large corpus consumes real GPU time and real cost, and losing an unprotected index means redoing all of it. Snapshots matter for the same reason: an index can be corrupted mid-update or mid-compaction, and the ability to roll back to a known-good state avoids a full re-ingestion.
The document store is what grounds a model's answer in something real rather than letting it improvise from training data alone—the manuals, policy documents, prior support tickets, or network documentation that retrieval-augmented generation pulls from at query time.
The I/O pattern looks similar to vector storage in shape, bulk sequential writes during ingestion, random reads at query time, but the redundancy decision depends on something different: where the data actually originates. If the store is a synced copy of content that lives authoritatively elsewhere, heavy backup investment buys little. If it holds original or curated content with no upstream copy, internal runbooks, or proprietary network documentation, it deserves the same treatment as primary storage, replicated and backed up in full.
It helps to walk through a concrete scenario end-to-end because the five workloads rarely show up one at a time in production. They show up together within the same few seconds for the same request.
A field technician opens a mobile app to ask why a particular cell site keeps dropping a sector. Behind that one question, an MEC-hosted inference stack typically has to: load (or already have loaded) the diagnostic model into GPU memory; run retrieval against a vector index built from the site's maintenance history and equipment manuals to find the relevant passages; pull the actual manual pages and prior ticket notes from the document store; generate a KV cache for the assembled context as the model reasons over it; and then append the exchange to that technician's session history in case they follow up with a second question a minute later.
Each of those five steps is hitting a different storage tier with a different latency budget. The vector lookup and document retrieval can tolerate a few extra milliseconds without the technician noticing. The KV cache generation cannot — prefill can take anywhere from hundreds of milliseconds to several seconds depending on context length, making it one of the most consequential steps in the entire response path, which is exactly why fast, local storage at the serving site matters more here than anywhere else. The session append, by contrast, can happen asynchronously after the answer is already on screen, because nothing about the user experience depends on it completing in real time.
Architected as one undifferentiated storage pool, this request inherits the latency profile of its slowest, most conservative component. Architected workload-by-workload, the technician gets an edge-fast answer while the durability-sensitive parts of the transaction settle in the background, which is the whole argument for treating these as five problems rather than one.
None of this points to a single winning interface across the board—the right one depends on the workload's latency tolerance and what's already deployed in the environment.
For most enterprise AI deployments, the choice between co-located and external storage is a cost-versus-convenience decision. For telco AI, particularly anything running at MEC sites, regional data centers, or RAN-adjacent compute, it's closer to a physics decision.
Co-located NVMe, sitting inside the same server as the GPU, delivers the lowest possible latency and is the only sensible home for KV cache in any deployment where prefill speed is user-visible, which, in telco contexts like real-time customer interaction or closed-loop network automation, is most of them. The tradeoff is that capacity is bound by chassis space and can't scale independently of compute.
External, networked storage—typically connected over RDMA—decouples capacity from compute and lets a storage tier be shared across multiple GPU servers or even multiple edge sites. The latency cost is real but shrinks considerably with RDMA, making it a sound choice for model weights, document stores, and vector indexes, where access patterns are far less latency-critical than KV cache.
In practice, telco deployments that get this right end up running both: KV cache pinned to local NVMe at the edge for latency, with model weights, documents, and vector indexes living on shared external storage that scales independently and serves multiple edge nodes. Done well, the inference stack treats this as one logical namespace—the split between co-located and external is an infrastructure decision, not something application teams need to reason about.
The instinct to simplify AI infrastructure by standardizing on one storage system is understandable, with fewer vendors, fewer contracts, fewer things to monitor. But inference doesn't simplify that easily. Model weights need bulk throughput and read parallelism. KV cache needs the shortest possible path to GPU memory. Conversational data needs durability and the ability to roll back. Vector indexes and document stores need protection for content that's genuinely expensive to rebuild.
For telco operators building out AI at the network edge, where the latency stakes are highest and the infrastructure is already distributed by nature, this workload-first thinking isn't an optimization—it's the baseline requirement for inference that actually performs the way the use case demands.
Get the workload analysis right first, and the storage architecture that follows from it tends to be obvious.
The author, Jagadish Mukku, writes on Medium. You can follow him here.
