This text was written using Google Translate.
I am not proposing that we equip cards with expensive AI-specific chips like HBM or HBF.
I currently use a 3060 OC.
I get frustrated every day due to a lack of memory.
Right now, consumer GPU prices refuse to drop because of HBM.
So, let’s try a different approach.
Let’s add an expansion slot to consumer GPUs.
We could install inexpensive memory into this slot.
Currently, the only option available to us is NVMe.
I am not proposing this for general PC storage expansion, like with the RTX 4060 Ti.
I want it to be used to expand that damn GPU memory.
It doesn’t even need to show up in Windows Device Manager or diskpart.
It would simply be used to load things like model caches.
The advantage is that memory is easy to acquire and can be swapped out based on the required model cache capacity.
The downside is that performance will vary depending on the NVMe module used.
I want an end to this situation where I can’t get an RTX 5060 Ti or better because of HBM.
I actually think the idea of expandable GPU memory is interesting, especially for AI workloads.
Using NVMe directly as VRAM would probably be too slow for actively used tensors, but it could make sense as a lower-tier cache for model weights or data that isn’t constantly accessed. CUDA already supports GPU memory oversubscription through Unified Memory, although performance depends heavily on how often data has to move between memory tiers.
What I would really like to see is a consumer GPU with modular VRAM, or a proper memory hierarchy where additional memory could be added without replacing the whole GPU.
Oh. It looks like there are more precedents for this idea than I expected:
I think the useful version of this idea is not really “make an NVMe SSD behave like another 16/32/64 GB of GDDR”, but rather:
fast GPU memory
active weights / active KV / active experts
↓
system RAM
staging / prefetched / near-term state
↓
NVMe / flash
colder / reusable / retained state
That kind of hierarchy is much less hypothetical than I initially thought.
There are historical hardware examples, current software implementations, research systems, and now even commercial products that treat flash as an additional AI memory capacity tier. The recurring pattern is that flash does not become equally fast VRAM; instead, software decides what can live in the slower tier and when it should be promoted back toward the GPU.
A surprisingly close current example is Phison’s Pascari aiDAPTIV. Phison describes its SSD as a flash-based capacity tier in an AI memory system, with GPU memory holding immediately active state, system RAM used for staging/prefetch, and flash retaining less-active state such as KV cache, colder MoE experts, or model/training data. It even has M.2 variants aimed at systems including discrete-GPU desktops. Phison: How Pascari aiDAPTIV works Phison: aiDAPTIV Cache Memory
So I think your basic idea has a real design space behind it. The important distinction is probably:
expandable fast VRAM and expandable GPU-managed backing/cache capacity do not have to be the same thing.
The second one looks much more practical with current technology.
One caveat: aiDAPTIV is useful here mainly as evidence that the architecture is real, not as a drop-in solution for your RTX 3060. Its currently published middleware is Linux-only, requires its supported cache-memory SSD setup, and the public aiDAPTIVLink 2 branch is primarily focused on post-training/fine-tuning; Phison says KV-cache inference features belong to aiDAPTIVLink 3. Their current compatibility guidance is also well above a 12 GB 3060 for the inference configurations they validate. aiDAPTIV official GitHub repository and current requirements
What could actually go into the slower tier?
I think this is where “model caches” needs to be split into a few cases. They have very different access patterns.
1. Downloaded model files
If this just means the Hugging Face model cache / GGUF files / checkpoints before execution, then ordinary NVMe storage already solves it. That is storage capacity, not GPU-memory expansion.
2. Model weights that are not needed simultaneously
This is much closer to a useful memory tier.
Hugging Face Accelerate already supports disk offload: weights can remain as memory-mapped arrays on disk and be moved to the execution device as they are needed during the forward pass. Hugging Face Accelerate: big-model disk offload
That is obviously slower than keeping the whole model resident in VRAM, but it demonstrates an important point: a model does not necessarily have to reside entirely in GPU memory for GPU execution to be useful.
Layer-by-layer streaming, prefetching, and overlap between I/O and compute then become the important engineering problems.
3. Reusable / completed KV cache
This may be one of the best fits.
Current vLLM has an OffloadingConnector specifically for moving completed KV blocks into slower but larger tiers. Its multi-tier design is:
GPU <-> CPU primary tier <-> filesystem / other secondary tiers
Only the CPU primary tier talks directly to GPU memory; secondary tiers are promoted through CPU memory before returning to the GPU. Transfers between GPU and pinned CPU memory are asynchronous. vLLM KV Offloading Usage Guide
That is very close conceptually to the hierarchy you are describing, except it is implemented at the runtime level rather than by presenting the SSD as additional VRAM.
It also illustrates why “cache” matters. A completed prefix that may be reused later can tolerate a slower backing tier much better than data needed immediately by the next matrix multiplication.
Phison’s current KV-cache material makes essentially the same distinction: GPU/unified memory remains the primary working memory while lower tiers retain eligible less-active KV state. Phison: KV Cache Extension & Reuse
4. Active decode KV cache
This is harder.
During generation, the model repeatedly needs the active attention state. If an access misses and the GPU must wait for SSD → RAM → GPU promotion, latency can quickly dominate.
So I would not expect NVMe to behave well as arbitrary live KV memory in the same sense as GDDR. A useful system would try to keep the active working set high in the hierarchy and use flash for retained/cold/reusable parts.
5. Mixture-of-Experts weights
MoE is especially interesting because a model can have a huge total expert pool while activating only a few experts for each token.
That gives software something exploitable: keep hot experts close to the GPU and colder experts lower down.
Phison is explicitly pursuing this with its Dynamic MoE design. Phison: Dynamic MoE
There is also a very relevant recent llama.cpp example involving an RTX 3060 12 GB. In that particular setup, expert weights were kept in system RAM and copied to the GPU as required. Profiling showed that serial host-to-device transfers, rather than GPU computation, were leaving the GPU idle. Using pinned memory and overlapping the next expert transfer with the current layer’s computation improved that test substantially.
The author is careful to say the measurement was on a particular older llama.cpp revision and had not yet been repeated against current master, so I would treat the numbers as one profiling result rather than a universal performance claim. But the mechanism is very relevant here: once capacity is moved out of VRAM, data movement and scheduling become the next bottleneck. llama.cpp issue #25859: RTX 3060 MoE offload profiling
There are also active llama.cpp proposals around persistent GPU expert caches plus pinned-RAM backing stores, which is basically the same “small fast tier + large slow tier” idea applied specifically to sparse experts. llama.cpp issue #20757: two-tier GPU + RAM expert cache
6. Arbitrary active tensors
This is the difficult case.
If the GPU may access arbitrary data at fine granularity, unpredictably, and needs it immediately, then an SSD miss behaves very differently from a GDDR access.
This is where simply exposing “500 GB of virtual VRAM” could become misleading: the address capacity may exist, but repeated movement between tiers can make the workload unusably slow.
So I think a useful implementation needs to understand at least some of the semantics of the workload: weights, KV blocks, experts, training state, etc., rather than treating every allocation identically.
There are older precedents too
The idea actually goes back quite a while.
AMD’s Radeon Pro SSG in 2017 combined a Vega GPU with 2 TB of onboard solid-state storage. AMD marketed it around removing local-memory limits for very large professional datasets. It was not simply “2 TB of normal VRAM,” but it is a very direct historical example of putting a large solid-state capacity tier physically on the graphics card. AMD Radeon Pro SSG announcement
AMD’s own annual report described it as providing 2 TB of onboard SSG memory alongside the Vega GPU. AMD 2017 annual report
There is also a large research literature around the same general problem: out-of-core GPU execution, SSD-backed tensors, GPU-initiated storage, layer streaming, and compiler/runtime-managed migration.
So the general problem is well established. What is less established on ordinary consumer GPUs is a simple, vendor-supported, user-expandable hardware interface where someone can insert an inexpensive capacity module and have normal AI runtimes automatically use it well.
The recent direction of the industry is interesting here too. At GTC 2026, NVIDIA described Storage-Next explicitly as a “new memory tier,” with an emphasis on very fine-grained access and IOPS per dollar/watt rather than treating storage only as large sequential files. NVIDIA GTC 2026: Storage-Next memory tier
That is data-center-oriented work, not a promise of an M.2 slot on the next GeForce card, but it suggests that the underlying distinction — storage versus memory — is becoming less binary for some GPU workloads.
Why the software side may matter as much as the slot
A removable M.2 slot would solve the capacity and replaceability part very nicely.
But performance would depend on more than the advertised sequential speed of the SSD.
A runtime would also need to care about things such as:
- which state is hot enough to keep in VRAM;
- what should be staged in RAM;
- what is safe to evict to flash;
- how far ahead it can prefetch;
- whether transfers can overlap GPU computation;
- how much data must move per generated token;
- cache hit / reuse rate;
- access size and random-I/O behavior;
- PCIe bandwidth and topology;
- eviction policy;
- SSD latency and endurance.
This is one reason the current implementations tend to expose a hierarchy rather than pretending all tiers are equivalent.
vLLM, for example, explicitly makes CPU memory the gateway between GPU memory and secondary tiers. Its current implementation even has separate tier managers, promotion behavior, and eviction protection. vLLM tiered-offload manager documentation
Likewise, Phison’s architecture separates immediate GPU state, RAM staging, and retained flash state instead of calling the whole thing one homogeneous pool. Phison aiDAPTIV overview
So if consumer GPU vendors ever did adopt something like your proposed slot, I suspect the really useful part would be not just the electrical M.2 connection but a supported contract between:
GPU/driver
↕
memory manager / AI runtime
↕
RAM staging
↕
flash capacity tier
Without that policy layer, a large nominal memory pool could spend too much time waiting for promotion from the slow tier.
There is also a small caveat to the Unified Memory point above.
Since you mentioned Windows Device Manager and diskpart, Windows may be relevant to your setup. CUDA does have Unified Memory and GPU-memory oversubscription on supported platforms, but the current CUDA Programming Guide documents an important Windows limitation: in the Windows / concurrentManagedAccess == 0 model described there, managed allocations cannot oversubscribe physical GPU memory, and fine-grained GPU page-fault migration is not available. So Unified Memory should not be read as “a Windows RTX 3060 can already transparently use arbitrary RAM as extra VRAM.” CUDA Programming Guide: Unified Memory on Windows/WSL
One other minor terminology point: the RTX 5060 Ti itself uses 8 GB or 16 GB of GDDR7, not HBM. NVIDIA RTX 5060 Ti specifications
That does not invalidate the broader frustration about GPU memory capacity or the effects of AI demand on the memory market; I would just separate that market question from the proposed architecture.
As for modular GDDR VRAM itself, I think that is a separate branch of the idea. A replaceable fast-memory module would be attractive, but engineering a socketed high-speed GPU-memory interface is a different problem from adding a replaceable NVMe capacity tier. The latter already has standardized, inexpensive, swappable hardware and a substantial software/research precedent behind it.
So if I had to pick the most realistic default version of your idea, it would be:
Keep normal high-bandwidth VRAM on the card.
Add a supported, replaceable lower-capacity tier
(M.2/NVMe or something derived from it).
Let the driver/runtime distinguish hot working state
from cold/reusable state and migrate/prefetch accordingly.
That would not turn a 12 GB RTX 3060 into a card with hundreds of gigabytes of equally fast VRAM.
But for workloads where only part of the total state has to be active at once — reusable KV cache, sparse MoE experts, staged model weights, some training state, etc. — it could make the difference between “does not fit at all” and “fits, with an I/O/performance tradeoff.”
And for anyone wanting to explore the idea in software before specialized hardware exists, there are already cheap approximations: Hugging Face Accelerate’s disk offload for weights, CPU offload, and current KV-tiering implementations can help show where a particular workload changes from capacity-bound to data-movement-bound. That seems like a more informative experiment than treating raw SSD bandwidth alone as the deciding number.
So overall: I think #4’s “proper memory hierarchy” framing is probably the right direction, but there is considerably more precedent behind it than I expected — including systems that are now explicitly treating flash as an AI memory tier rather than merely as ordinary file storage.