DeepSeek's flagship model has 671 billion parameters, and only 37 billion of them fire on any given token. So how much VRAM does it need? All 671 billion parameters' worth: roughly 382 GB even quantized to 4 bits, because a mixture-of-experts model must hold every expert in memory whether or not this token uses it. Active parameters buy you speed. They do not buy you memory, and that single misunderstanding is where more AI hardware budgets die than anywhere else.
VRAM is the one hard constraint in running language models: a card with too little does not run the model slowly, it does not run it at all, or offloads to system RAM and crawls. This guide gives you the complete answer for 2026: the ten-second formula that sizes any model, the reference table covering every major open model at every quantization level that matters, the KV cache math that catches long-context users off guard, and which GPU tier, from a 24 GB consumer card to a 96 GB RTX Pro 6000, actually fits your workload.
VRAM is the one hard constraint in running language models: a card with too little does not run the model slowly, it does not run it at all, or offloads to system RAM and crawls. This guide gives you the complete answer for 2026: the ten-second formula that sizes any model, the reference table covering every major open model at every quantization level that matters, the KV cache math that catches long-context users off guard, and which GPU tier, from a 24 GB consumer card to a 96 GB RTX Pro 6000, actually fits your workload.
The short answer
- The formula: VRAM for weights ≈ parameters (in billions) × bytes per parameter: 2.0 at FP16, 1.0 at Q8, 0.57 at Q4_K_M. Add KV cache and 15 to 20% overhead.
- Quick tiers (Q4 quantization): 8B models need ~6 GB; 32B models need ~20 GB (a 24 to 32 GB card); 70B models need ~40 GB (a 48 GB card); Llama 4 Scout needs ~60 GB (a 96 GB card, comfortably).
- The MoE trap: mixture-of-experts models load ALL parameters. DeepSeek-R1 671B needs ~382 GB at Q4 despite 37B active.
- The 2026 sweet spots: 24 GB runs excellent 32B-class models; 48 GB is where single-card 70B begins; 96 GB runs Llama 4 Scout on one card.
The formula: bytes per parameter decide everything
A model's weights are its parameters stored at some precision, so the arithmetic is genuinely this simple: parameters times bytes per parameter. At full FP16 precision each parameter takes 2 bytes; Q8 and FP8 store one byte; the GGUF quantizations most local setups run land lower still, about 0.68 bytes effective for Q5_K_M and 0.57 for Q4_K_M once their mixed-precision layers average out.
Work one example and the habit sticks: Llama 3.1 8B at Q4_K_M is 8 × 0.57 ≈ 4.6 GB of weights; add a modest context's cache and runtime overhead and it lands near 6 GB, which is exactly why it runs happily on an 8 GB card. The same arithmetic at the other end: Llama 3.3 70B is 140 GB at FP16 and about 40 GB at Q4_K_M, and in production serving its total footprint can pass 200 GB once cache and framework overhead pile on. Two more consumers share the card with the weights: the KV cache, which gets its own section below because it scales with context, and runtime overhead, the CUDA context, activations, and framework buffers worth a steady 1 to 2 GB, more under serving engines that pre-allocate.
The reference table: every major model, mid-2026
Weights-only figures at the three precisions that matter, with the card class each model fits at Q4. Models marked MoE load all parameters regardless of how few activate per token.
Three rows deserve a second look. Llama 3.3 70B at ~40 GB is the reason 48 GB cards changed the local AI market: the 70B class, where open models start competing with cloud APIs on quality, fits a single card for the first time. Llama 4 Scout (109B total, 17B active) at 55 to 62 GB runs on one 96 GB RTX Pro 6000 with room for long context, a frontier-class MoE on a workstation card. And gpt-oss 120B ships natively in MXFP4 at about 65 GB, built deliberately to fit a single 80 GB A100 or H100. The pattern across all three: the interesting action in 2026 is between 48 and 96 GB, the range where serious models stop needing servers.
KV cache: the hidden second consumer
The weights are only the entry fee. During inference the model stores attention state, the key-value cache, for every token in context, and that cache grows linearly with context length and with every concurrent user. It is the number-one reason a model that “fits” crashes the moment someone pastes a long document.
The worked numbers make it concrete. Llama 3.1 8B holds about half a GB of cache at 4K context, ~4.2 GB at 32K, and ~17 GB at 128K, at which point the cache outweighs the Q4 weights. The 70B version scales to ~42 GB of cache at full 128K context, a second GPU's worth of memory for state alone. Two mercies keep this manageable. Modern models ship with grouped-query attention: Llama 3.1 70B uses 8 KV heads instead of 64, an 8x cache reduction that is the quiet reason single-GPU 70B serving works at all. And the cache itself can be quantized to FP8 or INT8 for another 2x with minimal quality impact, natively supported in vLLM. Budget rule: whatever the weights need, reserve headroom for the context you actually intend to use, and multiply the cache by your concurrent users if you are serving.
What quantization costs you (and when it costs too much)
Q4_K_M is the local standard for a reason: it cuts memory 3.5x against FP16 while losing under 5% on quality benchmarks, a trade almost everyone should take. The formats you will meet: GGUF (the llama.cpp and Ollama family, quantization handled automatically), AWQ and GPTQ (GPU-serving formats common under vLLM), and increasingly native low precision, models like gpt-oss released directly in MXFP4 with no full-precision version intended for deployment. The one rule that saves people from a bad build: quality falls off a cliff below 4-bit, and a well-trained 8B at Q4 will beat a 14B squeezed to Q3 almost every time. Choose the largest model that fits at Q4 or better, not the largest number that technically loads. What that quality actually buys per model class is the subject of our best local LLMs rankings.
Ready to Start Mining?
Free worldwide DDP shipping. Professional hosting from $0.055/kWh.
Matching VRAM tiers to real hardware
The ladder has one rung that matters more than the others in 2026: 48 GB, where single-card 70B begins. Below it, the 16 to 24 GB consumer tier is genuinely capable, running the excellent 32B class (Qwen3 32B, Gemma 3 27B) that covers most personal and small-team use, and the 32 GB RTX 5090 adds the context headroom the 24 GB cards lack. Above it, 80 GB data-center cards bring FP8 serving and gpt-oss 120B, and the 96 GB RTX Pro 6000 runs Llama 4 Scout, frontier-class output, on a single workstation card. Which specific card wins at each tier on price and speed is what the GPU benchmark tool ranks across 78 cards, and the choosing guide turns into a decision.
The tier everyone asks about: a 48 GB card runs Llama 3.3 70B at Q4 with ~6 GB to spare for context. A 24 GB card cannot load it at all.
The tier everyone asks about: a 48 GB card runs Llama 3.3 70B at Q4 with ~6 GB to spare for context. A 24 GB card cannot load it at all.
The MoE trap: active parameters buy speed, not memory
Mixture-of-experts models dominate the 2026 frontier (DeepSeek-R1 and V3.2, Llama 4, Qwen3 235B, gpt-oss 120B), and they all share the property that breaks intuition: every expert must sit in VRAM, because the router picks different experts token by token. The active-parameter count sets your speed, since each token only computes through that subset, and the memory-bandwidth logic from our benchmarks explainer applies to the active slice. But the total parameter count sets your memory, full stop. DeepSeek-R1 671B runs surprisingly fast for its size precisely because only 37B parameters work per token; it still needs ~382 GB loaded, which in practice means an 8-GPU H200 server. When you evaluate any MoE, read the total, not the marketing number.
When one card is not enough: splitting and offloading
Two escape hatches exist when a model outgrows your VRAM, and they are not equal. Multi-GPU splitting (tensor or pipeline parallelism) shares the weights across cards and works well: two 48 GB cards hold what one 96 GB card holds, at the cost of inter-GPU traffic, which is why NVLink-connected setups covered in our multi-GPU server guide pull ahead of PCIe-only builds. CPU offloading spills layers to system RAM and is the option of last resort: system memory bandwidth is a tiny fraction of VRAM bandwidth, so every offloaded layer drags the whole pipeline toward CPU speed. A model 10% too big might lose half its speed. The honest hierarchy: fit it in one card if you can, split across GPUs if you must, and treat offloading as a way to test a model, not to run one. At fleet scale, the same constraint is why memory keeps headlining every new GPU generation.
What this table cannot decide for you
Three limits, stated plainly. First, the figures are weights at standard quantizations; your exact GGUF file, your engine's pre-allocation, and your context setting move the total, so treat the table as a floor plus the formula, not a guarantee, and check the model card's actual file sizes before buying anything. Second, fitting is not the same as performing: a model that loads with zero headroom will thrash the moment context grows, and the speed you get from a fitted model is a separate question, covered by the tokens-per-second math in the benchmarks methodology and measured per card in the tool.
Third, this guide covers inference; training and fine-tuning need several times more memory for gradients and optimizer states, a different budget entirely, sketched in our training GPU guide. Size for the workload you will actually run, then add the headroom you will inevitably need.
Third, this guide covers inference; training and fine-tuning need several times more memory for gradients and optimizer states, a different budget entirely, sketched in our training GPU guide. Size for the workload you will actually run, then add the headroom you will inevitably need.
The bottom line
VRAM sizing stops being guesswork the moment you hold the formula: parameters times bytes per parameter, plus cache, plus overhead. From there the 2026 map is short. An 8 GB card is a real entry point at 8B; 24 GB runs the 32B class that handles most real work; 48 GB is the new line that matters, putting 70B-class quality on a single card; 80 GB adds serving headroom and gpt-oss 120B; and 96 GB runs a frontier MoE on a workstation. Respect the MoE trap, budget the KV cache for the context you will actually use, and never plan around CPU offloading. Then match the tier to a specific card with the benchmark tool and price it in the GPU shop, and if the answer turns out to be “more than one card,” that is not a failure of planning; that is just what frontier-class models cost in silicon.
Frequently asked questions
How much VRAM do I need to run an LLM?
Depends on the model size and quantization. Quick guide at Q4_K_M (the local standard): 8B models need about 6 GB total, 14B about 10 GB, the 32B class about 20 to 22 GB, 70B models about 40 to 45 GB, and frontier MoE models like Llama 4 Scout about 60 GB. The formula for anything else: parameters in billions times bytes per parameter (2.0 at FP16, 1.0 at Q8, 0.57 at Q4_K_M), plus KV cache and 15 to 20% overhead.
How much VRAM does Llama 3.3 70B need?
About 40 GB at Q4_K_M, 74 GB at Q8, and 140 GB at FP16, weights only; add cache and overhead on top. In practice that means a single 48 GB card runs it at Q4 with modest context, an 80 GB card runs it at FP8 comfortably, and FP16 requires splitting across two 80 GB GPUs. This is the model class where 48 GB cards earn their price.
How much VRAM does DeepSeek-R1 need?
The distilled versions are modest: the 8B distill needs about 5 GB at Q4 and the 32B distill about 18 to 20 GB, fitting a 24 to 32 GB card. The full 671B model is another world: roughly 370 to 382 GB at Q4 and about 671 GB at FP8, requiring a multi-GPU server (typically 8x H200 for production). Only 37B parameters activate per token, which makes it fast, but every parameter must be loaded, which makes it enormous.
Can I run a 70B model on 24 GB of VRAM?
Not on the GPU alone. A 70B model needs about 40 GB even at Q4_K_M, so a 24 GB card must offload roughly half the layers to system RAM, which typically collapses speed to a few tokens per second. The realistic options: run the excellent 32B class instead (Qwen3 32B fits 24 GB and outperforms a heavily squeezed 70B), move to a 48 GB card, or split across two 24 GB GPUs.
What is the KV cache and how much VRAM does it use?
The cache of attention keys and values the model keeps for every token in context. It grows linearly with context length and with concurrent users: Llama 3.1 8B holds about 0.5 GB at 4K context but ~17 GB at 128K; the 70B version reaches ~42 GB at full context. Modern models reduce it 8x via grouped-query attention, and quantizing the cache to FP8 halves it again. Always budget cache for the context you actually plan to use.
Do MoE models need less VRAM?
No, and this is the most expensive misconception in local AI. Mixture-of-experts models compute through only their active parameters per token (which makes them fast) but must load every expert into VRAM (which makes them large). DeepSeek-R1 671B with 37B active still needs ~382 GB at Q4. When sizing an MoE, use the total parameter count; the active count predicts speed, not memory.
How much quality do I lose with Q4 quantization?
Typically under 5% on benchmarks against FP16, which is why Q4_K_M is the default for local deployment; most users cannot tell the difference in normal use. Quality degrades sharply below 4-bit, so the working rule is: run the largest model that fits at Q4 or better rather than crushing a bigger model to Q3 or Q2. A well-trained 8B at Q4 beats a 14B at Q3 almost every time.
Is 48 GB of VRAM worth it for local AI?
In 2026 it is the single most meaningful upgrade line, because it is where single-card 70B begins: Llama 3.3 70B and Qwen2.5 72B fit at Q4 with headroom, the class where open models start matching cloud APIs on quality. Below 48 GB you live in the (very good) 32B world; above it, 80 to 96 GB adds FP8 serving, gpt-oss 120B, and Llama 4 Scout. If 70B-class output is the goal, 48 GB is the ticket.
What happens if my model does not fit in VRAM?
Either an out-of-memory error or automatic CPU offloading, depending on the engine. Offloading spills layers to system RAM, whose bandwidth is a small fraction of VRAM's, so speed collapses toward CPU pace: a model 10% too large can lose half its throughput. Offloading is fine for testing whether a model suits you; for actual use, quantize smaller, pick a smaller model, or add VRAM via a bigger card or a multi-GPU split.
Do I need more VRAM for fine-tuning than for inference?
Substantially more. Training holds gradients and optimizer states alongside the weights, multiplying the footprint several-fold at full precision; even parameter-efficient methods like QLoRA, which freeze the base model at 4-bit, need meaningful headroom above inference. As a rough guide, comfortable QLoRA fine-tuning of a 70B model wants 48 to 80 GB where Q4 inference needs 40. Size training budgets separately from this table.
Sources and notes
Model memory figures compiled from published deployment guides, quantized release files, and framework documentation (llama.cpp/GGUF, vLLM), mid-2026; bytes-per-parameter values reflect standard GGUF quantization averages (Q4_K_M ≈ 0.57, Q5_K_M ≈ 0.68). KV cache figures use GQA-era architectures at FP16 cache precision and scale with the stated per-token formula. Exact requirements vary with the specific quantized file, engine pre-allocation, and context settings; check model-card file sizes before purchasing hardware. Q4 quality-loss figure reflects commonly reported benchmark deltas. Hero photo: Geekerwan, CC BY 3.0, via Wikimedia Commons. Body photo: PantheraLeo1359531, CC BY 4.0, via Wikimedia Commons. Product visual: MillionMiner catalog (RTX Pro 6000). Diagrams: original MillionMiner graphics, free to reuse with attribution and a link to this page. Informational content, not purchasing advi
Sources and notes
Model memory figures compiled from published deployment guides, quantized release files, and framework documentation (llama.cpp/GGUF, vLLM), mid-2026; bytes-per-parameter values reflect standard GGUF quantization averages (Q4_K_M ≈ 0.57, Q5_K_M ≈ 0.68). KV cache figures use GQA-era architectures at FP16 cache precision and scale with the stated per-token formula. Exact requirements vary with the specific quantized file, engine pre-allocation, and context settings; check model-card file sizes before purchasing hardware. Q4 quality-loss figure reflects commonly reported benchmark deltas. Hero photo: Geekerwan, CC BY 3.0, via Wikimedia Commons. Body photo: PantheraLeo1359531, CC BY 4.0, via Wikimedia Commons. Product visual: MillionMiner catalog (RTX Pro 6000). Diagrams: original MillionMiner graphics, free to reuse with attribution and a link to this page. Informational content, not purchasing advi
