AI Model Size / VRAM Calculator

About AI Model Size / VRAM Calculator

How VRAM Is Calculated

Component	Formula	Example (Llama 3.1 8B, Q4_K_M, 8K context)
Weight memory	Parameters x Bytes per parameter	8B x 0.57 = ~4.56 GB
KV cache	2 x Layers x Hidden size x Context x Bytes per value	Varies by architecture, typically 0.5-2 GB at 8K context
Runtime overhead	Total x 1.2 multiplier	Accounts for CUDA context, activations, framework memory
Total VRAM	Weight + KV cache + Overhead	~6-7 GB

Quantization Levels

Quantization	Bytes per Parameter	Size (7B model)	Quality vs FP16
FP32 (full precision)	4.0	~28 GB	Maximum (not used for inference)
FP16 / BF16	2.0	~14 GB	Near-identical to FP32
Q8_0	1.0	~7 GB	Minimal quality loss
Q6_K	~0.81	~5.7 GB	Very small quality loss
Q5_K_M	~0.69	~4.8 GB	Small quality loss
Q4_K_M	~0.57	~4.0 GB	Noticeable but acceptable; most popular choice
Q3_K_M	~0.44	~3.1 GB	More noticeable degradation
Q2_K	~0.31	~2.2 GB	Significant quality loss; use as last resort

Q4_K_M is the most commonly used quantization for local inference. It offers the best balance between file size and output quality for most use cases.

Popular Model Presets

Model	Parameters	FP16 Size	Q4_K_M Size	Min VRAM (Q4_K_M, 8K ctx)
Llama 3.1 8B	8B	~16 GB	~4.6 GB	~6-7 GB
Llama 3.1 70B	70B	~140 GB	~40 GB	~48 GB
Mistral 7B	7B	~14 GB	~4.1 GB	~5-6 GB
Gemma 2 9B	9B	~18 GB	~5.2 GB	~7 GB
Gemma 2 27B	27B	~54 GB	~15.5 GB	~19 GB
Phi-3 Mini (3.8B)	3.8B	~7.6 GB	~2.2 GB	~3 GB
Qwen2 7B	7B	~14 GB	~4.1 GB	~5-6 GB

GPU Compatibility

GPU	VRAM	Can Run (Q4_K_M)
RTX 3060	12 GB	Up to ~13B models
RTX 3080	10 GB	Up to ~8B models comfortably
RTX 3090 / 4090	24 GB	Up to ~30B models
RTX 4060 Ti 16GB	16 GB	Up to ~20B models
Apple M1/M2 (16 GB)	16 GB unified	Up to ~20B models (shared with system)
Apple M2/M3 Max (32 GB)	32 GB unified	Up to ~40B models
Apple M2/M3 Ultra (64 GB)	64 GB unified	70B models at Q4_K_M
A100 / H100	40-80 GB	70B+ models at higher quantization

KV Cache and Context Length

The KV (key-value) cache stores attention information for each token in the context window. Longer context lengths require more KV cache memory. Doubling the context length roughly doubles the KV cache size.

Context Length	Typical KV Cache (7B model, FP16 KV)	Use Case
2,048 tokens	~0.25 GB	Short conversations, simple prompts
4,096 tokens	~0.5 GB	Standard chat, most tasks
8,192 tokens	~1 GB	Longer documents, code review
32,768 tokens	~4 GB	Large documents, book analysis
131,072 tokens	~16 GB	Extremely long context (Llama 3.1)

Apple Silicon Unified Memory

Apple Silicon Macs share RAM between CPU and GPU, so the full system memory is available for model loading. The VRAM estimates still apply, but you can use total system RAM as your ceiling rather than a dedicated GPU VRAM figure. Performance may be slightly lower than dedicated GPU VRAM because the bandwidth is shared with the rest of the system. An M2 Ultra with 192 GB unified memory has roughly 800 GB/s of bandwidth, while a 128 GB M4 Max sits near 546 GB/s - both comfortably above what a single consumer NVIDIA card offers in absolute capacity, but below the RTX 5090's 1.79 TB/s GDDR7 bandwidth on a smaller pool.

Memory Bandwidth vs Capacity

Token generation speed on a loaded model is dominated by memory bandwidth, not compute. Each token generated requires streaming the entire weight tensor through the GPU's memory subsystem once, so tokens-per-second roughly equals memory bandwidth divided by model size in bytes. An 8B model at Q4_K_M (~4.6 GB) on an RTX 5090 (1.79 TB/s GDDR7 per NVIDIA's official specs) would ceiling at about 390 tokens/sec of pure memory throughput, though real-world inference loses some of that to kernel launch overhead, attention computation and sampling. On an RTX 3090 at roughly 936 GB/s the same model ceilings near 200 tokens/sec.

This is why Apple Silicon handles big models well despite slower compute: a Mac Studio M2 Ultra can load a Q4 70B model (~40 GB) where a 24 GB RTX 3090 or 4090 cannot, and still produce 10-15 tokens per second - slow but usable. For comparing the compute and token economics of API inference instead, the AI pricing calculator breaks down per-token costs across major providers.

How Much Quality Do You Lose at Q4?

Q4_K_M keeps roughly 95-97% of FP16 output quality for most conversational and coding tasks. Published llama.cpp perplexity benchmarks show Q4_K_M within about 3-4% of FP16 perplexity on Llama 2 13B, and comparative testing by Jarvis Labs in 2025 put Q4_K_M on HumanEval within 4% of baseline for code generation - matching AWQ and BitsandBytes despite its simpler scale-and-zero-point approach. For chat, summarisation and RAG workloads, the difference is usually imperceptible.

Quantization	Typical Perplexity Increase vs FP16	Practical Quality
Q8_0	under 0.5%	Indistinguishable from FP16
Q6_K	under 1%	Indistinguishable from FP16
Q5_K_M	1-2%	Very hard to notice in chat
Q4_K_M	3-4%	Strong sweet spot for most users
Q3_K_M	7-10%	Noticeable on complex reasoning
Q2_K	15-25%	Use only when Q3 won't fit

The same quantization hurts small models more than large ones. Q4 on a 70B model often still feels sharper than FP16 on a 7B model. If you have the VRAM for a bigger model at lower bit depth rather than a smaller model at higher bit depth, the bigger model usually wins on output quality.

CPU Offloading and Mixed Inference

When a model does not quite fit in VRAM, llama.cpp and similar runtimes can split the model between GPU and system RAM using the n_gpu_layers parameter. Every layer placed on the CPU slows generation roughly 10-20x because system DDR5 bandwidth (about 90-100 GB/s dual channel) is far below GPU bandwidth. A 70B Q4 model with 40 of its 80 layers offloaded to GPU typically runs at 2-4 tokens/sec instead of the 15-20 it would hit fully loaded. If the calculator shows a "Tight fit" badge, usually the right move is to drop one quantization level - going from Q5_K_M to Q4_K_M on a 13B model saves roughly 1.5 GB for minimal quality loss.

How Context Length Scales Memory

KV cache memory grows linearly with context length and with the number of KV attention heads. The formula the calculator uses is 2 (keys and values) x layers x KV heads x head dimension x 2 bytes (FP16) x context tokens. Two architectural choices dramatically affect this: Grouped Query Attention (GQA) reduces KV heads far below the number of attention heads (Llama 3.1 uses 8 KV heads for all sizes), and newer models with Multi-head Latent Attention cut KV cache by another 4-8x. On a Llama 3.1 70B model with 80 layers, 8 KV heads and 128 head dim, a 32K context adds about 20 GB of KV cache alone - roughly half the size of the quantized weights. For that reason, most practical local setups cap context at 8K-16K unless they have a specific long-document workload.

If you plan to run long-context summarisation or code analysis, budget KV cache separately before picking a model. The calculator's slider lets you see KV cache scale in real time, which is usually the most revealing part of the exercise for users new to local inference.

Common Planning Mistakes

The most frequent VRAM surprise is forgetting the KV cache scales linearly with context length. A 128K context on an 8B model can add 8-16 GB of KV cache, turning a comfortable fit into an out-of-memory error. Other common mistakes:

• Forgetting the runtime overhead. The 1.2x multiplier is conservative but real - CUDA context, PyTorch caches and activation tensors can easily add 15-25% over raw weight size.
• Treating system RAM as fungible on dedicated GPUs. On a Windows or Linux box with a discrete NVIDIA card, shared memory fallback is slow to the point of being unusable. Plan to fit everything in VRAM proper.
• Ignoring quantized KV cache. Flash Attention plus Q8 KV cache (available in recent llama.cpp builds) halves KV cache size for roughly zero quality loss - worth enabling on long-context use cases.
• Confusing active vs total parameters on MoE models. Llama 4 Scout is "17B active" per token but all 109B parameters must sit in memory. Mixture-of-experts models save compute, not memory.
• Chasing parameter count over quantization quality. A Q4 70B generally outperforms a Q8 13B for the same VRAM budget - size beats precision up to a point.

For counting input tokens before an inference run, the AI token counter estimates prompt length against common model tokenisers. To inspect a specific GGUF or Safetensors file's actual tensor layout and metadata rather than working from generic sizes, use the model file inspector. All calculations on this page run locally in your browser with no data sent anywhere.

Sources

• llama.cpp - Perplexity (Quality of Generation) Scores
• llama.cpp - Quantization README (GGUF types)
• NVIDIA - GeForce RTX 5090 Specifications
• Jarvis Labs - vLLM Quantization Benchmarks (2025)
• Hugging Face - Transformers Quantization Docs
• Apple - M2 Ultra Unified Memory Bandwidth