A practical 12GB GPU guide for choosing model size, quantization, and context settings without crashes or misleading benchmark claims.
Short answer: a 12GB GPU is a strong local-AI tier. Start with an 8B model at Q5 or Q6 for speed and headroom, or a 12B–14B model at Q4 for better capability. Do not treat a model file’s size as its total VRAM requirement: the runtime, KV cache, vision encoder, and context length also consume memory.
Our recommendation for most people: use Ministral 3 8B Instruct when you value responsiveness and multimodal input, Qwen3 14B at Q4 when you prefer a larger text model, or Gemma 4 12B when your workflow needs Google’s current multimodal model family. Begin at an 8K context window and increase it only after measuring actual memory use.
What fits in 12GB of VRAM?
The useful answer is a range, not a single parameter count. Quantized weights are only one part of the allocation. On a 12GB card, leaving roughly 1.5–2.5GB free for the KV cache, compute buffers, and display workload is a sensible starting policy. That makes a model around 5–9.5GB on disk comfortable; files close to the full 12GB budget can fail as soon as context grows.
Model classPractical format12GB fitBest use 7B–9B denseQ5_K_M or Q6_KComfortableFast chat, extraction, coding help 12B–14B denseQ4_K_MGood with moderate contextReasoning, writing, broader knowledge 20B–24B denseQ3 or partial GPU offloadTightOnly when latency is secondary 30B+ denseCPU/GPU splitNot a clean 12GB experienceExperimental use
If you are deciding between nearby hardware tiers, compare our guides for 8GB VRAM, 16GB VRAM, and 24GB coding models.
Best choices by workload
Best balanced option: Ministral 3 8B Instruct
Mistral describes Ministral 3 8B as an edge-deployment model with vision and multilingual capabilities. Its official GGUF model card says the model can fit in 12GB of VRAM at FP8 and use less when further quantized. That makes a Q5 or Q6 build a conservative choice with room for context and runtime overhead.
Choose it for everyday assistants, document work, multilingual prompts, and image-aware tasks. The important caveat is licensing: always read the specific model card and license before commercial deployment.
Best larger text model: Qwen3 14B Q4
Qwen publishes official GGUF variants for Qwen3 14B. Ollama’s 14B package is about 9.3GB, which leaves a limited but workable margin on a 12GB card. Use an 8K context first, close other GPU-heavy applications, and reduce context if the runtime offloads layers to system RAM.
Qwen3 supports thinking and non-thinking behavior. For interactive work, disabling extended thinking can improve response time; enable it when the task benefits from additional reasoning and you accept slower generation.
Best current multimodal family: Gemma 4 12B
Google positions Gemma 4 12B for laptops, desktops, and small servers and documents text, image, audio, and video inputs. The original BF16 checkpoint is around 24GB, so it does not fit a 12GB GPU as published. A supported 4-bit or comparable quantized build is required. Multimodal inputs add memory overhead, so reserve more headroom than for text-only prompts.
Best low-friction choice: an 8B model at Q5
If reliability matters more than maximizing parameter count, an 8B Q5 model is usually the sweet spot. It leaves enough room to increase context, run a desktop interface, and avoid CPU offload. A smaller model that stays fully on the GPU often feels better than a larger model that crosses the VRAM boundary.
Recommended settings for RTX 3060 12GB and similar GPUs
Start with Q5_K_M for 8B or Q4_K_M for 12B–14B. These are practical quality/size starting points, not universal winners.
Set context to 8,192 tokens. Increase to 16K only after observing peak VRAM. Advertised maximum context is not a promise that the maximum is economical locally.
Offload all layers only if memory remains stable. If the runtime crashes, reduce context before sacrificing quantization quality.
Keep 10–15% headroom. A display connected to the same GPU and other accelerated apps consume part of the nominal 12GB.
Measure first-token latency and tokens per second on your prompts. Community benchmark tables rarely match your runtime, driver, context, and sampling settings.
Ollama quick start
ollama run qwen3:14bIf memory pressure is high, test the 8B variant:
ollama run qwen3:8bCheck the model’s published tag and size before downloading because default tags can change. For a graphical workflow, see our Ollama vs LM Studio comparison.
How we selected these models
This guide does not claim private benchmark results. Recommendations are based on official model documentation, published package sizes, architecture, supported inputs, license notes, and conservative memory budgeting. We prefer first-party model cards over anonymous quant descriptions. “Best” means the most sensible fit for a stated workload—not a universal quality ranking.
Last verified: August 13, 2026. Model tags, licenses, and runtimes change; verify the linked primary source before deployment.
Frequently asked questions
Can a 14B model run entirely on a 12GB GPU?
Usually yes at a suitable 4-bit quantization and moderate context, but not at FP16/BF16. Total memory depends on architecture and runtime overhead, so a 9GB model file should be tested rather than assumed safe.
Is Q8 better than Q4 on a 12GB card?
Q8 retains more weight precision but consumes much more memory. For a 14B model it generally exceeds a clean 12GB budget; Q4 is more practical. For an 8B model, Q6 or sometimes Q8 may fit, but context headroom shrinks.
Does an RTX 4070 12GB run LLMs better than an RTX 3060 12GB?
Both have the same capacity, so similar model sizes fit. The 4070 generally processes faster, while memory capacity—not compute speed—sets the hard ceiling for full GPU offload.