How to Quantize Models: Q4 vs Q8 Explained Simply

If you've spent any time downloading local AI models, you've run into names like Q4_K_M, Q5_K_S, or Q8_0, and it looks like a wall of gibberish. It's not. Once you understand the one core idea behind it, the naming actually makes sense, and picking the right one for your hardware becomes an easy decision instead of a guessing game.

The Simple Idea Behind Quantization

Every number inside an AI model — every single one of its billions of parameters — has to be stored in some numeric format. By default, models are trained and stored using 16-bit floating-point numbers (FP16), which gives each parameter a lot of precision, like storing a measurement as 3.14159265.

Quantization is the process of storing those same numbers with less precision — rounding 3.14159265 down to something like 3.14, or even just 3. You lose a small amount of exactness, but the number takes up dramatically less space. Do that across billions of parameters, and a model that needed 14GB to store now needs 4-5GB — with only a small, often barely noticeable, dip in quality.

That's the whole concept. Everything else — Q4, Q8, K_M, IQ4_XS — is just naming for different ways of doing that rounding.

Reading the Alphabet Soup

Once you know what to look for, quantization names follow a predictable pattern: Q, followed by a number, followed sometimes by letters. Breaking down what each part means:

• The number (4, 5, 6, 8) → roughly how many bits are used to store each parameter. Lower number = smaller file, more rounding, more potential quality loss

• _K → indicates a "K-quant," a smarter method that groups weights into blocks and allocates more precision to the weights that matter most (typically attention layers), rather than treating every weight equally

• S / M / L → small, medium, or large variants of a K-quant, trading a bit more size for a bit more quality. M is almost always worth the small size increase over _S

• 0 / 1 → older, simpler "legacy" formats that quantize every weight uniformly, without the smarter block-based allocation K-quants use

• IQ (like IQ4_XS) → "importance quantization," an even more advanced method using lookup tables and calibration data to preserve quality at very low bit counts, at the cost of slightly slower decoding on some hardware

In plain terms: Q4_K_M means "roughly 4-bit precision, using the smarter K-quant method, medium-sized variant." That's the single most recommended format in the entire local AI community, and now you know exactly why its name looks the way it does.

Q4 vs Q8: The Actual Difference

Here's where the practical decision lives. Using a 7B model as a concrete example:

• FP16 (full precision) → around 14GB → the original, uncompressed size

• Q8_0 → around 7.7GB, roughly half the size of FP16 → quality loss is under 0.5%, essentially indistinguishable from full precision in real use

• Q5_K_M → a middle ground → noticeably smaller than Q8_0, with a small, occasionally noticeable quality dip on complex reasoning or coding tasks

• Q4_K_M → around 4.4-4.5GB, about 75% smaller than FP16 → quality loss in the range of 1-3% on standard benchmarks, which in practice reads as "essentially the same" for most everyday chat, writing, and coding use

That gap between Q4_K_M and Q8_0 is the one that actually matters for most people: roughly double the file size, for around a 3-4% quality difference. That's exactly why Q4_K_M has become the community default — it's the point where you stop paying much in file size for the last bit of quality.

Where Quality Loss Actually Shows Up

Quantization doesn't degrade every kind of task equally. Research on this has become fairly consistent:

• Commonsense reasoning and everyday conversation are highly resilient to quantization — you're unlikely to notice a difference even at Q4

• Logical and structured reasoning holds up well too, generally staying solid down to around Q4

• Arithmetic and precise multi-step math are the most sensitive — quality noticeably drops once you go below 4-bit quantization, since exact calculation depends on precision that gets rounded away more aggressively at lower bit counts

• Below Q3, degradation becomes broadly noticeable across most task types, not just math

A useful rule that follows from this: a bigger model quantized more aggressively often beats a smaller model kept at higher precision. A 70B model at Q4_K_M frequently outperforms a 13B model at full FP16, despite using similar memory — model size matters more than quantization level, so when you have a choice, lean toward the larger model at a lower quant rather than a smaller model at a higher one.

Which One Should You Actually Pick?

Text-style decision guide:

• Tight on VRAM (8GB GPU, or CPU-only) → Q4_K_M is your default. Best balance of size and quality for constrained hardware

• Have some room to spare (16GB+ VRAM) → Q5_K_M or Q6_K, if you want a noticeable quality step up, particularly for coding or complex reasoning

• Memory isn't a constraint (24GB+ VRAM, or you just want maximum fidelity) → Q8_0, which is close enough to lossless that there's little reason to go higher unless you're doing rigorous evaluation work

• Absolute last resort on very tight hardware → Q3_K_S or Q2_K, but test your specific use case carefully, since this is where quality loss becomes easy to notice, especially on anything math-heavy

• Squeezing extra quality out of a small file size → look for IQ4_XS or similar "I-quant" builds calibrated with an importance matrix, if one is available for your model

How to Actually Quantize a Model Yourself

If you're starting from a Hugging Face model rather than downloading an already-quantized GGUF, here's the real process using llama.cpp:

Step 1 — Convert the original model to GGUF format at full precision:

python convert_hf_to_gguf.py ./my-model --outfile model-f16.gguf --outtype f16

Step 2 — Quantize it to your target level:

./llama-quantize model-f16.gguf model-q4km.gguf Q4_K_M

Step 3 (optional, recommended for better quality at low bit counts) — Generate an importance matrix from calibration data, then quantize using it:

./llama-imatrix -m model-f16.gguf -f calibration-data.txt --chunk 512 -o imatrix.dat
./llama-quantize --imatrix imatrix.dat model-f16.gguf model-q4km-imatrix.gguf Q4_K_M

A few rules worth following closely:

• Always quantize starting from the F16 or F32 source weights, never from an already-quantized file. Quantizing a quantized model (Q8 to Q4, for example) compounds the rounding errors from both steps and produces noticeably worse results than quantizing directly from full precision

• You cannot convert a quantized model back up to a higher precision — once it's Q4_K_M, that's final; if you want Q5_K_M later, you need to re-quantize from the original F16 file

• If you'd rather not quantize anything yourself, reputable community uploaders on Hugging Face — bartowski, mradermacher, and Unsloth among them — publish pre-quantized, often imatrix-calibrated GGUF files for most popular models, which is the easier path for most people

If You're Using Ollama Instead

You don't need to touch llama.cpp directly if you're using Ollama — it handles quantization selection for you:

• Running ollama pull llama3.1:8b downloads the Q4_K_M variant by default

• To get a specific quantization level instead, append it to the tag: ollama pull llama3.1:8b-instruct-q5_K_M

• Available quantization tags for any given model are listed on that model's page at ollama.com/library

The Bottom Line

Quantization is just controlled rounding — trading a small amount of numeric precision for a large reduction in file size, and in most cases the tradeoff is barely noticeable in everyday use. Q4_K_M is the community default for a good reason: it delivers roughly 75% of the size savings of aggressive quantization while keeping quality within a few percent of full precision. Q8_0 is there for when memory truly isn't a constraint and you want as close to lossless as quantization gets. Start with Q4_K_M, move up to Q5_K_M or Q8_0 if you have the VRAM to spare and want the extra polish, and reserve anything below Q4 for situations where you have no other choice — and even then, test it against your actual use case rather than trusting the number alone.