Compare

Quantization vs a smaller model

Shrink the numbers or shrink the model, then let your evals decide.

The short answer

Quantization keeps the same model and stores each weight in fewer bits: 8 bits is close to lossless, and 4 bits with GPTQ or AWQ keeps most of the quality on 7B and larger models. A smaller model has fewer weights, so it is cheaper and faster at every batch size, but it knows less and reasons less well. At the same memory a larger model at 4 bits often beats a smaller one at 16, but only your own evals, run at your real batch sizes, can settle it.

Quantization

Use it when

You want this model, in less memory.
Decode is slow at small batches, where each step is mostly reading the weights.
You want a change you can test in a day, with no training run.

What it costs: No training: a calibration set of a few hundred real inputs and a tool that writes the new checkpoint. A 70B model's weights drop from 140 GB in BF16 to about 35 GB in INT4.
Watch out: Perplexity can look fine while maths, code or your own task gets worse. And when the KV cache is most of what each step reads, smaller weights save less than you'd expect.

A smaller model

Use it when

The task is simple, such as tagging.
You need lower cost and delay on every request, as well as less memory.
You can fine-tune it on your task, or route only the easy requests to it.

What it costs: Fewer weights cut memory, time per step and price per token together. Serving costs about 2 operations per parameter per token, so a tenth of the size is about a tenth of the compute.
Watch out: It knows less and reasons less well, and the gap shows on hard requests. Grade it on your own traffic, hard cases included, before you switch.

At a glance

Question	Quantization	A smaller model
What changes	Bits per weight; the model stays the same	The number of weights
Memory for weights	Half at 8 bits, about a quarter at 4	Shrinks with the parameter count
Speed	Faster decode at small batches, less gain at large ones	Faster at every batch size
Quality risk	Small at 8 bits; at 4 bits, maths and code suffer first	Weaker on hard requests across the board
Work to try it	A calibration set and a tool, no training	Pick or tune a model, then grade it on your traffic
Best when	You need this model's quality and memory is the limit	The task is easy enough that a smaller model already passes

How to choose

Write down the limit you're hitting: memory, delay or cost per request.
Build an eval from your own traffic, hard cases included, and score the current model as the baseline.
Try 8-bit weights first, since they are close to lossless, then 4 bits with GPTQ or AWQ if you need more room.
Try a smaller model on the same eval, plain and fine-tuned on your task if you can.
Compare quality first, then speed and cost at the batch sizes and prompt lengths you really serve.

When you need both

They stack. Teams often send easy requests to a small model and hard ones to a large one, and serve both quantized. A small model can also draft tokens for the large one to check, which is speculative decoding. Profile before either: in one worked example, reordering a prompt so the server could reuse its cached prefix saved far more GPU time than quantizing did.

Learn it properly

Questions

Is a 4-bit large model better than a 16-bit small one?

Often, at the same memory: research across several model families found that 4-bit weights gave the best accuracy for a fixed memory budget. It isn't guaranteed for your task, so compare both on your own evals, and watch maths and code, which tend to lose the most.

How much quality does quantization lose?

At 8 bits with a scale per row, very little. At 4 bits, plain rounding can hurt, but GPTQ and AWQ close most of the gap to 16-bit on 7B and larger models. Check your own task, since a gap of 1% in perplexity can hide a bigger loss on maths or code.

Does quantization make a model faster?

At small batches, yes: decode is limited by reading the weights, so fewer bytes per weight means faster steps. At large batches, or when the KV cache is most of what each step reads, the gain shrinks, and 4-bit weights that must be unpacked can even run slower than 8-bit ones.

How much memory does a 70B model need?

For the weights alone, about 140 GB in BF16, 70 GB in 8 bits and 35 GB in 4 bits. The KV cache and runtime buffers come on top, so a 35 GB model is a tight fit on a 40 GB GPU.