Large Model Quantization

As large language models grew from tens of billions to thousands of billions of parameters, serving them became increasi

As large language models grew from tens of billions to thousands of billions of parameters, serving them became increasingly difficult. The main bottlenecks were memory usage and inference cost. Quantization is one of the most widely used techniques for compressing such models. It reduces the numerical precision used for weights and sometimes activations, lowering both storage requirements and compute overhead.

The core idea is to use fewer bits to represent floating point values. A common linear quantization scheme can be written as:

  • q = round(x / s) + z
  • x ≈ s × (q − z)

Here, s is the scale and z is the zero point. Calibration data is often used to estimate these values, and their quality has a direct impact on quantized performance.

For a 7B parameter model, approximate memory usage can look like this:

  • FP16: 2 bytes per parameter, about 14 GB.
  • INT8: 1 byte per parameter, about 7 GB.
  • INT4: 0.5 bytes per parameter, about 3.5 GB.

Moving from FP16 to INT4 can reduce memory to roughly one quarter of the original size. That makes a 7B model more feasible on consumer-grade hardware.

Two main technical routes: PTQ and QAT

There are two broad families of quantization methods.

  • PTQ stands for post-training quantization. After training is complete, weights, and sometimes activations, are quantized. Calibration data helps guide the process. No retraining is required, so the cost is relatively low. It is a common choice in production.
  • QAT stands for quantization-aware training. Fake quantization operators are inserted during training or fine-tuning. The model learns within a low-bit constraint. Accuracy may improve, but the full training pipeline is much more expensive.

Because full training of large models is costly, PTQ and PTQ combined with parameter-efficient fine-tuning methods, such as QLoRA, have become popular.

Key design dimensions

Several design choices affect quantized model quality.

  1. Grain size: per-tensor, per-channel, or per-group. Per-tensor uses one value for a whole tensor, while per-channel or per-group methods use more structured values. Finer granularity can improve precision but adds complexity.
  2. Symmetric and asymmetric quantization: symmetric quantization fixes the zero point at zero, which is simple. Asymmetric quantization allows a nonzero zero point and may better match data distributions.
  3. Weights and activations: W4A16 keeps weights low-bit and activations higher precision. It is simple and compatible. W8A8 or W4A4 quantize both, which can exploit integer arithmetic better, but outlier handling becomes important.
  4. Data types: integer formats such as INT8 are common, while NF4 was designed for weights with near-normal distributions.

Methods at a glance

  • LLM.int8(): keeps a small number of outlier features in FP16 and uses INT8 for the rest. It was an early approach that achieved 8-bit quantization with little obvious accuracy loss.
  • GPTQ: minimizes quantization error layer by layer using second-order information. It is a classic W4A16 method.
  • AWQ: examines activation distribution and protects important weight channels. It is considered robust.
  • SmoothQuant: smooths activation outliers by moving them into weights. It supports W8A8 deployment.
  • GGUF / llama.cpp: offers a quantization ecosystem for CPU and edge devices, with options ranging from 2-bit to 8-bit.

These methods differ in implementation details and trade-offs. Their algorithmic nuances and accuracy comparisons are often covered in separate discussions.

Practical example: loading a 4-bit model

A practical starting point is Hugging Face transformers with bitsandbytes. The following snippet shows how to load a 7