Why compare quantization formats
Once you decide to run a quantized model locally, you face a practical choice: which format to download. The main options — GPTQ, AWQ, GGUF, and bitsandbytes — all shrink a model to roughly 4 bits, but they differ in which hardware they target, how fast they run, and how much accuracy they keep. Picking the right one is mostly about matching the format to your hardware.
GPTQ
GPTQ is a post-training method that quantizes the model one layer at a time, using a calibration dataset and second-order error information to choose weights that minimise the quality loss. It targets NVIDIA GPUs and is widely supported by GPU inference libraries.
- Strengths — strong 4-bit accuracy, fast GPU inference, broad tooling.
- Watch for — GPU-focused; less suited to pure CPU use.
AWQ
AWQ (Activation-aware Weight Quantization) is also a 4-bit GPU method, but its trick is to protect the weights that matter most for the activations rather than treating all weights equally. This often gives slightly better accuracy than GPTQ at the same bit-width, with very fast inference.
- Strengths — excellent accuracy-per-bit, fast, good for serving.
- Watch for — like GPTQ, it is built for GPUs.
GGUF
GGUF is the format used by llama.cpp and the ecosystem around it — Ollama, LM Studio, and similar tools. It is built for CPU and mixed CPU/GPU inference and offers many quantization levels (for example Q4_K_M, Q5_K_M, Q8_0), letting you trade size against quality finely.
- Strengths — runs almost anywhere, great on Apple Silicon and laptops, flexible bit-widths.
- Watch for — raw GPU throughput can trail GPTQ/AWQ on high-end NVIDIA cards.
bitsandbytes
bitsandbytes provides on-the-fly 8-bit and 4-bit (NF4) quantization directly inside Hugging Face Transformers. You load a standard model and quantize it as it loads — no separate converted file required.
- Strengths — simplest to use, integrated with the Transformers workflow, handy for experiments and fine-tuning (QLoRA).
- Watch for — typically a little slower and slightly less accurate than a purpose-built GPTQ or AWQ build.
Picking the right one
A simple decision guide:
- NVIDIA GPU, best speed and quality → GPTQ or AWQ (AWQ if available).
- CPU, Apple Silicon, or mixed hardware → GGUF via llama.cpp / Ollama.
- Easiest path inside Python / Transformers, or QLoRA fine-tuning → bitsandbytes.
At a given bit-width the accuracy differences are usually small, so let your hardware drive the choice. For the underlying concepts of precision and bit- widths, see What Is Model Quantization?.