What is Quantization?
Quantization reduces the precision of a model's weights (e.g., from 16-bit to 4-bit). This slashes memory requirements by 75% while retaining 95% of the model's intelligence.
GGUF: The CPU King
Designed for Llama.cpp, GGUF is perfect for running models on CPUs and MacBooks. It supports mixed quantization and is highly portable.
AWQ and GPTQ: GPU Warriors
If you are deploying on Nvidia GPUs, AWQ (Activation-aware Weight Quantization) and GPTQ offer incredibly fast generation speeds and are supported natively by vLLM.
