A Complete Guide to LLM Quantization

From floating point fundamentals to 1-bit models — everything you need to understand how large language models are compressed for the real world.

1. What is Quantization?

Neural network weights are just numbers. By default they are stored at high precision — 32-bit floats — which consume enormous amounts of memory. A 7-billion parameter model at FP32 occupies roughly 28 GB of RAM. Running it requires a server-grade GPU.

Quantization is the process of reducing the numerical precision of those weights to make the model smaller and faster — with minimal loss in quality.

FP32  →  7B model = 28 GB
FP16  →  7B model = 14 GB   (2× smaller)
INT8  →  7B model =  7 GB   (4× smaller)
INT4  →  7B model = 3.5 GB  (8× smaller)

The core trade-off: lower precision = smaller model = slightly degraded output quality.

Why it matters

What quantization does NOT do

Quantization does not change the model's architecture, parameter count, or behaviour. A quantized model still does exactly the same job — just more efficiently. It does not make the model learn new things (that is fine-tuning), and it does not give it access to new information (that is RAG).

2. The Number Formats

Float formats (FP32, FP16, BF16) all use the IEEE 754 standard — a binary version of scientific notation. Every single float number is physically stored as exactly three consecutive bit fields packed together in memory.

The three fields

┌─────────┬──────────────────────┬──────────────────────────────────────────────┐
│  Sign   │      Exponent        │               Mantissa                       │
│  1 bit  │  8 bits  (FP32)      │  23 bits  (FP32)                             │
└─────────┴──────────────────────┴──────────────────────────────────────────────┘
 bit 31        bits 30–23                      bits 22–0

Field 1 — Sign (S), always 1 bit Answers one question: positive or negative?

• S = 0 → positive
• S = 1 → negative
• Contribution to the final value: (−1)^S — gives either +1 or −1

Field 2 — Exponent (E), 8 bits in FP32 Controls the magnitude — how large or small the number is. Works like the power-of-10 in scientific notation, but in binary (power-of-2). A bias is subtracted from the stored bits so both tiny (negative true exponents) and huge (positive true exponents) numbers can be represented.

• Stored as an unsigned integer (0–255 for FP32)
• True exponent = stored E − bias
• FP32 bias = 127, so stored 10000000 (=128) means true exponent = 128 − 127 = 1
• Contribution: 2^(E − bias)

Field 3 — Mantissa (M), 23 bits in FP32 Controls the precision — the fine-grained detail of the number. Represents the fractional digits after an implicit "1." prefix that is never actually stored (called the hidden bit or implicit leading 1 — it is always assumed to be there).

• More mantissa bits = more decimal digits of accuracy
• FP32 has 23 bits → 2²³ = 8,388,608 possible steps between any two powers of 2
• FP16 has 10 bits → only 1,024 steps (much coarser)
• BF16 has 7 bits → only 128 steps (coarsest of the three)
• Contribution: (1 + M / 2^mantissa_bits)

The complete decoding formula

Multiply all three contributions together:

value  =  (−1)^S  ×  2^(E − bias)  ×  (1 + M / 2^mantissa_bits)
           ↑               ↑                      ↑
        sign part     magnitude part          precision part
        (+1 or −1)   (powers of 2)       (fine steps within each power)

The formula is identical for FP32, FP16, and BF16 — only the field sizes and bias differ:

Notice: FP16 and BF16 are both 16 bits but make opposite trade-offs. FP16 keeps more mantissa bits (better precision, narrower range). BF16 keeps the full 8-bit exponent from FP32 (same enormous range, coarser precision).

FP32 — Full Precision

The baseline. Every weight stored as a 32-bit IEEE 754 float. Used during initial model training where accuracy matters most.

Encoding and decoding w = 2.347 in FP32:

Step 1 — sign:
  2.347 is positive  →  S = 0

Step 2 — exponent:
  2.347 = 1.1735 × 2¹   →   true exponent = 1
  stored E = 1 + 127 = 128 = 10000000 in binary

Step 3 — mantissa:
  fractional part after "1." = 0.1735
  M = round(0.1735 × 8,388,608) = 1,455,440
  binary: 00101100001010001111011

Decode check:
  value = (−1)⁰ × 2^(128−127) × (1 + 1,455,440 / 8,388,608)
        = 1 × 2 × 1.17350
        = 2.347000   ← zero error

FP16 — Half Precision

16 bits — exactly half the memory of FP32. The exponent shrinks from 8 bits to 5 bits, which dramatically reduces the representable range (max ~65,504 instead of ~3.4×10³⁸).

Formula:

value = (−1)^S × 2^(E − 15) × (1 + M / 1024)

Encoding and decoding w = 2.347 in FP16:

Step 1 — sign:
  2.347 is positive  →  S = 0

Step 2 — exponent:
  2.347 = 1.1735 × 2¹   →   true exponent = 1
  stored E = 1 + 15 = 16 = 10000 in binary

Step 3 — mantissa:
  fractional part after "1." = 0.1735
  M = round(0.1735 × 1024) = 178
  binary: 0010110010

Decode check:
  value = (−1)⁰ × 2^(16−15) × (1 + 178/1024)
        = 1 × 2 × 1.17383
        = 2.34766   →   error = 0.00066  (~0.028%)

FP16 overflow — the hard limit

FP16's 5-bit exponent field can only store values 0–31. However, IEEE 754 reserves the all-ones pattern 11111 (= 31) as a special code, leaving only 11110 (= 30) as the maximum usable exponent:

Maximum normal FP16:
  S=0,  E = 11110 (=30),  M = 1111111111 (all ones = 1023)
  value = 1 × 2^(30−15) × (1 + 1023/1024)
        = 1 × 2¹⁵ × 1.9990234375
        = 32768 × 1.9990  =  65,504

What does the reserved 11111 pattern mean?

What happens when a value exceeds 65,504?

To store 66,504 you would need true exponent = 16 (since 66,504 ≈ 1.something × 2¹⁶). That requires stored E = 16 + 15 = 31 = 11111 — which is the reserved Infinity code. So 66,504 cannot be stored as a normal number:

66,504 in binary  =  10000001111001000  (17 bits — too wide for FP16)

Attempted stored E = 16 + 15 = 31 = 11111  →  reserved for Infinity!

Result:  66,504  →  stored as  +Infinity  (overflow)

Once Infinity appears in a weight or activation, every subsequent multiply and add propagates it — the entire layer becomes Infinity or NaN and training crashes. This is the training instability that makes FP16 dangerous for large model training.

BF16 — Brain Float 16

Google's direct solution to FP16's overflow problem. BF16 keeps the same 8-bit exponent as FP32 (same enormous range ±3.4×10³⁸), but sacrifices mantissa bits for it — cutting from 23 bits down to 7. BF16 is literally FP32 with the last 16 mantissa bits chopped off.

The overflow example above is exactly why BF16 was invented. A value like 66,504 stores without any problem:

66,504 in BF16:
  true exponent = 16  →  stored E = 16 + 127 = 143 = 10001111
  max storable exponent = 11111110 (=254)  →  true exp = 254−127 = 127
  66,504 fits comfortably — no overflow

Formula:

value = (−1)^S × 2^(E − 127) × (1 + M / 128)

Encoding and decoding w = 2.347 in BF16:

Step 1 — sign:
  2.347 is positive  →  S = 0

Step 2 — exponent:
  2.347 = 1.1735 × 2¹   →   true exponent = 1
  stored E = 1 + 127 = 128 = 10000000 in binary  (same as FP32)

Step 3 — mantissa:
  fractional part after "1." = 0.1735
  M = round(0.1735 × 128) = 22
  binary: 0010110

Decode check:
  value = (−1)⁰ × 2^(128−127) × (1 + 22/128)
        = 1 × 2 × 1.17188
        = 2.34375   →   error = 0.00325  (~0.14%)

The error is slightly larger than FP16 (0.003 vs 0.001) because BF16 has fewer mantissa steps (128 vs 1,024). For training, this trade-off is completely acceptable — stability matters far more than that extra precision.

FP16 vs BF16 — the key trade-off

The industry standard today: train in BF16, deploy in INT4 or INT8.

Important: floats do NOT use symmetric/asymmetric quantization

Float formats have their own built-in scaling mechanism — the exponent field. Every single number carries its own scale independently. There is no shared scale factor S, no zero point Z. The concepts of symmetric and asymmetric quantization simply do not apply to FP32, FP16, or BF16.

3. Integer Quantization

Integer formats (INT8, INT4, INT2) contain no exponent, no mantissa, no decimal point. They are plain whole numbers:

• INT8: 8 bits → values from −128 to +127 (256 possible values)
• INT4: 4 bits → values from −8 to +7 (16 possible values)
• INT2: 2 bits → values from −2 to +1 (4 possible values)

How integers encode decimal weights

A model weight like 2.347 is clearly not an integer. The trick is linear scaling — a shared scale factor S maps the continuous float range onto the discrete integer grid.

Quantize:    q  = clamp( round( w / S ),  q_min,  q_max )
Dequantize:  w̃  = q × S

The scale factor S

S measures how wide the weight range is — it is the size of one integer step in float space. Smaller S means finer precision; larger S means coarser.

S = max(|w|) / q_max

For INT8:  S = max absolute weight / 127
For INT4:  S = max absolute weight / 7

Absolute value notation — |w|

The |w| notation means absolute value — strip the sign, keep the magnitude. So |−2.634| = 2.634 and |+2.634| = 2.634. max(|w|) finds the weight that sits furthest from zero in either direction.

Clamp

clamp(value, min, max) is a safety guard that prevents computed integers from escaping the allowed range:

if value < min  →  return min
if value > max  →  return max
otherwise       →  return value unchanged

Without clamp, an edge-case weight could produce an integer like 169 which does not fit in an INT8 byte. Clamp guarantees the output always fits.

4. Symmetric vs Asymmetric Quantization

This is a choice you make per layer when quantizing to an integer format. It determines whether the integer grid is centred at zero or shifted to match where the data actually lives.

Symmetric Quantization

The integer range is split evenly around zero. Zero in float space maps exactly to zero in integer space.

Scale:     S = max(|w|) / q_max
Quantize:  q = clamp( round(w / S),  −q_max,  +q_max )
Dequant:   w̃ = q × S
Zero point: Z = 0  (always)

Example with weights [2.347, −1.821, 0.493, −0.156, 1.205, −2.634] → INT8:

S = 2.634 / 127 = 0.02074

2.347  →  round(2.347/0.02074)  = round(113.15) = 113
−1.821 →  round(−1.821/0.02074) = round(−87.8)  = −88
0.493  →  round(0.493/0.02074)  = round(23.8)   = 24
−0.156 →  round(−0.156/0.02074) = round(−7.5)   = −8
1.205  →  round(1.205/0.02074)  = round(58.1)   = 58
−2.634 →  round(−2.634/0.02074) = round(−127.0) = −127

Stored: [113, −88, 24, −8, 58, −127]  +  scale S=0.02074

Best for: weights in Linear and Conv layers — they are initialised near zero and stay roughly balanced around it during training.

Asymmetric Quantization

The integer grid is shifted so its full range covers exactly where the data lives — even if that range is not centred at zero.

Scale:      S = (max − min) / (q_max − q_min)
Zero point: Z = round( q_min − min / S )
Quantize:   q = clamp( round(w / S) + Z,  q_min,  q_max )
Dequant:    w̃ = S × (q − Z)

Example with skewed ReLU activations [0.1, 0.8, 1.5, 2.9, 4.1, 5.2] → INT8:

S = (5.2 − 0.0) / (127 − (−128)) = 5.2 / 255 = 0.02039
Z = round(−128 − 0.0/0.02039) = −128

Weight 2.9:  q = round(2.9/0.02039) + (−128) = 142 − 128 = 14
Dequant:     w̃ = 0.02039 × (14 − (−128)) = 0.02039 × 142 = 2.895

Best for: activations after ReLU (all ≥ 0), sigmoid (range 0–1), or any layer where data is skewed away from zero.

Zero Point

The zero point Z is the integer value that represents real-valued 0.0 in asymmetric quantization. In symmetric quantization, Z is always 0. In asymmetric, it shifts to match where real zero falls within the integer grid.

Symmetric:  Z = 0  →  INT 0  maps to  real 0.0
Asymmetric: Z = −128  →  INT −128 maps to real 0.0
            Z = +10   →  INT +10  maps to real 0.0

The decode formula w̃ = S × (q − Z) uses Z to correctly undo this shift.

When to use which

The practical rule: weights → symmetric. Activations → check the activation function.

Using symmetric on ReLU activations wastes half the INT range on negative values that never appear. Asymmetric fixes this and effectively doubles precision at no memory cost.

5. 1-bit and 1.58-bit Quantization

1-bit

The most extreme quantization possible. Each weight is stored as a single bit — only two possible values: 0 or 1. A 7B model at 1-bit occupies roughly 0.875 GB. In practice, quality loss is high unless the model was trained specifically for this constraint.

1.58-bit — Ternary (BitNet)

Microsoft Research's 2024 breakthrough. Each weight is one of three values: −1, 0, or +1. Three values require log₂(3) ≈ 1.58 bits of information per weight — hence the name.

Threshold:  α = mean( |weights| )
Quantize:   q = +1  if w > +α
            q = −1  if w < −α
            q =  0  otherwise
Dequantize: w̃ = q × α

Example with [2.347, −1.821, 0.493, −0.156, 1.205, −2.634]:

α = (2.347 + 1.821 + 0.493 + 0.156 + 1.205 + 2.634) / 6 = 1.443

2.347  > +1.443  →  +1
−1.821 < −1.443  →  −1
0.493  between   →   0
−0.156 between   →   0
1.205  between   →   0
−2.634 < −1.443  →  −1

Stored: [+1, −1, 0, 0, 0, −1]

Why 1.58-bit is so efficient

With ternary weights, matrix multiplication collapses into three branches:

• Weight is +1 → add the input (no multiply)
• Weight is 0 → skip entirely (free — ~50–60% of operations)
• Weight is −1 → subtract the input (no multiply)

No floating point multiplications at all — just additions and subtractions. This runs extremely fast on plain CPUs.

Native training vs post-quantization

The key insight: post-quantizing an FP32 model to 1.58-bit causes catastrophic quality loss because the model was trained expecting full float precision. Native training (training from scratch with ternary weights from day one) allows the model to develop internal representations that work within the constraint — producing surprisingly competitive quality.

Native 1.58-bit training is the most extreme form of Quantization-Aware Training (QAT) — the constraint is never relaxed at any point during training.

Real models using 1.58-bit

As of 2025, these are research/experimental models. The ecosystem is still maturing but the technology is real and downloadable today.

6. Mathematical Examples

All examples use the same six weights for direct comparison:

Original FP32: [2.347, −1.821, 0.493, −0.156, 1.205, −2.634]
max absolute value = 2.634

FP32 (baseline)

Formula:  value = (−1)^s × 2^(E−127) × (1 + M/2²³)
Result:   [2.347000, −1.821000, 0.493000, −0.156000, 1.205000, −2.634000]
Error:    0  (exact)
Storage:  24 bytes (6 × 4 bytes)

FP16

Formula:  value = (−1)^s × 2^(E−15) × (1 + M/1024)

2.347:  s=0, E=16, M=178  →  2×(1+178/1024) = 2.34766  error=0.00066
−1.821: s=1, E=15, M=849  →  −1×(1+849/1024) = −1.82910  error=0.00810
Storage: 12 bytes (6 × 2 bytes)

BF16

Formula:  value = (−1)^s × 2^(E−127) × (1 + M/128)

2.347:  s=0, E=128, M=22  →  2×(1+22/128) = 2.34375  error=0.00325
Storage: 12 bytes (6 × 2 bytes)

INT8 (symmetric)

S = 2.634 / 127 = 0.02074

Quantize:   q = clamp(round(w/S), −128, 127)
Dequantize: w̃ = q × S

2.347  → 113  → 113×0.02074 = 2.3436   error=0.0034
−1.821 → −88  → −88×0.02074 = −1.8251  error=0.0041
0.493  →  24  →  24×0.02074 = 0.4978   error=0.0048
−0.156 →  −8  →  −8×0.02074 = −0.1659  error=0.0099
1.205  →  58  →  58×0.02074 = 1.2029   error=0.0021
−2.634 → −127 → −127×0.02074 = −2.6340 error=0.0000

Storage: 10 bytes (6×1 byte INT8 + 4 bytes for scale S)

INT4 (symmetric)

S = 2.634 / 7 = 0.3763

2.347  →  6  →  6×0.3763 = 2.2578   error=0.0892
−1.821 → −5  → −5×0.3763 = −1.8815  error=0.0605
0.493  →  1  →  1×0.3763 = 0.3763   error=0.1167
−0.156 →  0  →  0×0.3763 = 0.0000   error=0.1560  ← small weight lost!
1.205  →  3  →  3×0.3763 = 1.1289   error=0.0761
−2.634 → −7  → −7×0.3763 = −2.6341  error=0.0001

Storage: 7 bytes (3 bytes packed INT4 + 4 bytes for scale)

1.58-bit (ternary)

α = 1.443

2.347  > +1.443  →  +1  →  +1×1.443 = 1.443  error=0.904
−1.821 < −1.443  →  −1  →  −1×1.443 = −1.443  error=0.378
0.493  between   →   0  →  0         error=0.493
−0.156 between   →   0  →  0         error=0.156
1.205  between   →   0  →  0         error=1.205  ← lost entirely
−2.634 < −1.443  →  −1  →  −1×1.443 = −1.443  error=1.191

Storage: ~1.2 bytes (6 ternary values + scale α)

Error comparison summary

Per-weight error and model-level quality are very different things. With billions of weights, small errors average out. Native-trained models compensate globally even when individual weights are aggressively approximated.

7. Quantization Methods

PTQ — Post-Training Quantization

Apply quantization after training is complete. No additional training required. Fast and simple, but quality degrades because the model was never aware of its coming compression.

Train fully (FP32)  →  Calibrate  →  Quantize  →  Deploy INT8

Best for: quick deployment, limited compute budget, INT8 (quality loss is manageable). Worst for: aggressive quantization like INT4 or lower without smart calibration.

QAT — Quantization-Aware Training

Insert fake quantization nodes into the model during training. The model experiences quantization noise on every batch and learns to be robust against it.

Start FP32  →  Train WITH fake quantization  →  S and Z learned by backprop  →  INT8 model

During the forward pass, fake quantization simulates INT8 rounding:

w_fake = S × round( clamp(w/S, −128, 127) )

During the backward pass, a straight-through estimator (STE) handles the non-differentiable round() function by passing gradients through unchanged:

∂L/∂w  =  ∂L/∂w_fake × 1

S itself is also a trainable parameter updated by backprop — the model finds its own optimal scale per layer.

QAT vs PTQ: QAT produces significantly better quality at the same bit-width. The cost is that it requires re-training the model (or at least fine-tuning it), which takes time and compute.

Native Training (Extreme QAT)

The quantization constraint is baked in from the very first gradient step. Weights are always ternary (−1, 0, +1) throughout the entire training run — there is no FP32 phase to compress from. This is what BitNet b1.58 uses.

8. Calibration

Calibration is not quantization — it is the measurement step that happens before PTQ to find the best possible scale factor S and zero point Z for each layer.

Why it is needed for PTQ

A PTQ model was trained in FP32 with no knowledge of quantization. It has no idea what scale each layer needs. Calibration fills this gap by running a small representative dataset (~100–1000 samples) through the model in FP32 and observing the actual range of activations flowing through each layer.

Why it is NOT needed for QAT

In QAT, S and Z are trainable parameters. Backpropagation updates them alongside the weights on every batch. By the end of training, every layer already knows its optimal quantization range — there is nothing left for calibration to measure.

Three calibration strategies

MinMax:

S = (observed_max − observed_min) / 255

Simple but sensitive to outliers. One extreme value forces a wide range and wastes most INT buckets on empty space.

Percentile:

S = (p99 − p1) / 255

Clips the top and bottom 1% of observed values. Far more robust to outliers. Most production pipelines use 99th percentile rather than true max.

Entropy (KL divergence): Finds the range that minimises information loss between the original FP32 distribution and the quantized INT8 distribution. Most accurate — this is what TensorRT uses by default.

The cost of bad calibration

Without calibration (or with a poorly chosen range), you are guessing S. A range that is too wide wastes INT buckets on empty space — effectively reducing precision. A range that is too narrow clips values — introducing large errors for weights near the boundary.

Good calibration is what separates a quantized model that performs well from one that degrades badly.

9. Decoding at Inference — Do You Need To?

You never manually decode anything. The inference engine (llama.cpp, PyTorch, TensorRT, etc.) handles it completely automatically.

However, three different strategies exist under the hood:

Approach 1 — Dequantize then compute

Load INT8 weight → multiply by S → get float approximation → do float matrix multiplication.

q=113, S=0.02074  →  w̃ = 113 × 0.02074 = 2.3436  →  multiply with input in FP16

The weight lives as INT8 only in memory. The moment it is loaded into a GPU register for computation, it becomes float. This is the approach used by llama.cpp's GGUF format.

Approach 2 — Pure integer arithmetic

Never decode at all. Multiply q_w × q_x entirely in integers (INT8 × INT8 → INT32 to avoid overflow), accumulate the whole dot product as integers, then apply scale correction once at the very end.

q_w=113, q_x=85  →  113 × 85 = 9605 (INT32)
rescale:  9605 × Sw × Sx = 9605 × 0.02074 × 0.01500 = 2.986

Faster because integer multiply units on modern GPUs are cheaper and more numerous than float units. Used by TensorRT and production quantized BERT deployments.

Approach 3 — 1.58-bit (no multiply at all)

if weight == +1  →  result += input   (add)
if weight ==  0  →  skip              (free)
if weight == −1  →  result -= input   (subtract)

No floating point multiplications anywhere. Runs efficiently on plain CPUs.

The mental model

Think of it like a compressed image file. A JPEG is stored compressed on disk, but when you display it the viewer decompresses it automatically. Quantized weights are identical — stored small, decompressed automatically at the exact moment needed, then discarded. The storage savings are real and permanent; the decode cost is tiny and ephemeral.

10. Quantization vs Fine-Tuning vs RAG

These three techniques are frequently confused. They solve completely different problems.

Quantization asks: "how efficiently can I store and run this model?"

Fine-tuning asks: "how does the model behave?" — it changes what the model knows or how it responds, by updating weight values through training. The model stays the same size.

RAG (Retrieval-Augmented Generation) asks: "what does the model know?" — it injects relevant documents into the prompt at query time. No training, no compression.

QLoRA — combining quantization and fine-tuning

The most famous combination, introduced in 2023:

1. Quantize the base model to 4-bit — fits in limited GPU memory
2. Freeze those quantized weights
3. Add small LoRA adapters — tiny trainable layers in FP16
4. Fine-tune only the adapters — the heavy base stays frozen and compressed

This lets you fine-tune a 70B model on a single consumer GPU that could never fit a full FP16 version.

The typical production pipeline

Pre-trained model (FP32/BF16)
        ↓
   Fine-tune           ← change behaviour (needs data + GPU time)
        ↓
   Calibrate           ← measure layer ranges (small data sample)
        ↓
   Quantize to INT4    ← compress for deployment
        ↓
 Deploy               ← small, fast, specialized

11. Practical Reference

Format selection guide

Symmetric vs asymmetric decision

Is your data balanced around zero?
├── YES → Symmetric  (weights, tanh activations)
└── NO  → Asymmetric (ReLU, sigmoid, any skewed activation)

PTQ vs QAT decision

Do you have budget to retrain?
├── NO  → PTQ  (fast, simple, good for INT8)
└── YES → QAT  (better quality, especially for INT4 and below)

Are you training from scratch at extreme compression (INT2 / 1.58-bit)?
└── YES → Native training (train with constraint baked in from day 1)

Calibration checklist

The complete formula reference

Float formats (FP32 / FP16 / BF16):

value = (−1)^s × 2^(E − bias) × (1 + M / 2^mantissa_bits)

FP32: bias=127, mantissa_bits=23
FP16: bias=15,  mantissa_bits=10
BF16: bias=127, mantissa_bits=7

Integer symmetric (INT8 / INT4):

S = max(|w|) / q_max
q = clamp( round(w / S),  −q_max,  +q_max )
w̃ = q × S
Z = 0 always

INT8: q_max=127,  range −128..+127
INT4: q_max=7,    range −8..+7

Integer asymmetric (INT8 / INT4):

S = (max − min) / (q_max − q_min)
Z = round( q_min − min/S )
q = clamp( round(w/S) + Z,  q_min,  q_max )
w̃ = S × (q − Z)

1.58-bit ternary:

α = mean( |weights| )
q = +1 if w > +α
q = −1 if w < −α
q =  0 otherwise
w̃ = q × α

Summary

Quantization is the art of making large models small enough to run on the hardware you have, at the quality level your application demands.

Float formats (FP32, FP16, BF16) reduce precision by truncating mantissa bits — no scaling required, each number self-describes its own range. Integer formats (INT8, INT4) reduce precision through linear mapping — a shared scale S and optional zero point Z translate between float space and integer buckets. The choice of symmetric vs asymmetric determines whether that mapping is centred at zero or shifted to match where the data actually lives.

Calibration finds the right S and Z before compression (PTQ). QAT learns them during training. Native training never needs them at all because the constraint was the foundation from the start.

The sweet spot for most production deployments today is INT4 weights with INT8 activations — fitting large models on consumer hardware with quality loss that is barely perceptible in practice.

All mathematical examples in this post use the same six example weights: [2.347, −1.821, 0.493, −0.156, 1.205, −2.634] throughout, so every method can be directly compared.