Capstone Part 4: Quantization, Fusion, Benchmarks, and Postmortem

An inference engine is not finished because it generates text. It is finished when you can measure latency, throughput, memory, quality impact, and the next bottleneck. Day 30 makes the capstone honest.

1. Add per-channel INT8 weight-only quantization to linear layers.
2. Optionally add group INT4 for weights.
3. Fuse one operation path or use torch.compile as a baseline.
4. Run a benchmark matrix for TTFT, TPOT, throughput, and peak memory.
5. Compare with a production engine on the same task.
6. Write a one-page engineering postmortem.

• TTFT is time to first token.
• ITL is inter-token latency.
• TPOT is time per output token.
• peak_mem is maximum device memory used during the run.
• speedup is baseline time divided by optimized time.

For a linear weight matrix [out_features, in_features], compute one scale per output row. Store int8_weight and fp16_scale. A [2048, 2048] FP16 matrix is 8,388,608 bytes; INT8 plus per-row FP16 scales is about 4,198,400 bytes, nearly 2x smaller.

Fusion reduces memory traffic and kernel launches. Good capstone targets are RMSNorm plus residual bookkeeping, or SwiGLU gate/up projection fusion. If custom kernels are too much, use torch.compile and measure eager versus compiled.

Do not fill this table from memory. The notebook writes the schema; the student fills it with measured values.

After comparison, identify the largest gap. Likely causes: no FlashAttention decode kernel, Python scheduler overhead, no CUDA Graphs, inefficient dequantization, padding waste, sampling overhead, or tokenizer/server overhead. A professional postmortem separates measured facts from guesses.

• Model loader works on real weights.
• Single-sequence forward verifies against reference logits.
• KV cache decode matches uncached outputs.
• Sampling loop supports greedy and at least one stochastic mode.
• Block pool and page table run multi-sequence scheduling.
• INT8 quantization reduces weight memory.
• One fusion or compile path is benchmarked.
• Benchmark table includes real hardware values.
• Postmortem states the biggest bottleneck and next optimization.

1. Implement per-channel INT8 quantization for all linear weights or a representative subset.
2. Benchmark FP16 versus INT8 memory and latency.
3. Fuse one hot path or run torch.compile and compare eager versus optimized.
4. Run the same model/task on one production engine available on your hardware.
5. Write a one-page postmortem with benchmark table and next optimization.

1. Why weight-only INT8 first? It is simple, gives about 2x weight memory reduction, and keeps activations precise.
2. What does fusion reduce? Intermediate memory traffic and kernel launch overhead.
3. Why compare to a production engine? It calibrates how far the educational engine is from real serving stacks.
4. What makes a benchmark professional? Pinned versions, same workload, warmups, real hardware metrics, and honest notes.
5. What is the final goal of the capstone? A correct, measured engine plus a clear explanation of remaining bottlenecks.