Your Quantized LLM Is Not Slow Because of the Quantization

▤ Summary

The article explains why a quantized LLM can still run slowly even after aggressive compression. It shows that the real bottleneck came from a large 778 MB output-projection copy performed on every token, not from the quantization scheme itself. The author demonstrates that profiling unmodified components and counting bytes moved can reveal hidden memory-traffic bottlenecks. The piece also describes a fix in the Candle framework that routes the transposed matrix directly into GEMM and removes the extra copy. It concludes with practical guidance for profiling and optimizing inference pipelines.

▦ Linked Companies