The economics of speculative decoding

▤ Summary

This article analyzes the economics of speculative decoding for large language model inference. It explains how mixture-of-experts routing and compressed attention reduce the free slack that speculative tokens used to exploit in dense transformers. It shows that acceptance and verification costs now vary with batch size and model architecture, which can make speculation less attractive in some serving regimes. The piece also argues that adaptive speculation can still improve throughput, especially when operators tune draft length to workload and model conditions.

▦ Linked Companies