Inference cost at scale with napkin math

▤ Summary

This article breaks down the cost of serving LLMs at scale using napkin math. It explains how matrix multiplication, attention, and KV-cache shape inference compute and memory requirements. It then estimates how many concurrent users a Blackwell-class GPU such as the NVIDIA B200 can support and what that means for throughput and cost per user. The piece also highlights practical optimizations like KV-cache reuse, grouped-query attention, and paged attention for production AI systems. Overall, it gives software teams a rough framework for planning AI infrastructure and unit economics.

▦ Linked Companies

NVIDIA Corporation

$1B+