When does fragmentation occur in the CUDA caching allocator?
Summary
This article explains why the CUDA caching allocator in PyTorch can fragment GPU memory and how allocation order affects reserved memory. It walks through how segments and blocks behave with and without expandable segments, showing why small-then-large allocation patterns can waste memory under the old model. It then shows how expandable segments use CUDA virtual memory APIs to keep allocations in one contiguous pool and reduce fragmentation for common workloads like CUDA graph recording and LLM serving. It also notes two important limits: live allocations can still create fragmentation, and crossing the 1 MiB boundary splits memory into separate pools. The piece is useful for teams optimizing GPU memory usage in PyTorch and planning LLM inference or CUDA graph workflows.
Classifications
AskAI Classifications
LabelsLinked Companies
