Understanding LLM Checkpoint/Restore I/O Strategies and Patterns

As large language models (LLMs) and foundation models (FMs) scale, checkpoint-restore has become critical pattern for both training and inferences. This is a difficult pattern: LLMs and FMs are distributed at scale using 3D parallelism (tensor, pipeline, data), resulting in a large number of processes, each holding a large number of tensors of different shapes and sizes, which need to be captured to stable storage (e.g. parallel file systems) frequently. As a consequence, checkpoint and restore becomes a big data problem, exhibiting volume, variery and velocity. A key challenge is the need to traverse an entire storage stack, from the GPUs, where the tensors live, through the host memory, local storage and finally external repositories such as parallel file systems, where the checkpoints need to be persisted. Despite efficient I/O techniques (e.g. asynchronous flushing/prefetching), the memory tiers of the storage stack exhibit orders of magnitdue difference in I/O performance, leading to I/O bottlenecks, especially under concurrency. Kernel-accelerated I/O libraries such as liburing aim to address some of these I/O bottlenecks compared with the aging POSIX I/O interface. However, their effectiveness for LLM and checkpoint and restore remains largely unexplored. To this end, we develop a series of microbenchmarks that study various trade-off when using liburing. Specifically, we evaluate how aggregation, alignment, and I/O coalescing interact under buffered and direct I/O modes. Results show that uncoalesced, small-buffer operations halve throughput relative to synthetic workloads, while aggregation restores bandwidth and reduces metadata overhead compared to the state-of-the-art LLM checkpointing engines, achieving up to 3.9 × higher write throughput than DataStates-LLM and 7.6 × higher than TorchSnapshot. These findings underscore the importance of adopting file system–aware aggregation and I/O coalescing strategies in LLM checkpointing to fully exploit modern storage and I/O backends.