Gradient Checkpointing in LLM Training
Imagine you are inside the GPU… not looking at code, but watching a system thinking. A model is loaded. Billions of parameters sit quietly in memory. But the moment training begins, something far more dynamic unfolds. Data flows in, passes
Imagine you are inside the GPU… not looking at code, but watching a system thinking. A model is loaded. Billions of parameters sit quietly in memory. But the moment training begins, something far more dynamic unfolds. Data flows in, passes layer by layer, and at each step, the model doesn’t just compute — it remembers. Every layer produces activations, and these are carefully stored, like footprints left behind. Because later, when the model learns, it must walk back the same path. This is the natural rhythm of training: forward to predict, backward to learn. But now imagine the scale. Not a small model, but something with billions of parameters. Suddenly, this act of remembering becomes expensive. The GPU is no longer just holding weights. It is holding gradients, optimizer states, and most heavily of all — these activations from every layer. The deeper the network, the longer the sequence, the larger the batch… the more memory is consumed. Quietly, invisibly, the system approaches its limit. And then comes the constraint. Memory runs out. Now instead of asking, “How do we train faster? ”, the question becomes more fundamental: What truly needs to be remembered… and what can be reconstructed? This is where the idea of gradient checkpointing enters, not as an optimization trick, but as a shift in philosophy. Imagine watching a long journey, but instead of recording every single step, you place markers at key points. The rest of the path is allowed to fade. During the forward pass, the model no longer holds on to every activation. It keeps only these selected checkpoints, letting the intermediate states go. At first, this feels like loss. Like forgetting. But the system knows something deeper. When the backward pass begins — when learning truly happens — and a missing activation is needed, the model doesn’t panic. It simply retraces its steps. Starting from the nearest checkpoint, it recomputes what was once discarded. Not stored, but regenerated. Not remembered, but rebuilt. So memory is saved… by exchanging storage for effort. You begin to see the trade forming clearly. The GPU breathes easier, holding far less than before. Larger models become trainable. Bigger batches become possible. But in return, the system must think more. It must redo parts of its own journey during learning. Time is sacrificed… to preserve space. And now, if you step back, the entire training process reveals something subtle. Learning in these systems was never just about moving forward. It always depended on the ability to go backward — to revisit, to recompute, to refine. Gradient checkpointing simply makes this truth explicit. It says: you don’t need to remember everything, as long as you know how to rebuild it when it matters. So what began as a memory optimization becomes something more philosophical. A model does not need to hold the entire past… only enough anchors to reconstruct it when learning demands it.