TechApr 9, 202615 minMegaTrain Trains a 120B-Parameter LLM on a Single GPU at Full PrecisionMegaTrain flips the GPU-centric paradigm by treating CPU memory as primary storage and the GPU as a transient compute device, enabling full-precision training of 100B+ LLMs on a single GPU with up to 12.2x throughput over DeepSpeed ZeRO-3.LLMMachine LearningGPUDeepSpeedMemory Optimization
