A Novel Memory-Efficient Deep Learning Training Framework via Error-Bounded Lossy Compression

Deep neural networks (DNNs) are becoming increasingly deeper, wider, and\nnon-linear due to the growing demands on prediction accuracy and analysis\nquality. When training a DNN model, the intermediate activation data must be\nsaved in the memory during forward propagation and then restored for backward\npropagation. However, state-of-the-art accelerators such as GPUs are only\nequipped with very limited memory capacities due to hardware design\nconstraints, which significantly limits the maximum batch size and hence\nperformance speedup when training large-scale DNNs.\n In this paper, we propose a novel memory-driven high performance DNN training\nframework that leverages error-bounded lossy compression to significantly\nreduce the memory requirement for training in order to allow training larger\nnetworks. Different from the state-of-the-art solutions that adopt image-based\nlossy compressors such as JPEG to compress the activation data, our framework\npurposely designs error-bounded lossy compression with a strict\nerror-controlling mechanism. Specifically, we provide theoretical analysis on\nthe compression error propagation from the altered activation data to the\ngradients, and then empirically investigate the impact of altered gradients\nover the entire training process. Based on these analyses, we then propose an\nimproved lossy compressor and an adaptive scheme to dynamically configure the\nlossy compression error-bound and adjust the training batch size to further\nutilize the saved memory space for additional speedup. We evaluate our design\nagainst state-of-the-art solutions with four popular DNNs and the ImageNet\ndataset. Results demonstrate that our proposed framework can significantly\nreduce the training memory consumption by up to 13.5x and 1.8x over the\nbaseline training and state-of-the-art framework with compression,\nrespectively, with little or no accuracy loss.\n