Ordering Chaos: Memory-Aware Scheduling of Irregularly Wired Neural Networks for Edge Devices

Recent advances demonstrate that irregularly wired neural networks from\nNeural Architecture Search (NAS) and Random Wiring can not only automate the\ndesign of deep neural networks but also emit models that outperform previous\nmanual designs. These designs are especially effective while designing neural\narchitectures under hard resource constraints (memory, MACs, . . . ) which\nhighlights the importance of this class of designing neural networks. However,\nsuch a move creates complication in the previously streamlined pattern of\nexecution. In fact one of the main challenges is that the order of such nodes\nin the neural network significantly effects the memory footprint of the\nintermediate activations. Current compilers do not schedule with regard to\nactivation memory footprint that it significantly increases its peak compared\nto the optimum, rendering it not applicable for edge devices. To address this\nstanding issue, we present a memory-aware compiler, dubbed SERENITY, that\nutilizes dynamic programming to find a sequence that finds a schedule with\noptimal memory footprint. Our solution also comprises of graph rewriting\ntechnique that allows further reduction beyond the optimum. As such, SERENITY\nachieves optimal peak memory, and the graph rewriting technique further\nimproves this resulting in 1.68x improvement with dynamic programming-based\nscheduler and 1.86x with graph rewriting, against TensorFlow Lite with less\nthan one minute overhead.\n