We propose an efficient and straightforward method for compressing deep\nconvolutional neural networks (CNNs) that uses basis filters to represent the\nconvolutional layers, and optimizes the performance of the compressed network\ndirectly in the basis space. Specifically, any spatial convolution layer of the\nCNN can be replaced by two successive convolution layers: the first is a set of\nthree-dimensional orthonormal basis filters, followed by a layer of\none-dimensional filters that represents the original spatial filters in the\nbasis space. We jointly fine-tune both the basis and the filter representation\nto directly mitigate any performance loss due to the truncation. Generality of\nthe proposed approach is demonstrated by applying it to several well known deep\nCNN architectures and data sets for image classification and object detection.\nWe also present the execution time and power usage at different compression\nlevels on the Xavier Jetson AGX processor.\n