Deep neural network (DNN) has emerged as the most important and popular\nartificial intelligent (AI) technique. The growth of model size poses a key\nenergy efficiency challenge for the underlying computing platform. Thus, model\ncompression becomes a crucial problem. However, the current approaches are\nlimited by various drawbacks. Specifically, network sparsification approach\nsuffers from irregularity, heuristic nature and large indexing overhead. On the\nother hand, the recent structured matrix-based approach (i.e., CirCNN) is\nlimited by the relatively complex arithmetic computation (i.e., FFT), less\nflexible compression ratio, and its inability to fully utilize input sparsity.\nTo address these drawbacks, this paper proposes PermDNN, a novel approach to\ngenerate and execute hardware-friendly structured sparse DNN models using\npermuted diagonal matrices. Compared with unstructured sparsification approach,\nPermDNN eliminates the drawbacks of indexing overhead, non-heuristic\ncompression effects and time-consuming retraining. Compared with circulant\nstructure-imposing approach, PermDNN enjoys the benefits of higher reduction in\ncomputational complexity, flexible compression ratio, simple arithmetic\ncomputation and full utilization of input sparsity. We propose PermDNN\narchitecture, a multi-processing element (PE) fully-connected (FC)\nlayer-targeted computing engine. The entire architecture is highly scalable and\nflexible, and hence it can support the needs of different applications with\ndifferent model configurations. We implement a 32-PE design using CMOS 28nm\ntechnology. Compared with EIE, PermDNN achieves 3.3x~4.8x higher throughout,\n5.9x~8.5x better area efficiency and 2.8x~4.0x better energy efficiency on\ndifferent workloads. Compared with CirCNN, PermDNN achieves 11.51x higher\nthroughput and 3.89x better energy efficiency.\n