This paper presents GPU performance optimization and scaling results for\ninference models of the Sparse Deep Neural Network Challenge 2020. Demands for\nnetwork quality have increased rapidly, pushing the size and thus the memory\nrequirements of many neural networks beyond the capacity of available\naccelerators. Sparse deep neural networks (SpDNN) have shown promise for\nreining in the memory footprint of large neural networks. However, there is\nroom for improvement in implementing SpDNN operations on GPUs. This work\npresents optimized sparse matrix multiplication kernels fused with the ReLU\nfunction. The optimized kernels reuse input feature maps from the shared memory\nand sparse weights from registers. For multi-GPU parallelism, our SpDNN\nimplementation duplicates weights and statically partition the feature maps\nacross GPUs. Results for the challenge benchmarks show that the proposed kernel\ndesign and multi-GPU parallelization achieve up to 180 tera-edges per second\ninference throughput. These results are up to 4.3x faster for a single GPU and\nan order of magnitude faster at full scale than those of the champion of the\n2019 Sparse Deep Neural Network Graph Challenge for the same generation of\nNVIDIA V100 GPUs. Using the same implementation, we also show single-GPU\nthroughput on NVIDIA A100 is 2.37$\\times$ faster than V100.\n