Summit: Scaling Deep Learning Interpretability by Visualizing Activation and Attribution Summarizations

Deep learning is increasingly used in decision-making tasks. However,\nunderstanding how neural networks produce final predictions remains a\nfundamental challenge. Existing work on interpreting neural network predictions\nfor images often focuses on explaining predictions for single images or\nneurons. As predictions are often computed from millions of weights that are\noptimized over millions of images, such explanations can easily miss a bigger\npicture. We present Summit, an interactive system that scalably and\nsystematically summarizes and visualizes what features a deep learning model\nhas learned and how those features interact to make predictions. Summit\nintroduces two new scalable summarization techniques: (1) activation\naggregation discovers important neurons, and (2) neuron-influence aggregation\nidentifies relationships among such neurons. Summit combines these techniques\nto create the novel attribution graph that reveals and summarizes crucial\nneuron associations and substructures that contribute to a model's outcomes.\nSummit scales to large data, such as the ImageNet dataset with 1.2M images, and\nleverages neural network feature visualization and dataset examples to help\nusers distill large, complex neural network models into compact, interactive\nvisualizations. We present neural network exploration scenarios where Summit\nhelps us discover multiple surprising insights into a prevalent, large-scale\nimage classifier's learned representations and informs future neural network\narchitecture design. The Summit visualization runs in modern web browsers and\nis open-sourced.\n