With the increasing computational demands of neural networks, many hardware\naccelerators for the neural networks have been proposed. Such existing neural\nnetwork accelerators often focus on popular neural network types such as\nconvolutional neural networks (CNNs) and recurrent neural networks (RNNs);\nhowever, not much attention has been paid to attention mechanisms, an emerging\nneural network primitive that enables neural networks to retrieve most relevant\ninformation from a knowledge-base, external memory, or past states. The\nattention mechanism is widely adopted by many state-of-the-art neural networks\nfor computer vision, natural language processing, and machine translation, and\naccounts for a large portion of total execution time. We observe today's\npractice of implementing this mechanism using matrix-vector multiplication is\nsuboptimal as the attention mechanism is semantically a content-based search\nwhere a large portion of computations ends up not being used. Based on this\nobservation, we design and architect A3, which accelerates attention mechanisms\nin neural networks with algorithmic approximation and hardware specialization.\nOur proposed accelerator achieves multiple orders of magnitude improvement in\nenergy efficiency (performance/watt) as well as substantial speedup over the\nstate-of-the-art conventional hardware.\n