The use of lower precision has emerged as a popular technique to optimize the compute and storage requirements of complex deep neural networks (DNNs). In the quest for lower precision, recent studies have shown that ternary DNNs (which represent weights and activations by signed ternary values) represent a promising sweet spot, achieving accuracy close to full-precision networks on complex tasks. We propose TiM-DNN, a programmable in-memory accelerator that is specifically designed to execute ternary DNNs. TiM-DNN supports various ternary representations including unweighted {−1, 0, 1}, symmetric weighted <inline-formula> <tex-math notation="LaTeX">$\{-a,0,a\}$ </tex-math></inline-formula>, and asymmetric weighted <inline-formula> <tex-math notation="LaTeX">$\{-a,0,b\}$ </tex-math></inline-formula> ternary systems. The building blocks of TiM-DNN are TiM tiles—specialized memory arrays that perform massively parallel signed ternary vector–matrix multiplications with a single access. TiM tiles are in turn composed of ternary processing cells (TPCs), bit-cells that function as both ternary storage units and signed ternary multiplication units. We evaluate an implementation of TiM-DNN in 32-nm technology using an architectural simulator calibrated with SPICE simulations and RTL synthesis. We evaluate TiM-DNN across a suite of state-of-the-art DNN benchmarks including both deep convolutional and recurrent neural networks. A 32-tile instance of TiM-DNN achieves a peak performance of 114 TOPs/s, consumes 0.9-W power, and occupies 1.96 mm<sup>2</sup> chip area, representing a <inline-formula> <tex-math notation="LaTeX">$300\times $ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$388\times $ </tex-math></inline-formula> improvement in TOPS/W and TOPS/mm<sup>2</sup>, respectively, compared to an NVIDIA Tesla V100 GPU. In comparison to specialized DNN accelerators, TiM-DNN achieves <inline-formula> <tex-math notation="LaTeX">$55\times $ </tex-math></inline-formula>-<inline-formula> <tex-math notation="LaTeX">$240\times $ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$160\times $ </tex-math></inline-formula>-<inline-formula> <tex-math notation="LaTeX">$291\times $ </tex-math></inline-formula> improvement in TOPS/W and TOPS/mm<sup>2</sup>, respectively. Finally, when compared to a well-optimized near-memory accelerator for ternary DNNs, TiM-DNN demonstrates <inline-formula> <tex-math notation="LaTeX">$3.9\times $ </tex-math></inline-formula>-<inline-formula> <tex-math notation="LaTeX">$4.7\times $ </tex-math></inline-formula> improvement in system-level energy and <inline-formula> <tex-math notation="LaTeX">$3.2\times $ </tex-math></inline-formula>-<inline-formula> <tex-math notation="LaTeX">$4.2\times $ </tex-math></inline-formula> speedup, underscoring the potential of in-memory computing for ternary DNNs.