Recent advancements in machine learning and signal processing domains have\nresulted in an extensive surge of interest in Deep Neural Networks (DNNs) due\nto their unprecedented performance and high accuracy for different and\nchallenging problems of significant engineering importance. However, when such\ndeep learning architectures are utilized for making critical decisions such as\nthe ones that involve human lives (e.g., in control systems and medical\napplications), it is of paramount importance to understand, trust, and in one\nword "explain" the argument behind deep models' decisions. In many\napplications, artificial neural networks (including DNNs) are considered as\nblack-box systems, which do not provide sufficient clue on their internal\nprocessing actions. Although some recent efforts have been initiated to explain\nthe behaviors and decisions of deep networks, explainable artificial\nintelligence (XAI) domain, which aims at reasoning about the behavior and\ndecisions of DNNs, is still in its infancy. The aim of this paper is to provide\na comprehensive overview on Understanding, Visualization, and Explanation of\nthe internal and overall behavior of DNNs.\n