Interpretability is an important aspect of the trustworthiness of a model's\npredictions. Transformer's predictions are widely explained by the attention\nweights, i.e., a probability distribution generated at its self-attention unit\n(head). Current empirical studies provide shreds of evidence that attention\nweights are not explanations by proving that they are not unique. A recent\nstudy showed theoretical justifications to this observation by proving the\nnon-identifiability of attention weights. For a given input to a head and its\noutput, if the attention weights generated in it are unique, we call the\nweights identifiable. In this work, we provide deeper theoretical analysis and\nempirical observations on the identifiability of attention weights. Ignored in\nthe previous works, we find the attention weights are more identifiable than we\ncurrently perceive by uncovering the hidden role of the key vector. However,\nthe weights are still prone to be non-unique attentions that make them unfit\nfor interpretation. To tackle this issue, we provide a variant of the encoder\nlayer that decouples the relationship between key and value vector and provides\nidentifiable weights up to the desired length of the input. We prove the\napplicability of such variations by providing empirical justifications on\nvaried text classification tasks. The implementations are available at\nhttps://github.com/declare-lab/identifiable-transformers.\n