Hate speech is an important problem in the management of user-generated\ncontent. To remove offensive content or ban misbehaving users, content\nmoderators need reliable hate speech detectors. Recently, deep neural networks\nbased on the transformer architecture, such as the (multilingual) BERT model,\nachieve superior performance in many natural language classification tasks,\nincluding hate speech detection. So far, these methods have not been able to\nquantify their output in terms of reliability. We propose a Bayesian method\nusing Monte Carlo dropout within the attention layers of the transformer models\nto provide well-calibrated reliability estimates. We evaluate and visualize the\nresults of the proposed approach on hate speech detection problems in several\nlanguages. Additionally, we test if affective dimensions can enhance the\ninformation extracted by the BERT model in hate speech classification. Our\nexperiments show that Monte Carlo dropout provides a viable mechanism for\nreliability estimation in transformer networks. Used within the BERT model, it\nofers state-of-the-art classification performance and can detect less trusted\npredictions. Also, it was observed that affective dimensions extracted using\nsentic computing methods can provide insights toward interpretation of emotions\ninvolved in hate speech. Our approach not only improves the classification\nperformance of the state-of-the-art multilingual BERT model but the computed\nreliability scores also significantly reduce the workload in an inspection of\nofending cases and reannotation campaigns. The provided visualization helps to\nunderstand the borderline outcomes.\n