Beyond assigning the correct class, an activity recognition model should also\nbe able to determine, how certain it is in its predictions. We present the\nfirst study of how welthe confidence values of modern action recognition\narchitectures indeed reflect the probability of the correct outcome and propose\na learning-based approach for improving it. First, we extend two popular action\nrecognition datasets with a reliability benchmark in form of the expected\ncalibration error and reliability diagrams. Since our evaluation highlights\nthat confidence values of standard action recognition architectures do not\nrepresent the uncertainty well, we introduce a new approach which learns to\ntransform the model output into realistic confidence estimates through an\nadditional calibration network. The main idea of our Calibrated Action\nRecognition with Input Guidance (CARING) model is to learn an optimal scaling\nparameter depending on the video representation. We compare our model with the\nnative action recognition networks and the temperature scaling approach - a\nwide spread calibration method utilized in image classification. While\ntemperature scaling alone drastically improves the reliability of the\nconfidence values, our CARING method consistently leads to the best uncertainty\nestimates in all benchmark settings.\n