Fusing Deep Learned and Hand-Crafted Features of Appearance, Shape, and Dynamics for Automatic Pain Estimation

Automatic continuous time, continuous value assessment of a patient's pain\nfrom face video is highly sought after by the medical profession. Despite the\nrecent advances in deep learning that attain impressive results in many\ndomains, pain estimation risks not being able to benefit from this due to the\ndifficulty in obtaining data sets of considerable size. In this work we propose\na combination of hand-crafted and deep-learned features that makes the most of\ndeep learning techniques in small sample settings. Encoding shape, appearance,\nand dynamics, our method significantly outperforms the current state of the\nart, attaining a RMSE error of less than 1 point on a 16-level pain scale,\nwhilst simultaneously scoring a 67.3% Pearson correlation coefficient between\nour predicted pain level time series and the ground truth.\n