Combining Image Inductive Bias and Self-Attention Mechanism for Accurate Isolated Sign Language Recognition

Abstract

Isolated sign language recognition has been an important part of breaking down communication bottlenecks for deaf-mute and others. While facing this problem, the purpose of this paper is to classify American isolated sign language video by modeling pose, hands and face keypoints representation. Specifically, this paper introduces a novel framework whose main components are the altered Dense Predictive Coding (DPC) pre-trained model and the Encoder pre-trained model. The DPC model is trained using self-supervised learning to obtain representation of pose and hands keypoints. The Encoder model is trained using supervised learning to obtain representation of face keypoints. Combining the altered DPC model with image inductive biases and the Encoder model with a self-attention mechanism, the final combined model achieves 0.81 on the test set of the ISAL dataset, outperforming the current open-source solution by a significant margin.