LipSyncNet: A Novel Deep Learning Approach for Visual Speech Recognition in Audio-Challenged Situations

Abstract

In recent lip-reading technologies, deep learning methodologies have emerged as the key, transcending the limitations of traditional hybrid Deep Neural Network-Hidden Markov Model (DNN-HMM) frameworks based on Discrete Cosine Transform (DCT) features. LipSyncNet comprises a three-dimensional-Convolutional Neural Network (3D-CNN) that consists of a maximum depth of four layers and is responsible for extracting visual features by integrating EfficientNetB0, which results in excellent feature extraction capabilities. Following this, the network architecture incorporates a backend that utilizes a Bidirectional Long Short-Term Memory (Bi-LSTM)—a component of the recurrent neural network family—combined with Connectionist Temporal Classification (CTC) loss, enhancing its ability to perform classification tasks. The effectiveness of the proposed method is demonstrated through the evaluation of the Graphics Research International Database (GRID) corpus, a challenging word-level lip-reading dataset. Initially, facial features are extracted from the mouth area of an individual’s face. Subsequently, these features are combined with available audio information to identify spoken words precisely. The lip-reading method aims to create a system that achieves accurate speech recognition by observing visual cues, thereby reducing the reliance on audio. The model utilizes information from various levels in a unified structure, enabling it to differentiate between words that sound alike and to improve its ability to handle changes in physical appearance.