Combining Phone Posteriorgrams from Strong and Weak Recognizers for Automatic Speech Assessment of People with Aphasia

Abstract

This paper presents an investigation on applying automatic speech recognition (ASR) to speech assessment of people with aphasia (PWA). A distinctive characteristic of PWA speech is paraphasia, which refers to frequent occurrence of phonemic errors, unintended words and non-verbal sounds. In view of the wide variety of paraphasias, we propose to view the ASR errors so caused as out-of-vocabulary (OOV) words. Inspired by previous research on OOV detection, paraphasias in PWA speech are captured by comparing the phone posteriorgrams of a strongly constrained speech recognizer and a weakly constrained one. The posteriorgrams also reveal other characteristics of impaired speech, e.g., change of speaking rate, voice abnormality. Siamese and 2-channel convolutional neural network (CNN) models are used for classifying the posteriorgram pairs and predicting the severity of aphasia. Experimental results on a Cantonese database of PWA speech confirm the effectiveness of the proposed methods. The best F1 score attained on binary classification (severe versus mild aphasia) is 0.891.