Alzheimer's Dementia Recognition Using Acoustic, Lexical, Disfluency and Speech Pause Features Robust to Noisy Inputs

We present two multimodal fusion-based deep learning models that consume ASR\ntranscribed speech and acoustic data simultaneously to classify whether a\nspeaker in a structured diagnostic task has Alzheimer's Disease and to what\ndegree, evaluating the ADReSSo challenge 2021 data. Our best model, a BiLSTM\nwith highway layers using words, word probabilities, disfluency features, pause\ninformation, and a variety of acoustic features, achieves an accuracy of 84%\nand RSME error prediction of 4.26 on MMSE cognitive scores. While predicting\ncognitive decline is more challenging, our models show improvement using the\nmultimodal approach and word probabilities, disfluency and pause information\nover word-only models. We show considerable gains for AD classification using\nmultimodal fusion and gating, which can effectively deal with noisy inputs from\nacoustic features and ASR hypotheses.\n