SpeakingFaces: A Large-Scale Multimodal Dataset of Voice Commands with Visual and Thermal Video Streams

We present SpeakingFaces as a publicly-available large-scale multimodal\ndataset developed to support machine learning research in contexts that utilize\na combination of thermal, visual, and audio data streams; examples include\nhuman-computer interaction, biometric authentication, recognition systems,\ndomain transfer, and speech recognition. SpeakingFaces is comprised of aligned\nhigh-resolution thermal and visual spectra image streams of fully-framed faces\nsynchronized with audio recordings of each subject speaking approximately 100\nimperative phrases. Data were collected from 142 subjects, yielding over 13,000\ninstances of synchronized data (~3.8 TB). For technical validation, we\ndemonstrate two baseline examples. The first baseline shows classification by\ngender, utilizing different combinations of the three data streams in both\nclean and noisy environments. The second example consists of thermal-to-visual\nfacial image translation, as an instance of domain transfer.\n