Multimodal Fusion for Disaster Event Classification on Social Media: A Deep Federated Learning Approach

Abstract

This paper explores the intersection of federated learning and disaster identification using a curated dataset of captioned images sourced from social media. Leveraging a federated learning framework, our methodology involves iterative client updates, server-side aggregation, and comprehensive testing to enhance the global model’s understanding of disaster-related multimedia content. The study incorporates deep embeddings extracted and encoded by BERT models with generic image features extracted by ResNet, which is followed by a late fusion strategy to formulate discriminating features from both textual and visual modalities. Through collaborative efforts among decentralized clients, the global model demonstrates improved accuracy and robustness in identifying and classifying diverse disaster- related scenarios. With an accuracy of 85.1% and F1-score of 85.2%, this multimodal deep federated model contributes to the evolving field of federated learning, highlighting the significance of adaptability, data privacy preservation, and iterative feature refinement in improving the performance of disaster event identification and analysis.