MAD-OOD: A Deep Learning Cluster-Driven Framework for an Out-of-Distribution Malware Detection and Classification

Out-of-distribution (OOD) detection in malware classification remains a significant challenge due to the high intra-class variability among malware variants within the same family. Existing deep learning approaches often overlook this intra-family variation, resulting in suboptimal detection performance. This research proposes a two-stage framework that addresses this limitation by incorporating Gaussian discriminant analysis (GDA) into deep neural networks to model spherical decision boundaries around malware families in the embedding space. The first stage employs unsupervised cluster analysis to determine whether a test sample is in-distribution or out-of-distribution, using z-score-based statistical analysis for reliable outlier detection. The second stage introduces a deep learning model trained on refined embeddings from the initial stage, using predictions from both the cluster analysis and a primary classifier to enhance final prediction accuracy. Evaluation on a dataset comprising 25 malware families and novel OOD samples demonstrates superior performance, achieving an AUC of 0.911 for OOD detection. This approach significantly improves the distinguishability of OOD samples and offers a scalable and statistically grounded method for robust out-of-distribution malware classification and anomaly detection in security contexts.