A Doubly Regularized Linear Discriminant Analysis Classifier with Automatic Parameter Selection

Linear discriminant analysis (LDA) based classifiers tend to falter in many\npractical settings where the training data size is smaller than, or comparable\nto, the number of features. As a remedy, different regularized LDA (RLDA)\nmethods have been proposed. These methods may still perform poorly depending on\nthe size and quality of the available training data. In particular, the test\ndata deviation from the training data model, for example, due to noise\ncontamination, can cause severe performance degradation. Moreover, these\nmethods commit further to the Gaussian assumption (upon which LDA is\nestablished) to tune their regularization parameters, which may compromise\naccuracy when dealing with real data. To address these issues, we propose a\ndoubly regularized LDA classifier that we denote as R2LDA. In the proposed\nR2LDA approach, the RLDA score function is converted into an inner product of\ntwo vectors. By substituting the expressions of the regularized estimators of\nthese vectors, we obtain the R2LDA score function that involves two\nregularization parameters. To set the values of these parameters, we adopt\nthree existing regularization techniques; the constrained perturbation\nregularization approach (COPRA), the bounded perturbation regularization (BPR)\nalgorithm, and the generalized cross-validation (GCV) method. These methods are\nused to tune the regularization parameters based on linear estimation models,\nwith the sample covariance matrix's square root being the linear operator.\nResults obtained from both synthetic and real data demonstrate the consistency\nand effectiveness of the proposed R2LDA approach, especially in scenarios\ninvolving test data contaminated with noise that is not observed during the\ntraining phase.\n