Feature Selection in the Contrastive Analysis Setting

Summary

The paper studied feature selection in the contrastive analysis setting. There are two types of datasets, the background dataset and the target dataset. The background dataset contains variations due to the uninteresting sources. The target dataset contains observations with variability from both ``interesting'' and ``uninteresting'' sources. The goal of the problem is to select interesting variables. Since interesting variables only exist in target variable, the feature selection is performed on the target dataset. To make use of the background dataset, the embedding parameters learned by the auto encoder on the background dataset is shared with the target dataset in feature selection. The problem is with practice interest. But the exact layers of background representation encoder, the background reconstruction and the target reconstruction network are not specified. The paper vaguely mentioned that it is similar to auto-encoder. Since the architecture is the main contributions of the paper, it is better to specify the networks. The theoretical analysis is about mutual information $I$, which is related with multiple variables. However, $I$ is not defined in the paper. Feature selection selects features that are most related with the target. However, the experiments are based on MNIST. There is no analysis about what features are selected. Though the motivation of the problem is about biomedical study, there is no experiments on related area.

Strengths

The paper is well-written. The motivation is strong. The paper proposed a method, made related analysis and conducted some experiments.

Weaknesses

1. The architecture is not clear to me, specially the representation / reconstruction networks. 2. There is no ablation study if there is no background dataset, the background representation encoder is initialized by random distribution. 3. There is no analysis about the impact of different population of background dataset and target dataset. 4. There is not enough experiments to validate the performance of the proposed method.

Questions

1. What is the definition of the mutual information $I$? 2. What is the impact of the population of background dataset and target dataset? 3. What does the networks $g$, $h$ and $f$ look like? 4. Is the feature space of the background dataset is a subset of the feature space of the target dataset? 5. What is the scale of population and feature dimension to make the proposed method work?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

1. The theoretical analysis is not clear which means the performance is not guaranteed. 2. Assume the background dataset is in large scale, how to select the background dataset related to the target dataset is important but not discussed here. 3.There are many assumptions to implement the algorithms in real-world applications, such as the population between background and target datasets, the number of selected features $s$ compared with the global feature space.

Summary

The paper aims to solve the problem of “salient” feature selection in a Contrastive Analysis setting, termed “Contrastive Feature Selection.” The authors' primary focus lies in uncovering distinctive features that highlight salient variations present in a specific target dataset that are absent in a standard background dataset. The paper discusses & compares the limitations of popular feature selection methods both in supervised & unsupervised methods settings and further attempts to formulate a “Contrastive Feature Selection” method to extract those salient features. The paper provides an initial empirical formulation based on intuitions, then formalizes the idea using principles of Information Theory (bounds on Mutual Information, Entropy) and finishes with empirical results on semi-synthetic (vision) and real-world (medical) datasets. Overall, the paper has made a novel attempt at feature selection in a weak supervision setting using a background and target dataset, with the goal being to learn features specific to the target dataset.

Strengths

1. The paper introduces a "contrastive feature selection" method. The authors provide experimental validation with popularly used baselines (both supervised+unsupervised) employed in feature selection. 2. The paper also provided intuitive, experimental, and mathematical proofs (based on Information Theoretic bounds) to justify their proposed method. 3. Another strength of the proposed approach is it relies only on weak supervision (whether a data sample is part of a background/target dataset). 4. The results show the superior performance of the proposed feature selection method for downstream tasks. 5. Code provided for reproducibility.

Weaknesses

1. The authors use only "Random Forest Classifiers" for their downstream tasks. It would be a good idea to add comparative results (perhaps in the Supplementary) using other popular classifiers (like Decision Trees, SVM, ANNs, XGBoosts, etc.) to show that the feature selection works across different classifiers. 2. The paper shows empirical results on the "Grassy-MNIST" and four other medical datasets. However, results on more practical classification tasks (>10 classes) would make more sense. Also, it would be interesting to see the performance variation for a particular dataset like GrassyMNIST as background vs. target dataset size is varied. 3. A more extensive ablation of varying dimensions of the latent {k,l} is missing. 4. The overall organization of the paper can be improved. It feels a bit unorganized going through the paper.

Questions

1. Is adding experiments based on Weakness Section Points 1-2 possible? 2. Also, is it possible to provide a theoretical explanation or intuitive insight why the two variants of CFS-SG and CFS-Pretrained result in a similar performance, but the top performer changes when tested across datasets? Is it related to some factor like dataset sizes/ number of classes for classification?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

1 poor

Contribution

2 fair

Limitations

1. Limited evaluation of real-life classification datasets. All datasets had <= ten classes. 2. Validation of the selected feature performance when using different classifier models.

Summary

The authors tackle the important problem of feature selection in a contrastive setting. Supervised and unsupervised feature selection problems have been extensively studied and have many high-impact applications. In this work, the authors design an algorithm for finding features that are more enriched in a target set than a background set (contrast). This type of comparison is widely used in biology, for example, in differential expression gene (DEG) analysis, where biologists look for differentially expressed genes across different medical conditions. The proposed method is based on a two-step procedure; in the first step, an autoencoder is trained to learn the latent information required to represent the background data. In the second step, a feature selection module is trained to identify the set of features required to capture the additional information needed to represent the target data. To sparsify the feature selection module, they use the recently proposed stochastic gates, which are continuously relaxed random variables. The new approach is motivation using a mutual information perspective. Several real and semi-synthetic datasets are used to evaluate the approach.

Strengths

The problem is important with many applications in bioinformatics. The proposed is new and seems highly appropriate for the suggested problem. The experimental results highlight that the approach can lead to advantages compared to existing schemes. The authors propose a few variations for training the method and compare their quality. The benefits of the two-stage approach are justified using mutual information under mild assumptions.

Weaknesses

Conceptually the problem could be solved using supervised feature selection; the authors also use supervised FS models as baselines. I understand that in terms of some applications, the setting is different. However, this is not well explained in the paper. Specifically, the authors should add information and results on how the selected features differ for the contrastive analysis case compared with the supervised (or unsupervised) case. Accuracy is important, but interpretability is vital in the examples motivated by the authors. This could be done using synthetic data or by benchmarking the method (and others) on a previously studied DEG problem. This will significantly improve the paper and help increase its impact and use in bioinformatics.

Questions

Some parts of the paper are very similar to [37], for example, two first sentences in the abstract and figure 1. Propper credit should be given here. Was the 80-20 percent split performed for FS or only for the classification part? This is not clear from the text. What are the architectures used for learning the background? Target? And also for STG and CAE? How are those tuned? While the authors demonstrate the merits of the approach for improving class separation (without access to class labels), I’m missing an evaluation that is tied to the motivation. Specifically, an example is that the approach is beneficial for DEG. As mentioned, this could be done using synthetic or real data with known gene markers. This would substantially improve the paper.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations of the method are not discussed in the paper.

Summary

They employed a contrastive analysis method for feature selection on the target dataset, which provides variation not found in the background dataset. Contrary to previous contrastive analysis that typically involves joint modelling of background and target datasets to identify shared variables, they drew from information-theoretic analysis to demonstrate that a two-step pre-training model outperforms joint modelling.

Strengths

Feature selection is a common requirement in health data to enhance interpretability. This paper clearly explained why the method proposed by them has a better performance than previous joint modelling and have good downstream accuracy when the number of features is limited.

Weaknesses

That method needs strong assumptions that may not hold in the real world. Assumption one states that the latent variables s, z are roughly independent, but the genes in scRNA-seq (usage case in its real data experiments) are mostly highly correlated.

Questions

In the real dataset, I noticed that all classes are larger than 1. What happens if there's only one class? Can the method still find variation that distinguishes it from the background? Regarding AML treatment, if the goal is to identify features that separate post-treatment from pre-treatment, could we use pre-treatment as the background and consider the selected features from post-treatment as the ones that have changed? Alternatively, would the method identify the same features as the current settings, which require healthy patients for the background? As for scenarios where the number of features exceeds 30, do all methods eventually achieve similar accuracy? From Figure 5, I noted that STG performs similarly to CFS in the Mice Protein and Epithelial Cells datasets, though it is slightly worse in the AML and ECCITE_seq datasets. Could you provide information on the accuracy when the number of features is increased to 50?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Same as above weakness. Hard to satisfy the assumption that independent features.

Response to authors

I thank the authors for responding to all my comments. My concerns have been properly addressed. I keep my score unchanged.