Leveraging sparse and shared feature activations for disentangled representation learning

Summary

The paper proposes a novel regularization technique for disentangled representation learning. This regularizer is based on two principles formalized in the paper: 1) sparsity and 2) minimality. It assumes access to multiple tasks during training where each task uses only a subset of features, and different tasks share some of the features. The paper also provides theoretical results showing that these two principles are enough for the identification of disentangled factors. The proposed approach is validated, showing the significance of both regularization terms. Empirical results show the benefits of the proposed feature learning technique over standard ERM training.

Strengths

- The proposed two formalized principles (sparse sufficiency and minimality) and their sufficiency for disentangled representation learning are valuable contributions to the field. - The empirical study assesses the quality of the learned representations beyond disentanglement, showing the practical benefits of the proposed approach. - The overall idea and motivation are sound.

Weaknesses

1. The paper lacks clarity; certain details and symbols' introductions are missing for better understanding (more in the question section). The writing can be improved. - For example, proposition 2.1 does not clearly state what S' and i are. Also, the symbol $f_t^*$ is not introduced. The proposition also misses subsequent discussion for better understanding. - The text has a number of mistakes. For example, L50: "a**n** entropy", L57: "co~~h~~exist", L77: "give**n**", L88: the formula has an extra parenthesis, L110: subscript $t$ is missing for $y$s. 2. The empirical evaluation is insufficient and lacks comparisons with other disentangled representation learning methods as well as few-shot/meta-learning methods. - Tab. 1 lacks other repr. learning methods. - Tab. 2 and 4 lack other few-shot/meta-learning methods. The main paper refers to Appendix for the latter mentioning a 10% improvement on MiniImageNet. However, Tab.15 in Appendix does not support the claim, with MetaOptNet being the best. - Fig. 5 lacks other domain adaptation methods. 3. It is not entirely clear how the sharing regularizer in (3) corresponds to the minimality principle introduced in Sec. 2.3. Also, it is unclear how exactly it favors clustering. Further clarifications are needed. (see questions for a more concrete question)

Questions

1. How the DCI metric is defined exactly? The paper refers to [22], which has three separate metrics. Is DCI a combination of those? If yes, how are they combined? 2. The Waterbirds dataset has only two classes; how is task distribution defined? 3. Is regularization used or not when learning a novel test task? If yes, where do multiple tasks need for it come from? 4. For the experiments on DomainBed, it is stated that pretrained RN50 is used. How the proposed method influences the features learned in this case? Do you finetune it? 5. In the definition of the two principles, what is the difference between $f_t^*/f^*/f^*_{S_t}$? They are not properly introduced. 6. See 3 from weaknesses. - How does the sharing term "forces features to cluster across tasks and therefore be reused by different tasks, when useful" (L92-93)? For example, consider 3 FoVs $f_i$ and 3 binary classification tasks $t_i$ (with logits for simplification, i.e., M=1) corresponding to each combination of only 2 features with weights +1/-1, i.e., $y = \sigma(f_{S_t[1]} - f_{S_t[2]})$, $S_t \in \{(1, 2), (1, 3), (2, 3)\}$ In this case, $\tilde{\phi}_m = 2/6 = 1/3$ (removing the extra $1/TC$ term, see minor comments), resulting in a uniform distribution that results in high entropy (maximum possible). On the other hand, the sparsity and minimality are satisfied. It seems that the distribution should be over $t,c$ instead of $m$. - How does it correspond to the minimality principle introduced in Sec 2.3? The latter seems to suggest that no (smaller? see 9 below) subset of features can solve the task. Why having an L1 loss is not enough to satisfy this? If using fewer features is possible, the corresponding weights will be set to 0 following the L1 regularizer. 7. While the overall idea is sound, it is unclear why the meta-learning framework is necessary. Can the same technique be applied to the standard multi-task setting where different linear heads are learned simultaneously with a shared feature extractor? Minor: 8. Why scaling by $1/TC$ in L88? $\tilde{\phi}_m$ is not a distribution in this case. 9. in L129 do you mean $S' \subset S_t$? Otherwise can include $S_t$ and be sufficient for classification.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Limitations are well-addressed.

Summary

The paper studies the problem of learning disentangled representations in a setting with access to multiple supervised tasks, each of which is assumed to depend only on a subset of the underlying factors of variation. While the factors of variation are never directly observed, two principles are incorporated to allow their identifiability; i) sparse sufficiency (“sparsity”) ii) minimality - encouraging features to be shared / reused across tasks. The proposed approach is demonstrated to achieve a higher degree of disentanglement on synthetic data compared to previous methods, as well as better out-of-distribution generalization in real image domains.

Strengths

1. The idea of studying disentanglement in the context of generalization to downstream tasks is well motivated, especially in real image domains. 2. While the “sparsity” regularization already introduced for disentanglement in previous works, including the concurrent work of [47], the second “minimality” term seems novel. Including this term is well justified for real world problems when the number of FoV is unknown, and the results in the paper highlight its effectiveness. 3. The experimental section contains both quantitative disentanglement evaluation on synthetic datasets, as well as appropriate assessment for domain generalization in real image data, where the ground truth for the factors of variation is unknown.

Weaknesses

1. The idea of multitask meta-learning still looks a bit “over” complicated to me. I think the paper would benefit from including the following simple baseline for comparison: combining all the supervision from all tasks into a single multilabel classification task. The feature layer before the classification head should still include the sparsity term. Can the authors elaborate on the ERM baseline? Is this one the “simple” baseline I just mentioned? 2. The dataset considered in [47] is 3Dshapes. However, I do not see a comparison on this benchmark against [47]. Could the authors report the DCI evaluation of the proposed method and [47] on 3Dshapes?

Questions

1. There are several generative approaches to learning disentangled representations which are worth mentioning e.g. [1, 2]. However, these methods rely on weak supervision e.g. observing some of the factors of variation for some of the images. Given the binary labels constructed in this paper for some of the dataset, I wonder if there is a way to “quantify” the amount of supervision provided to the proposed model. This is not trivial as no direct labels are used but as long as many of “binary” labels are constructed the effective level of supervision is increased. Could the authors elaborate on this perspective? [1] Locatello et al. “Disentangling factors of variations using few labels”. In ICLR, 2020. [2] Gabbay et al. “An Image is Worth More Than a Thousand Words: Towards Disentanglement in the Wild”. In NeurIPS 2021. 2. DCI metric is a combination of three different metrics: Disentanglement, Completeness and Informativeness. However, the results in this paper are presented as a single number. Is this the average of the three? or just the Disentanglement? I request the authors to present the metrics separately in the quantitative evaluation as they may shed more light on the performance of each of the considered methods. 3. Minor typos: * Line 45 - w.r.t to * Line 57 - “cohexist” → “coexist”

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have adequately addressed the limitations and potential negative societal impact of their work.

Summary

The foundation of the proposed work is a form of regularization motivated by the problem of disentanglement. A supervised task is deconstructed into randomly generated sub-tasks, and the same trained representations are used for all sub-tasks (followed by a linear head specific to each sub-task). Two regularizers are then introduced on the feature activations: one which encourages sparsity (L1) and one which encourages minimality by penalizing the entropy of the distribution of feature activations. The disentanglement motivation is such: If we assume each sub-task only requires information from a subset of some ground truth disentangled factors of variation of the data, it is argued that the training procedure and regularization will eventually disentangle the factors of variation completely. Disentangled factors confer various advantages in representation learning, one of which being increased robustness to shifts in the data distribution because spurious factors are able to be separated from the rest. This advantage forms a large part of the proposed work. After establishing the intended effect of the regularization on synthetic data where generative factors are known, the focus shifts to real data where domain shift is a challenge.

Strengths

The method is sensible while being nontrivial. It is presented well, and the experimental results, tackling either idealized disentanglement or practical domain shift, cover a lot of ground. It is interesting to leverage the ability to decompose label information as a way to condition representations and encourage structural properties of extracted factors of variation.

Weaknesses

There are two perspectives with which to view the introduced regularization and training procedure. The first is in terms of what it accomplishes with respect to disentanglement in synthetic scenarios, and the second in terms of its utility to address domain shift in real datasets (disregarding disentanglement as unrealistic). The support for both perspectives could be strengthened. From the disentanglement perspective, it’s not clear how much everything depends on Y containing full information about the generative factors. If Y depends on fewer factors than X (as is more often the case), how can Proposition 2.1 hold? For example, if the original labels (before any task deconstruction) have no “angle” information from DSprites, no sub-task should extract angle information, and even with unlimited samples from p(x,y), the “angle” generative factor will be ignored as spurious. If Y depends on the factors in an entangled way, e.g. I(xpos;Y)=I(ypos;Y)=0, but I(xpos,ypos;Y)>0, then there is no way to create sub-tasks out of Y that depend on xpos but not ypos. Even with unlimited samples there should be no disentanglement of the xpos and ypos factors. Thus it seems that the disentanglement is predicated on having I(Y;Z)=H(Z), ie all factor information in Y; if true, this is quite a severe constraint. More clarity around this point would be helpful, either through a constructed example on the synthetic dataset or elaboration about what’s expected of Y when discussing Proposition 2.1. From the real world, pragmatism perspective, the domain shift results are numerous yet somewhat insubstantial. The comparisons in the main text are predominantly to ERM, which does not seem particularly convincing. In the research cited to support using ERM as a baseline (Gulrajani and Lopez-Paz 2020), the competitiveness seemed to only come from re-tuning for ERM anew everything pertaining to training and model selection -- but I gather all conditions match for the ERM as the proposed method. Given two methods, identical except that one has two additional hyperparameters to tune, the one with additional hyperparameters should prevail. There are many more domain shift results in the Supp, though they are thrown together without much in the way of a unifying perspective. A significant number of baselines are compared to in App. D4,5, yet the results are presented without exposition. Curiously, the proposed method is a significant outlier, performing several standard deviations above all fourteen baselines, on a single category of each experiment of D4 (S in CLSV, C in PACS, C in ACPR), yet the performance on other categories is always middling or even the worst. Can the authors speak to this?

Questions

Can the authors please elaborate on the sub-task generation process for the real datasets? Take Camelyon17 as an example -- how were sub-tasks generated out of this binary classification problem?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Summary

The paper proposes to learn disentangled representations by leveraging knowledge from a diverse set of supervised task objectives. Specifically, the paper shows that disentangled representations can emerge without ground-truth annotations of the factors of variation under two assumptions: 1) sparse sufficiency, and 2) minimality. Practically, the paper implements a meta-learning objective for representation learning with the sparse sufficiency and minimality constraints. The authors also evaluate the learnt representations for out-of-distribution (OOD) generalization.

Strengths

- The paper is very well written and easy to follow. - The paper appropriately situates the proposed method among all the related works, including highly related concurrent work. - The paper has multiple significant contributions that would benefit the research community — it presents the theory as well shows experimentally that appropriate inductive biases can sufficiently recover the factors of variation from the data. - The disentanglement results are promising. The paper contains ablation studies to show the importance of individual components such as the feature sharing property. - The proposed method is evaluated for domain generalization and domain shift tasks on six different benchmarks, achieving convincing results.

Weaknesses

In my opinion, the paper does not have any major weaknesses. To me, barring a few grammatical mistakes, it already reads better than a lot of accepted papers. The only thing I could suggest is to make the caption of Figure 1 a bit more descriptive so that it conveys a full picture by itself and doesn’t rely on the main text.

Questions

I think the paper is already in a good and sufficient state to be accepted as it is. I do not have any further questions for the authors.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors have adequately addressed the limitations of the proposed method.

Reviewer's Response to Rebuttal

Thank you for the clarifications. Regarding the comparison with [47] on 3Dshapes, could you please report all DCI components separately as well? (Minor comment: Note that in the PDF rebuttal page the results in Table B are flipped between w/ and w/o regularization)