Summary
This work addresses the challenge deep neural networks face with systematic generalization. Systematic generalization refers to the ability to apply learned concepts to new, unobserved combinations. The authors draw on a cognitive science theory known as "iterated learning", a hypothesized process explaining how human language developed this representation ability. The theory is based on simultaneous pressures towards compressibility and expressivity when transmitting messages. The researchers propose using iterated learning, in combination with deep network models that incorporate simplicial embeddings, to generate nearly discrete messages, thereby improving systematic generalization. The effectiveness of this approach is demonstrated in vision tasks with known latent factors and in molecular graph prediction tasks with unknown latent structures, where it outperforms other methods.
Strengths
1. The paper is well-written. The motivation is clear and the problems are stated clearly.
2. The problem addressed in this work is important, and could be particularly interesting to many communities such as causal learning or domain generalization.
3. The empirical performance looks promising, where the gap between the baseline and the proposed approach is non-negligible.
Weaknesses
1. The method section is not described clearly. In particular, in section 3.1, the authors did not explain why IL can be used to improve the performance. Similarly, in section 3.2, I cannot understand the motivation of adopting SEM. Most of the details are deferred to Appendix B, however, it is still important to briefly explain why IL and SEM can lead to the desired results.
2. The main experiments are done with synthetic dataset such as 3dShapes or dSprites dataset. It would be interesting to see how the proposed problem can address the distribution shift problem in domain generalization benchmark such as DomainBed.
3. The model attempts to learn the minimal feature to perform prediction. This could fail when there is “short-cut” presenting in the dataset. In particular, there might be “easy feature” that is highly correlated with the true target feature, where the model could learn to use those easy features instead in the proposed setting.
4. The main theoretical results in Appendix B.3 are adopted from previous work [56], where they consider a distillation setting.
5. The main motivation is based on compressibility and expressivity. The same concept is also proposed in various previous works such as Variational Information Bottleneck (VIM), supervised contrastive learning, invariant risk minimization (IRM), or works in causal representation learning. Nevertheless, there is no discussion or comparison to previous works.
Questions
1. There are three analysis provided in Appendix B to explain IL. It is a bit confusing why we need so many explanation here. In particular, it is hard to related different explanation, e.g., connecting the Bayesian analysis with the KRR analysis.
2. How do we guarantee the predictor g is a good classifier in IL? Do we also train g in the IL procedure?
3. Why can’t IL alone improve the performance? According to Appendix B, it seems that IL alone can address the proposed problem in representation learning.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
There is no obvious potential negative social impact of this work.