Summary
The authors propose the use of the tensor product to model the interactions between object properties and their values, in contrast to the usual concatenation-based fusion for compositional representations. Extensive experiments on a large variety of image datasets are performed, where performance gains are often significantly higher than those of the baselines considered.
Strengths
- Modeling “multiplicative interactions” has proven to be a powerful mechanism for deep learning [1], but is largely neglected in modern times. This paper proposes an interesting way of fusing compositional representations of object properties and their values through the tensor product (and subsequent summation). Beyond the TPR’s success in this paper, I imagine the insights here through the proposed model form could help the design of more expressive deep learning architectures in future work more generally.
- The experiments are exhaustive, the technical exposition clear and crisp, and for the most part, every architectural design and decision is justified in great detail. Overall, the paper is remarkably digestible given the technical sophistication, yet offers many insights.
---
- [1]: Jayakumar, Siddhant M. et al. “Multiplicative Interactions and Where to Find Them.” *International Conference on Learning Representations* (2020).
Weaknesses
# [W1] Multiplicatively large dimensionality of the TPR representations & issues scaling to larger settings
My main criticism of the paper is the resultant TPR's dimensionality. In particular, the soft TPR representations live in a $D_f\cdot D_r$ dimensional space. Due to the use of the Kronecker product, the dimension of the TPR representations grows multiplicatively with the two terms.
Whilst the datasets studied in the paper are relatively simple and only 10 factors of variation are modeled, it seems prudent to acknowledge that the TPR size could grow prohibitively large for increased values of $D_f, D_r$ for more complex datasets outside of controlled settings, where significantly larger number of FoVs exist. In particular, the NeurIPS checklist states: `The authors should discuss the computational efficiency of the proposed algorithms`.
For example, I see from [L1008] that $N_R:=10$, and $N_F$ is as high as $106$ for Cars3D. Even in this regime with a very small number of roles, the TPR representations are (presumably?) larger than the concatenation-based representations of the baselines. A comparison of FLOPs/multiply-adds needed for the compared methods would be appreciated to better understand the methods’ drawbacks through the use of the (often computationally expensive) tensor product.
Questions
**[Q1]** As expanded upon in [L933], the “role” embedding matrix $M_{\xi_R}$ is initialized with $n$ orthogonal columns but not trained. We are told that it is “intuitive” that role embeddings do not need to convey semantic information, but I think it would be interesting to see an ablation study on this.
In particular: if it is the case that the embedding matrix does not need to learn semantic representations, why not make the simpler choice of initializing $M_{\xi_R}$ as a (truncated) identity matrix (this shares the same property of semi-orthogonality)? The decision not to make this parameter learnable is peculiar, and I would be interested in seeing its impact explored experimentally (for example, through the DCI score as in Table. 7).
Limitations
Limitations are discussed well throughout (and in further detail throughout the appendix), but an extra discussion of the potential drawbacks of increased computational costs should be made.