Response (1/2): Modeling Equivariance
Thank you for your response. We are glad to hear that you view our experimental results as convincing.
## Which Types of Equivariance can be modeled?
We apologize for the lack of clarity in our previous response. In the following, we will make two points: (1) We show that NIso can exactly model equivariance for any unitary transformations, which make up a large chunk of previously proposed equivariant ML architectures. (2) We provide detailed reasoning for our empirical results that clearly suggest that NIso can learn to be equivariant even to certain non-unitary groups.
### Equivariance to unitary transformations
NIso can learn to be equivariant to any transformation that preserves the norm of its input. Examples for this include:
Shifts on the torus, as demonstrated in Fig. 3 of the main paper.
SO(3) with spherical inputs, as demonstrated in Fig. 1 of the rebuttal PDF.
The group of 90 degree rotations mentioned by the reviewer, which we empirically validated but were not able to include in the rebuttalPDF due to space constraints (we will include this in the camera-ready paper!)
We note that prior work in equivariant ML at top conferences is often equivariant to only **a single one** of these transformations, with significant expert crafting. For instance, SO(3) is a non-abelian group and its IURs are substantially complex – nevertheless we are able to recover a basis and representation of transformations with *exactly* the same properties.
The fact that NIso learns to be equivariant to all these transformations is a significant strength - we are not aware of any previous work in symmetry discovery that has demonstrated this capacity.
We hope that this is absolutely un-ambiguous: **NIso can learn to be equivariant to a transformation that preserves a norm of its input**.
### Equivariance to non-unitary transformations
We agree that the statement “Any transformation can be approximated with a sufficiently strong encoder” was too loose. We specify it in the following.
As discussed above, NIso can closely approximate **unitary** transformations on its input.
For other more complex transformations, our experiments suggest that a sufficiently expressive autoencoder can, in fact, discover a latent space where the manifestation of the transformation is approximately unitary and even isometric. We believe this is a surprising and interesting result that is perhaps being overlooked. We note that it is not understood which transformations or groups can be embedded in this way, and point out that this is an exploration of fundamental math, beyond the scope of any ML paper.
However, a potential explanation of this phenomenon could be related to the fact that for many challenging transformations, there exists a domain in which these transformations are either unitary or isometric. For example, Mobius transformations are isomorphic to the isometries of the hyperbolic ball with their action on the sphere representing their restriction to the boundary. Similarly, camera motions in the image plane are in fact projections of the isometric action of SE(3) on the underlying 3D scene.
We hypothesize that by forcing the network to find an isometric representation for the observed transformations we are implicitly regularizing the learned latent space to discover information about this underlying geometry. We note that our ability to recover reasonable camera poses from the estimated transformations where other representation learners fail is indicative of this.
An additional illustrative example supporting this hypothesis can in fact be seen in Figure 7 of the supplement, where we show examples of failure cases in which the pose extraction from $\tau_{\Omega}$ fails catastrophically. Here we observe that these failure cases often coincide with sequences of frames where background and foreground objects are respectively far from and close to the camera – both cases where depth-estimation pipelines also often fail.
### Conclusion
All in all, NIso is a novel *first step* towards a new paradigm for symmetry discovery and equivariant ML. To the best of our knowledge, no prior work in symmetry discovery has been able to show, for instance, self-supervised discovery of approximate equivariance to a variety complex transformations, some of which do not even form a group in the observation space (such as the action of camera motions on the image plane).
Using the additional page of the paper, we will add above discussion to the “Methods” section of the paper - we agree with the reviewer that this will make the paper stronger!