Equivariant deep learning architectures exploit symmetries in learning problems to improve the sample efficiency of neural-network-based models and their ability to generalise. However, when modelling real-world data, learning problems are often not exactly equivariant, but only approximately. For example, when estimating the global temperature field from weather station observations, local topographical features like mountains break translation equivariance. In these scenarios, it is desirable to construct architectures that can flexibly depart from exact equivariance in a data-driven way. Current approaches to achieving this cannot usually be applied out-of-the-box to any architecture and symmetry group. In this paper, we develop a general approach to achieving this using existing equivariant architectures. Our approach is agnostic to both the choice of symmetry group and model architecture, making it widely applicable. We consider the use of approximately equivariant architectures in neural processes (NPs), a popular family of meta-learning models. We demonstrate the effectiveness of our approach on a number of synthetic and real-world regression experiments, showing that approximately equivariant NP models can outperform both their non-equivariant and strictly equivariant counterparts.
Paper
References (45)
Scroll for more · 33 remaining
Similar papers
Peer review
Summary
This paper implemented the approximately equivariant neural process (NP) by relaxing the equivariance of the NP decoder and the relaxation is conducted by adding several leanable parameters as fixed inputs fed with the data embeddings into the decoder.
Strengths
The proposed method is applicable regardless of architecture and easy to be implemented because it is simply adding some parameters as inputs.
Weaknesses
1. No supplemental material for a source code. 1. A concept figure will be helpful to understand the intuition of the paper. 1. Absent analysis for equivariance error of the trained models. Did the trained model really reflect the approximate equivariance inherited in the dataset? 1. The paper reported only the log-likelihood of (maybe) the target data, which may be overfitted by the target data and underfitted by the context data. It is necessary to report the log-likelihoods with respect to both context and target separately to show the strength of the suggested method accurately. 1. Analysis for the number of the fixed inputs is necessary in order to prove that the number of the fixed input really controls the magnitude of the approximate equivariance. 1. All included experiments are only regression tasks. Since NP is essentially a meta-learner, it would be better to add experiments beyond regression to demonstrate its generalization performance across various types of tasks.
Questions
1. Encoder should be equivariant? If so, doesn’t it lose non-equivariant information from the dataset D? 1. How to add the fixed inputs into the dataset embedding in ConvCNP? If we just sum all, the number of the fixed inputs is obviously meaningless. 1. When the decoder is ViT, how many model parameters are additionally necessary compared to the strictly equivariant case? 1. It seems that the suggested relaxation is applicable to not only NP. Why did the paper choose NP to prove the relaxation idea?
Rating
5
Confidence
4
Soundness
2
Presentation
1
Contribution
2
Limitations
There is no limitation of the contribution itself but experimental limitations are mentioned in the weakness.
Rebuttal
Thank you so much for the detailed clarification. I understood much of it, but I still have some concerns regarding the log-likelihood and the equivariance error. **The log-likelihood of reconstruction for the context data is important to evaluate the practical performance of neural processes.** For example, we often set the variance of the output as $0.1+0.9\cdot\text{Softplus}(\sigma)$ where $\sigma$ is the decoder output for variance. However, the Transformer Neural Process (TNP) sets it as $\exp(\sigma)$, which differed from its baselines. This setting results in high variance that can cover lots of the target data but remains high even for the context data, leading to significant underfitting in the context data. Indeed, TNP shows a log-likelihood in context reconstruction that is 10 times worse than its baselines. A good likelihood only in the target data may still seem favorable because its variance easily covers the true target data, but it simply maintain high uncertainty for all data, which is not practical as a meta-learner. For these reasons, many previous works also report the log-likelihood or error of context data reconstruction. Please refer to Figure 3 in ANP [1], Table 1 in BNP [2], and Table 1 in MPNP [3]. If you agree with my opinion, I hope you consider including the log-likelihood or error of context data reconstruction, which is not difficult to measure. [1] Kim, H. et al. (2019), Attentive Neural Processes. [2] Lee, J. et al. (2020), Bootstrapping Neural Processes. [3] Lee, H. et al. (2023), Martingale Posterior Neural Processes. **Equivariance error analysis, either in theory or experiment, is necessary.** What I mean by equivariance error analysis is, for example, adopting a simple function like $y=\text{CosineSimilarity}(\boldsymbol{x}\_1,\boldsymbol{x}\_2)$ (rotation-invariant), then weakly breaking the symmetry, such as by using $y = \text{CosineSimilarity}(\boldsymbol{x}\_1, \boldsymbol{x}\_2+0.01\cdot\boldsymbol{1})$, and generating a synthetic dataset. Note that since we already know the true function and the actual equivariance error, we can compare the equivariance error learned from the synthetic dataset with the true error. The paper theoretically showed that adding fixed learnable inputs breaks the equivariance but it does not showed that how much it breaks and whether it can indeed utilize the inductive bias of the weakly broken symmetry or not. If you show such analysis, you can say that the reason why your method works well is not because the method fully break the inductive bias and just empower the expressivity of the neural network. Please refer to Figure 3 in RPP [1] and Figure 3 in PER [2]. [1] Finzi, M. et al. (2021), Residual Pathway Priors for Soft Equivariance Constraints. [2] Kim, H. et al. (2023), Regularizing Towards Soft Equivariance Under Mixed Symmetries.
Answer Part 1
Thank you for taking the time to go through our rebuttal. We address your remaining concerns below. **Log-likelihood of context data.** The practical importance of neural processes is determined by how they are used in practice, after training. In the usual framework, after training, there will be an unseen context set $D^*_c$ and an unseen target set $D^*_t$, and practical utility of the neural process is determined by how well the model predicts $D^*_t$ given $D^*_c$, usually measured in terms of log-likelihood. The appropriate way of measuring this performance is to (a) hold out some context–target pairs from training; and (b), after training, to compute the log-likelihood of the held-out target sets given the context sets. In this framework, the probabilities $p(D_c | D_c)$ can be reasonably used as a diagnostic, but these probabilities do not necessarily correlate with good performance on the task of interest (predicting $D^*_t$ given $D^*_c$). For example, consider a NP perfectly trained on data sampled from a GP with a Matern kernel, and suppose $(D^*_c, D^*_t)$ is now sampled from a GP with a periodic kernel. Then the NP will achieve near-optimal $p(D^*_c | D^*_c)$, because the NP can reconstruct data it is conditioned on; but the NP will achieve disastrous $p(D^*_t | D^*_c)$, because the NP has never interpolated periodic data before. It is often the case that the sampling distributions of the context and target data are the same. In that case, a model that maximises $\log p(D_t | D_c)$ will be guaranteed to reconstruct the context data well, because a context set is like a target set. **“A good likelihood only in the target data may still seem favorable because its variance easily covers the true target data”** Because probability densities integrate to one, the likelihood will penalise both too high and too low variance. In fact, the optimal likelihood is achieved if and only if the variance is equal to the true variance and the mean equal to the true mean. The likelihood therefore cannot be “cheated” by artificially increasing the variance. To make this mathematically rigorous, see e.g. Prop 1 by Foong et al., 2020. In summary, the right way to measure practical performance is by computing log-likelihoods of held-out target data given context data, and this metric cannot be “cheated” by artificially increasing the variance. Nevertheless, we agree that the log-likelihood for reconstructing the context set can be used as a diagnostic metric. If you insist that this diagnostic metric is important, then we are willing to include it as an additional result in the supplementary material. Finally, as a clarification regarding implementation - **“(...) we often set the variance of the output as 0.1+0.9⋅Softplus($\sigma$), where $\sigma$ is the decoder output for variance. However, the Transformer Neural Process (TNP) sets it as $\exp(\sigma)$”** Our implementation of the TNP parameterises the standard deviation of the normal distribution as $\sigma_{min} + \text{Softplus}(z)^{1/2}$, where $z$ is the output of the decoder for the variance and $\sigma_{min}$ a hyperparameter usually set at 0.1. This is mentioned in the Appendix lines 700-701: “The decoder parameterises the mean and pre-softplus variance [correction: standard deviation] of a Gaussian likelihood with heterogeneous noise.” **References** Foong, Andrew Y. K. et al. Meta-Learning Stationary Stochastic Process Prediction with Convolutional Neural Processes. NeurIPS 2020.
Answer Part 2
**Equivariance error analysis.** **“(...) it does not showed that how much it breaks and whether it can indeed utilize the inductive bias of the weakly broken symmetry or not.”** In the experiments, we compare the approximately equivariant model to the strictly equivariant model obtained by setting the additional inputs to zero. This reveals exactly what the effect of the additional inputs is and in what way they break equivariance. In particular, the bottom row of Figure 1 shows predictions for both the approximately equivariant model and the equivariant model obtained by setting the additional inputs to zero. Note that the equivariant model unnecessarily inflates the uncertainty, because translation equivariance prevents the model from recognising a fixed $x$-location, so it cannot learn that a change-point always happens at $x=0$. The same comparison is done in Figure 2 for the environmental data experiment, (d) versus (e) and (g) versus (h). In the same figure, (f) and (i) show how much the predictions change as a consequence of the additional inputs. This difference exactly correlates with the missing information, the orography in (c). We believe that these comparisons show that the additional inputs break equivariance to an appropriate extent. For example, in the bottom row of Figure 1, note that the approximately equivariant and equivariant predictions are very similar before the length-scale change. Including the additional inputs only changes the predictions after the length-scale change. **“If you show such analysis, you can say that the reason why your method works well is not because the method fully break the inductive bias and just empower the expressivity of the neural network.”** The additional basis functions do strictly increase the expressivity of the model, exactly because the basis functions break equivariance of the model. That is, if the basis functions are zero, then the model is fully equivariant; and if the basis functions are non-zero, then the model is not equivariant. Because everything is continuous, the transition between these two “modes” is smooth and depends on the magnitude of the basis functions, which the model automatically learns. The main practical question that remains is to whether this indeed corresponds to useful, practical, and appropriate improvements. We believe that is clearly shown in Figures 1 and 2. We agree that it would be very interesting to more rigorously quantify the degree to which equivariance is broken with a metric like in RPP. However, for the current submission, we believe to have demonstrated that additional inputs (a) have a principled theoretical motivation, (b) break equivariance in interpretable and appropriate ways (as argued above), and (c) give considerable performance improvements on real-world data (experimental results). Although such an analysis would be interesting, we believe to have shown that additional inputs are a practical method that works in the intended way and have therefore left it for future work. Nevertheless, if you insist that a more rigorous quantification is essential, then we would be willing to compute a metric very similar to EquivError from RPP for Figures 1 and Figure 2. For example, for Figure 2, this would divide the norm for panel (f) by the average of the norms of panels (d) and (e). By the colour bars, we expect this value to be around 2%. We would include these numbers in the captions.
Thank you for providing more information, but most of it seems to carry the same nuance as the first rebuttal. **Context Log-Likelihood** As you mentioned, the context log-likelihood is not a direct performance metric. However, high uncertainty in the reconstruction of context indicates merely inferring the whole sequence from a pattern in the context, which is not a stochastic process. Instead, it is more like regression using a neural network that takes the context as input and produces the target as output. For example, in a task like prediction from irregularly sampled time series, the reason we use a stochastic process rather than other frameworks is to analyze the statistics of the time series, including both the context and target data, especially since the context itself may contain observational noise. If the goal were simply to build a model that reconstructs the target well, I would prefer using imputation models over neural processes. Target performance is not easily manipulated by variance, but it can be artificially enhanced by sacrificing context performance, as seen in TNP. **Equivariance Error** I understand that you showed the approximately equivariant model outperformed the equivariant model (with added inputs set to zero). However, my point is more about comparing the fully non-equivariant model with the approximately equivariant model. The approximately equivariant model should outperform the fully non-equivariant model because it leverages the inductive bias. While you did make this comparison, it does not guarantee that your model is indeed an approximately **equivariant** model, as you did not demonstrate that it truly reflects the actual approximate equivariance. I think your paper focused more on how to create an approximately **equivariant** model, rather than just breaking equivariance to overcome the strict constraints of equivariant architectures and add flexibility. If your paper is solely about breaking equivariance to introduce flexibility, then the experiments I suggested might not be necessary. To be honest, I do not understand why you do not report the log-likelihoods during rebuttal period, especially since you agree that that is also a diagnostic metric. In my experience, **measuring the context log-likelihood takes less than 10 minutes**. I think the paper should be reviewed after including the context log-likelihoods and the equivariance errors.
No comment notification
We have not received any notification after posting the previous two comments. With this comment, we just wanted to make sure that the reviewer gets notified about the new comments.
Here are the additional results for the environmental data. **Context log-likelihoods environmental data** Table 1: Average context loglikelihoods for the 2-D environmental regression experiment. Results are grouped together by model class. _________________________________ Model | Europe | US __________________ PT-TNP | 1.74 $\pm$ 0.01 | - | PT-TNP ($T$) | 1.66 $\pm$ 0.01 | 1.28 $\pm$ 0.01 PT-TNP ($\widetilde{T}$) | 1.76 $\pm$ 0.01 | 1.47 $\pm$ 0.01 ______________________________________ ConvCNP ($T$) | 1.20 $\pm$ 0.02 | 0.34 $\pm$ 0.02 ConvCNP ($\widetilde{T}$) | 1.50 $\pm$ 0.01 | 0.97 $\pm$ 0.01 RelaxedConvCNP ($\widetilde{T}$) | 1.29 $\pm$ 0.01 | 0.86 $\pm$ 0.01 __________________ EquivCNP ($E$) | 2.03 $\pm$ 0.01 | 1.76 $\pm$ 0.01 EquivCNP ($\widetilde{E}$) | 2.05 $\pm$ 0.01 | 1.69 $\pm$ 0.01 -------------------- Similar to the additional results for the synthetic 1-D regression experiments, these results show that all models achieve higher log-likelihood on the context data, demonstrating that our models are not “underfitting” to the context data. **Equivariance error environmental data** Table 2: Equivariance error on the 2-D environmental regression experiment. Note that all equivariance errors for the US are zero, as the additional fixed inputs are set to 0. _________________ Model | Europe _____________________ PT-TNP ($\widetilde{T}$) | 0.0406 $\pm$ 0.0005 ConvCNP ($\widetilde{T}$) | 0.0237 $\pm$ 0.0004 RelaxedConvCNP ($T$) | 0.0239 $\pm$ 0.0004 EquivCNP ($\widetilde{E}$) | 0.0242 $\pm$ 0.0005 _________________ These vary between 2-4%, again indicating that the model’s predictions only deviate slightly from the equivariant predictions. We hope that our previous comment, together with these additional results, have adequately addressed all of your concerns. If so, we would encourage you to reconsider your score.
Awaiting your response
Dear dBf7, This is a gentle reminder that we are still awaiting your response. Please note that the discussion period is ending in a few hours. In case it helps, we would like to clarify a technical point that we have left implicit. By showing that the approx. equiv. predictions are approximately equal to equivariant predictions, we show that `d(approx-np(D), equiv-np(D))` is small for all `D` (where `d` is some metric). Then, by the triangle inequality, we find that `d(T approx-np(D), approx-np(T D))` is small for all `T` and `D`: ``` d(T approx-np(D), approx-np(T D)) <= d(T approx-np(D), T equiv-np(D)) + d(T equiv-np(D), equiv-np(T D)) + d(equiv-np(T D)), approx-np(T D)) ``` where the first and third terms are small because `d(approx-np(D), equiv-np(D))` is small for all `D`, and the second term is zero because `equiv-np` is exactly equivariant. Therefore, the metric which we computed exactly implies equivariance in an approximate sense.
Thank you for your reply and we hope to address your remaining concerns by providing the metrics you requested. **Context Log-Likelihood** We computed the log-likelihoods for reconstructing the context data in the GP experiment, and we will provide the results on the environmental data by the end of the discussion period. Please see the results below, which we will include in the supplementary material. Table 1. Context log-likelihood for the GP experiment. Ground truth log-likelihood is $0.2806 \pm 0.0005$. | TNP | TNP ($T$) | TNP ($\widetilde{T}$) | | ------------------- | ------------------------------ | ------------------------------------ | | 0.2296 $\pm$ 0.0007 | 0.2396 $\pm$ 0.0013 | 0.2344 $\pm$ 0.0020 | | **ConvCNP ($T$)** | **ConvCNP ($\widetilde{T}$)** | **RelaxedConvCNP ($\widetilde{T}$)** | | 0.2362 $\pm$ 0.0007 | 0.2218 $\pm$ 0.0009 | 0.2381 $\pm$ 0.0008 | | **EquivCNP ($E$)** | **EquivCNP ($\widetilde{E}$)** | | | 0.2213 $\pm$ 0.0007 | 0.1992 $\pm$ 0.0010 | | | | | | Note that these log-likelihoods are much higher than target log-likelihoods reported in the main body, because the models can accurately reconstruct the context data. Moreover, they are close to the ground truth log-likelihood ($0.2806 \pm 0.0005$), indicating that underfitting is unlikely. **“However, high uncertainty in the reconstruction of context indicates merely inferring the whole sequence from a pattern in the context, which is not a stochastic process. (...) using imputation models over neural processes.”** A neural process is defined as a neural network architecture that maps a context set to a stochastic process. This stochastic process can then be evaluated at any target inputs to produce predictions for any target set. All models in our submission are of this form. Therefore, by construction, all neural process models derive their predictions from an underlying stochastic process, and this underlying stochastic process can always be queried to e.g. analyse statistics of the underlying time series. The premise of the neural process framework is that learning to predict the target sets given the context sets makes this underlying learned stochastic process converge to the “true stochastic process”. The sampling distributions of the context and target data should be set up in a way that enables this convergence in the limit of infinite data. **“Target performance is not easily manipulated by variance, but it can be artificially enhanced by sacrificing context performance, as seen in TNP.”** If all possible target sets include all possible context sets too, which is usually the case, then we would argue that this is false: in this case, worse context performance directly implies worse target performance, because context sets are target sets. A model could make a trade-off where it specifically performs worse in context set reconstruction in favour of better performance in interpolation further away, but this is unlikely to happen as the objective usually weights all target sets equally. In our experience, we have not seen this happen, unless the neural process severely underfits (which does happen for some architectures, like the original CNP based on just deep sets). In our submission, we do not believe that any of the models severely underfit in any of the experiments, which can be verified by the visualised predictions.
**Equivariance Error** While you acknowledge that we (1) show that the approximately equivariant models outperform the equivariant models (with added inputs set to zero) and (2) approximately equivariant model outperform the non-equivariant models too, we understand that your main concerns are that we (a) do not show that our model is equivariant in an approximate sense and (b) do not show that the model’s approximate equivariance reflects the “actual approximate equivariance”. We address these points in order. (a): Figures 1 and 2 show that the predictive means of the approximately equivariant model are equal to the predictive mean of the corresponding equivariant model plus a “small perturbation”. In other words, visually, the approximate equivariant predictive mean only slightly deviates from the corresponding equivariant predictive mean. Therefore, given that these figures are illustrative of the general behaviour of the models, we believe that we have clearly qualitatively shown that the predictions of the approximately equivariant models are indeed equivariant in an approximate sense. To also argue this quantitatively, we have computed equiv_deviation = $\mathbb{E}$[norm(equiv_mean - approx_equiv_mean) / norm(equiv_mean)], which quantifies exactly how much on average the predictive means deviate from the corresponding equivariant predictive mean. The results are as follows: Table 2. Equivariance error for the GP experiment. | TNP ($\widetilde{T})$ | ConvCNP ($\widetilde{T}$) | RelaxedConvCNP ($\widetilde{T}$) | EquivCNP ($\widetilde{T}$) | | --------------------- | ------------------------- | -------------------------------- | -------------------------- | | 0.0896 $\pm$ 0.0011 | 0.0823 $\pm$ 0.0005 | 0.1460 $\pm$ 0.0006 | 0.0825 $\pm$ 0.0006 | | | | | | These percentages are around 8-9% (only the RelaxedConvCNP shows 15% difference, which is based on the approach from Wang et al. [2022a], rather than our approach), which means that, on average, the models’ predictions deviate only slightly from the equivariant predictions. (b): Whether the models learn the “actual approximate equivariance” is hard to determine. For example, in the GP experiments, what would the actual approximate equivariance be? In addition, what would the actual approximate equivariance be in the climate experiments? While this is a hard question, we agree that it is an important question, which is why we attempted to answer this question in the following way: the models learn the “actual approximate equivariance” if the perturbation w.r.t. the corresponding equivariant prediction is “consistent with the structure of the problem”. In the GP experiments, we show that the perturbation exactly corresponds with the length-scale change. In the climate experiments, we show that the perturbation exactly corresponds with the key missing information that primarily breaks stationarity of the weather: orography. Therefore, we believe that we have reasonably demonstrated that the models approximate the “actual approximate equivariance”. By having provided the log-likelihoods for the reconstruction of the context data and having provided numerical evidence that the models' predictions are indeed equivariant in an approximate sense, we hope to have addressed your concerns and consequently hope that you will reconsider your score.
Summary
The paper describes a new framework for soft/approximate equivariance, based on the functional analysis of compact on hilbert spaces, assuming unitary group actions. This is then applied to equivariant neural processes.
Strengths
The proposed method is very general and conceptually well grounded, and to the extent I could verify they are also new. The experiments are well presented and seem promising in terms of accuracy (not sure about efficiency). Edit after rebuttal: the added experiments are convincing enough in terms of accuracy as well.
Weaknesses
Edit after rebuttal: I think that the method is a good addition to the literature, and my concerns 1-3 below have been adequately answered in the rebuttal and "global rebuttal" parts. ------------------- The main enigma about this paper, is: does the technique used really warrant putting "neural processes" in the paper's title? It seems like the authors develop a large setup for general approximate equivariance, and the application to neural processes is just one of many applications. What is restricting the applicability of the method, from general equivariant neural networks, to the class of neural processes? This (meaning, the emphasis on neural processes, and the paper's title) is a big distraction in reading the paper, as one tries to find a reason why the setting is restricted like that and one doesn't actually find a convincing answer. 1) The assumption of having tasks with underlying compact operators seems to be swept under the carpet, and not properly discussed. In functional analysis "compact" is roughly equivalent to "approximable by finite-rank, as pointed out at several points in the paper, however the authors don't discuss what limitation this could imply. If this is beyond the scope of this work, still I think that it needs highlighting and *at the very least* it requires pointing out very clearly that this is a future direction to be investigated. 2) when authors work with multilinear CNNs (see question 7 below), this seems not well described, and remains probably too mysterious. Also, I think that their implementation becomes clumsy and inefficient: is that so, or do the authors have a justification and complexity control of why not? This is not required to do in much detail, but maybe just to mention that in order not to hide possible underlying difficulties of the method. 3) the parameter $k_n$ from Theorem 3 is not well behaved, so it is not clear if this theorem is useful in practice.
Questions
0) see the un-numbered remark at the beginning of "weaknesses" section, about relevance to "neural processes" to the presented methods. 1) line 49-50 can the authors be more specific about what it means that "any equivariant [...] fixed inputs"? A reference to the actual result would suffice. 2) line 69-70 "represented with stochastic processes" -- is it not "represented by"? (this is a genuine question, not a rethorical one) 3) The statement of theorem 1 is not well written. If $\Phi$ has image in $C(\mathcal X,\Theta)$ and if eq. (2) holds, then $\rho$ has image in the same space, not in $\mathbb R^{2D}$. Similar incompatibilities also hold for the other domain / codomain spaces for operators $\rho, e, \psi$ in that theorem. Also, the discussion in lines 114-116 is incompatible in terms of domain/codomain spaces. This makes the theorem hard or impossible to understand. 4) Still for eq. (2), it is not clear why $\psi$ has two arguments, is it a kernel operator? the notation from (2) should be expanded and actually explained with a degree of consistency/precision.. pretty please.. 5) lines 130-133, one can just require that the action of $G$ is by linear unitary operators, which I think is equivalent to what the authors claim? Also, why do you say "acting linearly" at line 130, and then write the additivity explicitly at line 132, as if it were a new requirement and not part of the preceding one? 6) I think that before Proposition 1, or actually even in the introduction itself, one should emphasize the role of compact operators on Hilbert spaces. This is a strong requirement of this theory, and a strong limitation, so it should be highlighted. (I'll add an obvious remark: Don't worry, nobody will consider you worthless if you are fair about the limitations, on the contrary, it speaks highly of you if you do so!) 7) line 156: "$T\simeq CNN(\cdot, t_1,\dots,t_n)$" means what exactly? What is the kind of multilinear CNN's that this refers to? Any reference for them? I think that currently this kind of CNN is not well developed, and it is not that trivial as an extension of usual linear CNNs. I think this should be mentioned. If one looks at appendix C, lines 592 and following, that's just the simplest case of such CNNs, so I maybe the authors could spend some time to point this out, and to describe a little bit the difficulties and issues with such multilinear-CNN theory. 8) in the paragraph lines 205-211 I think that it should be stressed (or at the very least mentioned) that $T$ is assumed to be compact, and can't be more general than that. 9) same mention of "compact" should be inserted also in paragraph lines 249-261 10) same mention of "compact" operators should be inserted in seciton 6 (conclusions section)
Rating
8
Confidence
5
Soundness
3
Presentation
3
Contribution
4
Limitations
these are addressed in the "weaknesses" section. I think that the main limitations are in efficiency and scalability of multi-linear CNNs and in the fact that operators of interest are far from compact and not easy to approximate with finite-range ones.
No problem at all! **CNNs** So, as mentioned, we use a regular CNN which consists of multiple convolutional layers, each of which takes as input a $C_{in}$ dimensional feature map $f: \mathcal{X} \rightarrow \mathbb{R}^{C_{in}}$, where $\mathcal{X}$ denotes a discretised input domain (i.e. grid) and convolves it with a kernel $\psi: \mathcal{X} \rightarrow \mathbb{R}^{C_{out}\times C_{in}}$ : $$(f \ast \psi)(x)= \Sigma_{x'\in \mathcal{X}} f(x')\psi(x - x').$$ here, $C_{in}$ and $C_{out}$ denote the number of input and output channels, respectively, and $\ast$ denotes the convolution operation. In practice, the kernel has a finite receptive field, meaning that $\psi(x - x') = 0$ when $x - x'$ exceeds some value. After each convolutional layer there is a nonlinearity $\phi: \mathbb{R} \to \mathbb{R}$ which acts point-wise on the feature map value. We have omitted details such as strides, padding and bias, which are also used when implementing. Note that we provide precise details on our hyperparameters choices (e.g. number of layers, number of channels at each layer, receptive field) in the Appendix. When used in ConvCNPs, the input into the CNN is the output of the 'SetConv' encoder $e: \mathcal{S} \to \mathbb{H}$ which maps datasets $\mathcal{D} \in \mathcal{S}$ to feature maps $f: \mathcal{X} \to \mathbb{R}^{E} \in \mathbb{H}$ . **Additional fixed inputs** As we have described, there are many possible ways to include additional fixed inputs $t_i: \mathcal{X} \to \mathbb{R}^{E}$ . The simplest approach is to include them as additional channels into the first convolutional layer, which in practice is achieved by concatenating $f$ with the fixed inputs. The first convolutional layer is then $$(\operatorname{cat}(f, t_1, \ldots, t_B) \ast \psi)(x) = \sum_{x' \in \mathcal{X}} \operatorname{cat}(f, t_1, \ldots, t_B)(x') \psi(x - x').$$ Let $T_{\tau}f = f(\cdot - \tau)$. Note that $\operatorname{cat}(T_{\tau}f, t_1, \ldots, t_B) \neq T_{\tau}\operatorname{cat}(f, t_1, \ldots, t_B)$ , hence translation equivariance is broken. An alternative approach, which we take in the paper, is to add the additional channels to $f$: $$((f + t_1 + \ldots + t_B) \ast \psi)(x) = \sum_{x' \in \mathcal{X}} (f + t_1 + \ldots + t_B)(x') \psi(x - x').$$ Again, as $T_{\tau}f + t_1 + \ldots + t_B \neq T_{\tau}(f + t_1 + \ldots + t_B)$, translation equivariance is broken. Remarkably, two of the most prominent approaches to achieving approximate equivariance [1, 2] can be understood as alternative ways of including additional fixed inputs into a CNN to break equivariance. We discuss this in more detail in Appendix C. We believe that this marks an important contribution of our work: to first understand approximate equivariance, to unify existing methods, and to use this to develop a simple and effective approach to building approximate equivariance in *any* operator. **References** [1]: Wang, Rui, Robin Walters, and Rose Yu. "Approximately equivariant networks for imperfectly symmetric dynamics." International Conference on Machine Learning. PMLR, 2022. [2]: van der Ouderaa, Tycho, David W. Romero, and Mark van der Wilk. "Relaxing equivariance constraints with non-stationary continuous filters." Advances in Neural Information Processing Systems 35 (2022): 33818-33830.
Thanks for the clarification. Given your willingness to clarify the "compactness" part, the above clarifications, and the part of the "global rebuttal" on $k_n$ behavior, that we didn't discuss, I'm more confident to raise my score to 8.
We greatly appreciate your engagement with our work. Thank you again for taking the time to review the paper in detail—without doubt, your feedback has led to an improvement in the exposition of our research.
Summary
This work considers approximately equivariant models --- which may better model or learn real-world tasks than exactly equivariant models --- especially in the context of neural processes. A new approximately equivariant method is developed, which uses an exactly equivariant model along with fixed inputs that break equivariance. This method can be used to parameterize approximately equivariant neural processes, by modifying existing approaches in simple ways. Experiments on 1-D regression, smoke plumes, and environmental data show benefits of the approach.
Strengths
1. Nice flexible framework for approximate equivariance, which unifies existing methods. Similar ideas could be useful more broadly in learning with (approximate) symmetries. 2. The approach has nice ways of controlling the degree of equivariance, by number of inputs ("degrees of freedom" in Section 3.1), or empirically doing things like regularizing the effect of the additional inputs towards zero. The method of setting $t_b(x) = 0$ during test time is also nice. 3. Good empirical gains on several datasets, by making simple tweaks to several NP methods.
Weaknesses
1. Notation and definitions are heavy, which is somewhat understandable given the subject matter, but I do think it can be improved. For instance, compact operators are not defined. 2. From what I can tell, the smoke plumes experiment in Section 5.2 follows a similar setup to that of Wang et al. [2022a], but this is not mentioned. Also it would be good to note how you chose e.g. the parameters of the simulations and the choice to include an obstacle, or whether this was mostly arbitrary (not a big issue).
Questions
1. In equation (3), I think think the number of inputs should be larger than $n$, right? 2. Perhaps I missed this, but do you initialize the models to be equivariant? Via setting $t$ to be the zero function? 3. It would be worth noting connections of your approximate equivariance approach to some related prior work. You mention the connection to positional embeddings in Transformers on Page 5, but this also applies to other domains such as Vision Transformers [Dosovitskiy et al. 2021]; this is more specifically investigated in the context of approximate equivariance by Lim et al. 2023. Also, the approach of taking additional inputs is related to the symmetry breaking approach of Xie et al. 2024 and the probabilistically / approximately equivariant approach of Kim et al. 2023. References: * Wang et al. 2022a. Approximately Equivariant Networks for Imperfectly Symmetric Dynamics * Dosovitskiy et al. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale * Lim et al. 2023. Positional Encodings as Group Representations: A Unified Framework * Xie et al. 2024. Equivariant Symmetry Breaking Sets * Kim et al. 2023. Learning Probabilistic Symmetrization for Architecture Agnostic Equivariance
Rating
6
Confidence
2
Soundness
3
Presentation
3
Contribution
2
Limitations
Some discussion in conclusion
Summary
The work provides an alternative approach to obtain a loose equivariance constraint. The authors established an interesting relationship between equivariant and non-equivariant models, showing that equivariant models with enough fixed input can approximate any non-equivariant model. Subsequently, the authors suggest using additional fixed input in neural processes, which is a common practice in transformers and NeRFs. The proposed method achieved better performance on multiple benchmark datasets.
Strengths
1. The work provided a nice theoretical result that offers an interesting reinterpretation of existing practices. 2. The proposed technique is empirically evaluated on a wide range of datasets.
Weaknesses
The work provides a nicer reinterpretation of the existing practice of adding fixed/learnable positional embedding. Despite this reinterpretation, the work does not quantify the degree to which these additional fixed inputs hurt the equivariance property of the equivariant models. Nor does it provide a scheme to automatically learn the number of fixed inputs for controlling the equivariance restriction. Thus, the work indeed proposes an approximately equivariant model, however, with unknown approximation. This severely limits the contribution of the work. Line 91: different notation should be used to denote the group action on $X$ and $Y$ Line 102: the symbol $f$ is already used in line 87 to denote the group element. The use of different symbols will fascinate reading. Line 115: the notation of the encoder matches with a notation of the identity element of the group.
Questions
Line 103: how is the action of the group on the dataset $D$ defined? Equation 2, line 111: How is the G-invariance of the kernel defined? Line 143: $E_n$ is defined to take $n$ additional input; however, line 146 is defined to map on $H^{1+2n}$, which I think means $2n$ additional input. An explanation would be appreciated.
Rating
5
Confidence
3
Soundness
3
Presentation
2
Contribution
2
Limitations
yes
We thank the authors for their rebuttal. The clarifications are useful for me. On the Wang et al. [2022a] point, thanks for clarifying the differences. My point is that, given that you take inspiration from and use the same tools as the Wang et al. setup, you should note and cite this in the paper. Everything else looks good to me, I maintain my score.
Many thanks for your reply. You raise an important point that Wang et al. should be cited when detailing the set up for this experiment---we shall modify the paper accordingly. Thank you again for your feedback, and please don't hesitate to ask any remaining questions should they arise.
about the compactness + need more details for question 7
Thank you for the reply, I'm slowly going through the things you wrote back, so this may not be my only comment (sorry). About compactness, if you write something like the reply you put, I'll be satisfied. I was worried only about the "sweeping under the rug" nothing else. One part that i still do not understand is what are multilinear CNNs. a) is there a reference for multilinear CNNs in the literature? b) can you give a formula/pseudocode/details for a special case that is a multilinear CNN but not a standard CNN please? Because I'm not fully sure that I understand.
We greatly appreciate you taking the time to go through our rebuttal. Please don't hesitate to leave additional comments/questions. We are more than happy to answer them. **Compactness**: we will update the paper with the above discussion on compactness. **Multilinear CNNs**: With regards to multilinear CNNs: it seems as though there's potentially some communication error here. To be clear, we do not use, nor propose the use of, multilinear CNNs---we use regular (nonlinear) CNNs throughout. Admittedly, we hadn't heard of a "multilinear CNN" before it was mentioned in your review, and were hoping that you would be able to provide a reference. Our rebuttal assumed you implied the use of a single-layer CNN without any nonlinearities (which would indeed make it a multiple input CNN with linear operations on the input). The only reference we can find in the literature ([1]) takes this approach. [1] Pinson, Hannah, Joeri Lenaerts, and Vincent Ginis. "Linear CNNs discover the statistical structure of the dataset using only the most dominant frequencies." International Conference on Machine Learning. PMLR, 2023.
Ok sorry, I had forgotten that I was the one to use the term "multilinear" in the first place. I was assuming that the CNN was linear in $t_1,t_2,\dots$, but actually you never said it was. But anyway, can you give some details on the architecture of the CNN abbreviated at line 156 of the paper?
Response to the Authors
Thanks for the response. I want to maintain my score.
Decision
Accept (poster)