Summary
The paper presents a novel INR framework that leverages the Mixture of Experts (MoE) technique. The proposed strategy consists of an expert and a manager branch. Each branch has an encoder that processes the input coordinate and extracts an embedding. By processing the two encoder embeddings, the manager predicts the probability of which of the N experts should be used for extracting the signal. They show how the proposed INR framework achieves better reconstructions than SIREN on several modalities, such as audio, image, or 3D surfaces. They also propose a manager pre-training strategy, which is necessary to exploit all the experts effectively.
Strengths
- The original idea might be conducive to new research in this direction.
- The paper is well-written and easy to follow.
- Supervising experts with semantic losses and obtaining networks specialized in specific semantic areas of the input signal might unlock several applications for INRs and ease their interpretability.
- The proposed framework achieves good reconstruction performance.
- The ablations in Table 4 and Table 5 are very insightful.
Weaknesses
W1) The major weakness of the paper is the misalignment between the experimental results and the motivations of this research:
a- In the introduction (L26-28), the authors correctly point out that, in traditional INRs, each coordinate needs to be processed by the whole network. Even though they claim that this problem can be solved by MoE INRs, in the proposed architecture, the input coordinate needs to be processed by the full manager and the encoder of the expert branch. The only saved computation is the one from the final expert, a much smaller network than the others. Thus, the saved computation looks minimal to standard SIREN (considering the same total weights). Moreover, no experiments regarding computational efficiency and the advantages of parallelized inference are necessary to motivate this claim. Maybe, instead of talking about the absolute efficiency of the proposed approach, it is better to show the better trade-offs in terms of efficiency and reconstruction quality than SIREN.
b- In the introduction (L28-30), the authors claim that standard INRs extract features vastly different for close spatial coordinates (i.e., locality problem). I am unaware of studies that formally investigate this INR property. Thus, I suggest adding a reference work or validating it with experiments. Moreover, the authors claim MoE INRs can learn local piece-wise functions (L259 in the conclusion section). Thus, they do not suffer from the problem above. Yet, the experiments show something different. For instance, by looking at Figure 3c, different experts predict audio signals for close temporal coordinates. I can notice the same behavior in the last column of Figure 6, in which many distinct experts predict pixels in the upper part of the image.
W2 ) The idea of using MoE resembles the idea of KiloNeRF [1]. In that case, the routing strategy is not learned and depends only on the input 3D coordinate, and each expert focuses on a pre-determined spatial region. I think the authors should add this reference, explaining the pros and cons of the two kinds of approaches.
W3) In recent years, INR frameworks have been proposed to be faster and more efficient than MLP-only techniques such as SIREN. For instance, hybrid INRs such as InstantNGP [2] (hash grid + MLP) can be used as the base framework to speed up computation when learning INRs of images and surfaces or NeRFs. The paper should also include more recent competitors than SIREN.
[1] Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. ICCV 2021.
[2] Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG) 2022.
Questions
Q1) In Table 1, does the vanilla MoE baseline employ Softplus activations, while the final strategy uses Sine activations? In this case, the comparison is unfair and does not validate the superiority of their approach to the vanilla one.
Q2) Can the authors also include the Chamfer Distance metric in Table 3?
I like the paper's core idea. However, the author's response to my concerns will greatly influence my final rating.