Library

Generative Neural Fields by Mixtures of Neural Implicit Functions

We propose a novel approach to learning the generative neural fields represented by linear combinations of implicit basis networks. Our algorithm learns basis networks in the form of implicit neural representations and their coefficients in a latent space by either conducting meta-learning or adopting auto-decoding paradigms. The proposed method easily enlarges the capacity of generative neural fields by increasing the number of basis networks while maintaining the size of a network for inference to be small through their weighted model averaging. Consequently, sampling instances using the model is efficient in terms of latency and memory footprint. Moreover, we customize denoising diffusion probabilistic model for a target task to sample latent mixture coefficients, which allows our final model to generate unseen data effectively. Experiments show that our approach achieves competitive generation performance on diverse benchmarks for images, voxel data, and NeRF scenes without sophisticated designs for specific modalities and domains.

cs.LGcs.CVComputer ScienceMathematics
Source

Paper

References (42)

01Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance FieldsJ. Barron, B. Mildenhall, Dor Verbin et al.2023 · IEEE International Conference on Computer Vision · 873 citations In Library
02HyperDiffusion: Generating Implicit Neural Fields with Weight-Space DiffusionZiya Erkoç, Fangchang Ma, Qi Shan et al.2023 · IEEE International Conference on Computer Vision · 184 citations In Library
03Diffusion Probabilistic FieldsPeiye Zhuang, Samira Abnar, Jiatao Gu et al.2023 · International Conference on Learning Representations · 33 citations In Library
04Spatial Functa: Scaling Functa to ImageNet Classification and GenerationM. Bauer, Emilien Dupont, Andy Brock et al.2023 · arXiv.org · 59 citations In Library
05Neural Fourier Filter BankZhijie Wu, Yuhe Jin, K. M. Yi2022 · Computer Vision and Pattern Recognition · 37 citations In Library
06NeurMiPs: Neural Mixture of Planar Experts for View SynthesisZhi-Hao Lin, Wei-Chiu Ma, Hao-Yu Hsu et al.2022 · Computer Vision and Pattern Recognition · 35 citations In Library
07Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timeMitchell Wortsman, Gabriel Ilharco, S. Gadre et al.2022 · International Conference on Machine Learning · 1.7k citations In Library
08Generating Videos with Dynamics-aware Implicit Generative Adversarial NetworksSihyun Yu, Jihoon Tack, Sangwoo Mo et al.2022 · International Conference on Learning Representations · 233 citations In Library
09From data to functa: Your data point is a function and you can treat it like oneEmilien Dupont, Hyunjik Kim, S. Eslami et al.2022 · International Conference on Machine Learning · 228 citations In Library
10Instant neural graphics primitives with a multiresolution hash encodingT. Müller, Alex Evans, Christoph Schied et al.2022 · ACM Transactions on Graphics · 6.1k citations In Library
11Efficient Geometry-aware 3D Generative Adversarial NetworksEric Chan, Connor Z. Lin, Matthew A. Chan et al.2021 · Computer Vision and Pattern Recognition · 1.5k citations In Library
12Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPsSameera Ramasinghe, Simon Lucey2021 · European Conference on Computer Vision · 196 citations In Library

Scroll for more · 30 remaining

Similar papers

HyperDiffusion: Generating Implicit Neural Fields with Weight-Space DiffusionZiya Erkoç, Fangchang Ma, Qi Shan et al.2023 · IEEE International Conference on Computer Vision · 183 citations In Library
Learning Implicit Fields for Generative Shape ModelingZhiqin Chen, Hao Zhang2018 · Computer Vision and Pattern Recognition · 1.9k citations In Library
The Neural Coding Framework for Learning Generative ModelsAlexander G. Ororbia, Daniel Kifer2020 · arXiv (Cornell University) · 86 citations In Library
Neural Network based Explicit Mixture Models and Expectation-maximization based LearningDong Liu, Minh Thành Vu, Saikat Chatterjee et al.2020 In Library
Neural Network based Explicit Mixture Models and Expectation-maximization based LearningDong Liu, Minh Thành Vu, S. Chatterjee et al.2019 · IEEE International Joint Conference on Neural Network · 4 citations In Library
Stochastic Conditional Generative Networks with Basis DecompositionZe Wang, Xiuyuan Cheng, G. Sapiro et al.2019 · International Conference on Learning Representations · 19 citations In Library
Learning the Base Distribution in Implicit Generative ModelsCem Subakan, Oluwasanmi Koyejo, P. Smaragdis2018 · arXiv.org · 8 citations In Library
Learning in Implicit Generative ModelsS. Mohamed, Balaji Lakshminarayanan2016 · arXiv.org · 442 citations In Library
Gradient Origin NetworksSam Bond-Taylor, Chris G. Willcocks2020 · Durham Research Online (Durham University) · 19 citations In Library
Hyper-Transforming Latent Diffusion ModelsIgnacio Peis, Batuhan Koyuncu, Isabel Valera et al.2025 · arXiv (Cornell University) · 3 citations In Library

Peer review

Accept (poster)NeurIPS 2023 · via OpenReview
Reviewer YZcJ5/10 · confidence 4/52023-07-07

Summary

This paper proposes a new representation of neural fields based on a mixture of neural fields. Neural fields is another name for implicit neural representation (INR) which is a representation of data points (like images or 3d scenes), that maps an input of 2d or 3d position into the output of the RGB color and/or shape information. The paper builds on previous work like GASP [12], Functa [13] and GEM [1] and proposes a new representation on which the generative model is trained. As in [13] the generative model is a diffusion model and the novelty here is the representation which oi based on a mixture model.The mixture model representation proposed is implemented as a shared set of INRs which weights are averaged with a vector of mixture weights that are optimized per data-point. After describing the model, the paper demonstrates its effectiveness in different data modalities like images (Celeba-HQ), 3D shapes (ShapeNet) and 3D scenes (SRN Cars), where the model performs well compared to previous approaches. Then some additional ablation and analysis are presented.

Strengths

The main strength of this paper is the novel INR representation proposed. This representation follows a recent line of work on the topic, that aims to leverage the advantages of INRs into the setup of training generative models on different data modalities. A main challenge there is capturing 3D data in an efficient way, which is still not acheivable in standard direct methods. The proposed model is an interesting direction with a potential to contribute to the research of this topic.

Weaknesses

The main weaknesses of this paper are the following: 1. The description of the model and the comparisons to previous work is lacking. Although some of the aspects are simple and clearly presented and the comparison to previous models is extensive and contains many different metrics, there still remains some important aspects that are not sufficiently described (see questions below). 2. After describing the model, the authors "jump" to final metrics like PSNR and FIDs. Given that the main contribution of this work is the mixture representation itself, it is essential to show what this representation captures. Figure 5 is a good start in this direction, but what is still missing is some understanding of what the different components in the mixture are doing, and how they are combined. The authors could, for example, show the reconstructed images of different components in isolation as the number of components grow. Another component which is directly used in the model without investigation is the hierarchical mixture (mixture of every layer in the network). How does this affect the components? How are the differences in lower layer components manifested compared to differences in higher level components?

Questions

1. The meta learning algorithm presented in algorithm 1: why is $\phi$ randomaly initialized at every iteration? This is different than MAML that learns an initialization, and CAVIA [17] (which the authors state as reference for this) that keeps the initialization constant. This is an interesting difference that should be discussed. 2. Algorithm 2: The pseudo code is not clear about the indexes of $\phi$ how are they being updated through the iterations? 3. One aspect that is missing from all tables and comparisons is the size of the representation. For the proposed model this would be the number of mixture components times the number of layers. This is an important aspect of the model since it defines the rate-distortion tradeoff. The way the authors count parameters (learnable vs. inference) is interesting and of significant value, but adding the size of the representation is essential. 4. How are the previous models chosen for comparison? These models usually contain several options with different sizes of the network and the representation, where the reconstruction quality usually grows with the size of the network. 5. Line 115: What exactly is invariant to what? 6. The ablation in the first section of 5.3 is not very useful. As stated above, it would be better to try to understand what every component in the mixture captures. 6. In figure 5, I'm not sure I understand what the plots show. Are the matrices sorted somehow?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

The main limitation that is missing from the discussion in the paper is the relation of the quality of representation and its size (see weakness #2 and question #3).

Reviewer ZMK55/10 · confidence 5/52023-07-07

Summary

This paper deals with the problem of learning the latent space of data through the concept of meta-learning INRs (or neural fields). In contrast to prior works that consider either the adaptation of whole neural field parameters or some linear modulation parameters, the proposed method considers some basis neural field parameters as the shared parameters and adapts to coefficients for this basis to be adapted. The proposed method better encodes the given data of diverse modality, compared with prior INR-based data encoding scheme.

Strengths

- The observation that the linear combination of neural fields as basis function parameters is quite interesting and novel. - The quantitative results are better than prior works. - The paper is generally well-written and easy to follow, while some important details are missing in the current status (see the weakness below)

Weaknesses

- For generating new data from the proposed framework, specification of the latent sampling strategy (e.g., what is a prior distribution of $\phi$) is crucial, while the paper only states "off-the-shelf sampling method" in L179. Explaining how the sampling is performed would make the paper much stronger. - I guess this method requires much more GPU memory, as it uses multiple basis neural field parameters with the (second-order) gradient-based meta-learning method. In particular, this method requires about 10 times more parameters than Functa. - The experiments are mostly conducted under fine-grained datasets; I wonder if the method can be worked well with more complex datasets (e.g., CIFAR-10 or ImageNet-64x64) compared with other methods.

Questions

- Is there any constraints (or ranges) for the coefficients $\alpha_i$s?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The paper addresses the limitations and negative social impacts in the supplementary materials.

Reviewer 2vix7/10 · confidence 3/52023-07-08

Summary

This paper proposes an approach to increase the learning capacity of implicit neural networks (INRs). The idea is to define a set of basis implicit functions that can be mixed to produce an instance-specific INR for each instance in a dataset. Notably, the approach does not affect the inference performance as once an instance-specific INR is produced, the cost of coordinate sampling is the same as with any other INR. This method allows implicit neural networks to be generalized on entire datasets rather than overfitting to a single instance while still demonstrating high fidelity of samples. The training procedure is split into two steps, the first step being a meta learning or auto decoding problem to fit the basis INRs with a reconstruction loss to the training data, and the second step is training a latent model to generate samples that determine the mixing coefficients. The paper demonstrates solid results and evaluation on different datasets of varying modalities including images, voxels and NeRFs.

Strengths

The paper is well-written and easy to follow. The methodology is well explained and is simple but effective. The inference performance is significantly better than three state-of-the-art methods considered, and so are the extensive generation metrics like FID, precision, recall, coverage etc. The methodology and design choices are comprehensively evaluated on multiple datasets, and modalities with ablation studies.

Weaknesses

The paper makes a solid contribution and I do not see any major flaws.

Questions

Typos: Line 108: resepctively -> respectively Line 177: requires -> require I was wondering if the authors could add more interpolation results as in Figure 4, perhaps in the supplementary material. In particular, it would be interesting to see how interpolation works across ShapeNet categories.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

Yes, limitations and societal impact are sufficiently discussed in the supplementary material.

Reviewer uPSW6/10 · confidence 3/52023-07-09

Summary

The paper proposes a mixture of neural implicit functions for generating neural fields via model averaging of implicit representations trained through context adaptation (meta-learning/auto-decoding paradigm) and subsequently task-specific generalization (additional sampling strategy with respect to the latent mixture coefficient vector).

Strengths

This well-written paper clearly elucidates its contributions with convincing results.

Weaknesses

- The authors are requested to discuss limitations within the bounds of the main paper. - Minor Corrections - Line 108, resepctively → respectively - Line 226, the both → both the - Line 278, effective learn → effectively learns - Caption of Figure 1 ideally should elucidate the concept of neural radiance fields.

Questions

Experiments lack quantitative evaluation/baseline. The authors are requested to clarify/highlight baseline models in proposed evaluation protocols.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- The proposed approach does not scale. - The current model exhibits aliasing artifacts in the neural radiance field scene. - There’s potential for bias within the system.

Reviewer fjUK5/10 · confidence 5/52023-07-10

Summary

This paper introduces an approach for learning generative neural fields using linear combinations of implicit basis networks, called Mixture of Neural Implicit Functions (mNIF). The basis networks and their coefficients in mNIF can be learned through meta-learning or auto-decoding techniques. By this means, the capacity of generative neural fields can be increased by adding more basis networks while maintaining the size of each instance. The optimized network is further fine-tuned to make generalized predictions by optimizing with respect to latent vectors, enabling the representation of unseen data. Experimental results demonstrate performance across diverse benchmarks.

Strengths

+ The paper demonstrates strong writing, with clear notation and intuitively-rendered figures. The proposed method is well-motivated and easy to understand. + The experiments, although conducted on simple datasets like CelebA and Shapenet, effectively validate the method's effectiveness and parameter efficiency in both 2D and 3D domains. The proposed method achieves competitive performance in generating Implicit Neural Representations (INRs), surpassing the performance of all compared baselines according to metrics. The interpolation results show promise.

Weaknesses

- Some important related work and baselines are missing, including three papers (references [1][2][3]) that explore the idea of using Mixtures of Experts (MoE) in INRs, as well as reference [4] as an additional image generation baseline and reference [5] as a strong baseline for shape generation. As a result, the proposed method does not appear entirely novel in the field and resembles existing latent code-based implicit representations, differing primarily in the incorporation of latent information through different computation paradigms. The use of MoE is also common in the community. It is recommended to exhaustedly discuss the difference between the aforementioned works and highlight key contributions. - The motivation for using the mixture of Neural Implicit Fields (NIFs) is mentioned as better memory efficiency, but it is recommended to directly compare run-time memory with the baselines. Additionally, it would be beneficial to include inference time comparisons with the presented methods. [1] NeurMiPs: Neural Mixture of Planar Experts for View Synthesis [2] Neural Implicit Dictionary via Mixture-of-Expert Training [3] Switch-NeRF: Learning Scene Decomposition with Mixture of Experts for Large-scale Neural Radiance Fields [4] Adversarial Generation of Continuous Images [5] HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion

Questions

1. What training method is used throughout the experiments? 2. Why Functa is not compared in the voxel reconstruction/generation task on ShapeNet?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

4 excellent

Contribution

2 fair

Limitations

The main text of the paper does not address the limitations of the proposed method. From my understanding, scaling up the method to larger and more complex datasets could still present challenges.

Reviewer ZMK52023-08-11

Response

Thanks for the response. The provided response addressed most of my concerns. Please add the response of W2 in the final version of the manuscript if the paper is accepted. I slightly increase my score to 5, as the proposed method currently seems limited scalability to more complex dataset.

Authorsrebuttal2023-08-12

Thank you for your insightful feedback and for the improved rating on our work. We will address the issues previously mentioned. If you have any questions regarding our work, please feel free to leave comments.

Authorsrebuttal2023-08-20

To analyze the generation on CIFAR10, we introduce an alternative sampling scheme to replace the latent diffusion model described in our main paper. Drawing inspiration from the sampling strategy in generative manifold learning (GEM) [2], we employed latent space interpolation between two samples: one random sample from the training set and another random sample from its neighborhood. The resulting context vector, $\phi_\text{sample}$, is formed by a simple interpolation: $\phi_\text{sample} = \alpha \cdot \phi_i + (1- \alpha ) \cdot \phi_{\mathcal{N}(i)}$ where $\phi_i$ denotes context vector for $i$-th sample in training set, $\mathcal{N}(i)$ is a set of indices of the neighborhood of sample $i$ and $\alpha$ is sampled from the uniform distribution ranged between 0 to 1. method | reconstruction (PSNR) | sampling by interpolation (FID) | sampling from diffusion model (FID) | |----------|-------------------------------|-------------------------------------------|---------------------| ours | 29.89 | 59.67 | 84.64 | From the results, two major observations can be made: - Sampling through interpolation yields a superior FID score than the latent diffusion model, suggesting issues with the latter's interaction with context vectors. - The FID score for interpolation sampling, while better than diffusion, remains suboptimal. The observation that the context vector interpolation is not fully effective for the CIFAR10 model indicates a less smooth latent space in CIFAR10 compared to other datasets. This disparity could be impacting the generation quality. We hypothesize that the limitations of the SIREN [1] architecture might be at the root of these issues. Research [3] categorizes SIREN as an INR with predominantly frequency encodings and a lack of space resolution, leading to compromised content quality. A separate unpublished study [4] underlines this, stating SIREN's unsatisfactory generation performance (FID 78.2) when applied to CIFAR10. This research introduces a new SIREN architecture where biases in hidden layers of INR are varying with respect to spatial resolution. We anticipate that adopting an INR with spatial information as an alternative to SIREN will better accommodate diverse datasets, given that our approach—a linear combination of INR weights—is broadly compatible with various INR methods. References - [1] Implicit Neural Representations with Periodic Activation Functions, NeurIPS 19 - [2] Learning Signal-Agnostic Manifolds of Neural Fields, NeurIPS 21 - [3] Neural Fourier Filter Bank, CVPR 23 - [4] Spatial Functa: Scaling Functa to ImageNet Classification and Generation, ArXiv 2302.03130, February 6th 2023.

Reviewer uPSW2023-08-15

Response to Authors

Dear Authors, \ Appreciate your response - which addresses the major concern i.e., benchmarking. \ However, due to limitations concerning scalability, I hereby retain my original score of 6.

Authorsrebuttal2023-08-15

We are pleased to know that you appreciate our benchmark compared to the baselines over three distinct modalities and are grateful for your positive rating of our work. We are currently addressing the scalability concerns also mentioned by reviewer ZMK5 and plan to discuss this thoroughly before the end of the discussion period. Please don't hesitate to reach out if you have any further questions.

Authorsrebuttal2023-08-15

We truly appreciate your positive feedback on our work. We leave a comment to address the missing class information on the interpolation plots in Figure B of the attached PDF: For the objects in positions (a): they are chair, cabinet, display, and sofa. For those in positions (b): they are loudspeaker, sofa, airplane, and bench. And for positions (c): they are airplane, bench, car, and rifle. The order we're referencing is based on the four corners, starting from the top-left, moving to the top-right, then bottom-left, and finally bottom-right. Please don't hesitate to reach out if you have any further questions.

Reviewer 2vix2023-08-20

Thanks for the response and inter-category interpolation results which are promising and demonstrate the effectiveness of the method. I am retaining my score and would be happy to see this paper accepted.

Authorsrebuttal2023-08-20

Thank you for giving our paper a positive score. We are delighted to know that you appreciated the inter-class interpolation results in ShapeNet. We'll include a wider range of interpolation examples in the final version of our work.

Authorsrebuttal2023-08-15

We strive to provide comprehensive answers and hope our responses have addressed your queries. Nevertheless, we acknowledge that there might be moments of ambiguity or potential misunderstandings. Please don't hesitate to seek further clarification on any aspect of our work. We're also grateful for the points you raised concerning weakness 2. Your insights not only reinforce our motivation but also quantitatively demonstrate the efficiency of our model, particularly in the voxel and NeRF scenes. Here's a summary of the efficiency metrics from Table A in the attached PDF file: CelebA-HQ $64^2$ | Ours (light) | GASP | Ours (main paper) |GEM | Functa |--------------------------------|---------------|----------|-------------------------|---------|----------| Inference cost (GFLOPS) | 0.069 | 0.305 | 0.340 |3.299 | 8.602 Inference speed (fps) | 2841.0 | 1949.3 | 891.3 | 559.6 | 332.9 Inference memory (MB) | 10.16 | 16.35 | 24.42 |70.31 | 144.05 ShapeNet $64^3$ | Ours (light)| GASP | Ours (main paper) |GEM | Functa |--------------------------------|---------------|----------|--------------------------|---------|----------| Inference cost (GFLOPS) | 4.363 | 8.742 | 21.613 | 207.0 | 550.0 Inference speed (fps) | 191.5 | 180.9 | 69.6 | 16.7 | 6.3 Inference memory (MB) | 642.07 | 763.13 | 1513.26 | 4010.0 | 9000.0 SRN Cars | Ours (main paper) | Functa |--------------------------------|---------------------------|------------| Inference cost (TFLOPS) | 0.009 | 1.789 Inference speed (fps) | 97.7 | 2.0 Inference memory (GB) | 1.26 | 28.01 Our light configuration for images is given by $(M,D,W,H)=(256,4,64,512)$, and for voxels, it's $(M,D,W,H)=(256,4,64,1024)$. Here, $M,D,W,H$ represent the number of mixtures, hidden layers, hidden layer dimensions, and context vector dimensions, respectively. Additionally, to underscore the efficiency of our model during the training process, as requested by Reviewer ZMK5, we've measured memory consumption during meta-training process in context adaptation: Memory | Ours | Functa |-------------------------------|--------------------|-------------| Training (batch 32) | 9.3 GB | 40.2 GB Inference | 24.4 MB | 144.1 MB It's worth highlighting that our method utilizes 4.3 times less GPU memory for training and 5.9 times less for inference compared to Functa. While our training phase might slightly exceed our inference memory requirements, the efficiency gap between our method and Functa remains significant. This underlines that the small size of inference parameters can effectively mitigate computational burdens arising from handling large-scale coordinate input in both training and inference.

Reviewer fjUK2023-08-20

Thanks for rebuttal!

My major concerns lie in the lack of discussion on the existing works. The authors' rebuttal addresses this problem and now I'm lean toward acceptance. I expect authors to include the discussions above into the camera-ready version.

Authorsrebuttal2023-08-21

Thank you for your insightful feedback during the discussion phases and for improving the rating positively on our work. We will include mentioned comments and discussions in the final version of the paper.

Authorsrebuttal2023-08-15

We strive to provide concise and clear responses, but we understand if our previous reply may not have fully addressed your concerns or could have been misunderstood. Please feel free to ask any further questions. **Additional Comment on Q1** To elaborate on what might have been missing in our prior response regarding the random initialization of \phi: As previously discussed, during meta-training, our model begins with a random latent vector sampled from a zero-mean Gaussian distribution with a standard deviation of \sigma. This is a slightly different approach taken by CAVIA. Our intention in doing so is to diversify INR weights at the start phase of meta-training, ensuring that not all samples are represented with identical INR weights.

Reviewer YZcJ2023-08-17

Thank you for your answer. Regarding Q3 and the dimension of representation, I think this means you have a typo in line 139. Shouldn't the vector of $\alpha$'s be of dimension $M \times (L+2)$ rather than $\phi$? I still think the representation size is an important aspect when comparing the different methods, and should be part of the comparison tables.

Authorsrebuttal2023-08-17

We made a mistake that led you to misunderstand the content. Line 139 does have a typo; the correct notation should be $\phi \in \mathcal{R}^{H}$. The dimension $M \times (L+2)$ represents the total number of $\\{ \alpha_m^{(l)} \\}$. Our model used latent mixture coefficients, and the latent coefficient vector $\phi$ serves as the representation for a sample. So, the size of the coefficient vector $H$ becomes the representation size. As mentioned in answer to Q3, we set the representation size of our model as comparable to other models in the experiment tables of the main text. We are sorry for the mistakes and we will correct this error and explicitly mention the representation size in the tables, so it can be compared with other methods. We appreciate your careful and thorough review. Please don't hesitate to reach out if you have any further questions or concerns.

Authorsrebuttal2023-08-21

**Updating Table with representation size** To clarify your request, we instantiate the extended version of Table 1 of the main paper with the additional representation size denoted as $D(\phi)$. | Model | # Learnable Param | # Inference Param | $D(\phi)$ | PSNR | rFID | FID | Precision | Recall | F1 | |----------|-----------------|-----------------|-------------|---------|-------|-------|---------|----------|---------| Functa | 3.3 M | 2,629.6 K | 256 | 26.6 | 28.4 | 40.4 | 0.577 | 0.397 | 0.470 | Functa | 4.5 M | 2,629.6 K | 512 | 29.7 | 17.2 | - | - | - | - | Functa | 6.8 M | 2,629.6 K | 1024 | 32.4 | - | - | - | - | - | GEM | 99.0 M | 921.3 K | 1024 | - | - | 30.4 | 0.642 | 0.502 | 0.563 | GASP | 34.2 M | 83.1 K | 64 | - | - | 13.5 | 0.836 | 0.312 | 0.454 | DPF | 62.4 M | - | - | - | - | 13.2 | 0.866 | 0.347 | 0.495 | mNIF (main paper) | 33.4 M | 83.3 K | 512 | 34.5 | 9.2 | 13.2 | 0.902 | 0.544 | 0.679 | The performances of Functa, GEM, GASP, and DPF are adopted from the reported values in each paper.

Reviewer YZcJ2023-08-22

Thank you for your answers. This discussion removes some of the concerns I had regarding fair comparisons and lack of analysis, so I slightly raise my score.

Authorsrebuttal2023-08-22

Thank you for your feedback, which has highlighted the need for comparison and deeper analysis to enhance understanding of the model. We also appreciate your improved rating of our work. We will revise our paper to include discussions over the periods.

Authorsrebuttal2023-08-20

We deeply appreciate the insightful reviews and discussions throughout the review period. We will revise our paper to incorporate your valuable feedback. Below, we provide summaries of the reviews, our responses, and discussions. Our primary objective is to reconcile two seemingly contrasting goals: efficient inference and expansive capacity in an implicit neural representation (INR). Unlike other research on mixtures of INRs, our approach constructs mixtures within the INR's weight space. The computational cost of INR is dictated by the size of inference parameters, not by the total size of learnable parameters. But the sample quality is in favor of the size of the learnable parameter. Ensembling within the weight space maintains the size of the inference parameter compact while allowing us to easily increase learnable parameter size by adding more mixtures. This design allows our model to handle tremendous coordinate inputs with comparable visual quality. We evaluate the efficiency of our proposed method against INR baselines based on constructive feedback from reviewers fjUK and JMK5. We observe that our approach demonstrates a high degree of efficiency. Especially in NeRF scenes with a significant number of input coordinates, our method achieves 199 times fewer FLOPS, 49 times faster fps, and 22 times reduced memory than Functa during single view inference. Detailed metrics can be found in Table A of the attached file. SRN Cars | Ours | Functa |--------------------------------|---------|-----------| Inference speed (fps) | 97.7 | 2.0 Inference cost (TFLOPS) | 0.009 | 1.789 Inference memory (GB) | 1.26 | 28.01 Reconstruction PSNR | 25.9 | 24.2 Generation FID | 79.5 | 80.3 Reviewers fjUK and uPSW YZcJ raised questions regarding our choice of baselines. We have selected GEM, GASP, and Functa for this purpose. Our model and selected baselines focus on algorithms for generating neural fields, especially those algorithms handling various modalities such as 2D images, 3D voxels, and NeRF scenes. Our model exploits a two-step training process. Initially, we perform context adaptation on our INR model to obtain context vectors for the training set. Subsequently, we train a latent diffusion model for sampling context vectors. The reconstruction performance pertains to the first step, while the generation performance is influenced by both steps. As reviewer YZcJ noted, increasing the latent space's representation size improves reconstruction performance. But, generation performance may require a different optimal representation size. Upon request by reviewer YZcJ, we delved deeper into our model's internal states. Our observations indicate that information for each sample is evenly distributed across all mixtures within the hidden layers except the input and output layers. The main paper's Figure 5 showcases the diversity of learned weights in each mixture, implying that each mixture's neural implicit function contributes equally to representing a training set sample. Figure A(a) in the attached file further substantiates our model's distributed nature. As the number of mixtures increases, the quality of the reconstructed image steadily improves. It also explains that the information is distributed information over mixtures. Figure A (b) offers a visual representation of the impact of adding or subtracting coefficients from each layer. We can observe that information is predominantly located in the hidden layer, excluding the input and output layers. In Figure B of the attached file, we visualize the inter-class interpolations requested by reviewer 2vix. We plot three interpolations with four different classes correspondingly. (a) chair (top left), cabinet (top right), display (bottom left), sofa (bottom right) (b) loudspeaker, sofa, airplane, bench (c) airplane, bench, car, rifle. Figure B (c) shows smooth interpolation even with distinct shapes. We have one more day for discussions related to this work. We invite you to review our findings and share any further queries or concerns. Thank you for your time and feedback.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC