General Comments
In general the reviewers agreed that the paper’s method is clear, the problem of feedforward NeRF generation and accelerated convergence to out-of-distribution scenes are well motivated, and that the application of hypernetworks is interesting. The primary concerns raised by the reviewers are: (1) the limited quality/complexity of the scenes presented in the experiments, and (2) the relation of the work with respect to ATT3D. We address these concerns in an updated version and summarize these updates below.
# Scene quality/complexity
Please refer to Fig. 11 in the supplemental (or see the figure link [here](https://ibb.co/YbYnF7C)) for an example of single HyperFields model learning over high quality, complex scenes, by simply using a better teacher model such as ProlificDreamer. Our method’s generation quality is determined by the quality of the teacher NeRFs (see Fig. 2 in paper). The teacher NeRFs we use for the paper were generated with Stable-DreamFusion, which was the best publicly available model at the time of research. However, our method is general and capable of using any model as teacher NeRFs, and we can distill multiple scenes into a single HyperFields network all while maintaining the level of quality of the underlying model teacher NeRFs.
In addition to the scenes in Fig. 11 in the supplemental (or see the figure link [here](https://ibb.co/YbYnF7C)), we show 30 more scenes learned by a **single HyperFields** model in Fig. 12 in the supplement (or see the figure link [here](https://postimg.cc/QVVwy3x1)). In the limited time available to us we trained a **single HyperFields** model to learn in total 50+ scenes (we show 38 of those scenes Fig. 11, Fig. 12) depicting various complex poses of animals, thereby underscoring the ability of a single HyperFields model to learn multiple complex scenes.
### OOD Results
To add more variability to our OOD (out-of-distribution) results from Fig. 5 we add OOD animal prompts. We consider ‘wooden squirrel’ and ‘squirrel playing flute’, for both these prompts the geometry generated by HyperFields seems to be better than the baseline StableDreamFusion. The corresponding renders are in Fig 13 Appendix J and in this [link](https://ibb.co/fd64kVN). These prompts are OOD because the model has not seen the attribute wooden and playing flute in its training set. We optimize our HyperFields model using SDS loss.
# ATT3D
A direct comparison against ATT3D is not possible because neither the model nor the code has been made public. ATT3D’s visual quality and scene complexity is a consequence of the underlying text to 3D model, Magic3D, which uses proprietary diffusion models eDiff-I and Latent Diffusion Model (LDM) as guidance. These models are trained on far more resources (data & compute) than the publicly available StableDiffusion model we rely on for guidance.
We again emphasize that **with a stronger diffusion model** HyperFields is capable of packing and generating scenes of equivalent quality and complexity, which we demonstrate in the above [linked figure](https://ibb.co/YbYnF7C) and Section G of the supplemental.
## Novel technical contributions
1. We demonstrate the ability to generate out-of-distribution scenes with very few fine-tuning steps (Figure 5 in paper), whereas ATT3D only shows results on in-distribution scenes (convex combinations of training scenes). The ability to generate out-of-distribution scenes with minimal fine-tuning is key as it greatly saves time **on new out-in-the-wild prompts**, not simply prompts restricted to recombinations of the training set.
2. Furthermore, our proposed dynamic hypernetwork architecture is novel, as the predicted NeRF weights are generated based on the activations from the previous layers, leading to better flexibility of our model. This is key to our ability to pack multiple scenes with drastically varying geometric and color attributes (Ablation Fig 6a in the paper). On the contrary, ATT3D keeps the weights of the NeRF the same, and simply modulates the features of the Instant-NGP grid based on the text prompt. By keeping the MLP weights fixed and only updating the feature grid per prompt, they take advantage of spatial locality of common features across scenes. But it also induces a minor mode collapse, in which common features will be shared among unrelated scenes (e.g. the clothes/style objects in [Fig. 8 from ATT3D](https://research.nvidia.com/labs/toronto-ai/ATT3D/images/generalization_videos/animals_amortized_testing_12.5.mp4) are either entirely black/red, in contrast to the significant visual variation in the [per-prompt optimization](https://research.nvidia.com/labs/toronto-ai/ATT3D/images/generalization_videos/animals_per_prompt_test.mp4)). Our results in (Fig. 7) exhibit greater variation in color palettes, object poses, and geometric detail.