AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis

Summary

The paper introduces a new task called "real-world audio-visual scene synthesis", which involves synthesizing new videos with spatial audios along arbitrary novel camera trajectories in a scene. The proposed NeRF-based approach integrates a prior knowledge of audio into audio generation and associates it with the 3D visual environment. The proposed model is able to learn a sound source-centric acoustic field with the introduction of the coordinate transformation module that expresses a view direction relative to the sound source. Additionally, a high-quality Real-World Audio-Visual Scene (RWAVS) dataset is demonstrated along with the model.

Strengths

1. The paper presents an interesting new task that associates the audio with 3D geometry and material properties of scenes and the video demo demonstrates the effectiveness of the proposed method. 2. The paper compares the proposed method with similar methods on both the RWAVS dataset and the SoundSpaces dataset and surpasses the previous state-of-the-art method. 3. The method proposes a novel dataset that can be further utilized in similar tasks.

Weaknesses

1. The novelty of the proposed network seems to be limited. To tackle the new task, AV-NeRF combines an audio-NeRF and visual-NeRF and introduces an AV-mapper and the novelty mostly lies in the AV-mapper based on my current understanding. 2. The paper introduces a new task called "real-world audio-visual scene synthesis”. However, as mentioned in the paper, this work currently focuses on static scenes with a single fixed sound source, which is a bit far from the word “real-world” since in real-world, there can be much more sound sources.

Questions

1. While the method is spatially-aware, the method seems to be less effective when the distance of the sound source is changing from the demonstration of the video, could you provide more results? 2. The sound source demonstrated in the video is the same, could the sound source be changed, or the sound-source device has certain requirements? If so, the proposed method can be limited. 3. As mentioned in the weaknesses, please clarify the novelty of the proposed method again.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

As mentioned in the paper, this work currently focuses on static scenes with a single fixed sound source. To better suit the word “real-world”, the method needs to be improved and adapted to multiple sound-source situations.

Summary

The paper introduces a novel task called real-world audio-visual scene synthesis to generate novel views and spatial audio for any camera trajectories. The system contains a visual NeRF to synthesize novel view images, an audio NeRF to generate acoustic masks, and an audio-visual mapper to enable the visual information to enhance the A-NeRF. The authors also collect a new real-world dataset to demonstrate the task's importance and for benchmarking. Experiments on synthetic and real datasets show the effectiveness of the proposed method.

Strengths

1. The task of audio-visual scene synthesis in the real world is essential. To my best knowledge, it is the first work that aims to generate both video and spatial audio in a real environment. 2. Multimodal data collection for the real scene is challenging and requires lots of effort. I could see the new real data would be very useful for future research. 3. Each component in the model pipeline is well-motivated and makes sense. 4. The paper is generally well-written and easy to follow. 5. Extensive experiments and ablation studies are done on real and synthetic data to show the effectiveness of the proposed method.

Weaknesses

1. As the authors mention in the limitation, the current approach mainly works for static scenes with a single fixed sound source. However, as the initial exploration, the current setup makes sense. 2. It is often challenging to interpret and understand the quantitative results of generated audio. While the proposed method outperforms baselines across all quantitative metrics. If possible, I would suggest authors add human subjective evaluations in the future to see the perceptual difference. 3. As a new benchmark, it would be great to show some failure cases for the current method to indicate potential improvement in the future.

Questions

1. If I understand correctly, the source mono audio is fixed for each scene in the RWAVS dataset. Have you tried to use another source audio during the inference time, and does it sound as good as before? 2. Following the previous question, I guess one of the main differences between the real world and simulated data is that in the simulation platform, we could obtain dense impulse responses in arbitrary positions, but in the real world, it is obviously easier to play source audio and collect data along a trajectory continuously. However, how could we effectively validate the correctness at the inference stage if we are using new source audio without spatial audio ground truth? 3. Since the goal is to generate video frames and spatial audio, the current A-V mapper aims to extract useful information from visuals to help A-NeRF. Is it also possible to use it to extract useful information from the audio side to enhance the synthesized visual frame?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

N/A

Summary

Inspired by the task of novel view synthesis, this paper additionally considers the audio modality and thus proposes an interesting new task: real-world audio-visual scene synthesis. Specifically, the task is to synthesize new videos with corresponding spatial audio along arbitrary novel camera trajectories by learning from a video recording of an audio-visual scene. To solve this problem, the authors resort to AV-NeRF, an A-NeRF for spatial audio synthesis, and a V-NeRF for video synthesis, respectively. For V-NeRF, they directly use the Vallina NeRF model. For A-NeRF, they explore extracting 3D geometry and material properties from the corresponding visual cues of V-NeRF. Moreover, a coordinate transformation module is proposed for more accurate spatial sound reconstruction. Better performance on a self-collected real-world dataset RWAVS and a simulation-based dataset SoundSpaces demonstrates the effectiveness of the proposed method AV-NeRF.

Strengths

1. This paper proposes an interesting task: real-world audio-visual scene synthesis, which extends the task of novel view synthesis from only visual modality to multiple modalities. 2. The A-NeRF architecture introduced in this paper is well-designed. It considers sound propagation and view directions, which is quite reasonable. Besides, the coordinate transformation is also shown effective through comprehensive experiments. 3. Exhaustive experiments show the superior performance of AV-NeRF over several strong baseline methods. 4. The whole paper is well-written and presented in a coherent manner.

Weaknesses

1. The design of the most important module AV-Mapper is not reasonable. First, the authors just use a pretrained ResNet-18 network to extract RGB and depth features from RGB and depth images. I'm curious how the ResNet-18 network could obtain material information from RGB images since the semantics can't imply the material properties at all. For a black desk, I can't tell if it's made of wood or steel just from the appearance. Actually, the authors can do some testing experiments, such as conducting color jittering and seeing what happens, or cross-scene testing to see whether the ResNet-18 network can extract materials and geometry information. Second, depth images can't inform all the geometry information. For example, the occlusion, which significantly influences sound propagation, can't be reflected on depth maps. 2. For spatial audio modeling, a useful metric that can really reflect the necessity of distance and view directions is required. Can the distance help to determine the energy of sounds on the receiver side? It's impossible to tell only from MAG and ENV metrics. The quantitative result w.r.t. **a new useful metric** should be provided in the paper. Similarly, I can't tell whether the view directions are useful for modeling the relative energy of left and right channels. I also wonder which one is more important for spatial audio modeling.

Questions

1. How can the network obtain the distance between the camera and the sound source only from their positions? As we all know, distance calculation includes square and square roots, which are hard to approximate with simple neural networks. 2. The coordinate system in each scene. How to determine the origin?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

1. The biggest limitation is the inappropriate modeling of AV-Mapper. The authors claim this module can obtain the material and geometry information from V-NeRF. However, from the perspective of this reviewer, such important information can't be inferred from the current design. Please refer to the weaknesses for details. 2. The current evaluation protocols can't validate the effectiveness of some modules employed in the paper.

Summary

This paper describes an intriguing task that focuses on the synthesis of new perspectives of real-world audiovisual scenes. The task is to synthesize a new video with spatial audio along an arbitrary novel camera track in an audiovisual scene given a video recording of that audiovisual scene. A data acquisition system is built and a Real-World Audio-Visual Scene (RWAVS) dataset is collected in this paper. First, the paper proposes an acoustically aware audio generation module that integrates prior knowledge of audio propagation into NeRF, thus relating audio generation to the 3D geometry of the visual environment. In addition, a coordinate transformation module that represents the direction of observation with respect to the sound source is proposed. This directional transformation aids the model in learning the source-centered sound field. This paper's superiority is demonstrated by qualitative and quantitative results.

Strengths

1. The synthesis of new views on audiovisual scenes in the real world is an interesting task that contributes to a better understanding of the world and the dataset presented can contribute to the development of the field. 2. The paper is well written and each module is clearly presented and easy to understand. 3. The proposed AV-Mapper and the coordinate transformation mechanism can effectively fuse visual geometric information with audio information and effectively express sound direction.

Weaknesses

1. The vanilla NeRF [15] structure is used in this paper's V-Nerf module, which is a relatively slow training and sampling model. Existing acceleration models for Nerf can already significantly improve training and sampling speed, and a better model may be able to achieve higher accuracy in this paper. 2. How does the proposed model handle sounds that are not within the field of view? 3. There are numerous audio-visual datasets available, such as the AVSBench (https://github.com/OpenNLPLab/AVSBench). What is the proposed method's performance on this dataset?

Questions

As above.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

As above.

Summary

The paper proposes an audio-visual (AV-) NeRF model to synthesize binaural audio masks at novel poses by using audio-visual samples from a video walkthrough of a 3D scene. The synthesized audio masks can be convolved with any arbitrary anechoid audio signal to retrieve the corresponding binaural audio at the novel poses. Towards that goal, the paper proposes a model that uses the learned density information from a visual (V-) NeRF model to provide geometric and color information at a novel pose. Given this visual information, an audio (A-) NeRF model predicts masks that can be further used to generate the synthesize the binaural audio for a monaural sound source. Additionally, the paper proposes an alternate parameterization to the direction component of the camera pose parameters that accounts for the omnidirectionality of most sound sources. The paper also collects a realworld dataset for the task. The paper evaluates on both simulated and realworld data, compares against multiple baselines and reports impressive results. Post author-reviewer discussion: I have read the rebuttal. It addresses my concerns. I will keep my score and recommend accepting the paper.

Strengths

1, The paper proposes a new AV-NeRF model for binaural audio synthesis that leverages learned color and geometry information for novel poses from a V-NeRF model. 2. The alternate parameterization for the camera pose parameters is also a useful contribution that captures the physics of how sound travels from a omnidirectional sound source. 3. The paper also introduces a new realworld dataset for the task, which could be a valuable contribution for the community. 4. The paper evaluates on both simulated and realworld data, compares against multiple baselines and reports impressive results.

Weaknesses

1. The paper doesn't compare with recently published ViGAS [14] model. The paper argues that ViGAS needs GT images and is limited to a few viewpoints. However, the images rendered by a standalone V-Nerf can be used as input to ViGAS (since it's not a contribution of this paper anyway), and ViGAS can be trivially adapted to render spatial sounds at arbitrary points. Minor: 1. L131: [40] generalizes to new scenes, unlike the model proposed in this paper (also pointed out by the authors in limitations). Since there are no experiments on novel scenes, this statement looks unsubstantiated.

Questions

I would like to request the authors to address the concerns I mentioned in 'Weaknesses'.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper already discusses its limitations and societal impact.

Comment

Thank you for the clarification and the efforts you put into the rebuttal. All the questions are responded and the novelties are clarified in the response. The spatial awareness is demonstrated in the rebuttal and the effectiveness is proved. I would incline to raise my rating.

Post Rebuttal

Thank the authors for the detailed responses. My questions are well addressed. I will keep my score and recommend accepting the paper.

I have read authors' rebuttal

Thanks so much for your detailed response, which has already addressed most of my concerns. I'm happy to accept this paper and therefore decide to raise my score. Please include these additional results and discuss the limitations of AV-Mapper in the revision. By the way, for the distance between the camera and the sound source, I mean the root or square root operation may be hard to approximate with simple neural networks (just two layers for example), and wonder about the detailed neural network architectures.