AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis

Summary

This paper proposes a novel view acoustic synthesis approach based on 3D Gaussian Splatting scene representation. The framework consists of a 3D GS model, acoustics field network, and audio binauralizer. It first trains a 3D GS model to capture scene geometry. Then they use attributes from each Gaussian to initialize a learnable acoustic point representation, and then average the feature obtained from nearby Gaussian points for acoustic modeling. Through the experiments on the RWAVS and Soundspaces dataset, the authors demonstrate that the proposed method achieves state-of-the-art performance. They also perform the ablation study to ablate some components to demonstrate their technical advance.

Strengths

- The paper brings up a clever way to leverage point-based scene representation from 3DGS for acoustic modeling and the results are promising. - The authors perform thorough experiments and ablations to demonstrate its technical advance. The proposed method outperforms baselines on the quantitative evaluation.

Weaknesses

- The reviewer thinks there are several issues with the writing: - First, **the method part isn't clearly stated**. It takes a long time for the reviewer to fully understand the method. The method is simple, using pretrained 3DGS to initialize a learnable acoustic point representation. While the method figure doesn't quite help with understanding. - The reviewer strongly feels the paper is **overclaimed**. It claims it learns "holistic geometry-aware material-aware scene representation" while the experiments are only for acoustic modeling results. The author needs to use experiments to support this claim. One thought the reviewer has to visualize the PCA feature of learnable acoustic points to see if it has some cues. Otherwise, the reviewer is only convinced that this paper proposed a better acoustic modeling approach that leverages the GS features. - The major experimental **results for the baselines are directly brought from AV-NeRF and INRAS papers** while the authors don't mention this in the paper. Also, the authors need to confirm if they have guaranteed the same training and test split and evaluation code. - The reviewer has a question on the Position-guidance G condition. The authors use unit vectors which get rid of distance information. The reviewer is confused as to why remove the distance. Would it be an important cue? - In the paper [5] Real acoustic field, the authors found out that energy decay loss shows a powerful improvement on those energy-based metrics. Did authors consider using this loss to improve?

Questions

I have mentioned my concerns and questions in the weaknesses part and I hope the authors could address them during the rebuttal. A suggestion to the authors for paper writing: save the figures as PDF instead of screenshots or png.

Rating

6

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes, the authors adequately addressed the limitations

Summary

This paper proposes a new audio-visual Gaussian Splatting model for novel view acoustic synthesis. The proposed method explicitly models scene geometry and learns a point-based scene representation with an audio guidance parameter on locally initialized Gaussian points that takes the space relation from the listener and sound source into consideration. This work is the first that applies gaussian splatting to the novel view acoustic synthesis problem, and experiments on two datasets demonstrate the effectiness of the proposed method compared to prior work.

Strengths

- The idea makes sense, and I was expecting gaussian splatting to be applied to this problem by somebody sooner or later, and this paper is indeed doing that. - Generally, the paper makes good attempts to apply Gaussian Splatting to this recently proposed task, a direct extension on how it is used for visual rendering. The method is clearly formulated and defined with informative notations. - Experiments are conducted on two datasets following prior work AV-NeRF [17], and it shows noticeable gains compared to prior baselines. - Many alation studies are performed including ablation on the size of vicinity w.r.t the listener and sound source position, ablation on the choice of physical parameters, which is extensive.

Weaknesses

- There is a strong claim that the proposed model is learning material-aware priors, jointly modeling 3D geometry and material characteristics, holistic scene geometry and material information, etc. Then, I would actually expect materials are explicitly modeled, such as acoustic properties of surfaces in the form of some material parameters (e.g., absorption coefficients, Young’s modulus, etc.). However, all that is modeled is just a parameter alpha that claims to encapsulate material-specific characteristics of the scene. This is somewhat misleading as there is nothing constrains alpha to be physics-based or to be correlated with some material parameters. It's likely that it’s just learning some scene representation that may be more helpful for acoustic rendering. - Also the so-called acoustic field network is also nothing about audio by itself. By acoustic field network, I was thinking the method is explicitly modeling sound propagation by taking into account the material properties of surfaces and the geometry of the space. The audio binauralizer is basically following prior work AV-NeRF [17], and what changes is what the audio binauralizer conditions on. In AV-NeRF, it’s scene features from NeRF and in this work it’s the context scene features from gaussian splatting. - There were no qualitative examples given in supplementary materials, so it's really hard to appreciate the improvement that this method leads to. - There are also many typos or grammar mistakes here and there. For example: 1. L45, 3D Gaussian splatting-based *methods*? 2. Related work, mixed use of past tense and present tense. 3. L125, a brief ??

Questions

- See questions in the weakness section above. - Also, for the SoundSpaces synthetic dataset, why it only contains 6 indoor scenes. There are more scenes in the dataset, and it would be good to explain the reason why only these six are not used. - Is there any way to interpret the learned audio-guidance parameter to show that it’s material related?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

Yes some limitations are discussed in the end, though some might be missing as discussed above.

Summary

This work proposes an efficient and performant method for novel view acoustic synthesis using Gaussian Splatting representations of 3D scenes. The proposed model integrates geometry and material conditioning, and offers a way to bridge the gap between visually-useful 3DGS representations and acoustically useful ones, using “audio-guidance” which considers the local environment in Experiments show this outperforms existing NeRF-based approaches while being more efficient.

Strengths

Overall, I think this work makes a very useful contribution. Bridging Gaussian Splatting representations with audio rendering is likely to be of significant interest to the community, and the quantitative results and efficiency seem strong. The experiments and comparisons are helpful, and give me confidence that these results are quite robust.

Weaknesses

The value of $\tau_g$ seems somewhat arbitrarily defined, and the range of tested vicinity sizes is somewhat narrow; given the efficiency of the method. It’s hard to extrapolate what the performance would be with more context (i.e. would it keep going down?). Most of my other suggestions are relatively minor, about the writing and presentation: - Abstract: “at a 3D scene” -> for clarity, should this be “within” a 3D scene? - Introduction: “visual rendering of 3D scenes without spatial audio (i.e., deaf)” -> this seems like an inaccurate characterization. Deafness is not about rendering, but about sensation. Spatiality is also not clearly related to this. - Introduction: “realistic binaural audio synthesis is challenging, since the wavelengths of sound waves are much longer” -> this is not the only reason that this is challenging. Even if you use a simplified (e.g. ray-tracing) model, which would be inaccurate at lower frequencies due to wave-like behavior etc., you would still need a good model of the environment for physics-based rendering, or a large number of spatial measurements for a data-driven approach. - Introduction: “…modeling 3D geometry and material characteristics of the visual scene objects to instigate direction and distance awareness…” -> while geometry has a direct impact on “direction and distance”, material impacts spatial perception indirectly (through absorption, reflection, and diffusion). Could this be rephrased more clearly? - Related work: “Anton and Dinesh” should maybe be “Ratnarajah and Manocha”, i.e. last names of authors? - Related work: Under “Geometry and material conditioning”, the final statement differentiates the current work from prior work by pointing out the non-reliance on scene and geometry inputs, but doesn’t clarify why this is useful. E.g. it could be pointed out that, in the real world, such inputs are often not available, so this work allows generalizing to such settings. - Figure 2: “Differential Rasterizer” -> “Differentiable Rasterizer”? - Figure 4: Could there be a subfigure showing 15%, since that seems to give the best overall performance? It would be helpful context, since otherwise the reader needs to interpolate between (c) and (d). - Section 4.4: “intuitionally” -> “intuitively”?

Questions

- Is $S$ modeled as an omni-directional point source? I assume so because point source is the typical assumption, and an emission direction isn’t given (vs. listener direction d), but it’s worth clarifying. - I think the notation here could be clearer. For example, $V_C$ and $V_A$ don’t seem used, and it’s not clear to me what their role is? E.g. why is the observation $O_p$ = ($V_C$, $V_A$) instead of ($I$, $a_{bi}$)? If the pose $p$ captures the position and orientation, what do $V_C$ and $V_A$ add on top of this representation? Why is $I$ upper-case and $a$ lower-case here, when they both represent observations? Why is $a_{bi} = \{a_l, a_r\}$ notated as a set, but other sets are notated as tuples? Additionally, shouldn’t $k_l$ be $k_L$, similar for $S$? - Conceptually, there is a tension between considering the global scene context, as the paper argues for, and dropping points outside the local vicinity of the source and listener (since sound reflections can easily include points from other regions, in principle). Could the authors comment on this? It’s interesting that the optimal vicinity size is not the largest one tested. Is there some intuition for why this might be the case, given that it offers more information? Currently, the reasoning seems to be that this contains unnecessary information, but the useful information seems to be a subset of this? - The audio rendering task is cast as a binauralization task. I understand that this is practically necessary, i.e. the input is mono and output is stereo binaural. However, wouldn’t many of the problems in this paper still exist in the monaural listener context as well? E.g. compared with a binauralization task purely based on HRTFs or simplified room geometries (e.g. Richard et al., ICLR 2021), where the context $C$ is not as rich. Could this be clarified? - How was the $\tau_g$ value decided?

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

I think the authors have fairly represented the limitations. The broader impact statement is a little bit less clear to me, i.e. I’m not sure I understand the surveillance applications. Could this be clarified? I would be interested for the authors to consider the potential impact of the limitations listed. For example, the limitation of rendering larger scenes; what practical negative impacts could this have in deployment scenarios?

Summary

The paper proposes a 3D Gaussian splatting (3DGS) based method for improving the acoustic context modeling for novel view acoustic synthesis (NVAS). The proposed model outperforms multiple existing methods on both simulated and realworld data across different metrics. The method also trains and infers faster than the state-of-the-art AV-Nerf method. The paper also provides model ablations and analyses that help in better understanding the role of different model components.

Strengths

1. The idea of using 3D Gaussian Splatting for better modeling of finegrained geometry and material aware acoustic context for improving the NVAS performance is interesting, novel and works well in practice. 2. The paper reports strong results, where the proposed model outperforms existing methods on both simulated and realworld data 3. The provided model ablations and analyses help in better understanding the contributions of different model components and design choices

Weaknesses

1. Acoustic conditioning with G_a: the paper "averages the context across all points in G_a" (L156-7). Doesn't this lead to loss of finegrained point-specific information? A learned aggregation strategy (for example, by using the attention mechanism) would probably make more sense here? 2. L192-4, "L_v ... non-overlapping": i) the insight behind doing this is not clear from the text ii) there are no references to support these steps iii) there is no ablation for this loss, as well 3. L208, "The audio guidance ... random initialization": won't random initialization during point expansion lead to local 'discontuity'? Did the authors try warm initialization to preserve local 'continuity' by doing nearest neighbor search or bilinear interporation on the existing alphas? 4. Does the model generalize to more challenging scene datasets like Matterport3D [1]? 5. Audio examples: the paper does not seem to provide audio examples and qualitiatively compare their quality with that of the baselines 6. Minor: i) L274-5, "4 perspective views ... receiver's position": do these 4 views cover the full 360 degree, otherwise it's not comparable with the field of view of the AVGS model? ii) Para in L285-90 looks like repeated text References: [1] Matterport3D: Learning from RGB-D Data in Indoor Environments. Chang et al. 3DV 2017.

Questions

Could the rebuttal comment (more) on the following? 1. how the chosen strategy for aggregating the acoustic context in G_a affects model performance. See weakness 1 for details. 2. the rationale behind using L_m, and how the model performs without the loss. See weakness 2 for details. 3. initialization of alphas during point densification in G_a. See weakness 3 for details. 4. generalization of the model to more challenging datasets like Matterport3D. See weakness 4 for details. 5. my minor comments. See weakness 6 for details. Also, could the authors anonymously provide a few audio examples from the model and baselines?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper mentions limitations in Discussion but I could not find any discussion on societal impact.