Acoustic Volume Rendering for Neural Impulse Response Fields

Summary

This paper addresses the challenge of novel sound synthesis from arbitrary positions. To achieve this, acoustic fields are modeled using an implicit neural representation, integrating acoustic wave propagation rules. Subsequently, a specifically designed volume rendering technique is employed to ensure consistency across diverse positions. Results from both simulated and real datasets validate the effectiveness of our approach. Additionally, the authors have developed a new impulse response simulation platform.

Strengths

1. It is novel to employ volume rendering to ensure consistency across different positions, offering an advantage in capturing detailed waveform characteristics of impulse responses. 2. The proposed method demonstrates superior performance compared to state-of-the-art techniques. 3. The development of a new impulse response simulation platform is a valuable contribution.

Weaknesses

1. The network is not very big (6 layers), I do not understand why the training takes 24 hours. Actually, the cited paper [29] is very fast, could the training time be reduced? 2. I enjoyed listening to the attached audio, which exhibits a strong sense of three-dimensionality. Including synthesized audios from other methods in the final Supplementary Material would enhance the comprehensiveness of the submission.

Questions

See Weakness.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

See Weakness.

Summary

This method proposes a physics-based approach for learning neural impulse response field through implicit wave propagation modeling. This is the first work that really incorporates acoustic wave propagation principles for constructing an impulse response field, leveraging volume rendering techinques in analagy to how it's applied in NeRF. There is also a new acoustic simulation platform introduced for simulating more realistic impulse respones. Experiments show that the proposed method outperforms several previous learning based methods.

Strengths

- There are many so-called audio-NeRF in prior work, but none of them actually are. This work is the most physics-based neural impulse response method I have seen, and the closest to what an audio-NeRF should look like. - The problem is well-formulated, and it's an elegant formulation to implicitly model wave propagation, and use volume rendering for synthesizing room impulse responses. - There are very informative visualizations and ablation studies that shows the advantage of the proposed method, especially on modeling the phase of the signal. - The new simulation platform introduced will also be very useful for the community, and the comparison with SoundSpaces suggests that the community should start to use more realistic simulation platforms. - Experiments on both simulated and real datasets have demonstrated the proposed method outperforms a series of prior baselines. - Generally, the paper is also very well written with sufficent details and illustrations.

Weaknesses

- The paper introduces a new acoustic simulation platform, which is claimed as one major contribution of the paper. However, in the main text, there is no detailed description of the simulator, and only says it is based on Sionna [16]. I then found it in Supp, but it would be good to also briefly summarize the key characteristics of this new simulator in the main text. - Since it's the first method that implicity models wave propagation for neural impulse response rendering, it would be important to also compare with prior physics-based methods that is geometry-based to highlight the superity of modeling wave propogation, apart from those learning based methods. For example, how would the proposed method compare to this recent work https://arxiv.org/abs/2406.07532, that models the acoustic field using an image-source differentiably method. - It would also be good to have some separate comparisons for high-frequency components vs. low-frequency components, and far-field vs. near-field to better understand the limitations and strength of the proposed framework. For example, for equation 1, the 1 / d assumption might not hold well for near field. - The qualitative examples are somewhat lacking. It would be much more informative to include some qualitative comparisons to prior work.

Questions

Apart from the questions in the Weakness section, another question I have is related to the zero-shot binaural audio experiment. The paper claims that the proposed method can even render binaural audio even without HRTFs, which is a bit confusing to me. Even if the simulator can render binarual audio by synthesizing the sound heard at the left and right ears, there are other ear/head-related effect are not factored in, so how can binarual audio be directly generated?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

There are some limitations discussed, but it would be great to hear from the authors on more discussions on limitations especially on the compute efficiency (e.g., how much time it takes for the network to converge?) and memory usage compared to prior work, so that it can give peope the complete picture on the pros and cons of the proposed framework.

Summary

This work presents a method to render a neural impulse response field to generate room impulse responses at novel positions after being trained with a number of limited RIR samples. The method works similar to a Nerf, rays are cast from a sphere and sampled at discrete positions along the rays. Here the signal at each sample point is transferred into the frequency domain and a phase shift is applied to delay them in time. The goal is to render the acoustic signal transmitted and volume density for any observer-emitter pair with known orientations and locations. The approach uses among other methods two datasets to compare itself against common baselines in the domain. The authors emphasize having the best phase reconstruction compared to other results.

Strengths

The method has great potential to correctly learn and reconstruct the phase in synthesized RIRs. It uses physical properties of wave-propagation in a very explicit way, compared to black box approaches, which can also help interpretability of the results. The work potentially provides an improved simulator and, if certain points are improved, a good alternative to current simulators. The authors remark very interesting issues with SoundSpaces, such as the time-of-flight error. They also found a very intuitive illustration for the sound pressure and phase field in a room which can again help with interpretation of results and comparison with other approaches. The figures are very clear and the method is clearly described apart from notable omissions (see weaknesses and questions).

Weaknesses

There are several odd observations and design decisions in this paper which at least should be explained. Their impact on performance is potentially great as can be probably seen in their baseline comparison. They differ by a large amount from the existing literature. Also, runtime (inference time) and impact of design decisions on it are not discussed but probably a major hurdle for this method. Without further explanation about the comparison with the state of the art and the validity of the method this contribution can not be sufficiently judged: Especially since one of the arguments is about issues with sampling, the relation to the sampling rate used should not go uncommented. The authors decide to resample impulse responses to 16 kHz yet the RAF dataset is sampled at 48 kHz. Does that mean in Table 1 all other methods are also compared at 16 kHz? In the RAF paper, Table 7 shows results for this 16 kHz case for NAF and INRAS. They combine the furnished and empty case but for example, Inras has a T60 error of 5.34. In Table 1 of this paper, Inras is described as having errors of 11.3 and 10.6 in the furnished and empty case. Why is the baseline roughly twice as bad as reported in that paper? The EDT values of INRAS seem to have an opposite relation with 0.019 sec (RAF paper) vs. 57.4 MICROseconds and 62.9 MICROseconds. Either the authors mean milliseconds, in that case the results are worse without explanation, or they actually mean microseconds, in that case the results of Inras are way better in the RAF paper also without explaining why. Most of the other results are also not consistent with the RAF paper. The author need to explain why their baseline results seem to be so much worse than in the dataset paper. If the explanation is the different split the authors use, they should explain why they did not use the split of 80%, 5%, 15% for training validation and test as used in the RAF paper or why a difference of 5% in the split leads to these vastly different results. Another reason could be cutting the samples to 0.1 s. RAF compares samples at 0.32 s. Large churches may have over 10 seconds of RT60. A typical living room may have 0.5 s RT60. Limiting the method to 0.1 s seems arbitrary. The supplementary material plays music which sounds as if it starts towards the front left and then moves behind the head. Figure 7 shows a different route of the music. I did on purpose listen first to the audio before I read the description because expectation can change this experience. It is possible to imagine the sound being in the front or the back and both being plausible. This is only my experience and a number of test-listeners would be necessary to verify my findings. However, for me the audio experience is different than real binaural recordings which make me wonder if using a HRTF can really be omitted. If the authors simply create a time-delayed signal but without shadowing effects of the head and other important factors of the HRTF this is not binaural sound, this is stereo and then the directional ambiguity in perception can be explained. The authors should go into detail why they consider this zero-shot binaural sound. This is a minor point and I does not reflect at all on the quality of the manuscript or the research but I would like to suggest to the authors to rename N_azi and N_ele to N_az and N_el to be consistent with other notations such as N_pt and avoid the distraction which could stem from reading the variable name N_azi wrong. Little mistakes: - In equation 12 the font of L_time is inconsistent with line 205 - Line 503 has one additional "frequency" word - Line 509 starts with IIn

Questions

- Why is 0.1 s chosen as duration time? - What is the inference time of this method? - What is the expected computation time if a duration of 1 second would be used? - How is the inference speed dependent on the number of rays? Optimization seems to take 24 hours for 200 epochs which is not bad but if this can only be used for very short RIRs in very small rooms or with a reduced number of rays these are important limitations which should be discussed or measured and explained. - How do the authors choose the size of the sphere they sample on? For example is it room dependent? Do they need the measurements of the room? What about complex scenes like apartments with connected rooms that are not shoebox like? - Why do the authors not compare to AV-NeRF in table 1. The results from the RAF paper could have been taken to compare against a visual-audio method if split, sampling etc. would have been the same as in the RAF paper. - What is the intuition/reson to concatenate directions only into the last 3 layers of the MLP? - What is the bouncing depth of 30? Is that meters? Is that because 0.1 * speed of sound makes bouncing beyond 30 m irrelevant? - Why is the phase important for this use case? The authors argue with VR applications. Many audio and audio-visual tasks ignore phase. If the method, after correcting the issues mentioned with the evaluation, ends up being closer to the other methods or even worse, then what is the benefit of reconstructing phase? - To compare truly with the RAF paper, what is the STFT error? Having the waveform the authors could calculate the multi-resolution STFT loss to compare over the whole dataset instead of comparing waveforms on limited examples in figure 6.

Rating

7

Confidence

4

Soundness

1

Presentation

2

Contribution

3

Limitations

The biggest limitation coming from using a duration of 0.1 seconds seems unexplained. Inference time and impact of more complex rooms or longer durations seems important but is unexplained.

Summary

The authors reformulate neural fields to model the spatial interactions of sound, by learning to predict impulse response in a fixed transmitter / arbitrary receiver pose geometry. The method substantially outperforms contemporary work on simulated and real world datasets, and lays the groundwork for new immersive AR/VR applications.

Strengths

1. The methodology is well-motivated and creative in its incorporation of acoustic / wave propagation principles in the learning algorithm 2. Performance improvements over related work are nontrivial, in both simulated and real datasets 3. The proposed method stands out in its ability to recover the phase of synthesized waves, which is crucial for many time-critical applications 4. Frequency-domain rendering is a clever way to address the challenges in fractional time delay / discrete sampling in predicting IPR 5. A new simulation platform (when released) would be a significant enabler to other researchers

Weaknesses

1. I feel that the ablation study + associated commentary is limited (see my questions in the next section). This is exacerbated by the fact that performance metrics are reported without standard deviations. I saw that the authors' rationale in the checklist for omitting error bars was budget, but I still think that expanding this section to include experiments on different network sizes, loss coefficients, etc. would increase this work's utility to researchers with different computational budgets, and better elucidate the impact of each component of the method. It would be nice to expand the commentary in the discussion with the authors' hypothesized explanations for results as well. 2. The paper lacks a crucial discussion on the computational complexity and runtime of the proposed method compared to existing approaches.

Questions

1. How does the computational complexity of your method compare to existing approaches, particularly for real-time applications? 2. In lines 282-294 (discussion of zero-shot binaural audio rendering), are there any user studies / qualitative evaluations that could characterize the zero-shot performance vs. a network optimized for that particular task? 3. In table 3, it seems counterintuitive that removing raw signal loss improves phase error.. do the authors have any insight into this result? 4. In lines 302-304, its not clear to me why frequency-domain rendering gives significantly better results than time-domain rendering.. I understood the claim that the former is more compute-efficient, but don't see the claim that this should occur in Section 3.3 as the authors state -- could the authors clarify this?

Rating

8

Confidence

3

Soundness

3

Presentation

4

Contribution

4

Limitations

The authors do acknowledge memory consumption (due to omnidirectional ray sampling) as a limitation. It would be good to expand on other potential limitations like generalization across environments, or applicability to time-critical applications.