Self-Supervised Visual Acoustic Matching

Summary

The authors proposed a self-supervised approach to match acoustic conditions via visual information without the need of paired audio-visual data. With this approach, the in-the-wild web data or simulated data can be utilized. They show that this approach can outperforms the state-of-the-art on multiple datasets.

Strengths

- The proposed approach is novel in several aspects, 1. does not require paired audio-visual data, 2. Able to leverage more data from the web or from simulation. - The authors provide human perception study, which is important for their claims in perception accuracies. - This technique has potential applications in various audio domains such as domain adaptation, help improving in-the-wild acoustic event classification, etc.

Weaknesses

- The writing of this paper is not easy to follow and missing content, for example in line 200 it refers to Section 4 for details in "RT60 to allow generalization", which does not existing in Section 4, mostly focusing on datasets. In line 226, the off-the-self dereverberator is referred to Section 5, which is difficult to locate for the readers. - In Section 5, for all the implementation details, suggest the authors to organize and highlight with a table, what data is used to train in which step. Currently it is not easy to follow in the scripts and have clear understanding of the experimental designs. - The paragraph of line 188 seems to appear in the wrong place, referred Figure 4 is in the results section, while the narrative appears in the approach section.

Questions

- The use of RT60 needs more explanation, is the intuition to model just the acoustic environment and disentangled with the content involved? - Why is Table 1 on the far right, train with AVSpeech-Rooms and test with LibriSpeech does not include comparisons of STFT?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

- No potential social or ethical implications.

Summary

The paper introduces a self-supervised method for visual acoustic matching (VAM), where the training samples consist of only the target scene image and audio. The paper is well-written and easy to follow. The experimental results have validated the effectiveness of the proposed approach. We have the following comments: (1) The proposed system is somewhat simple as it incorporates existing methods. Therefore, the scientific depth presented in the paper may not be up to the standards presented at the prestigious conference NeurIPS. (2) It would be beneficial to include an analysis of the dereverberator. The authors can evaluate the dereverberator's performance using SRMR with different levels of reverberation. It would also be interesting to see the results with and without the De-biaser component based on SRMR. (3) Please provide results for environments with mild, moderate, and severe reverberant conditions. This will help demonstrate the effectiveness of the LeMARA approach more clearly. (4) It is crucial to showcase the impact of the De-biaser component. In addition to the spectrograms presented in Figure 4, I recommend that the authors perform additional experiments using supplementary metrics like SRMR or a readily available ASR model. This will provide further evidence and insight into the effectiveness of the De-biaser approach. (5) It would be informative to report the results of the LeMARA approach separately for seen and unseen images. (6) Figure 4 shows that LeMARA exhibits greater variation in RT60 compared to AViTAR. Please provide an explanation for this observation. (7) We suggest preparing an anonymous website where several sets of sound samples of the source, target, AViTAR, and LeMARA can be presented. (8) Figure 5 indicates that AViTAR outperforms LeMARA in the left-down example. Please provide an explanation for this result.

Strengths

The paper introduces a self-supervised method for visual acoustic matching (VAM), where the training samples consist of only the target scene image and audio. The paper is well-written and easy to follow. The relevance of the research task holds significant value in AR/VR systems and can serve as a crucial element. The experimental results have validated the effectiveness of the proposed approach.

Weaknesses

Some analyses, which can further demonstrate the effectiveness of the proposed approach, are missing, for example: (1) The authors can evaluate the dereverberator's performance using SRMR with different levels of reverberation. It would also be interesting to see the results with and without the De-biaser component based on SRMR. (2) Please provide results for environments with mild, moderate, and severe reverberant conditions. This will help demonstrate the effectiveness of the LeMARA approach more clearly. (3) It is crucial to showcase the impact of the De-biaser component. In addition to the spectrograms presented in Figure 4, I recommend that the authors perform additional experiments using supplementary metrics like SRMR or a readily available ASR model. This will provide further evidence and insight into the effectiveness of the De-biaser approach. (4) It would be informative to report the results of the LeMARA approach separately for seen and unseen images. (5) In Line 30, there is one reference missing. (6) In Fig. 2(b), D(G(A_T)) should be D(G(A_t)).

Questions

(1) A critical point is to showcase the effectiveness of the De-biaser component. Please provide additional experiments to demonstrate its performance. (2) It s interesting to see the achievable performance of LeMARA approach under mild, moderate, and severe reverberant conditions. (3) It is crucial to report the results of the LeMARA approach separately for seen and unseen images.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

(1) The proposed system is somewhat simple as it incorporates existing methods. Therefore, the scientific depth presented in the paper may not be up to the standards presented at the prestigious conference NeurIPS. (2) Figure 4 shows that LeMARA exhibits greater variation in RT60 compared to AViTAR. Please provide an explanation for this observation. (3) Figure 5 indicates that AViTAR outperforms LeMARA in the left-down example. Please provide an explanation for this result.

Summary

This paper addresses the task of "visual acoustic matching" (VAM): taking a source audio clip and target visual environment (i.e. an image), and modifying the source audio clip such that it sounds like the clip was recorded in the target environment. The paper proposes a self-supervised approach for training neural networks to solve this task, which can train on examples that only contain the target audio and target visual environment. The idea is to disentangle the room acoustics from the audio content, such that the source room acoustics can be removed, then an audio-visual model can transform this audio to match the target visual environment. The method uses a conditional GAN framework, where an "acoustic residue" metric is defined that measures the discrepancy between audio-only and audio-visual reverberation. The acoustic residue metric is used to train a discriminator that can be used to train a "de-biaser" generator model that strips additional reverberation from an initially dereverberated signal. The output of the de-biaser can then be fed to the audio-visual reverberator. Although the method can theoretically be applied to other types of signals, the paper focuses on clean speech as the audio signals. Datasets used are SoundSpaces-Speech (reverb impulse responses (RIRs) simulated from Matterport3D scans of homes with 3D-rendered human at source location and audio from LibriSpeech) and AVSpeech-Rooms (YouTube videos that mostly feature a single speaker with little background noise). Objective metrics are used to measure performance: MSE between magnitude spectrograms of predicted and ground-truth speech, and MSE between RT60 estimates of predicted and ground-truth speech. The proposed method achieves better objective scores compared to comparable methods. A perceptual study is also done, which shows users could identify the room from the proposed method 46.1% of the time, versus 34.7% for a baseline method.

Strengths

**S1)** The method provides a means of training self-supervised models for VAM. **S2)** Evaluation includes a human perceptual study. **S3)** Demo video is clear and helps with understanding the method. Also, thanks for providing audio demos, they are very useful for evaluating the method and how it compares to the AViTAR baseline.

Weaknesses

**W1)** The evaluation of the method is weak. Watching the demo video, it seems like this is certainly a difficult task to evaluate, since the effect can be rather subtle. Nevertheless, I think the evaluation metrics could be improved. First, there are better options than just measuring MSE on magnitude spectrograms. Human hearing is logarithmic, so MSE on linear magnitude is a poor match to human perception. An easy alternative is to measure MSE between log spectrograms, although this encounters an issue with where to set the floor on the log to avoid -infinity. A solution to this is to use magnitude raised to a power (e.g. 0.3), which approximates the log, but also goes to 0 where the magnitude is 0. This may help this this issue: "Models that use this dereverberator without further de-biasing will display artificially low STFT error when evaluated in-dataset" Second, there are other important properties of reverb impulse responses besides RT60, such as direct-to-reverberant ratio (DRR). DRR measures the ratio of energy of the direct path to the energy of the reverberant part, and can be an important cue for distance of the source. Also, distance of the source is not really discussed, and DRR is a crucial property to help measure this (see W3). Third, the comparison of the RT60 distributions in Figure 4 are not very convincing, because they are just box plots, and it's not clear how the proposed approach "more closely matches the ground truth target distribution.": the median is closer to ground-truth than the baseline, but it seems like a different type of plot would allow more detailed comparison of the distributions, like histograms or violin plots. **W2)** The paper restricts its focus to a clean single speaker, which suggests the method may be limited on real-world data, which can contain a great variety of non-speech sounds, and also multiple speakers. I don't think the proposed method could be applied directly to these scenarios, because all sounds would need to first be separated, then each processed by the pipeline (i.e. each sound dereverberated and re-reverberated) **W3)** The paper does not discuss the effect of distance of an object, nor the effect of distance on the reverberation. In particular, the DRR is an important property of reverberation that is totally ignored in this paper. At the very least, assumptions about how the method handles distance and location of sources needs to be clearly described and discussed, and also I think evaluation metrics should take distance into account (see W1). **Minor comments and typos** a) "data, we" -> "data, and we" b) "target space [?,..": update missing ref c) "In-the-wild Web" -> "In-the-wild web" d) "We focus on human speech in indoor settings": clearly specify that it's clean speech, without background noise e) "this leaves signals of the target environment": I think "residuals" would be better than "signals" f) "Unlike SRMR, DNSMOS [31], or any existing off-the-shelf metric that quantifies dereverberation,": I don't think DNS-MOS is trained to quantify reverberation specifically, it's overall quality, which can be affected by other properties such as background noise and/or artifacts. May be good to adjust this description

Questions

Q1) Looking at the spectrograms of de-biased audio in Figure 4 and listening in the demo video, de-biased audio seems like it is applying fairly aggressive suppression, perhaps removing some energy of the anechoic audio. This seems like a weakness of the method. Are there mechanisms to prevent over-suppression of the anechoic audio? Does this suggest a trade-off for the de-biaser between dereverberating and suppressing signal? Some discussion of this would be helpful. Q2) In Figure 2, where does "A" come from (lower right panel)? I guess this is the de-biased audio? Would be good to make this more clear

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Limitations described adequately.

Summary

This work proposes a method for visual acoustic matching, the task of processing an audio signal so the room acoustics are perceived as originating from within a certain room based on an image. While paired data is generally required for this task, the proposed approach is self-supervised and trained using “in-the-wild” videos of speakers in various rooms. To achieve this, a dereverberation system along with a de-biaser, a visually-conditioned reverberator, and a blind reverberator are used. The de-biaser is trained by optimizing a novel acoustic residue metric that measures the relative difference between the amount of acoustic information in two recordings. While similar to dereverberation, this de-biasing process can be seen as post-processing step that aims to further reduce information in the dereverberated output so the visually conditioned reverberator model must use cues from the supplied image to match the target room acoustics. To use this acoustic residue metric as an objective function, the de-biaser is trained in a MetricGAN fashion where the discriminator is trained to regress the true metric value. Two evaluation datasets are used and results indicate that the proposed method outperforms existing baseline systems. In addition, a human perception study also validates these results.

Strengths

1. The proposed method is original and focuses on combining multiple existing approaches along with a novel acoustic residue metric in order to achieve a system that can be trained without paired data. These approaches include the use of various pretrained submodules, including a dereverberation model and a RT60 estimation model, as well as the MetricGAN approach that enables the use of the acoustic residue metric during optimization. 2. As outlined by the authors, existing approaches are often greatly limited by the availability of paired data. Not only is the proposed approach capable of being trained without paired data, the evaluation indicates superior performance to existing methods.

Weaknesses

1. Some relevant references for work in audio-only acoustic matching are omitted. Works such as [1] are similar in nature but rely on conditioning from an audio signal instead of an image. Major components of the reverberator architecture are similar as well, such as the use of a WaveNet architecture for audio processing. This would be highly relevant in the context of the blind reverberator in this work. Also, other audio only approaches such as [2] would be relevant to include in the context of related work. It would also be relevant to address other GAN-based reverberation models such as [3, 4]. 2. While overall the manuscript is well organized and clear, further refinement with regards to the explanation of the process for training the complete system may help reduce the burden for readers. For example, perhaps an algorithm listing each step of the process and referencing the variables used in the text would be helpful. 3. While the evaluation considers two datasets and appears relatively sound, it is still somewhat difficult to understand the relative improvement afforded by the proposed method in comparison to the baselines. 4. While the authors are commended for including a human perception study, the design of the study may be somewhat flawed, making it difficult to make any conclusion from the results. [1] Su, Jiaqi, Zeyu Jin, and Adam Finkelstein. "Acoustic matching by embedding impulse responses." ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020. [2] Steinmetz, Christian J., Vamsi Krishna Ithapu, and Paul Calamia. "Filtered noise shaping for time domain room impulse response estimation from reverberant speech." 2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2021. [3] Ratnarajah, Anton, Zhenyu Tang, and Dinesh Manocha. "IR-GAN: room impulse response generator for speech augmentation." INTERSPEECH (2021). [4] Ratnarajah, Anton, Zhenyu Tang, and Dinesh Manocha. "Ts-rir: Translated synthetic room impulse responses for speech augmentation." 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021.

Questions

Is it fair to call the proposed approach “self-supervised”? While the approach does not require paired data directly, it appears that paired data is indirectly required since pretrained models, such as the RT60 estimator and dereverberation models. In other words, it appears that it would not be possible to train the proposed system in a fully self-supervised manner from scratch, without these pretrained modules, which in fact use labeled/paired data. Instead, it may be more accurate to label the proposed method as one capable of using an “unpaired” dataset instead of self-supervised. Perhaps terms such as “semi-supervised” or “weakly-supervised” are more appropriate? What is the motivation for using the MetricGAN approach to construct a differentiable proxy of the acoustic residue metric $\mathcal{M}$? Based on the text, it appears that the RT60 estimation model $\mathcal{RT}$ is implemented with a neural network and hence differentiable. While (3) involves non-differentiable functions such as the absolute value and $\max$, these are both approximately differentiably, indicating that it may be possible to directly backprop through the metric $\mathcal{M}$. Is this something the authors considered? If so, was the failure of this approach the motivation to use the MetricGAN approach? Some discussion on the motivation for the MetricGAN approach would be beneficial. Line 30: A reference has not been rendered properly “?”

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

It could be beneficial to include not only the mean RT60 error (RTE (s) as reported in Table 1, but also the bias and correlation coefficient, which is common practice in RT60 estimation evaluation. This can help to establish how the model is failing especially if the test dataset has a non uniform distribution of RT60 targets. One limitation of the use of the RT60 estimation in the proposed acoustic residue metric is that this measurement appears to consider only the broadband RT60. It is generally accepted that in most rooms the RT60 can vary quite significantly across frequency, for example, with the higher frequencies decaying at a significantly faster rate than the low frequencies (generally due to absorption from the material properties). As a result, this variation of RT60 across frequency often provides significant insight into the character of the room, which is often detectable by a human listener. Therefore, using only the broadband RT60 in the acoustic residue metric may lead to some cues about the bandwise RT60 remaining in the de-biased recording as long as the broadband RT60 is perturbed. Future work could consider a band-wise RT60 estimation.

Summary

This work aims to perform acoustic matching with unpaired training data, i.e. observing the audio only in the target environment without samples of the same audio in the source environment. The basic process involves the common process of dereverberator and reverberator. But one key point in this work is to introduce an acoustic residue metric to measure the residual information in a waveform to ensure the quality of dereverberated audio. Finally, experiments on SoundSpaces-Speech and AV Speech show its effectiveness.

Strengths

The presentation is very clear. The motivation to tackle the task is also intuitive and meaningful. The method is reasonable. I like this smart and simple manner proposed in this paper to tackle the key limitation in visual acoustic matching.

Weaknesses

(1) Is it limited to evaluate with only two metrics, STFT and RTE? (2) Although the number of methods using unpaired data may be few, the number of methods using paired data should be large. There could be some comparison with them to show the performance level of the proposed method. For example, comparing on other paired datasets with existing methods which use paired data could show the difference of the methods use and not use paired data.

Questions

(1) The error of STFT changes obviously in AVSpeech-Rooms compared with other settings, i.e. about 6~7 vs 1~2, what’s the possible reason? I t would be nice to give the clarification. (2) Two questions about the evaluation metrics and the comparison are as listed in weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

yes.

Response to rebuttal

Thanks to the authors for the detailed responses and rebuttal PDF. > If we have misunderstood the suggestion, we’d welcome a pointer on how to allow DRR as a reliable metric when one lacks ground truth RIRs. I guess I was thinking of blind DRR estimation, which is certainly not perfect, but could provide some insight. A number of blind DRR methods have been proposed that could be used, even from a single microphone. There is this classic Matlab implementation: https://www.mathworks.com/matlabcentral/fileexchange/32752-blind-direct-to-reverberant-energy-ratio-drr-estimation And here is a more recent paper on blind DRR estimation: https://www.isca-speech.org/archive_v0/Interspeech_2020/pdfs/2171.pdf But in any case, I appreciate the additional results on DRR versus distance. > We now provide histogram and violin plots of the source, predicted, and target RT60 distributions (Figures 9-12 in rebuttal pdf) Thanks, that's definitely an improvement. > We have added in log magnitude STFT loss as an additional metric in Tables 5, and 6 in the rebuttal pdf. I would still encourage the authors to consider MSE on magnitude spectrograms raised to the 0.3 power, i.e. $|X|^{0.3}$. The authors have addressed most of my concerns, so I am willing to raise my score.

Response to rebuttal

The authors have well addressed our concerns. We decided to raise our score.