ViSAGe: Video-to-Spatial Audio Generation

We sincerely thank Reviewer vsEj for the thoughtful feedback. --- ## vsEj-Q1. Diversity of the dataset Thank you for the suggestion. We included distribution statistics of YT-Ambigen in the Appendix G, which cover: - (a) The top-50 AudioSet label distribution predicted with PaSST [1] - (b) The top-50 COCO object class distribution of the most salient object per video with FPN [2] - (c) The center coordinates of each salient object’s bounding box - (d) The tracking of center pixels per video predicted with CoTracker [3] (randomly selected 1K samples for visibility). It is worth noting that the distribution reported in (a) is similar to that of AudioSet, which covers diverse real-world audio events. Among the 527 classes in AudioSet, our YT-Ambigen covers 314 classes, which account for 97.91% of the entire AudioSet videos. Please refer to __T5M6-Q2__ for further explanation. --- ## vsEj-Q2. Incorporating dynamic changes in rapidly changing scenes We have introduced the patchwise energy map to address this challenge, which effectively highlights dynamic changes in visual scenes. When objects move dynamically within a scene or when specific regions undergo temporal changes, these areas are represented by high energy values due to significant differences with their spatially and temporally neighboring patches. To provide further clarity, we have added qualitative examples of the patchwise energy map for video frames, alongside audio energy maps for generated audio in the Appendix F. Additionally, we have expanded our demonstration with more examples featuring dynamically moving scenes on the demo page. --- ## vsEj-Q3. Adherence to physical principles While we recognize the importance of incorporating physical principles, such as room impulse responses (RIRs), to ensure physical plausibility, our task operates in an open-domain setup with diverse and uncontrolled acoustic environments. Explicitly modeling these physical characteristics under such conditions would pose significant challenges. Instead, we adopted neural networks to implicitly capture acoustic characteristics and approximate physical principles, which are inherently embedded in ambisonics captured in the wild. Despite the lack of explicit enforcement, ViSAGe demonstrated strong adherence to physical principles when compared to baseline methods. For instance, as illustrated in Figure 5, ViSAGe avoided introducing implausible artifacts (as observed in Audio Spatialization) and did not produce nearly identical spectrograms for XYZ channels (as observed in Ambi Enc.). In future work, we aim to incorporate physical principles more explicitly into our model design and training process. For example, we plan to explore simulated environments such as SoundSpaces [4] to better integrate physical plausibility into spatial audio generation. --- [1] Koutini et al. Efficient Training of Audio Transformers with Patchout. In Interspeech 2022. [2] Lin et al. Feature Pyramid Networks for Object Detection. In CVPR 2017. [3] Karaev et al. CoTracker: It is Better to Track Together. In ECCV 2024. [4] Chen et al. SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning. In NeurIPS 2022.

We deeply appreciate Reviewer XV3v for the helpful suggestion. --- ## XV3v-Q1. Role of camera orientation parameter in enhancing spatial perception The orientation from which the visual information is captured significantly impacts the output ambisonics. As described in Sec 3.2, ambisonics capture the full three-dimensional sound field and are commonly used with panoramic videos. However, when paired with a field-of-view (FoV) video, ambiguity arises regarding the visual scene's placement within the three-dimensional space. While treating the FoV scene as a frontal view simplifies processing, it compromises the immersiveness and controllability of ambisonics generation since all sounds appear to originate from directly in front of the listener. To address this, we introduce a camera orientation parameter as an additional condition that specifies the visual scene's position within the three-dimensional sound field, enabling proper audio-visual spatial alignment. In practice, the camera orientation parameter guides the directivity of spatial audio generation. For instance, in a orchestra recording, if the camera faces front, the audio originates primarily from the front. If the camera turns left, the audio follows, enhancing spatial realism. To further illustrate this mechanism, we have added qualitative examples of the camera orientation parameter’s effect in Appendix F. --- ## XV3v-Q2. Influence of visual representations with stronger local perception (DINOv2) instead of CLIP The performance of using DINOv2 for visual representations is reported below. We followed the same training procedure explained in Section 5 other than using DINOv2 instead of CLIP. Using DINOv2 is not beneficial to performance for both semantic and spatial metrics. To ensure that the result is not confined to YT-Ambigen, we conducted identical experiments with VGGSound, a well-established benchmark for video-to-audio generation. DINOv2 as visual representations is also detrimental to performance in VGGSound, degrading both FAD (3.62 → 4.33) and KLD (2.23 → 2.25). This performance degradation potentially has to do with differences in dataset scales and pretext tasks during pretraining. | | FAD$_\text{dec}$ | KLD$_\text{dec}$ | FAD$_\text{avg}$ | CC$_\text{All}$ | CC$_\text{1fps}$ | CC$_\text{5fps}$ | AUC$_\text{All}$ | AUC$_\text{1fps}$ | AUC$_\text{5fps}$ | |--------|:----------------:|:----------------:|:----------------:|-----------------|------------------|------------------|------------------|-------------------|-------------------| | CLIP | 3.74 | 1.77 | 4.04 | 0.524 | 0.482 | 0.439 | 0.778 | 0.757 | 0.734 | | DINOv2 | 4.06 | 1.79 | 4.30 | 0.484 | 0.447 | 0.406 | 0.759 | 0.739 | 0.788 |

We greatly appreciate Reviewer NiUt for the constructive feedback. --- ## NiUt-Q1. Subjective test results Thank you for the feedback. We conducted human preference analysis with two-sample hypothesis testing of generated audio with respect to four subjective criteria: - __Naturalness__: Which audio sounds more natural? - __Relevance__: Which audio is more closely related to objects and surroundings in the video? - __Spatiality__: After observing different viewpoints of a 360° video by rotating, which audio better captures the spatial effects perceived in both ears? - __Overall preference__: Which audio do you prefer overall? Due to the characteristics of 360° videos and spatial audio, we recruited 12 participants in person instead of crowdsourcing (e.g., MTurk). Each annotator evaluated an average of 15 videos out of 30 randomly selected samples from the test split. The results are summarized below, showing that our samples are generally preferred over the prior arts across all four criteria. It is worth noting that the gap is particularly large for the spatiality criterion. |(a)SpecVQGAN|Win|Tie|Lose| |---|:-:|:-:|:-:| |Natural |43.33|25.56|31.11| |Relevant|50.00|27.78|22.22| |Spatial |52.22|31.11|16.67| |Overall |50.00|23.33|26.67| |(b)Diff-Foley|Win|Tie|Lose| |---|:-:|:-:|:-:| |Natural |44.44|14.44|41.11| |Relevant|40.00|23.33|36.67| |Spatial |42.22|30.00|27.78| |Overall |44.44|16.67|38.89| --- ## NiUt-Q2. Curating a clean subset of test split Thank you for the suggestion. We have applied a tighter set of conditions on the test split of YT-Ambigen to select 1.5K samples (i.e., about 15% of the test split) with improved cleanliness as _mini-test_: - Higher CAVP and PaSST filtering thresholds for improved audio-visual correspondence and audio event likelihood, respectively - Sampling ambisonics with clearer directivity by measuring inter-channel correlation. Using the mini-test set improves KLD (1.71 → 1.48), CC (0.635 → 0.718 ), and AUC (0.846 → 0.897) at the cost of FAD (3.86 → 6.10). This implies that a distribution shift may occur during the selection of the semantically localized subset, as the sound becomes less diffused, thereby enhancing the directivity-related metrics. We believe this mini-test split will also be useful for analyzing other aspects of the benchmark. As such, we will publicly release the mini-test split. --- ## NiUt-Q3. More intuitive qualitative examples We included qualitative examples from the clean subset explained above in our demo page. Please refer to __T5M6-Q3__ for additional qualitative examples. --- ## NiUt-Q4. Missing relevant references Both arXiv papers were released in late October, which is after the ICLR submission deadline. A key difference from our approach is that they use synthesized spatial audio for conditional audio generation, while we leverage real ambisonics captured in the wild. We will add these references in our final draft.

We sincerely thank Reviewer T5M6 for the thoughtful feedback and valuable suggestions. --- ## T5M6-Q1. Quality control of the dataset We established the reliability of our dataset's quality in two aspects. First, the quality of audio generated using YT-Ambigen is in line with well-established large-scale benchmarks, e.g., VGGSound. As reported in Table 2, the FAD (3.95 vs. 3.62) and KLD (1.77 vs. 2.23) scores of our model trained with YT-Ambigen's mono channel are similar to those of VGGSound. These metrics from our VGGSound-trained model are also comparable to the prior arts like SpecVQGAN and Diff-Foley, suggesting that YT-Ambigen offers a reliable data source for benchmarking video-to-audio generation. Second, using a combination of performant off-the-shelf multimodal discriminators for filtering is a well-established practice for constructing large-scale datasets with quality control [1, 2, 3], and could be more effective than manual filtering in some cases. For instance, in Table 2, our predecessor in discriminative audio-visual reasoning (YT360) reports an FAD metric of 15.91 for video-to-audio generation, despite being collected through manual filtering. --- ## T5M6-Q2. Distribution statistics of the dataset Thank you for your suggestion. We included the distribution statistics of YT-Ambigen in the Appendix G, which cover: - (a) The top-50 AudioSet label distribution predicted with PaSST [1] - (b) The top-50 COCO object class distribution of the most salient object per video with FPN [2] - (c) The center coordinates of each salient object’s bounding box - (d) The tracking of center pixels per video predicted with CoTracker [3] (randomly selected 1K samples for visibility). Our audio distribution is similar to that of AudioSet, where YT-Ambigen covers 314 out of 527 classes in AudioSet, accounting for 97.91% of the entire AudioSet videos. Moreover, the semantic, spatial, and temporal distributions of the most salient object per video are summarized in (b-d). These objects cover 79 out of 80 classes in COCO. Moreover, they are located in diverse positions within the field of view and often move around during five-second segments, creating more challenging scenarios for video-to-ambisonics generation. --- ## T5M6-Q3. More examples covering challenging scenarios We included more qualitative examples with non-centered or moving objects in our demo page and Appendix F. --- ## T5M6-Q4. Codebook generation with nine residual codebooks We apologize for the confusion. As shown in the legend of Figure 2, each block represents a codebook group rather than an individual code. All residual codes from the selected groups are generated at each sequence step. For example, with 9 RVQ codes per channel: - Step 1: 1 code from $W_p$ - Step 2: 8 codes from $W_r$ and 3x1 from $S_p$ (total: 11) - Step 3: 1 code from $W_p$ and 3x8 from $S_r$ (total: 25) - and so on. The figure caption has been updated to clarify this process. --- ## T5M6-Q5. Typos and visualization suggestions Thank you for your suggestions. We fixed the typos and updated visualizations in Figure 3 and 5. --- [1] Lee et al. Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning. In ICCV 2021. [2] Nagrani et al. Learning Audio-Video Modalities from Image Captions. In ECCV 2022. [3] Wang et al. A Large-scale Video-Text Dataset for Multimodal Understanding and Generation. In ICLR 2024.

We deeply appreciate Review Xpfx’s constructive feedback. --- ## Xpfx-Q1. Space-time complexity analysis The number of parameters and the inference time for each model are summarized below. Inference time is computed end-to-end for 320 samples using a batch size of 32, including all auxiliary computations to generate outputs like the Griffin-Lim Algorithm in Diff-Foley [1]. Our framework is up to 1.7x faster than prior arts while using similar or fewer parameters, presumably due to fewer decoder layers and the vocoder-less design. |Model|Trainable Parameters(M)|Overall Parameters(M)|Inference Time(s/it)| |--------------|--------------------------|-------------------------------------------------------------|-----------------------| |SpecVQGAN|307.0|ResNet50 (23.5) + Transformer (307.0) + VQGAN Decoder (42.8) + MelGAN (4.3)|3.830| |Diff-Foley|859.5|CAVP (32.7) + LDM (859.5) + Latent Decoder (49.5) + Guidance Classifier (11.7)|3.996| |Spatializer|80.5|CLIP (87.9) + U-Net (80.5)|0.043| |ViSAGe|358.6|CLIP (87.9) + Transformer (358.6) + DAC Decoder (56.5) |2.289| --- ## Xpfx-Q2. Adapting cross-modal contrastive learning as in VGG-SS To perform cross-modal contrastive learning in source localization tasks as in VGG-SS, the model requires ground-truth audio inputs, which are not available in video-to-audio generation tasks. One conceivable strategy to utilize knowledge from cross-modal contrastive learning (e.g., [1] or [2]) without ground-truth audio is to apply reranking among generated audio candidates, i.e., selecting the candidate with highest score as a positive. Model performance under diverse backbones are reported below. Although it takes several times longer to generate outputs, the quantitative metrics remain virtually unchanged with reranking among 10 candidates. | | FAD$_\text{dec}$ | KLD$_\text{dec}$ | FAD$_\text{avg}$ | CC$_\text{All}$ | CC$_\text{1fps}$ | CC$_\text{5fps}$ | AUC$_\text{All}$ | AUC$_\text{1fps}$ | AUC$_\text{5fps}$ | |---------------|:----------------:|:----------------:|:----------------:|-----------------|------------------|------------------|------------------|-------------------|-------------------| | w/o Rerank | 3.86 | 1.71 | 4.20 | 0.635 | 0.584 | 0.531 | 0.846 | 0.819 | 0.790 | | Rerank (CAVP [1]) | 3.85 | 1.71 | 4.25 | 0.635 | 0.581 | 0.526 | 0.846 | 0.818 | 0.788 | | Rerank (FNAC [2]) | 3.85 | 1.70 | 4.26 | 0.632 | 0.580 | 0.526 | 0.845 | 0.817 | 0.788 | --- [1] Luo et al., Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models. In NeurIPS 2023. [2] Sun et al., Learning Audio-Visual Source Localization via False Negative Aware Contrastive Learning. In CVPR 2023.

Dear Authors, Could you please address the following concern: Is it true that a significant portion of the examples used for demonstration purposes were directly sourced from the training dataset? If examples from the training set were used in the demo, it could raise concerns about the reliability of the results and the model's true capabilities. Best, Area Chair

Dear Yang Liu, Thank you for your interest and reporting this issue! Except for the four mistakenly included samples, we want to reassure that all our experimental analysis and conclusions in our paper are indeed fair and accurate, which are manifested through multiple demo examples that were unseen during training and through extensive experiments conducted under rigorous conditions. After thorough inspection of all qualitative samples, we confirmed that 4 out of 16 samples on the demo page were from train split and replaced these with additional examples from the validation or test splits. Best Regards, Authors