Learning Spatially-Aware Language and Audio Embeddings

Summary

This paper presents an approach to learning spatially aware language representations. The authors propose a contrastive representation model that integrates spatial context into language representations, aiming to enhance the performance of tasks that require spatial reasoning. The model combines visual and textual data to create embeddings that are sensitive to spatial relationships. The contributions include an architecture of the proposed model, extensive experimental results demonstrating improved performance on spatial reasoning benchmarks, and an analysis of the model's ability to generalize across different spatial contexts.

Strengths

1. The paper makes a strong contribution to spatial reasoning. The integration of spatial context into text-to-audio generation models is an important and underexplored area, and this work offers a novel and effective solution. 2. The experimental setup is rigorous, with well-designed experiments that effectively validate the model's performance. 3. The paper is well-written, with clear explanations of the methodology and results. 4. The findings have significant implications for improving spatial language understanding in various applications.

Weaknesses

1. The reliance on synthetic datasets may limit the generalizability of the findings. The authors could explore the way to train the model on in-the-wild data. 2. The current interpretation experiments (Sec. 5.4) only study a four-class classification ("left," "right," "up," "down"), which is insufficient for real-life scenarios. For instance, spatial audio applications often require more nuanced classifications, such as distance perception (e.g., strong/weak reverb in indoor/outdoor settings), which are critical for capturing and representing spatial information. The authors should consider extending the experiement to handle a wider range of spatial attributes to enhance its applicability in diverse settings. For example, the authors should consider using prompts like "xxx is making a sound in the distance" and "xxx is making a sound nearby" to figure out if the results are different. 3. The paper could benefit from a more detailed error analysis, identifying common failure cases and understanding why the model fails in certain scenarios. This analysis would provide insights for further improvement and refinement of the model. 4. While the model performs well on tasks like retrieval and source localization, its ability to generalize to spatial text-to-audio generation remains to be seen.

Questions

See weakness.

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors have addressed the limitations of their work by discussing the datasets used and the model's computational requirements. Additional limitations can be found in the Weaknesses section.

Summary

This paper describes a method for learning to represent spatial audio (and text). The proposed model is trained on synthetically spatialized audio data with corresponding text prompts. The authors evaluate the system on audio captioning/retrieval and localization tasks, showing that the proposed model effectively represents both semantics and locality of audio.

Strengths

The paper is well written, clearly organized, and easy to follow. The proposed method makes intuitive sense and appears to be effective. The empirical evaluation is reasonably thorough and the choice of tasks and baseline models seem appropriate. Spatial audio is a growing area (in the context of machine learning / event detection / representation learning), and I think the work described here does fill an unaddressed area of the literature in an interesting way. Overall I think they did a great job here.

Weaknesses

I don't have much to fault here, but there are a few points that I think could be expanded to improve clarity and help readers understand the contribution here. I'll elaborate on these points in the questions section, but the high-level gloss is: - While the spatial representation part of the work (ie FOA-derived input) is explained well, there is almost no explanation of how the spatialization was implemented. - There is little (if any) qualitative analysis of the results, only aggregate scores reported in tables.

Questions

- How was the spatialization implemented? I expect this was done via standard methods (ISM implemented by pyroomacoustics? something else?), but there is no mention of this in either the main text or the appendix. Additionally, I think some of the details from the appendix (Table A.2) should be mentioned directly in the main text, such as the range of room sizes, angle distributions, etc.; these details are important, and do not take much space. (If you do need to sacrifice anything, I don't think the definition of log-mel transformation is as critical to include since it is standard.) - Since TUT2018 is a finite vocabulary dataset, it would be informative (and entirely possible) to see a per-class and per-environment breakdown of the evaluations reported in table 1. This would be informative because it's not necessarily a given that your spatialization is equally effective across categories (or rooms / room sizes). If the model does turn out to perform consistently across categories - great! If not, it may suggest a weakness in either the spatial rendering or prompt generation. (If you do compute these results, it may or may not make sense to store in the appendix, depending on how interesting the results are.)

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The limitations sections seems sufficient to me. One potential caveat here is that the authors do not explicitly mention any limitations imposed by the accuracy of the spatialization process, e.g., whether it will only work well for simulated closed environments (shoebox model) or if it can accurately capture open environments. This I think would be easy to resolve with a bit more information about the process and an extra line the limitations (if necessary).

Summary

The paper presents ELSA (Embeddings for Language and Spatial Audio), a novel model designed to learn spatially-aware audio and text embeddings using multimodal contrastive learning. The primary aim is to address the limitations of existing audio foundation models, which lack spatial awareness, and sound event localization and detection models, which are constrained to a fixed number of classes and absolute positional descriptions. The authors spatially augment several classical open-source audio datasets in order to train ELSA. Results show that ELSA is able to capture spatial attributes and semantic meaning of the audio.

Strengths

- The focus of this paper is on learning spatial audio embeddings associated with natural language description, which is a very interesting and rewarding problem for which there is a lack of models. - These authors synthesize large amounts of non-spatial audio data under various spatial configurations, which is a valuable contribution to the field of spatial audio understanding.

Weaknesses

- For this paper, my biggest concern is the generalizability of the model to real scenarios. While the synthetic dataset is extensive, there is a risk that the model might not generalize well to real-world scenarios due to potential biases in simulated environments. To show the performance of model generalization to real scenarios, the experiments only on a small real-world dataset appear too thin. Would it be possible to test ELSA in other real scenarios, for example, in some of the tasks in the latest DCase competition, e.g. Sound Event Localization? - For paper writing, too much important information is put in appendices, such as the structure figure of the whole model. Perhaps the layout of the writing could be adjusted to make it easier to read. - The citation format of the article is highly problematic and needs to be standardized.

Questions

The experiments in Table 2 confuse me a lot. In Sec. 5.2, the authors mentioned that "The ELSA text embeddings for such captions are extracted from the pre-trained encoder and compared in a zero-shot fashion with ELSA audio embeddings for samples from the test set using cosine similarity. We classify the match as correct if the spatial attribute in the closest audio sample matches the spatial attribute of the query caption". However, the number of classes of a spatial attribute is very limited (For instance, there are only two classes "far" and "near" for the "distance" attribute), which means there are only two captions that will be used for the "distance" attribute? Wouldn't there be very few captions being used for testing totally? Hopefully, the authors can explain the experimental configuration a bit more. - To train ELSA on single-channel audio, the authors repeat the single channel 4 times to fake a 4-channel FOA audio and compute Intensity Vectors. However, the way IV is calculated possibly doesn't make sense for this kind of faked 4-channel audio. Why is it designed this way? Why not try to design a separate feature extractor for single-channel audio? - It is natural to understand computing bearing information from spatial audio, which is essentially a bit similar to calculating the "Time Difference of Arrival" based on different channels. But how to understand that the model can get distance information from spatial audio? In other words, where does the information about distance come from?

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

- As the authors mentioned, for creating a spatial audio caption dataset, using LLM to rewrite the caption might lead to hallucinations. - Model performance in real scenarios is yet to be verified. - ELSA looks very suitable to be used as a spatial audio encoder for a caption model to conduct spatial audio captioning, but unfortunately, the authors did not show this kind of capability in the paper.

Summary

The paper presents ELSA (EMbeddings for Language and Spatial Audio), a spatially aware-audio and text embedding model. The training data is created by synthesizing spatial audio in ambisonic format and augmenting text captions with spatial information. A small real world data is also collected for evaluations. The model training itself largely follows standard CLIP/CLAP training by using contrastive losses. Additional losses for direction and distances are added for the spatial part. Evaluations are done on both semantic retrieval tasks and spatial tasks.

Strengths

– The paper addresses a key part of multimodal contrastive embeddings. Sounds contain a significant amount of spatial information and humans naturally rely on directional information from sounds. Considering this it is expected that embeddings with spatial information are created. The paper is a good step in the right direction. – For the most part, the paper is well done. Spatial audio can present several challenges with respect to data (more so in multimodal settings, training approach). Considering the challenges around learning from spatial audio, the paper presents a good approach for learning spatially-aware language embeddings. The experiments are also reasonably good. – The paper is also well written and mostly clear. ------- Score increased after rebuttal.

Weaknesses

There are a few weaknesses which are worth addressing in the paper. – For table 2, I would be curious to see what CLAP on its own can achieve. It would be good to contrast this zero-shot classification on the spatial task. – How were the non-spatial audio-text pairs used in training (as shown in Table 3, last row) ? – Using non-spatial audio-text seems crucial for good semantic retrieval. This is evidenced by A.6 as well where the models training on just spatial audio-text pairs do not do well on semantic retrieval task. This is a bit surprising. The CLIP loss is still present in training, the semantics are also intact in spatial audio-text pairs. Why should there be a performance drop in that case ? it would be good to provide a good discussion and justification – In Table A.7, the performance of the model trained on spatial Clotho and Audiocaps is better on RWD data than even on Clotho and Audiocaps itself. That is a bit surprising. We would expect that the model would be better in it’s own domain. The difference also is pretty big. – The discussion in Section 5.4 is a bit adhoc. I would suggest not referring to anecdotal observations. The experiments could be better designed. – Several of the classification experiments end up using 3-4 layers MLP. I think a more shallower model (maybe even just linear classifier) would provide a better confirmation of what information the embeddings store. Otherwise such deeper networks are able to push the numbers on their and it’s not clear how good the embeddings are. – Some form of clustering and distance visualization would be good. It has been incorporated in some form in Table 2, but it would be good to explicitly show how the distances between embedding represent the spatial information. – All the spatial mapping in terms of the language is very discrete (A.2). The range for distance, direction etc. can appear a bit arbitrary and forced. While this is perhaps a good first attempt, a more continuous form of “spatial-language” is desirable. Another thing could be a perception driven approach can also be taken where the boundaries are decided by what people generally perceive as left or right w.r.t sound direction.

Questions

Please address the weaknesses above

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Please add some limitations.

Follow up on Exp. B.

Due to lack of space, we could not include the following tables for Exp. B. in the general response. 1. Direction-of-arrival error breakdown for azimuth ranges evaluated on test sets of Spatial Clotho and Spatial AudioCaps. ``` [-3.13, -2.51]: mean: 0.36, stddev: 0.44, median: 0.24, iqr: 0.21, min: 0.00, max: 3.02, n: 272 [-2.51, -1.88]: mean: 0.29, stddev: 0.46, median: 0.16, iqr: 0.17, min: 0.01, max: 3.06, n: 251 [-1.88, -1.25]: mean: 0.23, stddev: 0.43, median: 0.12, iqr: 0.16, min: 0.00, max: 3.09, n: 258 [-1.25, -0.63]: mean: 0.24, stddev: 0.37, median: 0.16, iqr: 0.17, min: 0.02, max: 3.03, n: 245 [-0.63, 0.00]: mean: 0.27, stddev: 0.31, median: 0.19, iqr: 0.16, min: 0.01, max: 2.33, n: 215 [0.00, 0.63]: mean: 0.23, stddev: 0.25, median: 0.18, iqr: 0.15, min: 0.01, max: 1.99, n: 224 [0.63, 1.26]: mean: 0.23, stddev: 0.24, median: 0.16, iqr: 0.15, min: 0.00, max: 1.70, n: 270 [1.26, 1.88]: mean: 0.19, stddev: 0.23, median: 0.12, iqr: 0.18, min: 0.00, max: 1.67, n: 220 [1.88, 2.51]: mean: 0.22, stddev: 0.22, median: 0.16, iqr: 0.16, min: 0.01, max: 1.81, n: 259 [2.51, 3.14]: mean: 0.24, stddev: 0.29, median: 0.18, iqr: 0.14, min: 0.01, max: 2.46, n: 244 ``` 2. Direction-of-arrival error breakdown for semantic classes evaluated on the test set of TUT Sound Events 2018. ``` [Drawer]: mean: 0.12, stddev: 0.16, median: 0.07, iqr: 0.15, min: 0.00, max: 0.98, n: 97 [Laughter]: mean: 0.16, stddev: 0.22, median: 0.11, iqr: 0.17, min: 0.00, max: 1.95, n: 95 [Cough]: mean: 0.16, stddev: 0.16, median: 0.09, iqr: 0.20, min: 0.00, max: 0.83, n: 87 [Clearthroat]: mean: 0.17, stddev: 0.29, median: 0.10, iqr: 0.14, min: 0.00, max: 1.95, n: 115 [Keyboard]: mean: 0.16, stddev: 0.15, median: 0.15, iqr: 0.19, min: 0.00, max: 1.19, n: 97 [Speech]: mean: 0.14, stddev: 0.17, median: 0.06, iqr: 0.13, min: 0.00, max: 0.88, n: 105 [Phone]: mean: 0.24, stddev: 0.30, median: 0.18, iqr: 0.18, min: 0.01, max: 2.21, n: 117 [Pageturn]: mean: 0.12, stddev: 0.15, median: 0.09, iqr: 0.12, min: 0.00, max: 1.32, n: 115 [Knock]: mean: 0.15, stddev: 0.17, median: 0.11, iqr: 0.13, min: 0.00, max: 1.10, n: 112 [Doorslam]: mean: 0.17, stddev: 0.17, median: 0.10, iqr: 0.18, min: 0.00, max: 0.85, n: 101 [Keysdrop]: mean: 0.15, stddev: 0.21, median: 0.09, iqr: 0.15, min: 0.00, max: 1.45, n: 111 ```

Erratum

We made a small mistake when addressing the following point: ***Several of the classification experiments end up using 3-4 layers MLP. I think a more shallower model (maybe even just linear classifier) would provide a better confirmation of what information the embeddings store.*** In our response, we only mention Section 4.3, but we meant to refer to both **Sections 4.3 and 5.4**. That is, all the downstream classifiers and regressors we reported in the paper had one hidden layer with 64 parameters. We will make sure to update the manuscript accordingly.