Sounding Bodies: Modeling 3D Spatial Sound of Humans Using Body Pose and Audio

Summary

This paper deals with the problem of predicting sound fields around human bodies and proposes a method that exploits binaural audio signals and human body motions. The proposed method first encodes binaural audio signals and human poses and then decodes them into audio signals that are supposed to be captured by surrounding microphones. Finally, the proposed method renders the predicted signals into sound fields. The proposed method also introduces a loss function that is robust against small time shifts of time-domain audio signals. Experimental evaluations with originally collected datasets demonstrate that the proposed method captured the ground-truth sound fields reasonably well and it requires full body poses instead of only head poses for obtaining good predictions.

Strengths

1. The problem dealt with in this paper is novel as far as I know. Although sound field prediction is one of the standard tasks in audio signal processing, sound field prediction with blindly moving microphones would be novel and challenging. This paper solves this problem by (1) assuming that sound is emitted from a human and the human wears microphones and (2) integrating pose estimation of the target human. 2. The proposed method is technically sound. Since the current problem setting is sensitive to time shifts, the proposed method handles audio signals in the time domain, which is not so popular in multi-modal and cross-modal analysis with deep learning. The proposed method achieves time-domain processing by incorporating WaveNet-like decoders. 3. The newly introduced loss function is also technically sound. Time-domain signals are more sensitive to time shifts than frequency-domain ones. The proposed loss function named the shift-\ell_2 loss effectively mitigates this problem while maintaining its discriminability. 4. The experimental evaluations demonstrate that (1) the proposed method reasonably worked well, (2) full-body poses contributed to the performance improvement of sound source localization compared with head poses, and (3) the proposed loss function was effective for non-speech sounds.

Weaknesses

1. The current experimental evaluations do not contain comparisons with other existing methods. I understand that the current problem setting is novel and thus there are no existing methods that can be directly applied to this problem. However, several previous methods for sound field prediction have already been known, and experimental comparisons with one of those methods will be informative for readers. 2. The problem setting seems to be too extreme. Sound field prediction and reconstruction are one of significant problems in audio signal processing. However, the current problem setting requires (1) humans as sound sources, (2) microphones worn by humans, and (3) external video cameras. In particular, the first constraint seems to be too strict. Some justifications and potential applications will be required.

Questions

None.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

I could not find any negative societal impacts of this work.

Summary

The paper introduces a novel model for a novel task to render a 3D spatial sound from human body motion and audio collected by headset microphones. The authors also present a novel dataset containing human body motion and audio for this task. The model takes encoded audio, pose features, and target microphone position to render the corresponding spatial sound. A novel shifted-l2 loss is also proposed to address the issue of big spikes in l2 error due to the small shifts of impulsive signals. Experiments and results demonstrate the effectiveness of the proposed method.

Strengths

1. The motivation and objectives are clear. To my best knowledge, it is indeed the first paper that aims to render 3D spatial sound from a human body pose. 2. The dataset itself is valuable. It could unlock various interesting research projects in the future. 3. The system is simple but effective. The proposed shifted-l2 loss is interesting and could be useful in other audio applications as well. 4. Analyses of the roles of body pose and head-mounted microphones are well-done in the ablation studies. 5. The phase is important and I am glad that the authors consider it in the optimization.

Weaknesses

1. While extensive quantitative evaluations and analyses are done, it is often challenging to understand the gap in the perceptual level. It would be great if there is a subjective evaluation to demonstrate both the importance of the proposed task/approach and the contribution of the technical components. 2. It would also be nice to have more samples in the supplementary material for a subjective evaluation of the proposed approach. 3. I wonder whether any failure cases are appearing in the current exploration stage. 4. audio and visual data time synchronization is often challenging but the details to achieve that are missing. 5. As the authors mentioned in the limitation, the current approach could not handle spatial sound for human-object interaction, mainly for far-field modeling, and require intensive computation resources.

Questions

1. Why do you use only 1s audio clips during training? Is there a bottleneck that avoids having a longer audio sequence during training? Or is it simply because of the high sampling rate (48khz)? 2. Would it be possible to have data with multi-person and human-object interaction in the future? What might be the challenges to record these data? 3. I am interested in the hyper-parameter choices for the shifted-l2 loss. How is the L=128 varied if the sampling rate differs?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This work introduces an approach for generating spatial audio from a 3D human pose and microphones placed close to the subject’s head, e.g. on a VR/AR headset, as is the case here. The authors first collect a multimodal dataset that contains 3D bodies and audio, recorded using multiple Kinects for body tracking and 345 microphones for audio acquisition. The dataset contains multiple participants in different outfits and poses, generating different sounds following a pre-defined script. To generate spatial audio, the authors propose a model that receives as input the audio recorded from the headset microphones and the 3D body pose and predicts the audio signal recorded at a microphone of the spherical capture array. Using the audio at each microphone and the locations of each microphone, the authors compute harmonic sound field coefficients that best represent the sound field.

Strengths

- Important problem with large impact: The authors tackle an important problem for AR/VR applications that has not yet received a lot of attention. Generating accurate sound fields promises to increase immersion and the realism of virtual experiences. The proposed dataset will be an important benchmark for the community and open up new and exciting avenues of research. - Novelty: The proposed dataset is an important contribution with unique data that are not publicly available. - A loss that is well-motivated by prior-work and adapted to the problem structure that leads to clear benefits. - A simple model that merges different types of information, using well-proven components. - Writing: The paper is well-written and easy to follow. - Experiments: The experimental analysis is clear, highlights the effects of each design choice and provides insights into what makes the method work.

Weaknesses

- The related work section lacks a discussion of recent methods that model sound fields using neural network fields, such as: - [Learning Neural Acoustic Fields](https://www.andrew.cmu.edu/user/afluo/Neural_Acoustic_Fields/). - [INRAS: Implicit Neural Representation for Audio Scenes](https://openreview.net/forum?id=7KBzV5IL7W) These could also be useful baselines for the model, both in terms of accuracy and in terms of computation compared to the proposed method. Furthermore, the following datasets could be a useful discussion point for the related work section: - SoundSpaces: Audio-Visual Navigation in 3D Environments: A discussion of whether a similar dataset as the one proposed in this work could be constructed by simulating audio propagation from artificial bodies. - Talking With Hands 16.2M: A Large-Scale Dataset of Synchronized Body-Finger Motion and Audio for Conversational Motion Analysis and Synthesis: I believe the related work section could benefit from a comparison with this paper in terms of granularity of pose representation and sound field, number of persons and type of conversations.

Questions

- Do you use only body keypoints? Or do you include hand keypoints from OpenPose? - I believe that a comment or discussion on the granularity of the representation of the human body would be useful. The ablation clearly shows that a more detailed representation of the human body helps improve the quality of the predicted sound field. I would like to hear the authors' answer on the following questions: - Is the robustness of the estimated keypoints important? Do noisy keypoints significantly affect the quality of the sound field? - Is the lower body important for sound quality? - Would a finer representation of the subject help? For example, would including hand or face keypoints from OpenPose help? Some implementation details that should be clarified: - How was the size of the window in loss function chosen? - How was the training sampling strategy selected? Maximization of GPU memory usage? - Finally, regarding the performance of the model, it would be useful to have a visualization of the distribution of errors around the sphere of microphones. Is there a subset of microphones where the audio signal is better reconstructed?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors present the limitations of the proposed dataset and method. They acknowledge that the current work is limited to modeling sound effects for a single person in an artificial scene and that it cannot model sound close to the body.

Summary

The paper tackles the task spatializing a mixture of speech and body sounds at different points in a sphere around a human body without explicitly recording or knowing the sound source location. Towards that goal, the paper captures a new dataset of humans speaking and making different body sounds in a very controlled and extensive setup. The paper proposes a model that takes as input the time-warped audio recorded by a VR headset and the body pose sequence, and renders the audio at different points on the sphere conditioned on the azimuth and elevation of the points. Furthermore, the paper proposes a novel and intuitloss function to accurately model both speech and short-lived body sounds. The paper compares its model with a heurisitcal baseline, evaluates different versions of the loss function, and also reports results for different microphone channel count and an ablation of the model. Post author-reviewer discussion: I have read the rebuttal. The responses are detailed and answer my questions. I have increased my score.

Strengths

1. The proposed task is interesting. 2. The proposed loss is intuitive and seems to be well-designed. 3. The dataset could also be a valuable contribution for the community.

Weaknesses

1. a) The dataset capture seems to be done in just 1 environment. There is no discussion or experiments on the model's generalization to other environments. b) The capture setup seems to be pretty complicated and could limit the applicability of the model to novel environments if training needs to be redone for new environments. 2. a) The capture scene seem to be simple in the sense that it doesn't contain other objects, is single-room in nature, etc. How would the model fare for more complex realworld scenes? b) Would the model be able to accurately render sounds at locations that don't have a path of direct sounds from the source? 3. How is the source location in L150-1 determined? Minor: 1. The shift-l2 loss doesn't seem to perform by itself in most cases (as also pointed out by the authors) even though the design is pretty intuitive.

Questions

I would like to request the authors to address the questions/concerns I raised in 'Weaknesses', in the rebuttal.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The paper discusses its limitations, and I can't think of any serious negative societal implications of the work that need to be discussed in the paper.

Post rebuttal

Thank the authors for the responses. Most of my questions are well addressed. Since the task is novel, I believe it makes sense that the initial settings are relatively simple to better understand the problem itself. I also appreciate that the proposed method is not overcomplicated. Thereby, I will keep my original rating and prefer to accept the paper.

Reply to rebuttal

I would like to thank the authors for their reply and for including the requested figures. My questions have all been answered by the reply and the provided rebuttal PDF. I will keep my original rating and recommend accepting the paper.