Answer to Reviewer A4oN
We appreciate the effort you've invested in reviewing our manuscript and providing detailed feedback. Your insightful suggestions have been incredibly valuable, and we're confident that by addressing the points you've raised, we can enhance the quality of our paper. Below, we've addressed each of the concerns you've highlighted in your review:
***Q1: As the authors mention, the dual-path RNN idea is not new to this problem. I understand that the authors chose SRU for their complexity-related considerations, but I also wonder if the audio processing module could benefit from its own self-attention mechanism, such as in the SepFormer model.***
**A1:** We experimented with different sequence processing architectures in Appendix D, which has now been updated to include several transformer designs. The self-attention mechanism leads to a decrease of 1 dB SI-SNRi, while significantly increasing the memory requirements of the model. Other transformer approaches resulted in extremely high parameter counts, which contradicts our aim of constructing a lightweight AVSS model.
***Q2: The spectral source separation module might be the weakest contribution, because complex masks have been extensively studied in the audio-only source separation literature.***
**A2:** Some research has been conducted in the AOSS field using complex masks, but such studies are relatively scarce in the context of audio-visual speech separation. Additionally, to our knowledge, current state-of-the-art audio-only source separation methods, such as Sepformer, still employ real-valued masks. Indeed, TF-GridNet, the current AOSS SOTA method, does not use masks at all and directly computes the separated sources. We provided a study on all these different methods in Table 4. It is important to note that results in AOSS do not necessarily translate to the AVSS field.
***Q3: I wish the paper provides more details on the TDANet block for video processing, which is relegated to the reference in the current version.***
**A3:** We have included an explanation of the video preprocessing block (VP BLOCK) in Appendix A.
***Q4: The authors chose to "add" $\mathbf{f}_1$ and $\mathbf{f}_2$ (eq 11) after the CAF processing. I think it's a little abrupt in the sense that there might be other choices that preserve the unique information that each vector learns, such as concatenation. Have the authors considered other ways to combine the two vectors?***
**A4:** In Table 2 of our paper we compare the CAF block with pure concatenation (CTCNet Fusion (adapted)) of the audio and visual features. We observe that despite our CAF Block using only 3.6% ofthe parameters and 1.3% of the MACs, it outperformed pure concatenation by a large margin.
The purpose for the CAF Block is to produce a low-parameter and MAC alternative to CTCNet’s fusion approach. If we directly concatenated $\mathbf{f}_1$ and $\mathbf{f}_2$, and then downscaled to the correct channel dimension using a convolution, this would be the same parameter and MAC count as the pure concatenation discussed above and in Table 2 of our paper. However, in order to fully answer your question, we ran an experiment where we concatenated $\mathbf{f}_1$ and $\mathbf{f}_2$ and then downscaled using a group convolution. Using a group convolution does not avoid the memory increase that comes from the concatenation, but it does avoid increasing the parameter and MAC counts significantly. The results show that this is an ineffective operation, and that a simple summation should be preferred.
| Method | Params (K) | MACs (M) | LRS2-2Mix SI-SNRi | LRS2-2Mix SDRi |
| --------------- | ------------- | --------------- | ------------- | ------------- |
| Add (ours) | 14.1 | 14.3 | 739 | 21,896 |
| Concat | 13.7 | 13.9 | 741 | 21,913 |