A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation

Theoretical Background on Mixture of Noise Levels

## Theoretical Background on Mixture of Noise Levels $ $ ### 1. Theoretical Background on Multimodal Learning In *"A Theory of Multimodal Learning"* [1], multimodal learning is shown to offer a superior **generalization bound** compared to unimodal learning, with an improvement factor of **$O(\sqrt{n})$**, where **$n$** denotes the sample size. This benefit relies on **connection** and **heterogeneity** between modalities: - **Connection**: The bound depends on learned connections between (**$\mathcal{X}$**) and (**$\mathcal{Y}$**). - **Heterogeneity**: Describes how modalities, **$\mathcal{X}$** and **$\mathcal{Y}$**, diverge and complement. If connection and heterogeneity are missing, ill-conditioned scenarios can arise. For instance, if **$x \equiv y$**, perfect connection suggests no need for learning about **$\mathcal{Y}$**. On the other hand, if **$x$** is random noise, there is heterogeneity but no meaningful connection between **$\mathcal{X}$** and **$\mathcal{Y}$**, making non-trivial learning on **$\mathcal{X}$** alone impractical. The theory also highlights that and learning effective connections between modalities via **generative models** can enhance multimodal learning. This forms the basis for our **Mixture of Noise Levels (MoNL)** approach, which is particularly suited for multimodal learning with **audio** and **video** data. [1] Zhou Lu. *"A Theory of Multimodal Learning."* NeurIPS 2023. $ $ ### 2. Advantages of Mixture of Noise Level Training (MoNL) Our **Mixture of Noise Level (MoNL)** training method offers significant benefits for multimodal learning, especially with **audio** and **video** data: - **Heterogeneity and Connection**: Audio and video are naturally heterogeneous. For example, a video of a person speaking includes **audio** of spoken words and **video** of lip movements and facial expressions. MoNL uses **variable noise levels** to enhance learning by capturing the **generic transition matrix** across the **temporal axis**. $$p _{\mathbf{\theta}}([\mathbf{z} _{t ^{(1,1)}-1} ^{(1,1)}, \ldots, \mathbf{z} _{t ^{(M, N)}-1} ^{(M, N)}] \mid[\mathbf{z} _{t ^{(1,1)}} ^{(1,1)}, \ldots, \mathbf{z} _{t ^{(M, N)}} ^{(M, N)}]) \qquad \text{(Eq. (4))}$$ where $M$, $N$ are the number of modalities and time-segments, respectively. - **Enhanced Connectivity**: MoNL improves **connectivity** between **audio** and **video** modalities. Our experiments show that MoNL often surpasses **task-specific learning** approaches by fostering better connections between modalities, adapting its focus more effectively. $ $ ### 3. Enhanced Connectivity - Comparison with Existing Methods - **MoNL vs. Joint Learning in MMD** [2]: Unlike joint learning methods that focus on the joint distribution $p_{\mathbf{\theta}}(\mathbf{z} _{t-1} \mid\mathbf{z} _{t})$, MoNL trains across **multiple conditioning**, enabling better connections by varying its focus. This is evidenced by MoNL outperforming the Vanilla (see Table 1) and MMD models (see Tables 2 and 3). - **MoNL vs. Per-Modality Training**: MoNL goes beyond per-modality training in UniDiffuser [3], which uses variable noise between modalities i.e., learning $p_{\mathbf{\theta}}([\mathbf{z} _{t ^{(1)}-1} ^{(1)}, \ldots, \mathbf{z} _{t ^{(M)}-1} ^{(M)}] \mid[\mathbf{z} _{t ^{(1)}} ^{(1)}, \ldots, \mathbf{z} _{t ^{(M)}} ^{(M)}])$. MoNL introduces variable noise across different **time segments**, learning connections across **temporal dynamics** as well. This advantage is demonstrated in Table 1. - **MoNL vs. Masked Training** [4]: Diffusion models often obscure high-frequency details with low noise and low-frequency structures with high noise [5]. MoNL employs variable noise levels to explore diverse **frequency components**, enhancing the model's ability to correlate high and low-frequency elements. This is in contrast to masked self-supervised learning, which limits frequency-specific connections by masking entire elements. [2] Ruan et al. *"MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation."* CVPR 2023. [3] Bao et al. *"One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale."* ICML 2023. [4] Voleti et al. *"MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation."* NeurIPS 2022. [5] Sander Dieleman. *"Noise Schedules Considered Harmful."* [Link](https://sander.ai/2024/06/14/noise-schedules). $ $ ### Conclusion In summary, the effectiveness of **MoNL** for **multimodal diffusion models**, particularly with **audio** and **video** data, stems from its strategic use of **connection** and **heterogeneity**. By applying **variable noise levels**, MoNL enhances **connectivity** between modalities and better adapts to diverse **temporal** and **frequency components**, leading to superior performance compared to existing multimodal learning methods.

Clarification on Resource Constraints and Additional Theoretical Contributions

Dear Reviewer 4xRp, Thank you for your feedback and for considering our initial response. We acknowledge the importance of validating our model on publicly available datasets. Unfortunately, training on these datasets with the necessary computational resources (requiring approximately $35k per model) presented a significant challenge at the moment. However, we plan to proceed the experiments as the permit is approved and believe our work with the Monologues dataset, which contains 16M samples with a diverse range of audiovisual cues, provides valuable insights and demonstrates the robustness of our approach. In addition, we have newly provided the **"Theoretical Background on Mixture of Noise Levels,"** which we hope you will consider as part of our ongoing effort to strengthen our work and contribute to the field. We hope you understand our constraints and continue to see the value in the contributions we have made with the resources available. We remain committed to advancing research in **audiovisual generation** and plan to release the Monologues dataset to further support reproducibility in the future and encourage future work. Thank you for your consideration!

Clarifications and Additional Insights on the Monologues Dataset and MoNL

Dear Reviewer 4xRp, Thank you for your valuable feedback. We appreciate your interest in the Monologues dataset and your insights into the theoretical aspects of our work. **Dataset Construction** To address your concerns about dataset construction, we want to emphasize that the Monologues dataset was meticulously curated to adhere to ethical and legal standards. We have implemented robust measures to protect user privacy and avoid copyright infringement. The dataset consists exclusively of public links (and timestamps) to videos that are: * Including those that are **older than 90 days** * Free from **copyright claims** * Excluding **harmful or inappropriate material** (e.g., nudity, violence) * Verified to contain **motion** (excluding static images) * In the **English language** with a **single visible speaker** * Framed with **torso up** The dataset link list is automatically updated on a daily basis to ensure ongoing compliance with these criteria. The Monologues dataset is a valuable resource for the research community, providing a rich audiovisual benchmark for developing and evaluating multimodal models. **Theoretical Background** While formally proving the connectivity benefits of MoNL presents a significant challenge, our empirical results provide compelling evidence of its effectiveness. Table 1 and Figures 2 and 7 demonstrate MoNL's superior performance across diverse audiovisual tasks, generating more temproally-consistent audiovisuals compared to joint learning and per-modality method, suggesting enhanced intra- and inter-modality connectivity. We recognize the need for a deeper theoretical understanding and plan to explore this area in future research. As highlighted in *"A Theory of Multimodal Learning."* (Zhou et al. NeurIPS 2023), theoretical foundations in multimodal learning remain relatively underdeveloped. Our work serves as a **crucial experimental foundation** for future theoretical investigations in this field. We believe our work represents a **substantial step forward in audiovisual and multimodal research**. **We believe these clarifications provide valuable insights into our work and would be grateful for your consideration of an improved score.** Sincerely, The Authors

Additional analysis of the objectives and effectiveness of our approach

To enhance the understanding of how MoNL contributes to building a unified model for diverse audiovisual tasks—including cross-modal inference and multimodal interpolation— and improve connectivity between modalities, we provide a detailed analysis of the objectives and effectiveness of our approach, considering a simplified case with two modalities and two time-segments. $ $ ### Objective Functions To determine the optimal objective function for learning a model $\theta$ that estimates: - **Cross-modal inferences:** $P(X|Y)$ and $P(Y|X)$ - **Multimodal interpolations:** $P([\mathbf{x}_2, \mathbf{y}_2] \mid [\mathbf{x}_1, \mathbf{y}_1])$, $P([\mathbf{x}_1, \mathbf{y}_1] \mid [\mathbf{x}_2, \mathbf{y}_2])$, $P([\mathbf{x}_1, \mathbf{y}_2] \mid [\mathbf{x}_2, \mathbf{y}_1])$, and $P([\mathbf{x}_2, \mathbf{y}_1] \mid [\mathbf{x}_1, \mathbf{y}_2])$ we evaluate the following objectives for heterogeneous multimodal data $X$ and $Y$, where $X = [\mathbf{x}_1, \mathbf{x}_2]$ and $Y = [\mathbf{y}_1, \mathbf{y}_2]$. 1. **Task-Specific Method** $$ \min \mathbb{E}[P_\theta(X \mid Y)] $$ This approach focuses solely on one direction of inference, potentially missing significant interactions between modalities. 2. **Joint Learning Method** $$ \min \mathbb{E}[P_\theta(X, Y)] $$ This method captures both modalities simultaneously but may fail to model temporal and multimodal interactions, which are crucial for conditional tasks. 3. **Per-Modality Method** $$ \min \mathbb{E}\left[P_\theta(X, Y) + P_\theta(X \mid Y) + \alpha P_\theta(Y \mid X)\right] $$ This method improves upon the task-specific approach by incorporating reverse inference between modalities. However, it may not effectively capture temporal interactions, which are important for interpolation tasks. 4. **Mixture of Noise Levels (MoNL)** $$ \min \mathbb{E}\left[P_\theta(X, Y) + \beta_1 P_\theta(X \mid Y) + \beta_2 P_\theta(Y \mid X) + \beta_3 P([\mathbf{x}_2, \mathbf{y}_2] \mid [\mathbf{x}_1, \mathbf{y}_1]) + \beta_4 P([\mathbf{x}_1, \mathbf{y}_1] \mid [\mathbf{x}_2, \mathbf{y}_2]) + \beta_5 P([\mathbf{x}_1, \mathbf{y}_2] \mid [\mathbf{x}_2, \mathbf{y}_1]) + \beta_6 P([\mathbf{x}_2, \mathbf{y}_1] \mid [\mathbf{x}_1, \mathbf{y}_2])\right] $$ MoNL integrates joint distributions, cross-modal inferences, and multimodal interpolations by learning a transition matrix that captures complex interactions between modalities. $ $ ### Heterogeneous and Connected Multimodality For heterogeneous multimodal data (e.g., audio and video), where $X$ and $Y$ are interconnected but distinct, MoNL excels by capturing intricate interactions through: - **Direct Relationships:** Modeled by $P_\theta(X, Y)$ - **Cross-Modal Inferences:** Modeled by $P_\theta(X \mid Y)$ and $P_\theta(Y \mid X)$ - **Multimodal Interpolations:** Modeled by $P([\mathbf{x}_2, \mathbf{y}_2] \mid [\mathbf{x}_1, \mathbf{y}_1])$, $P([\mathbf{x}_1, \mathbf{y}_1] \mid [\mathbf{x}_2, \mathbf{y}_2])$ and similar terms. MoNL effectively models the connectivity and complex interactions between modalities, significantly enhancing performance in various audiovisual tasks. $ $ We hope this analysis is helpful and will be incorporated into the final version.

Theoretical Background on Mixture of Noise Levels

## Theoretical Background on Mixture of Noise Levels $ $ ### 1. Theoretical Background on Multimodal Learning In *"A Theory of Multimodal Learning"* [1], multimodal learning is shown to offer a superior **generalization bound** compared to unimodal learning, with an improvement factor of **$O(\sqrt{n})$**, where **$n$** denotes the sample size. This benefit relies on **connection** and **heterogeneity** between modalities: - **Connection**: The bound depends on learned connections between (**$\mathcal{X}$**) and (**$\mathcal{Y}$**). - **Heterogeneity**: Describes how modalities, **$\mathcal{X}$** and **$\mathcal{Y}$**, diverge and complement. If connection and heterogeneity are missing, ill-conditioned scenarios can arise. For instance, if **$x \equiv y$**, perfect connection suggests no need for learning about **$\mathcal{Y}$**. On the other hand, if **$x$** is random noise, there is heterogeneity but no meaningful connection between **$\mathcal{X}$** and **$\mathcal{Y}$**, making non-trivial learning on **$\mathcal{X}$** alone impractical. The theory also highlights that and learning effective connections between modalities via **generative models** can enhance multimodal learning. This forms the basis for our **Mixture of Noise Levels (MoNL)** approach, which is particularly suited for multimodal learning with **audio** and **video** data. [1] Zhou Lu. *"A Theory of Multimodal Learning."* NeurIPS 2023. $ $ ### 2. Advantages of Mixture of Noise Level Training (MoNL) Our **Mixture of Noise Level (MoNL)** training method offers significant benefits for multimodal learning, especially with **audio** and **video** data: - **Heterogeneity and Connection**: Audio and video are naturally heterogeneous. For example, a video of a person speaking includes **audio** of spoken words and **video** of lip movements and facial expressions. MoNL uses **variable noise levels** to enhance learning by capturing the **generic transition matrix** across the **temporal axis**. $$p _{\mathbf{\theta}}([\mathbf{z} _{t ^{(1,1)}-1} ^{(1,1)}, \ldots, \mathbf{z} _{t ^{(M, N)}-1} ^{(M, N)}] \mid[\mathbf{z} _{t ^{(1,1)}} ^{(1,1)}, \ldots, \mathbf{z} _{t ^{(M, N)}} ^{(M, N)}]) \qquad \text{(Eq. (4))}$$ where $M$, $N$ are the number of modalities and time-segments, respectively. - **Enhanced Connectivity**: MoNL improves **connectivity** between **audio** and **video** modalities. Our experiments show that MoNL often surpasses **task-specific learning** approaches by fostering better connections between modalities, adapting its focus more effectively. $ $ ### 3. Enhanced Connectivity - Comparison with Existing Methods - **MoNL vs. Joint Learning in MMD** [2]: Unlike joint learning methods that focus on the joint distribution $p_{\mathbf{\theta}}(\mathbf{z} _{t-1} \mid\mathbf{z} _{t})$, MoNL trains across **multiple conditioning**, enabling better connections by varying its focus. This is evidenced by MoNL outperforming the Vanilla (see Table 1) and MMD models (see Tables 2 and 3). - **MoNL vs. Per-Modality Training**: MoNL goes beyond per-modality training in UniDiffuser [3], which uses variable noise between modalities i.e., learning $p_{\mathbf{\theta}}([\mathbf{z} _{t ^{(1)}-1} ^{(1)}, \ldots, \mathbf{z} _{t ^{(M)}-1} ^{(M)}] \mid[\mathbf{z} _{t ^{(1)}} ^{(1)}, \ldots, \mathbf{z} _{t ^{(M)}} ^{(M)}])$. MoNL introduces variable noise across different **time segments**, learning connections across **temporal dynamics** as well. This advantage is demonstrated in Table 1. - **MoNL vs. Masked Training** [4]: Diffusion models often obscure high-frequency details with low noise and low-frequency structures with high noise [5]. MoNL employs variable noise levels to explore diverse **frequency components**, enhancing the model's ability to correlate high and low-frequency elements. This is in contrast to masked self-supervised learning, which limits frequency-specific connections by masking entire elements. [2] Ruan et al. *"MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation."* CVPR 2023. [3] Bao et al. *"One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale."* ICML 2023. [4] Voleti et al. *"MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation."* NeurIPS 2022. [5] Sander Dieleman. *"Noise Schedules Considered Harmful."* [Link](https://sander.ai/2024/06/14/noise-schedules). $ $ ### Conclusion In summary, the effectiveness of **MoNL** for **multimodal diffusion models**, particularly with **audio** and **video** data, stems from its strategic use of **connection** and **heterogeneity**. By applying **variable noise levels**, MoNL enhances **connectivity** between modalities and better adapts to diverse **temporal** and **frequency components**, leading to superior performance compared to existing multimodal learning methods.

Theoretical Background on Mixture of Noise Levels

## Theoretical Background on Mixture of Noise Levels $ $ ### 1. Theoretical Background on Multimodal Learning In *"A Theory of Multimodal Learning"* [1], multimodal learning is shown to offer a superior **generalization bound** compared to unimodal learning, with an improvement factor of **$O(\sqrt{n})$**, where **$n$** denotes the sample size. This benefit relies on **connection** and **heterogeneity** between modalities: - **Connection**: The bound depends on learned connections between (**$\mathcal{X}$**) and (**$\mathcal{Y}$**). - **Heterogeneity**: Describes how modalities, **$\mathcal{X}$** and **$\mathcal{Y}$**, diverge and complement. If connection and heterogeneity are missing, ill-conditioned scenarios can arise. For instance, if **$x \equiv y$**, perfect connection suggests no need for learning about **$\mathcal{Y}$**. On the other hand, if **$x$** is random noise, there is heterogeneity but no meaningful connection between **$\mathcal{X}$** and **$\mathcal{Y}$**, making non-trivial learning on **$\mathcal{X}$** alone impractical. The theory also highlights that and learning effective connections between modalities via **generative models** can enhance multimodal learning. This forms the basis for our **Mixture of Noise Levels (MoNL)** approach, which is particularly suited for multimodal learning with **audio** and **video** data. [1] Zhou Lu. *"A Theory of Multimodal Learning."* NeurIPS 2023. $ $ ### 2. Advantages of Mixture of Noise Level Training (MoNL) Our **Mixture of Noise Level (MoNL)** training method offers significant benefits for multimodal learning, especially with **audio** and **video** data: - **Heterogeneity and Connection**: Audio and video are naturally heterogeneous. For example, a video of a person speaking includes **audio** of spoken words and **video** of lip movements and facial expressions. MoNL uses **variable noise levels** to enhance learning by capturing the **generic transition matrix** across the **temporal axis**. $$p _{\mathbf{\theta}}([\mathbf{z} _{t ^{(1,1)}-1} ^{(1,1)}, \ldots, \mathbf{z} _{t ^{(M, N)}-1} ^{(M, N)}] \mid[\mathbf{z} _{t ^{(1,1)}} ^{(1,1)}, \ldots, \mathbf{z} _{t ^{(M, N)}} ^{(M, N)}]) \qquad \text{(Eq. (4))}$$ where $M$, $N$ are the number of modalities and time-segments, respectively. - **Enhanced Connectivity**: MoNL improves **connectivity** between **audio** and **video** modalities. Our experiments show that MoNL often surpasses **task-specific learning** approaches by fostering better connections between modalities, adapting its focus more effectively. $ $ ### 3. Enhanced Connectivity - Comparison with Existing Methods - **MoNL vs. Joint Learning in MMD** [2]: Unlike joint learning methods that focus on the joint distribution $p_{\mathbf{\theta}}(\mathbf{z} _{t-1} \mid\mathbf{z} _{t})$, MoNL trains across **multiple conditioning**, enabling better connections by varying its focus. This is evidenced by MoNL outperforming the Vanilla (see Table 1) and MMD models (see Tables 2 and 3). - **MoNL vs. Per-Modality Training**: MoNL goes beyond per-modality training in UniDiffuser [3], which uses variable noise between modalities i.e., learning $p_{\mathbf{\theta}}([\mathbf{z} _{t ^{(1)}-1} ^{(1)}, \ldots, \mathbf{z} _{t ^{(M)}-1} ^{(M)}] \mid[\mathbf{z} _{t ^{(1)}} ^{(1)}, \ldots, \mathbf{z} _{t ^{(M)}} ^{(M)}])$. MoNL introduces variable noise across different **time segments**, learning connections across **temporal dynamics** as well. This advantage is demonstrated in Table 1. - **MoNL vs. Masked Training** [4]: Diffusion models often obscure high-frequency details with low noise and low-frequency structures with high noise [5]. MoNL employs variable noise levels to explore diverse **frequency components**, enhancing the model's ability to correlate high and low-frequency elements. This is in contrast to masked self-supervised learning, which limits frequency-specific connections by masking entire elements. [2] Ruan et al. *"MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation."* CVPR 2023. [3] Bao et al. *"One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale."* ICML 2023. [4] Voleti et al. *"MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation."* NeurIPS 2022. [5] Sander Dieleman. *"Noise Schedules Considered Harmful."* [Link](https://sander.ai/2024/06/14/noise-schedules). $ $ ### Conclusion In summary, the effectiveness of **MoNL** for **multimodal diffusion models**, particularly with **audio** and **video** data, stems from its strategic use of **connection** and **heterogeneity**. By applying **variable noise levels**, MoNL enhances **connectivity** between modalities and better adapts to diverse **temporal** and **frequency components**, leading to superior performance compared to existing multimodal learning methods.

Request for Final Review: Response and Theoretical Clarifications

Dear Reviewer tkoE, We kindly request that you review our responses, as we have only **two days** left for further **discussion**. We have addressed your comments thoughtfully and provided a thorough **theoretical background** of our method. We sincerely appreciate the time and effort you have dedicated to reviewing our paper and your constructive and insightful feedback. Thank you once again. Best regards, The Authors

Thank you for your time and effort

Dear Reviewer tkoE, We sincerely appreciate your time and effort in reviewing our paper. We are glad that you found our work to be a good contribution to the field of audiovisual generation. We believe that our AVDiT with MoNL effectively models temporal dependencies, handles diverse multimodal tasks, and demonstrates superior performance compared to existing approaches, offering significant advancements beyond prior art. We believe these contributions can be valuable to the community! We are committed to fostering reproducibility in our research and are happy to make our trained models and code publicly available upon acceptance of the paper. Thank you again for your valuable insights. Sincerely, The Authors

Theoretical Background on Mixture of Noise Levels

## Theoretical Background on Mixture of Noise Levels $ $ ### 1. Theoretical Background on Multimodal Learning In *"A Theory of Multimodal Learning"* [1], multimodal learning is shown to offer a superior **generalization bound** compared to unimodal learning, with an improvement factor of **$O(\sqrt{n})$**, where **$n$** denotes the sample size. This benefit relies on **connection** and **heterogeneity** between modalities: - **Connection**: The bound depends on learned connections between (**$\mathcal{X}$**) and (**$\mathcal{Y}$**). - **Heterogeneity**: Describes how modalities, **$\mathcal{X}$** and **$\mathcal{Y}$**, diverge and complement. If connection and heterogeneity are missing, ill-conditioned scenarios can arise. For instance, if **$x \equiv y$**, perfect connection suggests no need for learning about **$\mathcal{Y}$**. On the other hand, if **$x$** is random noise, there is heterogeneity but no meaningful connection between **$\mathcal{X}$** and **$\mathcal{Y}$**, making non-trivial learning on **$\mathcal{X}$** alone impractical. The theory also highlights that and learning effective connections between modalities via **generative models** can enhance multimodal learning. This forms the basis for our **Mixture of Noise Levels (MoNL)** approach, which is particularly suited for multimodal learning with **audio** and **video** data. [1] Zhou Lu. *"A Theory of Multimodal Learning."* NeurIPS 2023. $ $ ### 2. Advantages of Mixture of Noise Level Training (MoNL) Our **Mixture of Noise Level (MoNL)** training method offers significant benefits for multimodal learning, especially with **audio** and **video** data: - **Heterogeneity and Connection**: Audio and video are naturally heterogeneous. For example, a video of a person speaking includes **audio** of spoken words and **video** of lip movements and facial expressions. MoNL uses **variable noise levels** to enhance learning by capturing the **generic transition matrix** across the **temporal axis**. $$p _{\mathbf{\theta}}([\mathbf{z} _{t ^{(1,1)}-1} ^{(1,1)}, \ldots, \mathbf{z} _{t ^{(M, N)}-1} ^{(M, N)}] \mid[\mathbf{z} _{t ^{(1,1)}} ^{(1,1)}, \ldots, \mathbf{z} _{t ^{(M, N)}} ^{(M, N)}]) \qquad \text{(Eq. (4))}$$ where $M$, $N$ are the number of modalities and time-segments, respectively. - **Enhanced Connectivity**: MoNL improves **connectivity** between **audio** and **video** modalities. Our experiments show that MoNL often surpasses **task-specific learning** approaches by fostering better connections between modalities, adapting its focus more effectively. $ $ ### 3. Enhanced Connectivity - Comparison with Existing Methods - **MoNL vs. Joint Learning in MMD** [2]: Unlike joint learning methods that focus on the joint distribution $p_{\mathbf{\theta}}(\mathbf{z} _{t-1} \mid\mathbf{z} _{t})$, MoNL trains across **multiple conditioning**, enabling better connections by varying its focus. This is evidenced by MoNL outperforming the Vanilla (see Table 1) and MMD models (see Tables 2 and 3). - **MoNL vs. Per-Modality Training**: MoNL goes beyond per-modality training in UniDiffuser [3], which uses variable noise between modalities i.e., learning $p_{\mathbf{\theta}}([\mathbf{z} _{t ^{(1)}-1} ^{(1)}, \ldots, \mathbf{z} _{t ^{(M)}-1} ^{(M)}] \mid[\mathbf{z} _{t ^{(1)}} ^{(1)}, \ldots, \mathbf{z} _{t ^{(M)}} ^{(M)}])$. MoNL introduces variable noise across different **time segments**, learning connections across **temporal dynamics** as well. This advantage is demonstrated in Table 1. - **MoNL vs. Masked Training** [4]: Diffusion models often obscure high-frequency details with low noise and low-frequency structures with high noise [5]. MoNL employs variable noise levels to explore diverse **frequency components**, enhancing the model's ability to correlate high and low-frequency elements. This is in contrast to masked self-supervised learning, which limits frequency-specific connections by masking entire elements. [2] Ruan et al. *"MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation."* CVPR 2023. [3] Bao et al. *"One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale."* ICML 2023. [4] Voleti et al. *"MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation."* NeurIPS 2022. [5] Sander Dieleman. *"Noise Schedules Considered Harmful."* [Link](https://sander.ai/2024/06/14/noise-schedules). $ $ ### Conclusion In summary, the effectiveness of **MoNL** for **multimodal diffusion models**, particularly with **audio** and **video** data, stems from its strategic use of **connection** and **heterogeneity**. By applying **variable noise levels**, MoNL enhances **connectivity** between modalities and better adapts to diverse **temporal** and **frequency components**, leading to superior performance compared to existing multimodal learning methods.

Request for Final Review: Response and Theoretical Clarifications

Dear Reviewer bNvz, We kindly request that you review our responses, as we have only **two days** left for further **discussion**. We have addressed your comments thoughtfully and provided a thorough **theoretical background** of our method. We sincerely appreciate the time and effort you have dedicated to reviewing our paper and your constructive and insightful feedback. Thank you once again. Best regards, The Authors