Thanks for your reply! We appreciate your time and efforts to read our responses and paper. Please see our new response below.
---
> **Q1: For Transformer models that do not include adding a causal attention mask, how do they possess the ability for causal modeling? Furthermore, based on the current definition of "causal modeling," can we consider CNNs as a form of causal modeling as well?**
A1: It should be first clarified that standard Transformers or CNNs can not learn to perform causal modeling without a causal attention mask. In mathematical terms, given an input token sequence $x_1, ..., x_n$, causal modeling aims to maximize the likelihood of the correct next token given previous tokens in the sequence, which can be formulated as
$$\max_\theta\sum_{i=1}^{n-1}\log p(x_{i+1}|x_1, ..., x_i;\theta),$$
where $\theta$ refers to the parameter set of the model.
For Transformers and CNNs, a causal mask is necessary to restrict access to future tokens. Without a causal mask, Transformers can access the entire input sequence, i.e., they will use $x_1, ..., x_n$ as the probability condition rather than $x_1, ..., x_i$. Similarly, CNNs can access neighborhood tokens in the local window, i.e., they will use $x_{i-w}, ..., x_{i+w}$ as the probability condition, where $w$ is the local window width. These conditions both include the next token, which leads to information leakage. Therefore, standard Transformers and CNNs will rely on such shortcuts, failing to learn to perform causal modeling effectively.
---
> **Q2: Building on the last question, if existing models already inherently possess causal modeling capabilities, would the proposed method be considered merely an enhancement?**
A2: Based on the reasons mentioned above, existing models do not inherently have causal modeling abilities, and our De-focus Attention Networks demonstrate that causal models equipped with the "de-focus attention" strategy can achieve comparable performances with non-causal models.
---
> **Q3: The operational mechanism of the Large Drop Path Rate seems similar to large spatial dropout/dropblock or randomly dropping masked image tokens. Why is it that the Large Drop Path Rate can achieve more advantageous results? What are the reasons that lead to its superior performance? Observation from the table in your reply.**
A3: Large spatial dropout or dropping masked image tokens could theoretically introduce some regularization effects. However, there are nuanced differences in their mechanisms and impacts on model training compared to large drop path rate.
- Large spatial dropout only drops portions of the tokens, so some tokens can still access information from neighboring tokens. Therefore, the network can still learn to focus on neighboring tokens and rely on depth to gather information.
- Dropping masked image tokens completely removes some input tokens, so the model may fail to learn to pay attention to tokens that are distant in positions. As a result, it does not prevent the model from leveraging depth to learn representations. Its regularization effect is thus not significant.
---
We hope that our response can address your concerns, and any further discussion is welcomed.