Further Response to Reviewer KTaQ
Dear Reviewer `KTaQ`,
Thank you for your further comments. We address your questions below:
**Q: Could you please give a pointer in the paper, where you describe how it affects the training of the network, in particular, how it affects gradient computation?**
**A:** Thank you for bringing this to our attention. In lines 177 - 188 and 210 - 217, we have elaborated on the method for computing the robust attention mechanism and its integration during the model training phase. Currently, comprehensive details of the model training can be accessed in the provided codebase. In future versions of our manuscript, we will expand on how the robust attention mechanism impacts gradient computation. To directly address your feedback, we've outlined our responses below:
For weight-based methods, the weight set $\omega$ is kept separate from the gradient updates of the Transformer models. Specifically, the `requires_grad` attribute is set to `False` for the weight tensor. The determination of this set of weights is achieved via the iterative methods elaborated upon in our paper. This design choice is motivated by two principal reasons:
(1) Allowing gradient updates for the weight tensors considerably decelerates the training process and, in our observations, doesn't produce any discernible improvements in the end results.
(2) Permitting gradient updates for these tensors could potentially disrupt the gradient flow, making it unstable.
However, this weight set does affect the learned parameters of Transformers. Should there be outliers in the training dataset, this auxiliary weight set introduced would generate attention vectors that differ from those produced by the standard softmax attention mechanism. This, in turn, will influence the model parameters learned during the training phase.
For the Median-of-Means-based method, determining the median block is non-differentiable. Once this block is identified and employed to compute the robust attention vectors, gradient updates focus solely on a subset of the parameter matrix, derived from the selected indices of the input sequence's median block. As a result, the attention vector, and by extension the model's training process, is minimally impacted by the input sequence's outlier regions, which are typically omitted from the median block.
In summary, neither of our proposed methods introduces extra differentiable parameters. However, they do affect the model parameters learned during training via the resultant robust attention vectors.
**Q: Could you please clarify, what is dimension of $q$? And what is assumed about the probability density $p$ we are estimating? As far as I know even in the classical case of density estimation, the error bound on $L_{\infty}$ norm depends dramatically on the dimension.**
**A:** We apologize for the ambiguity about the assumptions on the theoretical result. We will carefully clarify these assumptions in the revision.
For your question, the dimension of query $q$ is $D$, which is also the dimension of the key. For the probability density function $p$, we assume that it is differentiable up to order $m + 1$ where $m$ is a constant in the statement of Theorem 1 in Appendix C of the paper.
From the statement of Theorem 1, by choosing the optimal radius $R$ to balance the bias, which is at the order $O(1/R^{m + 1})$, and the variance, which is at the order of $O(R^{D}/ N)$ (up to some logarithmic function of $R$) in the upper bound for the $L_{\infty}$ norm, the optimal choice of $R$ is $O(N^{1/(D+ 2(m + 1))})$, which leads to the rate $N^{-(m + 1)/(D + 2(m + 1))}$ for the Median-of-Means estimator in Appendix C.
We note that this rate is standard and similar to the rate of density estimation under the standard Transformer (without any outliers in the data), which is essentially the rate of kernel density estimator with Gaussian kernel.
Please let us know if you have additional questions.
Sincerely,
Authors