Thank you for the response
We thank the reviewer for participating in the discussion phase. We proceed to address the remaining issues:
### Systematic Generalization in [2]
Considering that we agree on that a systematic generalization benchmark needs to exhibit entirely novel combinations of attributes in the test set samples, we disagree that [2] examines such a case. This is already evident in Fig. 1. Specifically, Fig 1.a presents examples from the training set. Last row contains samples of digit 9, while it suggests that the biasing/dominant color for 9 is fuschia. Notice how a cyan 9 (which is not the dominant color) also appears in the training set. At the same time Fig 9.c presents samples from the systematic generalization test set. Observe that it also contains a cyan 9 in the last row.
This is not a coincidence. The cited line from the paper in the comment just means that the test set contains only bias-conflicting samples; that is samples which are generated by combinations of (digit, not biasing color of that digit), like non-fuschia 9s. It does not claim, however, that such combinations are not exposed in the training set. In fact, "own biasing color" can just mean "highly correlated with the digit in the training set", as it is the case in [2] and for example in [8]. To underline this, we repeat here the training set generation procedure as it is presented in [2], and also mentioned in our rebuttal:
> COLOURED MNIST: Consider an illustrative dataset with coloured MNIST digits. For the training set, Tr, MNIST digits are coloured with a set of digit-correlated “biasing” colours 80% of the time, and with ten random colours that are different from the biasing colours the remaining 20% of the time.
We hope that this makes clear that **the training set of [2] is exposed to all possible combinations of digits and colors**, and hence under their training set construction there can be no possible test set containing samples from entirely novel combinations. Thus, in contrast to our work (see Fig 2.d and Appendix D.1 of our work), [2] does not actually examine a systematic generalization task, but an easier group-robustness case as it is further elaborated in our rebuttal.
[8] Nam, Junhyun, et al. "Learning from failure: De-biasing classifier from biased classifier." NeurIPS. 2020.
### In relation to [4] and [5]
[4] operates under the additional assumption that input data is available in pairs of (true context, bias source). In their case (VQA tasks) true context refers to an image and the biasing source is the textual question. In our work, true predictive features and bias features are not already "disentangled" like that in an input observation $x$. In that, our method is *simpler and more generic*.
On the other hand, [5] does not have a mechanism which prevents the biased model from fitting exactly the biased training set. If the biased networks fits entirely the training set, consequent logit adjustment will not be effective as it no longer a proxy for $p(y|z)$, but on the training samples it is simply acts as look-up table for the training set. To prevent that, [5] depends on early-stopping hyperparameters for the biased network, which are tuned using unspecified data resources, possibly a balanced validation set. In contrast, our work tunes hyperparameters based on our proposed robust model selection criterion using iid data, as well as our pretraining approach based on SSL and linear probing yields decreased sensitivity to the number of training steps of the biased network (Left plot Fig. 2 of rebuttal material).