Thank you for your continued engagement. We provide responses to your questions below.
## Question 1
We appreciate the catch; we agree that our sentence in lines 158-159 was unclear. We will rewrite the sentence to say “…it can in fact improve WGA conditional on the **lack of** a small minority group within the minority class.” Thus, the claim is consistent with Figure 1 and lines 188-189.
## Question 2
We agree with the reviewer that a more controlled investigation would be beneficial. To show that our conclusions hold in a controlled environment, we extend the synthetic dataset of [Sagawa et al., 2020: An investigation of why overparameterization exacerbates spurious correlations] to our class-imbalanced setting.
We generate a dataset of $100000$ points with labels $y\in \\{-1,1\\}$ and spurious attributes $s\in\\{-1,1\\}$ as follows. Each $(y, s)$ group has its own distribution over input features $x = [x_{core},x_{spu}]\in\mathbb{R}^{2d}$, corresponding to core features in $\mathbb{R}^d$ generated from the label $y$ and spurious features in $\mathbb{R}^d$ generated from the spurious attribute $s$:
$$x_{core}\sim \mathcal{N}(y\vec{1}, \sigma^2_{core} \mathbf{I}\_d)\qquad x_{spu}\sim \mathcal{N}(s\vec{1}, \sigma^2_{spu} \mathbf{I}_d).$$
We set $d=100$, $\sigma^2_{core}=100$, and $\sigma^2_{spu}=1$ following [Sagawa et al., 2020]. Different from their setup, we generate the data according to a class-imbalanced distribution where a small minority group is present within the majority class. To do so, we introduce a variable $\alpha\in [0.5, 1.0)$ which controls the size of the minority group within the majority class. Our dataset composition is detailed below:
| Group (y, s) | Number of data |
|--------------|----------------------|
| (-1, -1) | $45000 \alpha$ |
| (-1, 1) | $45000 (1 - \alpha)$ |
| (1, -1) | $5000$ |
| (1, 1) | $5000$ |
We train a two-layer ReLU neural network with hidden dimension $64$ using full-batch gradient descent with learning rate $0.01$ and momentum $0.9$. We class-balance with the subsetting method and evaluate on a held-out dataset of $100000$ points balanced across groups. Our WGA results are detailed below, averaged over ten seeds:
| $\alpha$ | Subsetting | No class-balancing |
|----------|------------|--------------------|
| 0.5 | 45.2 | 2.3 |
| 0.6 | 37.6 | 1.1 |
| 0.7 | 28.1 | 0.71 |
| 0.8 | 23.1 | 0.35 |
| 0.9 | 8.5 | 0.1 |
As seen in the table, the smaller the size of the minority group in the majority class (i.e., the larger $\alpha$ is), the worse the subsetting method performance becomes. We believe this contributes to the justification of our conclusions in a controlled environment, and we would be happy to discuss further if the reviewer has additional concerns.
## Question 3
We appreciate the reviewer’s concern. We believe the sensitivity to the degree of imbalance is not just a property of the dataset, but of the model as well. In Figure 3 in the global response PDF, we show that Swin Transformer Base exhibits a larger degree of sensitivity on Waterbirds compared to ConvNeXt-V2. Below, we include a table with a more explicit comparison (WGA averaged over 3 seeds):
| Architecture | Mixture balancing | Upweighting | Upsampling | No class-balancing | Subsetting |
|-----------------------|-------------------|-------------|------------|--------------------|------------|
| ConvNeXt-V2 Base | 81.1 | 80.2 | 79.9 | 80.4 | 67.5 |
| Swin Transformer Base | 90.5 | 88.8 | 87.0 | 85.7 | 82.2 |
Contrary to the reviewer’s concern that “the Waterbirds dataset shows high WGA for no class-balancing methods and the minimal difference compared to upsampling,” the suboptimality of no-class balancing and upsampling on Waterbirds is clearly observed for Swin Transformer Base. Together with the strong precedent in the literature of evaluating class-balancing methods on Waterbirds [3, 7], we believe Waterbirds is well-suited for our investigation.
## Question 4
We would like to clarify that upsampling **does not** lead to more iterations compared to training without class-balancing. In upsampling, we sample the mini-batches uniformly over the classes, but each mini-batch contains the same amount of data and we train for the same amount of steps as without class-balancing. We will make this more explicit in the paper.
## Question 5
We agree that these results will be important for the community. In Figures 2 and 3 in the global response PDF, we replicated our class-balancing and scaling experiments for different class-balancing methods using the ResNet family (including ResNet50) and Swin Transformer. We will include more comprehensive ResNet50 results in the updated version of the paper.