Summary of Results in Rebuttal
## New Ablations (as suggested by reviewer 4YtT)
### Choice of cost function
We run $||u||_\beta$ with $\beta =1, 2, 4$, and infinity, for fashion MNIST in both the federated and invariant setting. As discussed in Section 3.3, $\beta = 1$ recovers penalty based methods.
**Federated Learning**
| $\beta$ | Mean Acc | IQR | Max Range |
|------------|----------|-------|-----------|
| 1.0 | $93.3$ | $2.5$ | $49.5$ |
| 2.0 | $93.4$ | $2.4$ | $50.6$ |
| 4.0 | $93.4$ | $2.6$ | $45.6$ |
| $\infty$ | $92.7$ | $3.8$ | $28.1$ |
**Response Table 4**:*Cost function ablation for the heterogeneus federated learning in fashion-MNIST using 100 clients, 3 minority classes, an imbalance ratio of $\rho=10$ and dirichlet allocation with parameter $d=0.3$. We report the mean, interquartile range and range (maximum value minus minimum value) test accuracy across clients.*
**Invariance**
| $\beta$ | Partially Rotated | Translated | Scaled |
|----------|-------------------|-----------------|-----------------|
| 1.0 | $86.08\pm 0.38$ | $86.85\pm 0.20$ | $85.02\pm 0.46$ |
| 2.0 | $85.37\pm 0.17$ | $86.65\pm 0.25$ | $84.92\pm 0.39$ |
| 4.0 | $85.23\pm 0.20$ | $86.64\pm 0.11$ | $84.65\pm 0.68$ |
| $\infty$ | $82.94\pm 0.17$ | $85.47\pm 0.19$ | $83.35\pm 0.49$ |
**Response Table 5**:*Cost function ablation for invariant fashion-MNIST datasets. We compute the mean and standard deviation in test accuracy across three independent runs.*
### Learning Rates
We run an ablation on $\eta_u, \eta_\lambda$, the perturbation and dual learning rates over a small grid of 12 values, and find that, in this range, the performance of the algorithm is not overly sensitive to this choice. We also observe that that the rates that were used in the paper ($\eta_u = 0.1, \eta_\lambda = 2$) are not optimal in this setup, and thus further improvements in performance could be obtained through more extensive hyperparameter tuning, but this was not the focus of our experiments.
| $\eta_u$ \ $\eta_\lambda$ | 0.1 | 0.5 | 1 | 2 |
|-----------------------|------|------|------|------|
| 0.1 | 81.4 | 81.6 | 81.8 | 81.8 |
| 0.5 | 81.4 | 80.9 | 81.4 | 81.1 |
| 1 | 81.6 | 81.6 | 81.6 | 81.7 |
**Response Table 6**:*Dual and resilient learning rate ablation in Heterogenous federated learning setting. We report mean Test Accuracy for CIFAR100 using 100 clients, 3 minority classes, an imbalance ratio of $\rho=10$ and dirichlet allocation with parameter $d=0.3$.*
## Comparisons to constrained learning and discussion (reviewers *69dA*, *iyNP* and *4YtT*)
In addition to the quantitative metrics presented above, the results included both in the experimental section and appendix mainly aim to highlight the differences between our approach and existing constrained learning approaches (reviewers *69dA* and *iyNP*) . In doing so, we discuss the inherent trade-offs or limitations of out approach (reviewers *69dA* and *4YtT*). Here we provide a brief summary and expand on how these address the fundamental aspects of our method.
**Constraint Relaxation and Relative difficulty**:
This illustrates how our method adapts $u$ depending on how difficult it is to satisfy the constraint, whereas the constrained approach enforces the constraint ($u=0$) regardless of the price to pay in performance. Since the impact on performance depends on the data distribution, the most imbalanced clients (Figure 2 in main text) are relaxed more. This shows empirically that our method behaves as intended *in a learning setup*. The same happens for invariances that are not present in the dataset (Figure 8 in Appendix).
**Controlling the performance vs. relaxation trade-off**:
These experiments (Figure 2 in main text, Figure 4 in the Appendix) illustrate the inherent trade-off between imposing constraints and having better performance in terms of statistical risk in the objective.