Soft ascent-descent as a stable and flexible alternative to flooding

Summary

Flooding method is a previous method which aims at improving the generalization performance by making the average loss equal to a given threshold. This paper makes two changes of the 'flooding' method: 1. Rather than making the average loss equal to a given threshold, they make the point-wise loss equal to a given threshold. 2. To make them equal, previous work uses the absolute function to measure the difference, this work uses a softer function. Experiments are conducted to support their method.

Strengths

This paper is easy to follow. The organization of this paper is clear. The proposed link between flooding and sharpness, as shown in 2.2, is interesting and insightful. It explains why flooding method works from the flatness perspective.

Weaknesses

Although the generalization gap of the proposed method is smaller (shown in Table 1), the accuracy is not larger than SAM (shown in Figure 4). What are the advantages of the proposed method compared with SAM?

Questions

None

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

None

Summary

The flooding method sets a threshold for the average surrogate loss within a mini-batch during training: if above it do gradient descent as usual but if below it switch to gradient ascent. This paper updates the flooding method in 2 ways: 1) it invert the order of the sign function and the aggregation making it a pointwise method rather than an average method and 2) it replaces the sign function with a continuous, bounded, monotonic function. It provides numerical results to demonstrate the behavior of the new proposed method in Section 3.1 and show theoretical results to compare the convergence properties of the new method with flooding in Section 3.2. Finally, experiments show that the proposed method have similar accuracy compared with flooding/SAM while having much lower generalization gap and smaller model norms (despite not using any explicit shrinkage methods).

Strengths

- The soft truncation is a nice idea to improve the flooding method. The behavior shown in Fig. 1 & 2 is intuitive. - Stationarity guarantees are provided in the theoretical studies and the intuition is provided in Remark 4. - Experiments show softAD has the same level of test accuracy (SAM may be slightly better but at the double the gradient cost) while achieving better level of surrogate test loss. Furthermore, the generalization gap is much smaller and model norms are usually smaller than other baselines. The flood level comparisons are interesting: the softAD allows for a larger flood level. - The discussions about links to sharpness and OCE-like criteria are interesting and informative. - The story, organization, and presentation is clear and easy to read. There are a lot of informative information in the appendix as well.

Weaknesses

- A similar pointwise idea was proposed in "iFlood: A Stable and Effective Regularizer" (ICLR 2022). It would be helpful to have some discussions about iFlood and explain if there are any differences in the pointwise idea. - The experiments are informative, clearly showing the benefits of the approach. However, given that the pointwise idea is already proposed, as an ablation study, it would be important to further check which parts are more effective: the pointwise component or the soft truncation component. If the paper can show that the soft truncation is important, that would make the paper's contribution clear.

Questions

(I wrote my main suggestions in the "Weaknesses section". Here I provide some other comments/questions.) - Having a low generalization gap and better surrogate loss seems to be one strong benefit of the approach. I wonder if the paper can check calibration (such as ECE) which is often correlated with lower test losses. - It is interesting that softAD allows a higher theta threshold. Is it possible to provide some discussions on why the proposed method allows a higher theta?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

I believe there is no immediate negative societal impact of this work. The paper discuss that hyper-parameter selection remains and provide future directions to remove this aspect.

Summary

This paper proposes a method called SoftAD (soft ascent-descent) which aims to improve the "flooding" method. Specifically, it downweights points on the borderline, limits the effects of outliers, and retains the ascent-descent effect of flooding, with no additional computational overhead. Stationarity guarantees of these two methods are provided. Empirical results illustrate that SoftAD can have comparable classification accuracy with flooding while enjoying a smaller loss generalization gap and model norm.

Strengths

1. Overall, this paper is well written. 2. The experimental results demonstrate the merit of the proposed method SoftAD over two baselines (i.e., flooding and SAM). Specifically, SoftAD has competitive classification accuracy with these two methods while enjoying a smaller loss generalization gap and model norm.

Weaknesses

1. The motivation might be confusing. As the authors claimed, there are situations where we are interested in other metrics such as model complexity or average surrogate loss at test time. I can not imagine these situations except the generalization analyses of classification accuracy might involve these factors. 2. The proposed method is heuristic which lacks theoretical support. More specifically, generalization analyses are missing to support the claims. 3. Although it is good to give the optimization convergence guarantees of SoftAD and flooding, they can be not directly comparable because one is high probability bound and another is expectation bound.

Questions

Please give the situations where we are interested in other metrics such as model complexity or average surrogate loss at test time.

Rating

4

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

Please see the Weaknesses part.

Summary

The paper proposes a new optimization technique, Soft Ascent-Descent (SoftAD), aimed at enhancing the stability and performance of machine learning models. The method balances loss minimization and model complexity, positioning itself as an improvement over existing techniques like Flooding and SAM. The paper provides a theoretical foundation, empirical studies across multiple datasets, and a comparison of convergence properties with existing methods, claiming superior stability and often better test loss outcomes.

Strengths

The paper discusses SoftAD designed to balance minimizing loss and controlling model complexity. This method is compared to existing optimization techniques like Flooding and SAM and is shown to provide better stability and often superior test loss outcomes. The theoretical foundation of SoftAD is robust, with clear mathematical formulations and comparisons to existing methods. Empirical validation includes extensive experiments on multiple datasets, such as CIFAR-10, CIFAR-100, FashionMNIST, and SVHN, demonstrating the effectiveness of SoftAD in practical applications. The paper offers a comprehensive analysis of model norms and loss trajectories, underscoring the practical benefits of the method. Additionally, the document is well-structured, featuring clear explanations, figures, and tables that effectively support its claims and contributions.

Weaknesses

1. The SoftAD method may be more complex to implement compared to traditional methods, which could limit its adoption. Implementing SoftAD involves managing both ascent and descent phases, which adds to the complexity. This requires additional code for conditionally switching between ascent and descent based on the loss value and the threshold θ. The complexity is implied by the need to incorporate equations (3) and (4) for the ascent and descent phases into the optimization loop. 2. The method's performance depends on the setting of the threshold parameter θ, which can be non-trivial to optimize. The threshold θ determines when the algorithm switches between ascent and descent phases, directly impacting the performance. Choosing an inappropriate θ can lead to suboptimal results. 3. While the experiments are thorough, additional datasets and real-world applications could further validate the robustness of SoftAD. The experiments are conducted on CIFAR-10, CIFAR-100, FashionMNIST, and SVHN. While these datasets are standard benchmarks, they do not cover the full range of possible applications, especially large-scale or domain-specific datasets. The scope of the experiments is detailed in Section 4, where the datasets used are listed. 4. The paper does not provide a detailed analysis of how sensitive the performance is to hyperparameters other than θ. Changes in other hyperparameters can significantly affect the performance of optimization algorithms. The focus on θ without detailed analysis of other hyperparameters is evident in Section 5, where the experiments primarily vary θ. 5. The paper acknowledges the difficulty in setting the threshold θ but does not provide a concrete solution for it, which might be a crucial limitation for practical implementation. 6. While SoftAD shows better performance in terms of test loss and model norms, it is not always superior in terms of test accuracy when compared to SAM, as noted in the empirical results.

Questions

1. How do you suggest optimizing the threshold parameter θ in practice? Are there any heuristics or automated methods that can be employed? 2. How does SoftAD perform on larger datasets and models, such as those used in natural language processing or large-scale image recognition tasks? 3. Besides θ, are there other critical hyperparameters that significantly affect the performance of SoftAD? How sensitive is the method to these parameters? 4. Can you provide examples or case studies where SoftAD has been successfully applied in real-world scenarios?

Rating

5

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

Refer to Weaknesses

need your reply

To Reviewer JmzP, Please reply to the authors' comments because your score is low compared to the other reviewers. If you are going to keep the current score, please explain in detail why. best Area Chair,