Incorporating Surrogate Gradient Norm to Improve Offline Optimization Techniques

Summary

The article presents a method to improve offline optimization techniques by integrating the concept of model sharpness into the training. A constraint is introduced that limits the model sharpness to not exceed a user-specified threshold.

Strengths

1. The approach is model-agnostic, making it applicable across different types of models and optimization tasks without needing specific adjustments for each model or task. 2. The method is backed by a solid theoretical analysis, providing a robust framework for understanding and predicting the behavior of the regularization effect on offline optimization. 3. The method has been empirically tested on a variety of tasks, showing significant performance improvements, thus validating the theoretical predictions.

Weaknesses

1. The effectiveness of the method heavily depends on the correct setting of hyperparameters like the sharpness threshold, which can be tricky to optimize without extensive experimentation. 2. The computation complexity is increased. Can the author give the complexity analysis? 3. How to ensure the proposed method converge to a stationary point of the original problem?

Questions

see weakness

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

NA

Summary

This paper proposes a novel model-agnostic approach to enhance offline optimization methods by incorporating surrogate gradient norms. The paper provides a thorough review of existing literature, a clear problem definition, and detailed descriptions of the proposed methods and their implementation. The experiments are well-designed, covering various benchmarks and baselines, and the results are robustly analyzed.

Strengths

- a new regularizer based on surrogate sharpness, characterized by the surrogate's maximum output change under low-energy parameter perturbation. This regularizer is designed to be model-agnostic, making it broadly applicable across different surrogate and search models. - a practical approximation that reduces the surrogate sharpness measurement to a function of the surrogate’s gradient norm. This allows the optimization task to be transformed into a constrained optimization problem, which can be solved using existing optimization solvers. - a theoretical analysis demonstrating that reducing surrogate sharpness on an offline dataset provably reduces its generalized sharpness on unseen data. - extensive experimentation on a diverse range of optimization tasks shows that reducing surrogate sharpness often leads to performance improvements.

Weaknesses

- As the paper mentioned that they draw inspiration from the sharpness-aware minimization (SAM), the paper’s novelty could be better articulated in the context of existing work on surrogate model regularization and sharpness-aware optimization. Specifically, discuss how the proposed surrogate sharpness measure offers advantages over sharpness-aware minimization (SAM). - Assumption 2 (positive minimum eigenvalue of the parameter Hessian) might be too restrictive. It would be more helpful if the authors can provide more intuitive explanations or empirical evidence to justify these assumptions and discuss the implications if these assumptions are violated and how the method’s performance might be affected. - The proposed method involves computationally expensive operations, such as gradient norm calculations and constrained optimization. I'm wondering how expensive the proposed method is compared to existing base approaches in terms of the training time.

Questions

See above.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes. In Appendix H

Summary

This paper introduce a sharpness-aware optimization to improve out-of-distribution generalization. While inspired by SAM, the major difference is that this work considers the sharpness of predictor outputs rather than loss landscape. With the proposed notion of sharpness, practical algorithm, IGNITE, is developed through an empirical approximation (Eq. 6) and Taylor approximation of the sharpness constraint.

Strengths

- This material is well written and easy to follow. - Generalization-aware optimizer is very relevant. - Technical development appears viable and clear.

Weaknesses

- It is not immediately clear, on an intuition level, why this new notion of sharpness is better than loss sharpness in SAM [8]. - Table 2 only presents SAM on two algorithms (REINFORCE and GA) with no error interval reported. A more throughout benchmarking could improve the empirical evaluation.

Questions

- As illustrated in Figure 1, the intuition is that if the oracle is included in the perturbation neighborhood, a smoother predictor tends to have smaller error. However, Table 1 observes that IGNITE applied to some sub-optimal predictors, for example REINFORCE in Ant Morphology, it still achieves good performance. Since the suboptimal prediction performance may imply that $\hat{\omega}$ is not proximal to $\omega^\star$ hence the local approximation may break in practice. I am curious that why IGNITE should still work in this case, I appreciate if the authors could provide some high-level intuitions.

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

See above. --- post-rebuttal: -> 5

Summary

The paper studies the problem of offline optimization for material design problems. The paper proposes a model-agnostic method that changes the parameters of a model by constraining the sharpness of the model's predictions. Since the model generates smoother predictions, the error between the predictions and the oracle tends to be smaller and therefore the performance of the offline model is potentially more accurate. The paper gives a theoretical analysis and shows that the empirical sharpness can bound the theoretical sharpness. The paper applies the method to various offline optimization method and shows improvements in most of the cases.

Strengths

1. The paper starts with strong intuition, proposes a general optimization objective, and proposes a feasible algorithm to solve the objective efficiently. The paper also gives a theoretical analysis that connects the empirical objective to the theoretical sharpness. 2. The proposed method is model agnostic and therefore the method can be applied to many existing offline optimization methods to get improved performance. 3. The proposed method is shown to achieve good performance across a variety of tasks and is applied with various offline optimization methods.

Weaknesses

1. After certain approximations, the objective is optimized using BDMM. It would be interesting to analyze the specific structures of the objective and see if more effective optimization techniques can be applied. This is also related to the model-agnostic feature of the proposed method. While having the model-agnostic property is convenient, it may also miss the specific structures of the problems/models so that the performance is not ideal. Also, there is no empirical evidence showing whether the optimization is effective or how well the optimization converges in practice. 2. The comparisons with baseline methods are only compared for a limited number of tasks. To fully demonstrate the effectiveness, I think a thorough comparison as done in Table 1 would be necessary. 3. The paper very briefly mentions the background of material design. I think a more comprehensive introduction could be beneficial and probably the paper should add detailed descriptions of the tasks and datasets.

Questions

Please address the weaknesses part.

Rating

7

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

Limitations are properly addressed. No obvious negative societal impact.

Summary

This paper introduces a model-agnostic regularization method that reduces surrogate model sharpness to improve generalization in offline optimization. By incorporating a surrogate sharpness measure into the training loss, they provide theoretical proof and extensive experimental validation showing that this approach enhances performance on unseen data, achieving up to a 9.6% improvement in various optimization tasks. The proposed algorithm, IGNITE, demonstrates the effectiveness of this technique across different surrogate models and tasks.

Strengths

1. This paper introduces a novel concept of surrogate sharpness for offline optimization, providing a new robust optimization approach that differs from previous loss-based sharpness-aware minimization methods. 2. Based on surrogate sharpness, the authors propose a new algorithm, IGNITE, and provide corresponding theoretical analysis, demonstrating its ability to bound the worst-case generalized surrogate sharpness. 3. The proposed method can be combined with other offline optimization techniques, consistently enhancing generalization performance when integrated.

Weaknesses

1. The Introduction section does not clearly emphasize the difference between surrogate sharpness and loss sharpness. Figure 1 could be positioned earlier in the paper. Additionally, the annotations and explanations in Figure 1a are somewhat confusing; the concepts of \(\sigma\) and \(\sigma^*\) are not adequately explained in the figure and caption, which could lead to misunderstandings. 2. The Theoretical Analysis section (Section 4) should include more insightful discussions based on Theorems 1 and 2, ideally tying them to the results in the experimental section. The formatting of Equation 16 could also be improved. 3. In the experimental section, the authors mention that the IGNITE algorithm introduces five hyperparameters, but the ablation study only analyzes the effects of a subset of these hyperparameters. Many experimental results are placed in the supplementary materials, but they are not referenced in the main text.

Questions

Please see the weaknesses part.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Please see the weaknesses part.

Summary

The paper proposed IGNITE, a promising method for solving out-of-distribution issue in offline training by introducing model sharpness into the training loss of the surrogate as a regularizer. The key innovation of IGNITE lies on incorperating sharpness-aware minimization into offline training and developing theoretical analysis to show that reducing surrogate sharpness on the offline dataset provably reduces its generalized sharpness on unseen data. Experiments on benchmark datasets show the promising performance.

Strengths

1. Well motivated. 2. Easy to follow. I like the style of writing. 3. Experimental results seem promissing (9.6% performance boost). 4. Solid theoretical analysis are provided.

Weaknesses

Although the paper has many strengths, there are still some weaknesses: 1. Sharpness-aware minimization (SAM) is widely explored in many fields. I think this submission lacks sufficient introducing of SAM, especially on its development. Take some recent works as an example: Enhancing sharpness-aware optimization through variance suppression. (NeurIPS'23) Normalization layers are all that sharpness-aware minimization needs. (NeurIPS'23) Domain-Inspired Sharpness-Aware Minimization Under Domain Shifts. (ICLR'24) Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimization. (ICML'24) 2. The gain on performance is not always stable, as shown in the Table 4 and 5. 3. Lacking sharpness visualization of objectives on unseen data.

Questions

1. I recommend the authors to extent the contends on the related works of sharpness-aware minimization. 2. Can the authors explain the potential reason on the performance drop shown in Table 4 and 5? 3. Can the authors show the sharpness of objectives on unseen data before and after using your algorithm?

Rating

6

Confidence

3

Soundness

3

Presentation

4

Contribution

2

Limitations

I don't think there are any negative societal impacts of this work.

Summary

This paper focuses on offline optimization, presenting a novel approach to enhance the surrogate landscape's sharpness, thereby improving generalization. The authors introduce a gradient norm as an approximation of surrogate sharpness through a first-order Taylor expansion, resulting in a Lagrangian formulation. They employ two widely adopted methods—BDMM and a fixed multiplier as solvers for the Lagrangian. Theoretically, the authors provide a generalization bound for the worst-case prediction change. Empirically, they validate their approach using the widely adopted offline optimization dataset, design-bench, demonstrating relatively good performance.

Strengths

1. The idea of using a sharpness-aware method in offline optimization is well-motivated. 2. The proposed method performs well on some datasets with specific baselines. 3. The authors provide a generalization analysis.

Weaknesses

Though I agree with the ideas of using a sharpness-aware method in offline optimization, I think more details should be justified: 1. I am somewhat unsure whether the authors analyze the generalization bound on $\mathcal{R}_{X}(\omega)$ properly. Does the generalization over $R_{X}(\omega)$ help to improve the generalization of $L_{X}(\omega)$, which is the objective we want to generalize. Why not analyze the generalization bound w,r,t $L_{X}(\omega)$? 2. Is the bound tight? I understand that the original SAM paper [8] also provides a similar bound. But if it is not tight, we do not know whether it can guide the design of our algorithm. 3. Moreover, the authors should add more details to their proofs to help readers better understand. For example, in line 416, the authors should present it as a lemma rather than directly stating the results of [8] as there are some assumptions in their proofs which might not be discussed in your proof. In line 458, the authors say that they use Taylor's remainder term, but it seems that the remainder term is directly eliminated. Could the authors discuss more details? 4. Another concern is that I am not very sure whether Assumption 2 is a strong assumption. If the assumption holds, then the Hessian is positive definite. Since the proof highly relies on it, I am afraid the generalization bound is not established in the general stationary point (the Hessian is positive semi-definite rather than positive definite). 5. Though the proposed method shows relatively higher performance than some corresponding baselines on some datasets, the overall improvements are marginal and sometimes even decrease, as shown in table 1. Moreover, different percentages show vastly different performance. For example, reinforce+IGNITE shows a 6.5\% gain over reinforce at the 100th percentile level, but a -4.6\% at the 80th percentile level. How can this be explained? Does it mean that the proposed method is not robust? 6. Is there any analysis on memory and time? The term includes the gradient of the gradient $\nabla_{\omega}|| \nabla_{\omega} h\left(\omega^t\right) ||$, which I am concerned is both time and memory-consuming. In all, I think the idea is interesting, and the empirical and theoretical results are promising. But I still think more details should be added before I can vote for acceptance.

Questions

Please answer my question mentioned above.

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

Please see the weaknesses part.

Follow-up

Dear Reviewers, Thank you again for the detailed reviews. Please let us know if you have additional questions for us. We will be more than happy to provide further clarification and discussion. Reviewers CQYH and xjU2: May we know if our responses have addressed your concerns sufficiently? We hope our responses have improved your overall assessment of our work. Best regards, Authors

Response to author

Thank you for the detailed response from the author. The major concern has already been addressed. I will keep my score.

Thank you very much for increasing the score to 5

Dear Reviewer, We sincerely appreciate your prompt decision and your support for our work. We are very grateful for the time you dedicated to discussing our research and the valuable feedback that has greatly improved our work. Best regards, Authors

Follow-up

Dear Reviewer CQYH, May we know if our response has addressed your questions? Thank you very much for the interesting questions and suggestions. Best regards, Authors

Thanks for the rebuttal

Thanks for the rebuttal. I don't have further questions.

Post-rebuttal summary

Dear AC and reviewers, We would like to express our gratitude to the Area Chair for coordinating the review process of our paper. Following the rebuttal discussion, we are very happy that all reviewers have recognized our contribution and provided acceptance scores. We sincerely thank **Reviewers WGWE, tYoe, qZmG, and kVLN** for maintaining their acceptance scores. We appreciate **Reviewer qc6B** for raising the score to 7 in support of our paper's acceptance. We are also thankful to **Reviewers xjU2 and CQYH** for increasing their scores toward acceptance. Overall, based on the post-rebuttal feedback from **Reviewers WGWE, tYoe, kVLN, qc6B, xjU2, CQYH** and the sustained acceptance rating (7) from **Reviewer qZmG**, we believe our rebuttal has addressed all concerns raised by the reviewers. -- A summary of our responses to address these concerns is provided below: 1. We provided further elaboration on the advantage of surrogate sharpness over loss sharpness, emphasizing the conceptual novelty of our work (**tYoe**,**qc6B**, **xjU2**, **CQYH**). 2. We demonstrated that Assumption 2 is not overly restrictive (**qc6B**, **xjU2**) and provided detailed complexity analysis of IGNITE (**qc6B**, **kVLN**, **xjU2**). 3. We showed the reduction in surrogate sharpness on unseen data after conditioning with IGNITE (**WGWE**). 4. We illustrated the convergence and effectiveness of the BDMM algorithm by plotting the training loss and sharpness value during the surrogate fitting process (**qZmG**). 5. We provided a detailed explanation of the tightness of the generalization bound (**xjU2**). 6. We clarified how IGNITE can enhance even base optimizers with poor performance (**CQYH**). Once again, we would like to thank the Area Chair and all reviewers for their time and effort in reviewing our paper. Best regards, Authors