Parallel-mentoring for Offline Model-based Optimization

Summary

This paper introduces a novel offline model-based optimization (MBO) framework that incorporates a fine-tuning strategy for ensemble proxy models. Instead of directly using three independent proxy models, the paper suggests a "voting" strategy. In this scheme, based on the majority order of two (near) inputs, each model is fine-tuned using binary cross-entropy (BCE) loss before it is utilized for the optimization problem. The paper contends that this approach enables the proxy models to mentor each other, thereby leading to improved performance during the input optimization stage.

Strengths

- The paper is well-written and easy to follow. - The experimental results are evaluated extensively, as well as showing quite a good performance compared with prior offline MBO methods. - To best of my knowledge, exploring fine-tuning strategy to improve the performance of ensemble models is relatively under-explored in offline MBO literature. In this respect, I think this paper investigates important but overlooked problem.

Weaknesses

- My primary concern revolves around hyperparameter selection. It appears the method introduces a considerable number of additional hyperparameters, including, but not limited to, learning rates for fine-tuning soft-labels and proxy models, the variance $\delta^2$ for selecting an adjacent input from the current solution, and the number of near inputs, $K$. Although the authors demonstrated the stability and effect of these hyperparameters (e.g., $K$), an analysis of some other hyperparameters, such as $\delta^2$ or fine-tuning learning rates, is still missing. Given that the paper addresses "offline" MBO, the experiments should detail how these hyperparameters can be determined (e.g., $\delta^2$ might be defined as a function of input dimension or a constant). Alternatively, the authors should demonstrate that the proposed method is robust to variations in these hyperparameters. - The proposed fine-tuning strategy lacks the original regression objective; the model might produce inaccurate values at training points after fine-tuning. - Although it's not a major weakness, any theoretical guarantee or supporting intuition would significantly enhance the manuscript.

Questions

- The paper deals with discrete tasks as well, while the optimization strategy is based on gradient ascent. How the input optimization is performed? - I wonder if the authors tried to incorporate the method on prior offline MBO method, e.g., COMs to pre-trained the model first and then fine-tune models via the proposed fine-tuning strategy. - I wonder if the model is fine-tuned sequentially or not. Specifically, once $f_A$ is fine-tuned, then is fine-tuned $f_A$ used for fine-tuning $f_B$ or the original $f_A$ before fine-tuning used? - Why the fine-tuning step is set as 1? Did the authors try more steps?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors discussed the limitations and potential negative impact in their manuscript.

Summary

The aim of the paper is to tackle the out-of-distribution problem in offline model-based optimization methods. To this end, the authors drew inspiration from recent studies that proposed better ensemble proxies and weak ranking supervision signals. They proposed a new approach called parallel-mentoring, which consists of two main modules. The first module, a voting-based pairwise supervision module, generates consensus labels. The second module, an adaptive soft-labeling module, aims to learn more accurate soft labels. The authors conducted experiments on both continuous and discrete tasks, and their approach achieved the highest ranking in the results.

Strengths

1. The paper proposed a new approach, which settles the out-of-distribution problem in offline model-based optimization. 2. The paper is clear and easy to understand. 3. The paper conducts extensive experiments on the design-bench.

Weaknesses

1. The research question is poorly defined, and the statement of the importance of the research is lacking. The out-of-distribution problem is not clearly defined, and its significance is not convincingly conveyed. 2. There is no experimental evidence to support the assertion that the tri-mentoring approach can effectively address the out-of-distribution problem.

Questions

1. How does leveraging the knowledge of the static dataset help to learn more accurate soft-labels?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The authors have adequately addressed the limitations.

Summary

This paper introduces an innovative study that revolutionizes offline model-based optimization (offline MBO) by introducing a novel ensemble method. The proposed approach addresses a critical challenge in offline MBO, namely the handling of potentially inaccurate pseudo-labels generated by proxy models, particularly in out-of-distribution scenarios. It achieves this by harnessing the power of multiple parallel proxies, utilizing voting-based supervision, and incorporating adaptive pseudo-labeling. One of the notable strengths of this method is its intuitive ensemble framework, which effectively mitigates the inherent inaccuracies of proxy models. Despite the technical complexity and meticulous design tailored specifically for the offline MBO task, this approach demonstrates significant performance improvements across various benchmark datasets. I recommend accepting this paper with some minor revisions.

Strengths

1. This paper presents a pioneering method that effectively mitigates the issue of proxy model inaccuracy in out-of-distribution scenarios, offering a novel solution to this longstanding challenge. 2. The bi-level formulation utilized in this study demonstrates technical robustness and holds great potential for widespread application across various domains, highlighting its versatility and reliability. 3. The approach of employing a three-proxy ensemble may appear simple at first glance, but its effectiveness in improving results is remarkable, showcasing the power of this streamlined yet impactful technique. 4. The performance of the proposed method shows great promise, achieving impressive results not only at the 100th percentile but also at the 50th percentile, indicating consistent and reliable advancements across the entire distribution. 5. The ablation study presented in this paper is exemplary in its clarity and thoroughness, providing a comprehensive analysis of the individual components and their contributions to the overall method, enhancing the scientific rigor and understanding of the approach.

Weaknesses

1. The analysis lacks consideration for ensemble techniques, which could have enhanced the effectiveness of the offline MBO. Ensembles, being statistical methods, offer the opportunity for deeper analysis and intuition, especially from a statistical perspective. Incorporating such analysis and intuition into the manuscript would greatly improve its quality. 2. The algorithm appears to be overly complex, introducing numerous degrees of freedom. This complexity could limit the practicality of the offline MBO approach since there is no opportunity to fine-tune hyperparameters using an oracle function. Simplifying the algorithm and reducing the number of degrees of freedom would make it more practical and applicable in real-world scenarios. 3. It would be beneficial to provide a more comprehensive discussion on ensemble techniques in similar domains, such as offline biosequential design [1,2]. Exploring the applications of ensemble methods in these domains would strengthen the manuscript's relevance and provide valuable insights for researchers in related fields. [1] Kim, Minsu, et al. "Bootstrapped Training of Score-Conditioned Generator for Offline Design of Biological Sequences." arXiv preprint arXiv:2306.03111 (2023). [2] Jain, Moksh, et al. "Biological sequence design with gflownets." International Conference on Machine Learning. PMLR, 2022.

Questions

1. Can this method effectively solve high-dimensional biological sequence optimization problems, such as optimizing GFP and UTR sequences? 2. Could you please provide an approximate estimate of the training time required for this method? This information would be valuable for future researchers interested in implementing the approach. 3. Does the performance of the method depend on the number of proxies used? 4. What is the rationale behind using Tri-mentoring? How would leveraging bi-mentoring between the individuals involved affect the outcomes? 5. What are the expected outcomes when incorporating conservatism (e.g., COMs) or a Gaussian prior (e.g., ROMA) into the proxy model and integrating them with the tri-mentoring idea? Can these methods be orthogonal to each other?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

Authors already put the limitation part in the paper which is really valuable.

Summary

The paper studies offline model-based optimization, which maximizes a black-box objective with a dataset of designs and scores. Most approaches rely on training a dataset proxy to approximate the black-box objective and perform SGD on the objective. This paper proposes using three proxies and voting-based supervision to generate labels. In addition, they use adaptive soft-labeling to make the proposed method more robust to label noises.

Strengths

+ A clear description of the problem background and the proposed algorithm procedure; + Extensive ablation studies on various components, such as the voting based pairwise supervision;

Weaknesses

- It's weak and less general that the proposed method constrains itself into the three parallel mentoring cases. I'd suggest the writing changed to a much more general form of any number of the parallel proxies; - The adaptive soft labeling seems to directly apply the meta-learning method /bi-level optimization method on the pseudo-label/soft label. The novelty seems not very justifiable.

Questions

Please address the two weaknesses above.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

NA

Response

Thank you for the detailed response. The provided response addressed most of my concerns well. Hence, I raise my score to 7. Please add the related discussion in the final manuscript if the paper is accepted.

Looking forward to your feedback

We sincerely appreciate the time and effort you have invested in reviewing our work. We have made diligent efforts to address all the concerns you listed, including: - Clarified the research question and the out-of-distribution problem and emphasized the significance of both. - Provided experimental evidence to demonstrate the efficacy of our tri-mentoring approach in handling out-of-distribution issues - Explained how leveraging the knowledge of the static dataset aids in the learning of more accurate soft-labels Could you please provide further clarification if there are any aspects that remain unclear? In the past week, we have received constructive responses from the other three reviewers. We value your perspective and are eagerly looking forward to your feedback. We stand ready to make any necessary revisions to enhance our paper.