Guided Trajectory Generation with Diffusion Models for Offline Model-based Optimization

Summary

The paper proposes a novel conditional generative modeling approach using diffusion models for offline model-based optimization (MBO). The method constructs synthetic trajectories toward high-scoring regions, trains a conditional diffusion model, and samples multiple trajectories to explore and select high-fidelity designs.

Strengths

1. Introduces a new approach to generating trajectories that consistently move towards high-scoring regions. 2. Leverages the powerful capabilities of diffusion models to handle complex and high-dimensional data. 3. The method utilizes classifier-free guidance and context conditioning during the sampling process, enhancing the exploration of high-scoring regions.

Weaknesses

1. The authors have adopted the Diffusion model for offline model-based optimization, which is a current trending approach. It is recommended that the authors emphasize their unique contributions and clarify the distinctions between their method and previous approaches. 2. The model filters candidates through a trained proxy model. The authors should elaborate on how data handling is managed during experiments to ensure that there is no label leakage and discuss whether the comparisons with other models are conducted fairly. 3. The authors should provide a more detailed description and discussion of the experimental results.

Questions

Please see weaknesses

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

n/a

Summary

The paper introduces Guided Trajectory Generation (GTG) for offline model-based optimization (MBO). GTG creates synthetic trajectories from offline data, using locality bias to ensure consistent improvement. A conditional diffusion model generates these trajectories based on their scores. The method then uses guided sampling and a proxy function to identify the best designs. GTG shows superior performance on the Design-Bench benchmark compared to other methods.

Strengths

- The authors effectively motivate their approach with a toy 2D Branin example, highlighting the differences from previous synthetic trajectory-based methods. - Extensive ablation studies demonstrate the robustness and effectiveness of the proposed method. - The paper is well-written and easy to follow, with a straightforward and effectively communicated idea.

Weaknesses

- TFBind8 and TFBind10 are very similar. It would be better to see experiments on more varied discrete benchmarks like NAS. Also, why do the results for TFBind10 have such high variance? - It’s not clear how this approach deals with out-of-distribution (OOD) problems. Diffusion models usually generate in-distribution data, so where does the robustness against OOD desings come from? - The proxy function uses the standard regression approach, which can be quite fragile when estimating OOD designs. Have you investigated using COMS or ROMA models as a proxy? - There isn't a principled mathematical explanation for why this approach works. - The paper doesn’t include 50th percentile results, which are a standard part of evaluation in this field.

Questions

- Please check the weaknesses - For the discrete tasks, Is a latent space used or is everything kept discrete during diffusion and proxy training? - How many gradient steps are used for gradient ascent methods like COMs and RoMA? Is it 64? And is the trajectory length the same for BONET and PGS? - Were the hyperparameters picked using the proxy or the oracle? - Are designs picked at the end of the generation process, or can they be selected from the middle of the trajectory? Is the filtering applied to results generated by BONET or PGS? How does this affect their outcomes? - How many designs are initially considered, and how many are selected at the end (128 out of how many)? - Why is the perturbation epsilon different for DKitty?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors address the limitations and potential negative impacts in their paper.

Summary

The paper consider the problem of offline optimization where the goal is to find the optima of a black-box function in a zero-shot manner without online evaluations. The key idea is to generate trajectories with a locality biased heuristic and employ a conditional diffusion model to learn the distribution of these trajectories. Experiments are performed on multiple tasks from design bench benchmark.

Strengths

- I think looking at construction of trajectories is a really interesting problem in the context of offline optimization. Other than the two references mentioned in the paper, another recent ICML paper (listed below) also uses monotonic trajectories in the context of this problem that shows this is a recurring theme. Hoang, M., Fadhel, A., Deshwal, A., Doppa, J., & Hoang, T. N. (2024). Learning Surrogates for Offline Black-Box Optimization via Gradient Matching. In Forty-first International Conference on Machine Learning. - I like the ablation experiments in the experimental section that shows the efficacy of different components of the proposed approach.

Weaknesses

- This is not specific to the paper as such but the tasks in design-bench benchmark have multiple issues and as a result, some of the results are not reliable. For example, the offline dataset in superconductor task has multiple copies of the same inputs but with different outputs. As a result, the random forest oracle which is fit on this offline data is not reliable. - The paper calls existing trajectory generation approaches (like sorted monotonic trajectories) as heuristics but the proposed approach is also a heuristic. Is there more principled/theoretical explanation for the proposed approach?

Questions

Please see weaknesses section.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Please see weaknesses section.

Summary

This paper introduces Guided Trajectory Generation (GTG), a novel conditional generative modeling approach to solve the MBO problem. GTG consists of three parts, including trajectories construction, model training and trajectories sampling and filtering. Experimental results on various tasks, including a toy 2D experiment and Design-Bench variants, demonstrate the method's effectiveness.

Strengths

1. This paper is well-written with a great motivation and description. Especially, authors introduce the conditional guidance to explore high-scoring regions. 2. The effectiveness of the proposed method is verified through experiments on multiple tasks and a detailed comparison with existing methods is presented. 3. The paper analyzes the hyperparameter settings and explores the effects of different hyperparameter settings on the model performance.

Weaknesses

1. The method proposed is similar to Decision Diffuser [1]. The method proposed focuses on the offline model-based optimization problem while Decision Diffuser is used to sovle the offline RL problem. 2. It is not clear that in loss function, only x is used to calculate loss or both x and y are used. 3. In this paper, the concept of filtering in Section 3.4 and Section 5 are not clarified clearly. [1] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657, 2022.

Questions

1. Please clarify the difference between the method proposed in this paper and Decision Diffuser [1]. 2. In Section 3.4 and Appendix D.3, the distribution of data used to train models must be close to the real distribution. Otherwise, out of distribution could occur, leading to an error evaluation of trajectories. 3. Please clarify the concept of filtering in Section 3.4 and Section 5. Whether they are the same operation or not? [1] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657, 2022.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the authors have discussed the limitations and potential negative societal impact of their work.

Response to Rebuttal

Thanks for your response to my questions. I am fairly convinced that this will be a good addition to the offline model based optimization literature. I would request to add a discussion about following things: - I appreciate the special focus on fixing the seed and reproducing the baselines but the Superconductor task has one specific issue that it has multiple copies of the same inputs but with different outputs. As a result, the oracle is not reliable even though we can reproduce the results. Please add this while presenting the results about this task. - Hoang, M., Fadhel, A., Deshwal, A., Doppa, J., & Hoang, T. N. (2024). Learning Surrogates for Offline Black-Box Optimization via Gradient Matching. In Forty-first International Conference on Machine Learning. is another recent paper that constructs trajectories. It will be good to add discussion about this paper in the related work section.