Learning to Edit Visual Programs with Self-Supervision

Summary

This paper studied the task of visual program induction (VPI), which involves inferring programs given visual targets. Compared to existing methods using one-shot generation (iteratively predicting program tokens until completion), this work proposed an Edit Network that can predict local edits that improve visual programs toward a target. The one-shot generation model and the edit model are trained jointly, which can create better-bootstrapped data for self-supervised learning. The results on three VPI domains demonstrate that the edit network can help the model's performance.

Strengths

The proposed method is technically sound, and based on the quantitative and qualitative results, the edit network can improve performance.

Weaknesses

1. **Lack of baselines**: This paper only compares two models, with and without the edit network, both of which are implemented by the authors. Although the model performs better with editing, it is difficult to evaluate the effectiveness of the proposed method without the context of previous methods or some other reasonable baselines, such as supervised results, zero/few-shot prompting, fine-tuning of large multimodal models, etc. In addition, I feel the improvement from Edit Network is marginal, especially for Layout and 3DCSG. 2. **Editing Operations require manual efforts**: designing the set of editing operations requires human domain knowledge. This is actually a type of supervision as humans define the set of possible edit operations, which brings extra advantages (priors) to the proposed system. In this case, the proposed method may not generalize to other domains, as you need to adjust the Editing Operations for a different problem. 3. **Reproducitity**: Many details are needed to implement this method, such as the findEdits algorithm. Even after skimming the appendix, it is still not clear to me how this algorithm works, which may pose difficulties when reproducing the results in this paper.

Questions

1. Why not conduct the ablations on all three domains? 2. Can this edit network idea be applied to more general code generation tasks?

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors discussed the limitations in the Section 5.

Summary

This paper focuses on a problem in visual program induction. It proposes a new method for visual program editing, which trains an edit network along with a one-shot network with boostrapped finetuning.

Strengths

- The authors motivate and define the problem well. - The paper is well-written with clear illustrations, e.g. of the proposed method’s novelty (Algorithm 1) compared to the previous PLAD method. - The authors conducted comprehensive ablation studies on alternative formulations of their method, showing that their method achieves the best result. - The authors performed evaluations across three different visual programming domains.

Weaknesses

- The proposed method is somewhat but not significantly novel. - While the authors show performance gains of their method compared to alternatives in the ablations study, some of these gains (e.g. compared to No one shot FT and no edit FT) are surprisingly very small (<0.1). It’s not clear that all components of the proposed method are important. it would be great if the authors could provide additional analysis on why one shot FT and edit FT don't contribute much to the performance.

Questions

- Why did the authors choose to perform ablations on the Layout domain but not the other two?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

- Yes, the authors discussed a few limitations of their method, including additional training time, expensive program search, and the requirement of a domain-aware program.

Summary

The paper proposes a learning framework for editing visual programs. The learned edit network can be combined with the classical one-shot network, achieving better visual program induction performance under the same compute budget.

Strengths

* The paper is well-motivated, with the insight that editing operations are typically local and learning such network is data efficient. * The framework gracefully integrates with one-shot learning frameworks based on self boostrapping proposed in prior literature. * The paper shows results on various visual domains.

Weaknesses

* The visual domains in the experiments are still restricted, typically limited to simple combinations of primitives (e.g. 2D house) or a single category (e.g. chair). How to scale up the method to more complex data? For example, other categories from ShapeNet? * The fact that editing operations are local makes the framework data efficient, but it also means that after the predicted editing operations are carried out, the program can only be improved by a small margin. This poses challenges to applying the framework to more complex domains or to scenarios where curated synthetic data have a large distribution gap compared to the target visual inputs. How to address these challenges? * The framework requires a heuristics (findEdits Algorithm) to curate data. This seems reasonable, but as it can have a large impact on performance, more explanations on why designing the algorithm this way would be helpful. * The findEdits Algorithm assumes that there are two types of functions: transforms and combinators. Is this a generic enough formulation? Are there cases when there are other types of functions, or even helper functions, for a DSL with richer syntax?

Questions

* Why is Modify transform (MT) necessary? Why is it not equivalent to Remove transform (RT) followed by Add transform (AT)? * How to sample start programs and end programs? Are they randomly sampled subexpressions from the DSL?

Rating

6

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

Limitations are discussed.