Im-Promptu: In-Context Composition from Image Prompts

Summary

This paper presents a comprehensive study of in-context learning of compositionality in images. It introduces a simple benchmark of compositional language-driven visual transformations and explores under what conditions a model can perform in-context analogies from image pairs.

Strengths

Very interesting study, with relevant conclusions that can be easily turned into hypotheses for examining how one can achieve in-context analogical reasoning in a more real-world, less synthetic context. The scope of the study is well delineated, and the hypotheses that are tested are very clear. This is a very important topic beyond the narrow setting of 'can we do visual analogies', because of the widespread believe that analogy making is a great proxy for broader understanding of the visual world. This study further reinforces recent observations from a number of researchers that object or attribute-centric representations are very effective at enabling higher level visual reasoning and deserve more attention (pun intended), and confirms that cross-attention is a strong contender for enabling compositional learning beyond just a single modality.

Weaknesses

One could argue that results obtained on synthetic data always come with the question of whether the findings will transfer to more real-world settings. In this case, the diversity of synthetic data sources partially addresses the concern. Nits: - Fig 1: girafe has two 'f' in English. - Suggest shortening refs [17] and [19] if permitted byt the NeurIPS style guide. A whole printed page for a single reference is a bit over the top.

Questions

None.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Limitations of the work are adequately spelled out.

Summary

This paper focuses on the problem of using composable elements of visual modality to perform in-context referring for analogical reasoning like LLMs do. Firstly, the authors constructed three benchmarks to evaluate the generalization capacities of visual in-context learner. Then, the authors unified the formula of in-context analogy learning, and designed a framework called Im-Promptu. Different levels of agents, ranging from simple pixel-space rules to object-centric learners are trained for exploring the granularity of composition in visual modality. Lastly, the paper pointed out the characteristics of different degrees of compositionality via experiments.

Strengths

1. The paper proposes an interesting topic of considering the in-context composition learning from visual prompts. Similar to the composition skills of LLMs, visual generative models’ compositionality is largely under explored. The motivation of this paper is meaningful and insightful. 2. This paper is of great novelty. First, it formulates the notion of analogy-based in-context learner and proposed a novel training framework called Im-Promptu based on that. What’s more, the paper provides a detailed and insightful analysis of different granularities’ influence on the compositionality and whether analogical reasoning enables in-context generalization. 3. The paper contributes three benchmarks for evaluating the generalization properties of visual in-context learners.

Weaknesses

1. There is no obvious weakness I have found in this paper.

Questions

1. Just one question: The proposed three benchmarks are very different from actual scenes, are there any plans to introduce more real-scene images to the benchmark?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors discussed limitations and there is no potential negative societal impact of their work from my perspective.

Summary

This work proposes a framework for in-context image generation/composition from image prompts. The in-context learner resembles analogy completion and is simply optimized with a reconstruction objective. Several variants in terms of visual representations (pixels/latent vectors), the compositionality of the vectors (monolithic/object-centric), and the scheme of integrating prompts (cross-attention/simple concatenation) are explored to indicate the key ingredients of a successful visual in-context learning model. In addition, a benchmark consisting of three object/attribute-combinatorial datasets is created to evaluate visual in-context learning tasks.

Strengths

+ The main idea of introducing in-context inference ability (which is text-free) into the object-centric learning framework is interesting. + The proposed Object-Centric Learner (OCL) is technically sound and the experimental results demonstrate its validity for in-context composition from image prompts. + Extensive evaluations regarding different levels of compositionality and different means of context aggregation are investigated.

Weaknesses

- The main concern comes from the confusing contribution clarification of this paper. I do not see which specific/major contributions the authors want to emphasize. Text-free in-context generation compared to text-to-image generation models such as DALL-E? Endowing SLATE with the in-context composition capability by inserting a context encoder transformer? (a) If the answer is the first one, I would say it is over-claimed. Although Figure 6 to some extent showcases that DALL-E requires tedious prompt engineering and some external techniques like inpainting to achieve scene composition, the proposed text-free in-context generation method is not more appealing since 1) it can only be applied in the toy scenario (while DALL-E supports open-vocabulary generations with high fidelity), and 2) one needs to have each component in advance (e.g., an image of a rubber purple cube in the desired position), and this is not feasible in practical scenarios. (b) If the answer is the latter, I would say the claimed contribution is not well supported. The proposed OCL demonstrates its ability of in-context composition from image prompts, but SLATE also supports compositional generation using the concept library built from the training datasets. I think some comparisons and discussions with SLATE would help contextualize this paper. - Overall, the paper is not easy to follow because: 1) there are many long sentences with less specific descriptions, and 2) many technical details of the in-context learner are missing and Section 4 is redundant (e.g., Equations 2 and 3 could be moved to the appendix). In addition, the authors should consider adding some preliminaries of slot attention and SLATE, making the paper more self-contained. - The proposed method is only evaluated on toy synthetic datasets. Although previous related works (e.g., SLATE) generally struggle to generalize to real-world data with diverse scenes, I would like to see some experiments on natural images.

Questions

In terms of the experiments, I have two questions: (a) How to implement the multi-shot in-context generation? (b) In Figure 6, the objects in the generated image seem to be placed in the same position as they are in the image prompt. How is this position-preserving functionality realized by OCL? I would like to see some insights into this from the authors.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Yes

Summary

This work investigates whether analogical reasoning can enable in-context composition over composable elements of visual stimuli. First, they introduce a suite of three benchmarks to test the generalization properties of a visual in-context learner. They formalize the notion of an analogy-based in-context learner and use it to design a meta-learning framework called Im-Promptu. To this end, they use Im-Promptu to train multiple agents with different levels of compositionality, including vector representations, patch representations, and object slots. Lastly, they demonstrate a use case of Im-Promptu as an intuitive programming interface for image generation.

Strengths

In-context learning is a novel learning paradigm recently. While in-context learning is popular with large language models (LLM), it has not been discovered in computer vision fields. This paper provides three benchmarks for in-context learning for computer vision and introduces some algorithms for proposed benchmarks. Since in-context learning is open for computer vision, the attempt is valuable and makes further research possible.

Weaknesses

In-context learning is a general learning paradigm for diverse tasks. In the computer vision field, there are a lot of tasks (e.g., classification, object detection, segmentation). However, this paper only considers three tasks: 3D Shapes Image Prompts, BitMoji Faces Image Prompts, CLEVr Objects Image Prompts. Thus, this benchmark may be not sufficiently comprehensive from this perspective. On the other hand, the performance versus the number of in-context examples is one important aspect of in-context learning. This paper doesn't provide any insight from this view.

Questions

I have the following questions: 1. Since the number of in-context demonstrations is one important parameter in in-context learning fields, is there any ablation study for the number of demonstrations? 2. The input and output of these three tasks are all images. One question is can the text be the output of the task (e.g., image classification)? 3. In the proposed method, there are some components involved. What's the difference if we use models from scratch instead of pre-trained?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Compared with in-context learning in language models, this paper only considers three proposed computer vision tasks. Thus the setting of in-context learning may be limited.

Summary

The paper investigates whether analogical reasoning can enable in-context compositional generalization over visual entities. The authors construct three benchmarks to test the compositional generalization on visual analogy-making, including 3D shapes, BitMoji Faces, and CLEVR. The authors also present a visual analogy-making algorithm called Im-Promptu. The authors test the proposed method on the constructed benchmarks with various visual representations, including vector representations, patch representations, and object slots. The experiments demonstrate the tradeoffs between extrapolation abilities and the compositionality degree of the visual representations.

Strengths

1. The paper constructs three benchmarks to test the compositional generalization on visual analogy-making, including 3D shapes, BitMoji Faces, and CLEVR. 2. The paper conducts experiments using various visual representations, including vector representations, patch representations, and object slots, and provides insights into the impact of the compositionality degree of the visual representations on compositional generalization. The results demonstrate the effectiveness of the object-centric representation for the compositional visual analogy-making.

Weaknesses

1. The technical novelty of the proposed framework (Im-Promptu) is limited. The formulation in Section 4 is similar to visual analogy-making [1,2]. It is inappropriate to rename it as in-context learning or Im-Promptu learning, if there is no fundamental difference. Section 5 lists several model variants for visual analogy-making by composing existing modules. It is unclear what is the paper's contribution to the methodology. 2. The constructed benchmarks are all synthetic domains with little variance. This is a weakness considering that previous works [2,3] have already studied similar tasks using natural images. 3. The experimental results are inadequate. The authors only evaluated the proposed framework on the self-constructed benchmarks, while previous works [1,2,3] have introduced several benchmarks for visual analogy-making. It is unclear whether the proposed framework generalizes to other realistic benchmarks. 4. The paper lacks reference to important related works [2,3]. [1] Reed, Scott E., et al. "Deep visual analogy-making." Advances in neural information processing systems 28 (2015). [2] Sadeghi, Fereshteh, C. Lawrence Zitnick, and Ali Farhadi. "Visalogy: Answering visual analogy questions." Advances in Neural Information Processing Systems 28 (2015). [3] Bar, Amir, et al. "Visual prompting via image inpainting." Advances in Neural Information Processing Systems 35 (2022): 25005-25017.

Questions

1. What is the fundamental difference between the proposed Im-Promptu learning in Section 4 and visual analogy-making? 2. What is the main technical contribution of this paper? 3. The method of this paper works better on spatially consistent datasets, so how to reflect the statement "produces a more generalized composition beyond spatial relations" in the paper? 4. Why are there some results with lower MSE and higher FID?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

1 poor

Limitations

The authors do not discuss the limitations of the proposed method.

Post-rebuttal

After reading other reviewers' reviews and authors' rebuttals, I decide to keep my positive score. Even though the setting in the paper is not comprehensive, I still think in-context learning is an interesting topic in computer vision and this paper can provide benchmarks and some insights for the community.

more comments

Thanks authors for their responses. The rebuttal partially addresses my concerns, but I lean towards rejection given that (1) the contribution is not well justified; (2) empowering SLATE with the ability of in-context generation via explicit image prompts is interesting but the technical contribution is limited (as pointed by Reviewer 4dpY); (3) the evaluation on toy datasets is not adequate in my view; (4) the writing and presentation need substantial improvements. Thanks.