Summary
In this paper, the authors present, Apollo, a zero-shot method to improve generation of a language model with multiple expert models (including ones that can incorporate multimodal information), an extension from ZeroCap which only allowed one multimodal expert. The idea is to iteratively update the context of the language model using losses computed from the expert models. This approach can be used to generate stylized image captions using a GPT-2 model with 2 expert models: CLIP for image-text proximity and a text style classifier for style guidance. The authors showed that their method is able to achieve state-of-the-art results on stylized image captioning while remaining zero-shot. The approach is further examined on image captioning with accompanying audio, the authors showed that the resulting zero-shot generated captions reflect information from both audio and image.
Strengths
The authors introduced a simple but effective approach of using expert models to guide an LM to perform certain tasks in a zero-shot manner. More importantly, the authors showed that this approach can be used for multimodal tasks (by having multimodal models as experts) and can incorporate guidance from multiple experts simultaneously. I believe this approach could have potential: with a stronger LM and better experts, we may be able to zero-shot more challenging multimodal tasks.
Weaknesses
Some experiment design choices are questionable. For example, for the experiment in section 3, the only sound included is "children laughter", which is not enough to convince us that this approach will work on any other image/audio combinations. There should be at least a few more different audio for the result to be more convincing. Also, using models as both the expert model and the evaluation metric can be a little bit controversial, so maybe the authors should include some other metrics (such as human rating of relevance between the caption and image/audio).
It is unclear which of "Expert summation", "Expert product" and "Decentralization" works the best. The author introduced all 3 methods, but failed to provide any summary or conclusion on which one should be used. In Section 2 ablation study, the author claimed that "Decentralization" seemed to be more fluent and meets all criterion, but from Table 1 it seems that "Product" seems to have more fluency overall; moreover, in section 3, only Apollo-Cap-P (i.e. "Product") is used, thus make us question why "Decentralization" is needed/introduced.
Even though I believe that this approach could have potential in many challenging multimodal tasks if we can scale up the LM as well as the experts, the potential of the approach as well as possible future directions is not discussed in the paper.
The presentation of the paper needs improvement. For example,
(1) The term "lambda" in Eq(1) and Eq(2,4) seems inconsistent and represents different weights.
(2) The fonts of most Tables and Figures are quite small. Moreover, the fonts of Tables 1,2 and table in Figure 4 are all different and inconsistent. Also, in some tables (Table1 and Fig 4) the top performances are bolded, while in Table 2 there is no bolding.
Questions
1. How is "expert model" defined? Conventionally, we would define the generating LM as the "learner" and the guiding models as "experts" (i.e. GPT-2 is learner and CLIP/CLAP/style model are experts). But in section 1.1, it seems like all transformers are "experts" (i.e. GPT-2 is an expert too).
2. How computationally costly is this model? Approaches that attempts to control LM generation outputs through nested-loop gradient descent are generally quite computationally expensive.
Rating
5: marginally below the acceptance threshold
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.