Emergent Communication in Interactive Sketch Question Answering

Summary

The work focuses on multi-turn sketch-based emergent communication. Authors propose a novel two-round interactive task, named Interactive Sketch Question Answering (ISQA). They suggest an architecture and an implementation, based mainly on existing components (MCAN, Fast-RCNN) while incorporating several novel ideas such as 1) dynamically restricting the channel capacity by controlling the number of transmitted pixels, and 2) providing feedback from receiver to sender via focus boxes. They suggest a triangular evaluation method that seeks a balance between human interoperability and task accuracy.

Strengths

The main strength of this paper is by suggesting a two-turn visual communication game that nicely models the need for two parties to communicate, with partial observability, to solve a task. In addition, the paper demonstrates a method to achieve a nice balance between interoperability and pragmatism. The most interesting observation, to my mind, is provided in lines 304-306 where the authors show that when the complexity is too low, the reasoning module cannot infer sufficient useful information in a single round and thus needs to request a more focused information (a clarification question). The way the architecture is composed and implemented for modeling the problem at hand is not trivial and interesting.

Weaknesses

The authors assess human interpretability using the CLIP model. Doing an actual human survey of the results would be more appropriate. The interoperability/pragmatism balance is essentially solved by adding a CLIP-based loss that provides additional supervision towards human interpretability, which is not aligned with the intention to model communication emergence. Experimental datasets are not described in enough detail. For example, it is unclear how the three reported tasks (Yes-No, Number, Other) correspond to the two described datasets. Results are not totally consistent (for example, in the Yes-No task where the PraGeo is lower than both the geometric and pragmatic models) and more experiments over more datasets seems needed. Notations and explanations can be further worked out to assist the reader. See some examples in the Questions section. Maybe worth mentioning references: Pragmatic inference and visual abstraction enable contextual flexibility during visual communication, by Judith E. Fana, Robert X.D. Hawkins, Mike Wub and Noah D. Goodman,

Questions

In section 3.1 what are the dimensions of H_i and A_i? (explained later) Line 150 - will be good to stress the fact that b_i is a ratio (explained later) Line 244 – will be good to explain what proposals are. Lines 293-297 the x-axis is not easily defined (line 293) and then referred to as 0.1N, 0.3N, which are hard to find in the graphs. Can’t you use the 0.xN scale? or at least add the values you refer to as labels to the x-axis? Datasets are missing the explanation of complexity/difficulty of tasks, namely yes/no, number and other which you refer to in the figure. A baseline random accuracy can also be helpful to add or mention. Table-1: is the lower the better? Worth mentioning.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

limitation section is provided.

Summary

This paper proposed a new multi-round visual communication task with an interactive system for emergent communication. During the game, the sender needs to sketch on the canvas to communicate a target image, while the receiver needs to answer a question regarding the target image and give feedback on the sender’s sketch. The training framework balances task accuracy, drawing complexity, and human interpretability. Experimental results show that the agents can communicate successfully using sketches and feedback. The emerged sketches can maintain interpretability and high accuracy with lower complexity. And the feedback given by the receiver can effectively enhance the performance.

Strengths

1. This paper proposed a novel setting where each of the agents can only observe a partial environment that necessitates the feedback of the receiver. And the feedback is smartly provided in a sender-understandable way (bounding boxes). Compared with the previous work, this environment enables bi-directional communication where both agents can “draw” on the canvas. 2. The training framework considers triangle optimization – task accuracy, drawing complexity, and human interpretability.

Weaknesses

1. Complexity B: for the complexity in section 5.3, what is the specification for $b_i$ and $h_i$ separately? It will be interesting to know how which agents contribute more to the efficiency – while achieving high accuracy, is the high efficiency due to the sender drawing less or the receiver giving more accurate feedback? 2. The maximum round: only models trained with two-round are reported. Is there a reason that the maximum round is set to 2? Given more rounds, the performance change can help us understand whether one round of feedback from the receiver is enough to finish the task.

Questions

Why is the $\sum b_\tau$ given to the sketch model? Would $b_i$ be sufficient?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

It will be interesting if the agents can control the complexity of the sketch based on the target image and the receiver’s feedback. Similarly for the number of bounding boxes at the receiver’s side.

Summary

In this paper, the authors proposed a new task about emergent communication by tackling visual question answering as an iterative sketch question answering process. The authors proposed a three-factor evaluation metric, including question answering performance, drawing complexity and human-interpretability. A framework consisting of Sender and Receiver is proposed to perform multi-round interaction to tackle the proposed task. VQAv2 is used for empirical evaluation of the proposed framework.

Strengths

1. The problem setting is very interesting. 2. The proposed method is intuitive and straightforward. 3. The paper is presented clearly and easy to follow.

Weaknesses

1. The new insight is limited. a. Visual question answering is indeed a new task compared to classification. But what is unique in terms of emergent communication when visual question answering is used as the target task? From the current manuscript, there is not really a metric and any empirical evidence showing the improve communication quality over [14]. b. Despite the authors target at multi-round interaction, the two settings evaluated are one-round and two-round.From the visualizations, the sketch used is usually the sketch of the main object. There doesn't seem to be a pattern in terms of communication with only one/two round of communication. c. More fundamentally, how does communication emerge and how does communication gets better/more efficient when the task is more complex? The reviewer feels these fundamental questions still left unsolved and the current manuscript didn't show any potential of helping solve these problems. 2. Empirically, current evaluation is not sufficient enough. Currently, the communication quality is mainly measured through automatic metric like CLIP-based score. There should be some quantitative analysis verifying the correlation between automatic score and manual measurement.

Questions

Please check the weakness for details.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

1 poor

Presentation

3 good

Contribution

1 poor

Limitations

Need more discussion on the fundamental questions of emergent communication.

Summary

The authors present a new problem setup for sketch-based emergent communication, distinguishing itself from existing work primarily through communication taking place iteratively over multiple rounds. The authors also argue that the reliance on downstream tasks for evaluations allows for communication protocols to develop which are not necessarily easily interpretable by humans, and thus fail to fulfill this important goal of EC research. Evaluating the behavior of various design choices, the authors show that they can prioritize different aspects of the problem: performance, but also human interpretability and drawing complexity (automated metrics)

Strengths

- The idea of multi-round EC is very exciting! I would love to see research move in this direction and, given just the difficulty of agents who learn when to talk (and talk with resonable sparsity), I imagine there are plenty of interesting problems to solve in that space. - On that topic, the ability for the sketch model to generate very different sketches using the same image (when the question demands it) is demonstrated here and is a perfect example of what I would expect as sort of a main contribution from an EC model in this space. - The problem setup and task are novel

Weaknesses

- I won't dwell on this too much since it's primarily a track issue, and the paper could be considered by other merits, but it is difficult for me to consider this emergent communication at all. It does however mean that a lot of EC motivation cited here doesn't seem very relevant upon reaching the experimental design section and understanding the learning problem. - While the authors argue that existing work relied too much on downstream tasks for evaluation, regardless of this point, evaluating downstream did serve an important purpose in that it helped demonstrate some potentially useful application of the learned protocol. Here the protocal seems rather contrived. Of course something like a referential game is also rather contrived and I concede that point. However, I'm willing to accept contrived environments if what emerges in the language is itself interesting and gives us some insight on what sorts of less contrived environments we may consider in future work. - The drawer is vastly simplified when compared to the existing visual referential game work (cited here). The authors state, "Vision-based EC systems have been developed with the aim of imitating the pictographic systems that were used in early human communication", but a pictographic system tends to abstract important visual features, sometimes caricaturing them for the purpose of clarity in communication. I'm not convinced that this drawer is an appropriate substitute for this process. If I want a giraffe drawn in 3 strokes vs. 8 strokes, we see the important visual features that are most characteristic of the giraffe. If we are essentially revealing areas of an edge/depth-detected version of a real image, it seems very different. From the examples of the sketches produced by various modes (pragmatic, geometric, prageo), none strike me as very similar in creating some simplified version of the original high-res image of the object, and a case should be made why this process could be considered an imitation of those systems in early human communication. - While generating very different images from the same image when the question differs - Is the binary flag model of [14] really that different from what occurs in this work? Of course, time not considered, the listener would like to continue receiving new information until the end. That seems like optimal policy. So whether the listener conveys to the speaker what it would like to see, or if the speaker already has a priority order in which it would reveal / detail more parts of the image, that's not a hugely important distinction in my mind, unless the speaker and listener have very different perceptual abilites, or goals in mind. It would of course be good to be user-centric in many cases, but how important is it? I would have liked to see a comparison. - In comparison to existing work, and bearing in mind the emphasis on human interpretability in the paper narrative, I would have liked to see this method compete with [13]/[14] with a human substituting as the listener, or at least trying to solve the task (and perhaps no communicating). Without being able to play with the models directly, the previous work seems more interpretable with fewer strokes. I really find it surprising that humans aren't involved in the measuring of human interpretability, and I think that fact hints that there may be a more suitable name for what is being measured. Overall I think there is some promising work in how the task is setup, but deviations from the sketch model and the region-based (rather than complexity-based) of existing work seem like steps backwards. No direct comparisons to previous work, or adaptations of existing work to this, is detrimental both to placing it in the larger research context, and understanding the relative strengths/weaknesses of the proposed approach. Other comments: Paragraph 1: These claims seem speculative / opinion-based.

Questions

N/A

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Concerns remain

Thanks for the rebuttal. 1. The reviewer understands that the proposed model is not directly comparable with [14]. This doesn't necessarily mean that comparing with other tasks is not feasible. For example, the authors could build a classification version of the dataset by only using the labels of object categories of the COCO dataset where VQA is built upon. Further degraded version of the tasks could be built by choosing different ways of communication (sketch or binary). Based on this, apple-to-apple comparisons can then be done. 2. The reviewer still thinks two rounds are not enough to really show the complexity of emergent communication. Changing the focus the sketch based on the feedback doesn't necessarily reflect the complexity of communication. The reviewer thinks there is some relationships between the communication bandwidth (B) and the number of rounds towards accomplishing the task. The current evaluation really is too limited in terms of one important aspect of communication: communication is done by modeling a sequence of signals. 3. The reviewer also has some follow-up questions on the human evaluation. How is the evaluation done exactly? Is there any training process for the human evaluators? How is each criteria defined? 4. Can the authors comment on fundamental questions of emergent communication and which are directly answered in this paper?