Summary
This work proposed a new benchmark for the referring expression task, a system (DuMoGa) that models the task using the two modality semantic graphs and attempts to align them. Moreover, in-depth analyses are shown based on the dataset and system that provides findings.
Strengths
- It is not just the dataset a good contribution to the referring expression community. But also, the way that the dataset is constructed by using ChatGPT to enrich the expression and reduce the human workforce necessary to annotate this complex task is clearly a new trend and a good thing to use.
- The dataset is bigger than the previous dataset in the quantity of images and the length of the queries.
Weaknesses
- Not sure about the system statement Sec 3: "Unlike classic dense predictions models with complex design in RIS, which take expensive computational power to make inference, the graph learning-based architecture provides efficient training process and promising results". Graph learning-based architecture needs a couple of other methods that are complex to get the scene graph and the dependency trees. The computational power to make inferences per image requires three methods to run over the image (DP, SGG and the DuMoGa). I consider this a bold statement with no supporting experiments in the paper regarding inference or training time and computational requirements.
- The in-depth analysis needs more experiments that could support the statements and bring dome insights.
Questions
- I reckon ChatGPT4v can obtain more contextual information from the images, which could help to solve those ambiguous references. Chapter 2, step-3.
- Although queries are semantically richer than other datasets, images come from combining two previously existing datasets (VG and COCO). Could it be that the lengths of queries in previous datasets are short because the complexity of the images is less? I wonder if reannotating those datasets is enough to create a good benchmark for RIS. It is a step forward, but it would be good to know how complex are the images in the previously mentioned datasets. Maybe report the average amount of detected objects in the images, similar to the statement "85.3% text queries have short forms (i.e., with length < 5 words)" but on the image side.
- "all annotators underwent rigorous training before commencing the actual annotation process." what is the rigorous training?
- " Even input two different queries which refer to the same object but contain information in different granularities, current RIS models will output different regions." no qualitative or quantitative experiments show this issue or, at least, a reference. In this regard, an analysis of semantically similar queries using METEOR or SPICE and then show how many of the queries that are similar by certain threshold ground different objects would support that statement.
Rating
3: reject, not good enough
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.