Summary
This paper proposes a multi-task learning framework to jointly learn multiple vision: object detection, instance segmentation and semantic segmentation, and vision-language tasks: image captioning. The proposed model is built on top of YOLOR and ELAN. Though based in a multi-task learning setting, the paper assumes the prior knowledge of training tasks, and propose some additional task-specific designs: instance segmentation task relies on the prediction of object detection task; image captioning task relies on all the vision tasks. Finally, the author provides all the engineering tricks and optimisation strategies and show the model can outperform some task-specific models in these tasks.
Weaknesses
In general, I feel like the paper is quite rushed to finish. The major contribution is heavily relying on YOLOR, YOLOv7 and ELAN which were all published by Wang et al very recently in 2023, which is fine. But the paper itself does not provide all the necessary background knowledge for these papers, and therefore it’s very difficult to understand this paper itself without really understanding these other papers. Here are some other detailed comments.
1. The paper did a fairly poor job in the literature review. The proposed method is heavily related to multi-task learning and vision-language models, but did not cover major advances in both fields and therefore missed quite a lot of important baselines. In MTL, it’s important to discuss some popular model design strategies such as in MTAN (Liu et al 2019), Cross-Stitch (Misra et al 2016), Adashare (Sun et al, 2019) for standard dense prediction tasks in computer vision, and more recently UnifiedIO, BeIT-3, Flamingo, for unifying vision and vision language tasks.
2. The architectural design details are completely missing, and the visualisation is not clear. From Fig. 2, the image captioning task seems to be predicted from the P3 block only. But in section 3.1, the paper describes that all vision tasks are merged into image captioning block. In general, there’s no explanation of what each colour means in Fig. 2 and 3, and have absolutely no explanation of important design components like ProtoNet, P3/4/5. Coefficient, that being used in the network. Most texts in the Architecture section are simply confusing and possibly wrong, for example:
- “Similar to the Transformer, we train the image encoder and text decoder together. ” Only recently proposed VLMs have these architecture designs but with no citations.
- “This allows the model to be lightweight and efficient in addition to reducing training resource consumption. ” Compared to what design? Transformers are known to be not efficient since they require quadrative memory depending on the input length.
- “Intuitively, it mimics human learning – explicit and subconscious (implicit). ” This sentence simply does not make sense for an academic paper.
3. The experiment design does not cover important baselines, considering all evaluated tasks are well-designed and heavily evaluated within the community. In the image captioning task for example, the author only compared with an open-source model published on github, and did not compare or discuss any important VLMs published in the community. To name a few: BLIP ½, GIT, Flamingo, PaLI, SimVLM, CoCA all supposedly should have a very strong performance.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.