**Q1. "Alpaca converges easily" does not really convince me on its importance in the system design. Are there any references on instruction-following LLMs are easier to optimize and converge?**
**A1:** We would like to discuss this issue from two perspectives as follows:
(1) _**The LLM with instruction-following and parsing capabilities is able to effectively interpret the vision task instructions, which helps to reduce the loss during the early stage of model training.**_ To preserve the language ability of the LLM itself, we freeze the weights LLM during training. If we use LLMs without instruction-following, the frozen LLM struggles to learn this ability. Therefore, we directly utilize the instruction-following LLMs. The example below demonstrates that Alpaca (instruction-following LLaMA) can parse category mapping relationships and the required output format for the task, while LLaMA cannot achieve this.
```
System message:
"Below is an instruction that describes a task. Write a response that appropriately completes the request.
Input: {input}
Output:"
Input:
"class set: {'person': <c0>, 'car': <c1>, 'table': <c2>, 'cat': <c3>, 'television': <c4>, 'a man in a black hat': <c5>}. What is the class index associated with the class 'a man in a black hat' in the given class set?"
# Alpaca output
Output:
"The class index associated with the class 'a man in a black hat' in the given class set is 5."
# LLaMA output
"Answer: 0 Input: Given an array of strings, remove the last element of the array. The returned array should be a new array with the last element removed."
```
(2) For _**the detailed loss curves**_ (since we cannot include figures at this time, we provide the loss values every 2000 iterations), the model trained using Alpaca reduces demonstrates faster loss reduction, particularly in the early stages. We do not rule out the possibility that LLaMA can converge if it is unfrozen and allowed to train with more epochs, but at least under our current experimental setting (frozen LLMs, 50 epochs), Alpaca is better at following instructions than LLaMA.
```
# LLaMA loss
6.08, 4.61, 4.46, 4.41, 4.33, 4.22, 4.15, 4.10, 4.04, 4.01, 3.95, 3.87, 3.85, 3.98, 4.36, 4.63
# Alpaca loss
5.26, 4.56, 3.97, 2.85, 2.77, 2.69, 2.60, 2.62, 2.54, 2.56, 2.51, 2.50, 2.50, 2.49, 2.47, 2.45
```
We will make it clearer in our revised version.
**Q2: Uni-Perceiver-V2 / Pix2Seq v2 are much stronger at segmentation tasks (when using the same ViT-B backbone). Why?**
**A2:** Firstly, _**VisionLLM is distinct from models for pre-defined tasks (e.g., Uni-Perceiver-V2, Pix2Seq v2), as it has the capability to handle open-ended tasks customized by users, providing greater flexibility and versatility.**_ To be able to unify and flexibly customize tasks, we have made some design and trade-offs in terms of task formulation, model selection, and training methods, which may result in some performance losses, particularly on segmentation tasks.
Compared to Uni-Perceiver-V2, our model utilizes polygons with discrete coordinates to represent instance masks, ensuring a uniform task output. _**This results in two levels of performance loss**_: (1) the conversion from mask to polygon representation results in performance degradation, and (2) the conversion of polygon coordinates to integers also incurs performance loss (see L347-L353).
While Pix2Seq v2 also employs polygons to represent masks, they use _**128 polygon points**_ (at least 4 times more than our model) to enhance segmentation performance. Additionally, they utilize ensemble methods to _**merge the results of 8 inference runs**_ and _**adopt a crop-then-segment approach**_ to further improve segmentation performance. In contrast, VisionLLM requires an LLM-based decoder to support general user instruction parsing. This restricts us from using more than 32 polygon points on existing hardware, and we have not implemented generalist-model-agnostic techniques (such as ensemble and crop-then-segment) to enhance our segmentation results.
We will clarify this in our revised version.