We apologize for the confusion.
In the rebuttal, we had written, *“We would add… ”* referring to the inference time but a more precise description would be: *“We would add… **into the system prompt**.”*
Here, we give more details on the calculation differences between: (1) Yo’LLaVA and (2) Image Prompting, across three aspects: (A) Approach, (B) Training, and (C) Inference.
**A - Approach:**
* Yo’LLaVA: Learns to embed visual details about the subject into a personalized learnable soft-prompt:
"$<sks>$ is $<token_1>...<token_k>$".
In the experiments, we choose $k=16$, which results in 17 tokens ($<sks>$, $<token_1>$, $<token_{16}>$) used for a personalized subject.
* Image Prompting: Uses $n$ image(s) to represent the personalized subject:
"Here is a photo(s) of $<sks><image_1>...<image_n>$".
For simplicity of this computation, we use $n=1$ (In Table 5, we tried GPT-4V + Image Prompting with $n=1$ and $n=5$ images).
In Chameleon [1], an image $<image_i>$ is represented by 1024 tokens; thus, in this case, 1024 tokens are used to represent the personalized subject. (We use Chameleon because the number of tokens used to represent an image in GPT-4V is unknown.)
In both approaches, the personalized prompt is added to the system prompt.
**B - Training**
- Yo’LLaVA: learns 17 tokens (in which, token $<sks>$ is added into the vocabulary (Please refer to Line 152-157 for further details)), which takes roughly 40 minutes to train (measured with LLaVA-1.5-7B on an A6000 machine).
- Image Prompting: no training needed.
**C - Inference**
C.1. Consider we only have one personalized subject called $<sks^1>$
- Yo’LLaVA: Add "$<sks^1>$ is $<token^1_1>...<token^1_{16}>$" to the system prompt.
In this case, the number of tokens used to represent a personalized subject is: 1 (subject) x 17 (tokens/subject) = **17 tokens**.
- Image Prompting: Add "Here is a photo of $<sks^1><image^1_1>$”.
In this case, the number of tokens used to represent a personalized subject with a single image $<image^1_1>$ is: 1 (subject) x 1 (reference image) x 1024 (tokens/image) = **1024 tokens**.
C.2. Consider we have 10 personalized subjects $<sks^1>$, …, $<sks^{10}>$
- Yo’LLaVA: Add to system prompt:
“$<sks^1> is <token^1_1>...<token^1_{16}>$.
$<sks^2> is <token^2_1>...<token^2_{16}>$.
...
$<sks^{10}> is <token^{10}_1>...<token^{10}_{16}>$”.
In this case, we use 10 (subject) x 17 (tokens/subject) = **170 tokens** to represent 10 personalized subjects.
- Image Prompting: Add to system prompt
"Here is a photo of $<sks^1><image^1_1>$.
Here is a photo of $<sks^2><image^2_1>$
….
Here is a photo of $<sks^{10}><image^{10}_1>$”.
In this case, we use 1 (reference image/ subject) x 10 (subject) x 1024 (tokens/image) = **10240 tokens** to represent 10 personalized subjects!
---
We agree that Image Prompting is a strong baseline, and indeed, Table 5 shows that GPT-4V + Image Prompting (5 images) yields competitive or better results than Yo’LLaVA in both Recognition Accuracy (0.925 vs. 0.924) and Text Question Answering (0.987 vs. 0.883). However, it uses significantly more tokens (~5k vs. 17)! We hypothesize that integrating Yo’LLaVA with GPT-4V could further improve performance while being more token-efficient. Unfortunately, we cannot do this because GPT-4V is a closed-source model.
It’s also worth noting that in our early experiments, we empirically find that Yo’LLaVA can be integrated with other multimodal models (e.g., Chameleon) and offers advantages in image generation quality (e.g., the generated images are much closer to the personalized subject, as shown in the Rebuttal file, Fig. 1).
---
We thank reviewer for bringing up this discussion. We will include this in our revision to enhance the understanding between Yo'LLaVA and Image Prompting.
We are happy to discuss and clarify any further questions or doubts.
*Reference:*
*[1] Chameleon Team, Chameleon: Mixed-Modal Early-Fusion Foundation Models, arXiv, 2024*