Unveiling Encoder-Free Vision-Language Models

New reply and clarification to reviewer Z41h

We are pleased to reach a consensus on key insights of EVE. We understand your concerns regarding the terms "encoder-free" and "lossless" description, and we would like to clarify our rationale behind these choices: `Q1: On a side note, I still do not like the term "encoder-free" and "lossless". In my opinion, it is more accurate to say that this work explores joint training of the LLM and a light-weight vision encoder from scratch.` **(1) While there's nothing wrong with regarding EVE as joint training of the LLM and a light-weight vision layer, we intentionally adopted the terms "encoder-free" and "lossless" to highlight the different design principles that distinguish EVE from existing modular VLMs.** Actually, we have explored the simplest setup. By removing the PAL and introducing the PEL via a streamlined Conv1 (14×14×1024)-GeLU-Conv2 (2×2×4096) structure, we ensured that the total number of activation values remains unchanged. This alignment with the "encoder-free" and "lossless" concepts reflects our goal to chase maximum simplicity without the need for a conventional, standalone vision encoder. Our findings were as follows: - The LLM-guided Prealigning Stage effectively prevents model collapse, which is a challenging issue in building encoder-free VLMs. - The minimalist PEL and drop of PAL initially impact training efficiency, particularly during early convergence (about 2.1-4.4\% under 5M high-resolution pretraining data). However, this gap gradually decreases as the data scale increases. This behavior underscores the LLM's great capability to learn visual perception and establish multimodal alignment from scratch, even in the absence of a vision encoder. The current design, although a trade-off, was chosen to prove the feasibility of an encoder-free VLM that can rival encoder-based ones, especially given the previously limited data and device resources. **(2) Once again, we would like to stress that the core idea behind EVE is to challenge the prevailing inductive biases present in encoder-based VLMs by allowing the model maximum freedom to discover more rational patterns on its own, inspired by 'The Bitter Lesson' (Rich Sutton).** This design philosophy aligns with the new trend in VLM research (GPT-4o [a], "discrete" Chameleon [b] and MoMa [c], 'continuous' Fuyu-8B [d] and SOLO [e]), where the focus is on reducing architectural constraints to develop an end-to-end VLM and process textual and visual inputs through a unified network for better scaling efficiency. **(3) We have preliminarily demonstrated that EVE is not only feasible but also promising for future advancements in VLMs. This innovation is what we believe sets EVE apart and warrants a re-evaluation of our proposed terminology.** We respectfully hope that this clarification will prompt you to reconsider your assessment and potentially raise the score of our submission. Should you have any further questions or require additional details, we would be happy to provide them. [a] GPT-4o System Card. OpenAI. August 8, 2024. [b] Chameleon: Mixed-Modal Early-Fusion Foundation Models. Meta FAIR. arXiv 2405.09818. [c] MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts. Meta FAIR. arXiv 2407.21770. [d] Fuyu-8B: A Multimodal Architecture for AI Agents. Adept AI. October 17, 2023. [e] SOLO: A Single Transformer for Scalable Vision-Language Modeling. UIUC. arXiv 2407.06438.

New reply and clarification to reviewer VqDP

Thank you for considering our response. We appreciate the opportunity to further clarify our contributions and address the concerns raised. `Q1: They addressed some of my concerns, although I still think it is over-claimed.` **(1) We respectfully reiterate that the key innovation of EVE is its preliminary validation of the feasibility of eliminating the inductive biases associated with encoder-based vision-language models (VLMs).** By removing these biases, EVE provides a clear path for constructing encoder-free VLMs that do not require an encoder pretrained on an intermediate task with biases in representations and resolutions. Thus, EVE maximizes the model's autonomy in learning vision perception and aligning multimodal patterns for better scaling efficiency, inspired by **'The Bitter Lesson' (Rich Sutton)**. **(2) Our EVE aligns with recent trends in VLM research, where the focus is shifting towards models that reduce architectural biases, construct a unified backbone, and improve scaling efficiency.** Notable examples of this trend are listed in time order: - Fuyu-8B: A Multimodal Architecture for AI Agents. Adept AI. October 17, 2023. - Chameleon: Mixed-Modal Early-Fusion Foundation Models. Meta FAIR. arXiv 2405.09818. - SOLO: A Single Transformer for Scalable Vision-Language Modeling. UIUC. arXiv 2407.06438. - PaliGemma: A versatile 3B VLM for transfer. Google DeepMind. arXiv 2407.07726. (Its encoder-free version) - MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts. Meta FAIR. arXiv 2407.21770. - GPT-4o System Card. OpenAI. August 8, 2024. These influential works exemplify a broader shift towards reducing architectural constraints in VLMs, aiming to develop end-to-end models that process textual and visual inputs through a unified network, thereby enhancing scaling efficiency. **(3) Our preliminary results demonstrate that EVE is not only feasible but also holds good promise for advancing VLM development.** We respectfully hope that this clarification would convince you to reconsider our proposed terminology and potentially improve your evaluation score based on the clarified context. We are open to providing further details or addressing any additional questions you may have. `Q2: In fact, I am asking about the detailed description task (LLaVA Bench), where the model is asked to provide a detailed description given an image (much longer and more comprehensive and diverse than image captioning). Have you tried the model with this dataset? It should be convenient as LLaVA already provided similar evaluation scripts.` Since LLaVA-Bench requires the GPT-4 API, we didn't include it in our earlier validation. We conducted further experiments using EVE-7B (HD) trained with LLaVA-mix-665 on LLaVA-Wide, achieving a 56.9% score, which is lower than LLaVA's 65.4%. This performance gap may be due to the QA-oriented SFT data and the limited vision perception learned from the current million-scale pretraining data. We appreciate you bringing attention to the detailed description task like LLaVA-Bench. Moving forward, we will focus on enhancing the model's capabilities by incorporating more training data and developing well-constructed instruction data.

New reply and clarification to reviewer nP6r

Thank you for considering our response and for your valuable feedback. We appreciate the opportunity to further address the concerns raised. `Q1: "It becomes less crucial as the data scale increases" is not convincing. VE can be trained easily with noisy large-scale web data; while generative VLM does require higher quality data, which makes it less easy to scale the VLM training data. I.e., "This indicates that eliminating the vision encoder for a fully encoder-free VLM is practical" might not be true.` **(1) In the context of the rebuttal, "*it* becomes less crucial ..." and "eliminating the *vision encoder* for a ..." refer specifically to the distillation/supervision of the vision encoder, not the vision encoder used in encoder-based VLMs.** In Figure 1 of the rebuttal PDF, vision encoder distillation does help early convergence with a moderate data scale, but its importance diminishes with larger data sets (beyond 24M in our experiments). This suggests that removing the vision encoder supervision in EVE is feasible with extensive pre-training data. **(2) Removing the vision encoder in encoder-based VLMs proves practical even with noisy large-scale data.** Only with 1B mixture data (including large noisy WebLI, CC3M-35L, and etc), Paligemma's encoder-free model nearly matches the encoder-based one (with 40B data for separate SigLIP) across about 13 of 37 datasets (e.g., ChartQA, GQA, RefCOCO/+/g, RSVQA, etc.). The scaling trend reflects the potential for further bridging their performance gap, indicating that a pre-trained vision encoder may not be essential and that using noisy data (especially interleave data [a]) to train generative VLMs is acceptable. **(3) We agree that high-quality data is crucial for VLMs compared to noisy data. However, scaling up high-quality data is not as difficult as it might appear.** For example, with LLaVA1.5-13B via the SGLang package, we can caption about 4-5M images in one day using two A100 (40G) nodes. With more devices, the process becomes even faster. Additionally, there are many open-source re-captioning data sources (e.g., CapFusion [b], Recap-Datacomp-1B [c]) and caption engines (e.g., ShareGPT4v [d], DenseFusion [e]) available for constructing pre-training data. Besides, in the text-to-image generation field, building large high-quality datasets is commonly developed in the research and industry. Thus, we do not view this as a significant bottleneck in real-world scenarios. [a] VILA: On Pre-training for Visual Language Models. CVPR2024 [b] CapsFusion: Rethinking Image-Text Data at Scale. CVPR2024 [c] What If We Recaption Billions of Web Images with LLaMA-3? arXiv 2406.08478 [d] ShareGPT4V: Improving Large Multi-Modal Models with Better Captions. ECCV2024 [e] DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception. arXiv 2407.08303 `Q2: Elaborate "Scaling performance of EVE with or without vision encoder supervision"` For fairness, we retain the Patch Embedding Layer (PEL) and remove the Patch Aligning Layer (PAL) (i.e., VE supervision) at all stages, keeping other variables unchanged. - Stage 1: We train both PEL and PAL for EVE-7B and EVE-7B-HD, and only PEL for versions without VE supervision, using 16M out of 33M pretraining data at the largest 672-pixel resolution. - Stage 2: We unfreeze all modules for these versions, utilizing all 33M pretraining data at the largest 672-pixel resolution. - Stage 3: We train the full model for EVE-7B (w/ and w/o VE supervision) using LLaVA-mix-665K, and for EVE-7B-HD (w/ and w/o VE supervision) using extra 1.2M SFT data at a 1344-pixel resolution. `Q3: What is learned in this stage when "without vision encoder supervision"?` Through text supervision, EVE aims to learn visual perception and establish an initial vision-language connection. This provides a better starting point for subsequent large-scale pretraining and avoiding model collapse. `Q4: Have you tried maintaining language ability in stage 2 or 3?` Yes, we conducted experiments using our multimodal data and text-only FineWeb data in 7:3, 5:5, and 3:7 ratios. - We used 5M mixed data for Stages 1 and 2, followed by LLaVA-mix-665K for Stage 3, all with a 1344-pixel image resolution, and removed vision encoder supervision throughout all stages. - We observed that a higher proportion of language-only data preserved better language capability (SQA scores improved from 63.5 to 64.7 to 65.2), but led to a slower increase in multimodal capability (GQA scores decreased from 58.3 to 57.4 to 55.2). - We suggest a 5:5 ratio for pre-training with high-resolution images. While we haven’t tested a 672-pixel resolution, a 7:3 (multimodal:text-only samples) ratio might be a good choice for balancing image-text token ratios, which is a better consideration factor. We respectfully hope that these explanations can convince you to potentially enhance your evaluation score. We are available to address any additional questions you may have.