LongVILA: Scaling Long-Context Visual Language Models for Long Videos

Response to Reviewer MTsu (Part 1)

We sincerely appreciate your insightful comments. Below, we provide responses to each concern. **Q1: “More baselines and video QA benchmarks.”** **A:** Thank you for the reminder. Regarding baselines, we included additional strong models for comparison, such as PLLaVA [1], Flash-VStream [2], Video-LLaMA-2 [3], LLaVA One-Vision [4], and proprietary models like GPT-4V [5], GPT-4o [6], and Gemini-1.5-Pro [7]. For benchmarks, we added 8 more benchmarks for evaluation: ActivityNet-QA [8], EgoSchema [9], EventBench [10], LongVideoBench [11], PerceptionTest [12], MVBench [13], NExT-QA [14], and VNBench [15]. Due to the inaccessibility of the [MoVQA repository](https://github.com/OpenGVLab/MoVQA) at this period, we will consider it for future evaluation when their repository is available. Our LongVILA-7B model demonstrates strong performance across these benchmarks, as presented in **Table 3 in the revision**. | Model | LLM Size | ActNetQA | EgoSchema | MVBench | PercepTest | NExTQA | LVideoBench | EventBench | VNBench | VideoMME | VideoMME | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | | | test | test | test | val | mc | val | val | val | w/o sub. | w/ sub. | | GPT-4V | - | 57.0 | - | 43.5 | - | - | 61.3 | 32.6 | - | 59.9 | 63.3 | | GPT-4o | - | - | - | - | - | - | 66.7 | 53.3 | 64.4 | 71.9 | 77.2 | | Gemini-1.5-Pro | - | 57.5 | 72.2 | - | - | - | 64.0 | 43.2 | 66.7 | 75.0 | 81.3 | | | | | | | | | | | | | | | Video-LLaVA | 7B | 45.3 | 38.4 | 43.5 | - | - | 37.6 | 5.9 | 12.4 | 39.9 | 41.6 | | Flash-VStream | 7B | 51.9 | - | - | - | 61.6 | - | - | - | - | - | | ShareGPT4Video | 8B | 50.8 | - | 51.2 | - | - | 41.8 | - | - | 39.9 | 43.6 | | VideoLLaMA2 | 7B | 50.2 | 51.7 | 54.6 | 51.4 | - | - | 6.9 | 4.5 | 47.9 | 50.3 | | VideoLLaMA2.1 | 7B | 53.0 | 53.1 | 57.3 | 54.9 | - | - | - | - | 54.9 | 56.4 | | Kangaroo | 8B | - | 62.7 | 61.1 | - | - | 54.8 | - | - | 56.0 | 57.6 | | PLLaVA | 7B | 56.3 | - | 46.6 | - | - | 39.2 | 28.2 | - | - | - | | LLaVA-OV | 7B | 56.7 | 60.1 | 56.7 | 57.1 | 79.4 | 56.4 | - | 51.8 | 58.2 | 61.5 | | **LongVILA** | **7B** | **59.5** | **67.7** | **67.1** | **58.1** | **80.7** | **57.1** | **58.0** | **63.0** | **60.1** | **65.1** |

Response to Reviewer MTsu (Part 2)

**Q2: “Model complexity analysis.”** **A:** Thank you for the helpful suggestion. We conducted a detailed **model complexity** analysis of LongVILA-1.5B and LongVILA-7B, presented in **Table 10 in the revision**. This analysis includes parameters, FLOPs, and latency (ms) across different numbers of frames, context lengths, and model components. We categorize the LongVILA models into four types of components: - **Image Encoder:** the vision tower and multi-modal projector. - **LLM Linears:** k/q/v/o projectors and linear layers in the decoder. - **LLM Attention:** attention computation. - **LLM Others:** other components such as embeddings, output heads, and normalization layers. Latency measurements were performed with **fp16** data type and **Flash-Attention2** [16] on a single A100 GPU. This profiling provides a comprehensive view of the computational demands of LongVILA-1.5B and 7B in various settings. As the number of frames increases, the FLOPs and latency of LLM Attention grow quadratically, whereas other components increase linearly. LLM Attention represents the predominant computational cost in long video understanding, highlighting the necessity of the Multi-Modal Sequence Parallelism (MM-SP) system. **The model complexity profile of LongVILA-1.5B** | Frames | Context | Metric | ImageEncoder | LLM Linears | LLM Attention | LLM Others | | --- | --- | --- | --- | --- | --- | --- | | | | Params | 0.44B | 1.31B | - | 0.23B | | 32 | 6415 | Latency | 196.0 | 109.3 | 27.0 | 28.9 | | | | TFLOPs | 5.13 | 4.20 | 1.77 | 0.75 | | 64 | 12719 | Latency | 375.2 | 208.1 | 80.7 | 50.9 | | | | TFLOPs | 10.26 | 8.33 | 6.96 | 1.48 | | 128 | 25327 | Latency | 740.4 | 403.2 | 288.1 | 97.1 | | | | TFLOPs | 20.52 | 16.60 | 27.59 | 2.95 | | 256 | 50543 | Latency | 1456.0 | 811.2 | 1087.7 | 192.5 | | | | TFLOPs | 41.05 | 33.12 | 109.89 | 5.88 | | 512 | 100975 | Latency | 2921.0 | 1660.9 | 4359.2 | 389.4 | | | | TFLOPs | 82.09 | 66.17 | 438.59 | 11.75 | **The model complexity profile of LongVILA-7B** | Frames | Context | Metric | Image Encoder | LLM Linears | LLM Attention | LLM Others | | --- | --- | --- | --- | --- | --- | --- | | | | Params | 0.45B | 6.53B | - | 1.09B | | 32 | 6415 | Latency | 201.0 | 426.2 | 51.1 | 42.1 | | | | TFLOPs | 5.22 | 20.93 | 4.13 | 1.74 | | 64 | 12719 | Latency | 375.8 | 831.1 | 173.8 | 80.8 | | | | TFLOPs | 10.45 | 41.50 | 16.24 | 3.46 | | 128 | 25327 | Latency | 755.1 | 1642.4 | 644.9 | 158.3 | | | | TFLOPs | 20.89 | 82.64 | 64.38 | 6.89 | | 256 | 50543 | Latency | 1476.3 | 3308.2 | 2529.6 | 325.1 | | | | TFLOPs | 41.78 | 164.92 | 256.40 | 13.74 | | 512 | 100975 | Latency | 2980.2 | 6675.1 | 10149.3 | 653.4 | | | | TFLOPs | 83.57 | 329.48 | 1023.37 | 27.45 | [1] Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See-Kiong Ng, Jiashi Feng: PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning. 2024 [2] Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, Xiaojie Jin: Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams. 2024 [3] Zesen Cheng, et. al.: VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs. 2024 [4] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, Chunyuan Li: LLaVA-OneVision: Easy Visual Task Transfer. 2024 [5] OpenAI. Gpt-4v. 2023 [6] OpenAI. Hello gpt-4o. 2024 [7] Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. [8] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, Dacheng Tao: ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering. AAAI 2019 [9] Karttikeya Mangalam, Raiymbek Akshulakov, Jitendra Malik: EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding. NeurIPS 2023 [10] Yifan Du, Kun Zhou, Yuqi Huo, Yifan Li, Wayne Xin Zhao, Haoyu Lu, Zijia Zhao, Bingning Wang, Weipeng Chen, Ji-Rong Wen: Towards Event-oriented Long Video Understanding. 2024 [11] Haoning Wu, Dongxu Li, Bei Chen, Junnan Li: LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding. 2024 [12] Viorica Patraucean, et. al: Perception Test: A Diagnostic Benchmark for Multimodal Video Models. NeurIPS 2023 [13] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Lou, Limin Wang, Yu Qiao: MVBench: A Comprehensive Multi-modal Video Understanding Benchmark. CVPR 2024 [14] Junbin Xiao, Xindi Shang, Angela Yao, Tat-Seng Chua: NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions. CVPR 2021 [15] Zijia Zhao, Haoyu Lu, Yuqi Huo, Yifan Du, Tongtian Yue, Longteng Guo, Bingning Wang, Weipeng Chen, Jing Liu: Needle In A Video Haystack: A Scalable Synthetic Framework for Benchmarking Video MLLMs. 2024 [16] Tri Dao: FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. ICLR 2024

Response to Reviewer 7Lyg

We are truly appreciated for your valuable comments. In the following, we provide responses to the concerns. **Q1: “Performance metrics on general video understanding benchmarks, such as the PerceptionTest or EgoSchema.”** **A:** Thank you for the constructive suggestion. We have included 8 additional video benchmarks for evaluation, with ActivityNet [1], EgoSchema [2], PerceptionTest [3], MVBench [4], and NExT-QA [5] serving as general video understanding benchmarks. LongVILA-7B demonstrates promising performance on these benchmarks compared to strong baselines such as LLaVA-OneVision [6]. This is **Table 3 in the revision**. | Model | LLM Size | ActNetQA | EgoSchema | PercepTest | MVBench | NExTQA | LVideoBench | EventBench | VNBench | VideoMME | VideoMME | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | | | test | test | val | test | mc | val | val | val | w/o sub. | w/ sub. | | GPT-4V | - | 57.0 | - | - | 43.5 | - | 61.3 | 32.6 | - | 59.9 | 63.3 | | GPT-4o | - | - | - | - | - | - | 66.7 | 53.3 | 64.4 | 71.9 | 77.2 | | Gemini-1.5-Pro | - | 57.5 | 72.2 | - | - | - | 64.0 | 43.2 | 66.7 | 75.0 | 81.3 | | | | | | | | | | | | | | | Video-LLaVA | 7B | 45.3 | 38.4 | - | 43.5 | - | 37.6 | 5.9 | 12.4 | 39.9 | 41.6 | | Flash-VStream | 7B | 51.9 | - | - | - | 61.6 | - | - | - | - | - | | ShareGPT4Video | 8B | 50.8 | - | - | 51.2 | - | 41.8 | - | - | 39.9 | 43.6 | | VideoLLaMA2 | 7B | 50.2 | 51.7 | 51.4 | 54.6 | - | - | 6.9 | 4.5 | 47.9 | 50.3 | | VideoLLaMA2.1 | 7B | 53.0 | 53.1 | 54.9 | 57.3 | - | - | - | - | 54.9 | 56.4 | | Kangaroo | 8B | - | 62.7 | - | 61.1 | - | 54.8 | - | - | 56.0 | 57.6 | | PLLaVA | 7B | 56.3 | - | - | 46.6 | - | 39.2 | 28.2 | - | - | - | | LLaVA-OV | 7B | 56.7 | 60.1 | 57.1 | 56.7 | 79.4 | 56.4 | - | 51.8 | 58.2 | 61.5 | | **LongVILA** | **7B** | **59.5** | **67.7** | **58.1** | **67.1** | **80.7** | **57.1** | **58.0** | **63.0** | **60.1** | **65.1** | **Q2: “The proposed Multi-Modal Sequence Parallelism design and the source code.”** **A:** The source code, including the Multi-Modal Sequence Parallelism implementation, will definitely be released. For your convenience, we provide an [anonymous link](https://anonymous.4open.science/r/LongVILA) of our internal source code, where Multi-Modal Sequence Parallelism training can be easily enabled using the script with “llava/train/train_hybrid.py” and the argument “--seq_parallel_size”. [1] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, Dacheng Tao: ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering. AAAI 2019 [2] Karttikeya Mangalam, Raiymbek Akshulakov, Jitendra Malik: EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding. NeurIPS 2023 [3] Viorica Patraucean, et. al: Perception Test: A Diagnostic Benchmark for Multimodal Video Models. NeurIPS 2023 [4] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Lou, Limin Wang, Yu Qiao: MVBench: A Comprehensive Multi-modal Video Understanding Benchmark. CVPR 2024 [5] Junbin Xiao, Xindi Shang, Angela Yao, Tat-Seng Chua: NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions. CVPR 2021 [6] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, Chunyuan Li: LLaVA-OneVision: Easy Visual Task Transfer. 2024

Response to Reviewer DDGN

We sincerely appreciate your valuable comments. Below, we provide responses to each concern. **Q1: “Additional benchmarks focused on long video analysis, such as those introduced in recent works.”** **A:** We have evaluated LongVILA on 8 additional benchmarks, including the two you suggested: EventBench (arXiv:2406.14129) [1] and VNBench (arXiv:2406.09367) [2], which focus on long video analysis. LongVILA-7B demonstrates strong performance across these benchmarks, achieving the best results on EventBench. On VNBench, LongVILA-7B is slightly outperformed by GPT-4o [3] and Gemini-1.5-Pro [4], but surpasses other open models. These results are presented in **Table 3 in the revision**. | Model | LLM Size | EventBench | VNBench | ActNetQA | EgoSchema | LVideoBench | PercepTest | MVBench | NExTQA | VideoMME | VideoMME | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | | | val | val | test | test | val | val | test | mc | w/o sub. | w/ sub. | | GPT-4V | - | 32.6 | - | 57.0 | - | 61.3 | - | 43.5 | - | 59.9 | 63.3 | | GPT-4o | - | 53.3 | 64.4 | - | - | 66.7 | - | - | - | 71.9 | 77.2 | | Gemini-1.5-Pro | - | 43.2 | 66.7 | 57.5 | 72.2 | 64.0 | - | - | - | 75.0 | 81.3 | | | | | | | | | | | | | | | Video-LLaVA | 7B | 5.9 | 12.4 | 45.3 | 38.4 | 37.6 | - | 43.5 | - | 39.9 | 41.6 | | Flash-VStream | 7B | - | - | 51.9 | - | - | - | - | 61.6 | - | - | | ShareGPT4Video | 8B | - | - | 50.8 | - | 41.8 | - | 51.2 | - | 39.9 | 43.6 | | VideoLLaMA2 | 7B | 6.9 | 4.5 | 50.2 | 51.7 | - | 51.4 | 54.6 | - | 47.9 | 50.3 | | VideoLLaMA2.1 | 7B | - | - | 53.0 | 53.1 | - | 54.9 | 57.3 | - | 54.9 | 56.4 | | Kangaroo | 8B | - | - | - | 62.7 | 54.8 | - | 61.1 | - | 56.0 | 57.6 | | PLLaVA | 7B | 28.2 | - | 56.3 | - | 39.2 | - | 46.6 | - | - | - | | LLaVA-OV | 7B | - | 51.8 | 56.7 | 60.1 | 56.4 | 57.1 | 56.7 | 79.4 | 58.2 | 61.5 | | LongVILA | 7B | **58.0** | **63.0** | **59.5** | **67.7** | **57.1** | **58.1** | **67.1** | **80.7** | **60.1** | **65.1** | **Q2: “To include performance metrics for 512 and 1024 frames in Figure 2(a).”** **A:** Thank you for the suggestion. We have revised the figure (now **Figure 3**) to include performance metrics for 512 and 1024 frames, showing sustained growth as the number of frames increases. **Q3: “The performance change of stages 1-2-4-(3&5), 4-1-2-3-5, 4-1-2-(3&5).”** **A:** Thank you for this insightful question. We conducted additional ablation studies on these training stage configurations, using VideoMME without subtitles for evaluation. Mixing stage 3 with stage 5 (i.e., **1-2-4-(3&5)**) leads to large performance drops on short and medium videos. When context extension is prioritized (i.e., **4-1-2-3-5**), performance degrades for long videos, likely due to the extensive use of short data in stages 2 and 3, which diminishes long context capabilities. Further mixing stage 3 with stage 5 (**4-1-2-(3&5**) again results in performance drops for short and medium videos. These results are provided in **Table 1** **in the revision.** In addition to performance, the primary reason for selecting **1-2-3-4-5** is that **1-2-3** represents a widely adopted VLM training pipeline. Leveraging pre-trained, off-the-shelf VLMs provides two key advantages: 1. It avoids the costly re-training of stages 2 (VLM pretraining) and 3 (VLM supervised fine-tuning). 2. Stages 1, 2, and 3 are often difficult to train for public community, as open VLMs do not always provide access to their datasets (e.g., Qwen2-VL, LLaMA3.2-VL). Fixing **1-2-3** ensures greater consistency and reliability for long-term research, which is crucial for AI conferences. | Training Stages | Average | Short | Medium | Long | | --- | --- | --- | --- | --- | | 1-2-3-4-5 | 57.5 | 69.3 | 56.1 | 47.0 | | 1-2-4-(3&5) | 55.9 | 67.4 | 54.1 | 46.1 | | 4-1-2-3-5 | 56.0 | 69.2 | 54.1 | 44.5 | | 4-1-2-(3&5) | 55.3 | 67.2 | 53.6 | 45.1 | [1] Yifan Du, Kun Zhou, Yuqi Huo, Yifan Li, Wayne Xin Zhao, Haoyu Lu, Zijia Zhao, Bingning Wang, Weipeng Chen, Ji-Rong Wen: Towards Event-oriented Long Video Understanding. 2024 [2] Zijia Zhao, Haoyu Lu, Yuqi Huo, Yifan Du, Tongtian Yue, Longteng Guo, Bingning Wang, Weipeng Chen, Jing Liu: Needle In A Video Haystack: A Scalable Synthetic Framework for Benchmarking Video MLLMs. 2024 [3] OpenAI. Hello gpt-4o. 2024 [4] Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.