Vector-ICL: In-context Learning with Continuous Vector Representations

Thank you for your insightful feedback. We appreciate your thoughtful analysis and would like to address your points: **Encoder-Decoder Context** You raise a fair point about the connection to encoder-decoder work like Flan-T5. While our method does involve encoding and projection, there are key differences: Our focus is on enabling LLMs to process continuous vectors through lightweight projectors, rather than training a full encoder-decoder architecture. Unlike Flan-T5 which requires extensive instruction tuning, our method only trains small projectors while keeping the LLM frozen. We extend beyond text to handle arbitrary continuous vectors from various modalities We have added Flan-T5 in related work to better position our contributions in this context. **Limitations and Failure Cases** We appreciate this suggestion to provide a more balanced analysis. We agree that Vector-ICL wouldn’t be the choice for a lot of tasks, such as logic reasoning or code generation, where symbolic correctness plays an important role. But the key message of this work is that: language models can now process continuous vector context from any kind of input. We think this would open doors to lots of other interesting research and applications.

Thank you for your thoughtful review and constructive feedback. We appreciate your detailed analysis and would like to address each point: **We don’t necessarily require task finetuning** We appreciate this insightful observation. Our intention is to enable LLMs to process continuous context via simple projections. To clarify: Vector-ICL without supervised training outperformed few-shot ICL on 4 out of the 6 tasks when direct comparison is avaliable (numbers, molecule, brain decoding, brain classification) and corresponding baselines. We added more baselines for few-shot ICL on multi-modal inputs, and demonstrated what we expected before, that few-shot ICL is unable to directly work with such cross-modal data. And we added experiments with pretraining-only projectors on time-series with synthetic data, and it outperformed both few-shot ICL and the baseline, demonstrating the usefulness of our approach. **Relationship with Soft Prompt Tuning** Our method shares conceptual similarities with soft prompt tuning in that both involve learning embeddings that influence the LLM's behavior. However, there are key distinctions. Soft prompt tuning optimizes continuous prompts to elicit specific behaviors from LLMs on text-based tasks. Our approach learns projectors that map any form of data into the LLM's embedding space, enabling it to better handle previously challenging inputs like large numbers and molecule strings and process entirely new data modalities. We’ve added a baseline paragraph in section 4 to explain how we are using soft-prompt as a baseline for textual tasks with task-specific training. **Additional Baselines for Non-textual Inputs** We thank you for your suggestion, following this idea, we added few-shot ICL baselines by supplying the inputs in their text form (we explained how we did it in the baseline paragraph as well, in Sec. 4). We showed that the language models cannot comprehend such data as one might expect. As for directly throwing the data into LMs, we think it’s possible but not without significant limitations. For example, vision LLMs still require an encoder to encode image patches. There has been research on directly processing the pixels (PiT [1]), but any regular-sized image would impose impossible the compute and memory requirements. So we don’t think this is the significantly more promising approach compared to our encoder-based Vector-ICL. **Additional Baselines for Time-series/graphs** The current baselines are obtained via tuning the domain-specific models. We agree more domain-specific baselines would make our work better but the key message of our research is not “Vector-ICL would beat SOTA models on each domain” but rather “language models can understand continuous context via Vector-ICL”. **For your questions:** **Q1** *How does the proposed approach differ from soft prompt tuning (Li and Liang, 2021)?* We have provided our answer to this question in the last section. To reiterate, soft prompt tuning optimizes continuous prompts to elicit specific behaviors from LLMs on text-based tasks. Our approach learns projectors that map any form of data into the LLM's embedding space, enabling it to better handle previously challenging inputs like large numbers and molecule strings and process entirely new data modalities. **Q2** *In 3.3 it is mentioned that the LLM would be trained with ``conditional generation loss''. However, in Figure 2b that is referred to, the LLM is supposed to be frozen. Could you clarify this point and explain the training process in more detail, including which components are frozen and which are updated during a) pre-training and b) fine-tuning?* The encoders and LLM are always kept frozen. We thank you for pointing this out, we have made an edit to prevent confusion. **Q3** *For Time Series (l 346), it is mentioned that you use the output of the last time step from Chronos-base. Doesn't this mean that the model is not able to use the full context of the time series but just the final state? What are the trade-offs between using the full context of the time series and just the final state?* Chronos models the time-series are with T5’s encoder-decoder architecture. So we think the final state should still have all the information about the full context. [1] Nguyen, Duy-Kien, et al. "An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels." arXiv preprint arXiv:2406.09415 (2024).

Thank you again for your thoughtful and detailed reviews of our paper. As the discussion period is approaching the end, we wanted to follow up to ensure we've fully addressed your feedback. Your comments have made the methodology and contribution clearer for our work. And if you feel we have adequately addressed your major criticisms, we would appreciate you considering updating your score accordingly.

Thank you for your thorough review and constructive criticism of our paper. We appreciate the detailed feedback and would like to address your key concerns: **Baselines** We apologize for any confusion regarding the baselines. To clarify: for text tasks, we compare finetuned projectors against soft prompt tuning, which is indeed a supervised baseline requiring similar computational resources as our method, and pretrained projector with few-shot ICL. For non-text tasks, we compare finetuned projectors against tuned domain-specific encoders (like Graphformer for graphs, Chronos for time-series) as detailed in Section 4. We added few-shot ICL as baselines by transforming the inputs into text (see added Baseline paragraph in Sec. 4, we also try to make the comparison clear over there), and a pretrained projector for time-series (see details on synthetic pretraining corpus in Sec 6.3). So Vector-ICL without supervised training outperformed few-shot ICL on 4 out of the 6 tasks when direct comparison is possible (numbers, molecule, brain decoding, brain classification), which shows its potential in lots of scenarios when the input doesn’t have a natural text form. **Motivation** Thanks for your suggestions. We’ve added few-shot ICL baselines for these cross-modality tasks, and from the results we can see models can not make good use of it. We also added experiments with pretraining-only projectors on time-series, and it significantly outperformed over few-shot ICL and the baseline, demonstrating the usefulness of a simple module. And to reiterate Vector-ICL without supervised training outperformed few-shot ICL in a lot of cases. We think this provides enough motivation for this work. **Writing** We appreciate your feedback on the paper's organization. We have added a paragraph to clarify what baselines are we comparing to and reorganized the tables for better readability. **For your questions:** **Q1** *Figure 4a: Why does the correlation vary so much between datasets and models of similar size?* We think the correlation variation likely comes from: * Dataset characteristics: Some datasets (e.g., IMDB) contain longer texts with more complex semantic structures, making the relationship between reconstruction and downstream performance less direct * Model architectures: Despite similar sizes, different LLMs have varying attention patterns and token processing mechanisms that affect how well they utilize the projected embeddings * Encoder-LLM alignment: Some encoder-LLM pairs naturally align better in their embedding spaces (same base model, similar training data etc.), leading to more consistent performance correlations **Q2** *Figure 4b: What do the axes represent?* The axes in Figure 4b represent numbers of uniformly sampled numbers from 0 to 1e10, we’ve updated the graph with x/y axis. Thanks for bringing this up. **Q3** *lines 477-479: Can you expand on the block patterns and how they are explained?* The block patterns in the distance matrix emerge from our number representation scheme and how it interacts with the projector's learning. Each number is represented as a concatenated one-hot encoding of its digits - for a 10-digit number, this creates a 10×10 matrix (one-hot vector for each digit), flattened into a 100-dimensional vector. The projector learns to preserve these numerical relationships during training, maintaining the hierarchical structure of the decimal system. This organization helps the LLM process numerical relationships more effectively, as the distance patterns reflect natural place-value relationships between numbers. We’ve added additional explanations in the analysis section as well. **Q4** *How does the finetuning work precisely? Do you finetune with multiple shots already? If so, how many?* The process of finetuning can be found in Sec 3.3. It is as simple as training for any conditional generation data, where we freeze encoder and LLM, and train the projector with task-specific datasets with the conditional generation loss. Yes, we generally use 128-shot in finetuning,

Thank you for your follow-up. Yes, we controlled the soft-prompt tuning to have the same contextual examples as Vector-ICL/few-shot ICL. We've updated Appendix A.3 to clarify this experimental setup. We appreciate your thorough review and we look forward to addressing any remaining questions or concerns you might have.

Thank you again for taking the time to review our paper and provide thoughtful feedback! As the discussion period is coming to an end, we wanted to check if there are any remaining concerns or clarifications we can address to further improve our work. Your comments have helped us significantly improve our work, particularly in improving how we motivate our work and clarifying how we compare against the baselines. If we’ve already addressed your major concerns, we kindly hope you’ll consider updating your rating to reflect this.

Thank you for your careful review and insightful feedback on our paper. We greatly appreciate your recognition of the novelty and promise of our approach, as well as your constructive suggestions for improvement. We would like to address your key points: **Single-Vector v.s. Multi-Vector** We agree that the single vector approach may not be optimal, and we have included it as our future direction in the discussion section (*“Also, we only experimented with single-token encoders, while there exist encoders that produce variable-sized embeddings, that can potentially be more powerful and flexible.”*). Overlooking from the RAG lines of work, chunking the input, and encoding them as multiple vectors is definitely possible but non-trivial. We want this paper to be a starting point for exploiting continuous vector context, which opens up research questions like this. **Instruction Tuning & Baselines** We thank you for the suggestion. Firstly, we created such a synthetic instruction dataset with time-series data, where natural text pairs are difficult to obtain. We utilized the statistical properties of the time series such as its stability and overall trend (more details in Sec 6.3). We then use this dataset to pretrain the projector for time-series, turns out the pretrained projector + LLM can generalize to the classification task (see the new Figure 4). Secondly, we added more baselines for few-shot ICL on multi-modal inputs, and demonstrated what we expected before, that few-shot ICL is unable to directly work with such data. So to clarify, Vector-ICL with pretraining is more performant than few-shot ICL on 4 out of the 6 tasks when direct comparison is possible (numbers, molecule, brain decoding, brain classification), which shows its potential in lots of scenarios when the input doesn’t have a natural text form. **For your questions:** **Q1** *How do you see vector-ICL applied in practical scenarios, considering decreasing inference cost and increasing context length of LLMs? Given your experiments show that without task-specific fine-tuning, vector-ICL consistently outperforms tokens-based ICL?* We have clarified the performance issue. And we want to highlight that our work is not about efficiency per se (compressing long inputs into single tokens), it is about how any form of data can be utilized by LLMs via encoding and projection. As we have shown in the baselines, supplying time-series/graphs/brain fMRI naively into the language models does not work very well. So we are confident that Vector-ICL can be of practical use in many such scenarios. **Q2** *Did you explore instruction tuning after pre-training? Authors should discuss the tradeoffs between task-specific finetuning and more general approaches like instruction tuning/ human preference training.* Thank you for this suggestion. We have explored instruction tuning through a synthetic instruction corpus for time-series data, where natural text-data pairs are difficult to obtain. Our pipeline generates instruction-like data by analyzing statistical properties (trend, anomalies, stability) of time-series and converting them into natural language QA pairs. Results show that projectors pretrained on this synthetic corpus already outperform both few-shot ICL and domain-specific baselines. We’ll leave the text-based instruction tuning / RLHF for the future, as there’s lots of variability in such studies.

Thank you for the further clarifications.