Thank you for your insightful review! 🙂
About (1) technique novelty, we agree with your review. Other than showing how to do behavior finetuning (using content and behavior instruction tokens), we do not intend to go for any other technique novelty in this paper because we find existing techniques are good enough to train our model well. Rather, we focus on the following contributions:
1. Incorporating behavior tokens in content and behavior simulations not only enhances behavior and content simulation (as anticipated) but also unexpectedly improves behavior domain-adaptation (Table-6), behavior-understanding (Table 4, Figs. 7, 8), and content understanding (Table-5). Notably, LCBM exhibits enhancements in these tasks despite not being explicitly trained for them. This aligns with how LLMs, trained for next word prediction, showcase proficiency in various tasks such as question answering, summarization, and natural language inference. We demonstrate these capabilities across three behavior domain datasets (YouTube, Emails, and LVU) and five tasks spanning two content domain datasets.
2. Presently, large language models focus on training with content tokens (text, vision, videos, etc.) while disregarding behavior tokens (Biderman et al 2023, Penedo et al 2023). Consequently, LLMs lack behavior simulation (Table 1, 2), content simulation (generation) (Table 3), and behavior understanding (Table 4) capabilities.
3. In-context learning, although successful in other domains, proves ineffective in the behavior domain. Large in-context-finetuned language models (GPT-3.5, GPT-4, etc.) exhibit performance comparable to a random baseline, indicating their lack of exposure to behavior tokens.
4. Next, to spur research on the topic of content and behavior models, we release behavior and content tokens collected over 40,000 public domain YouTube videos containing a variety of behavior tokens.
5. Additionally, subsequent to paper submission, we've compiled a large-scale content behavior dataset comprising images and corresponding human behavior extracted from 168 million tweets across 10135 enterprise Twitter accounts from 2007 to 2023. This dataset includes account names, tweet text, associated media verbalizations (including image captions, keywords, colors, and tones), tweet timestamps, and like counts. We plan to release both the YouTube and Twitter corpora. Our experiments with behavior simulation, domain adaptation, and content simulation on this corpus yield results consistent with those reported in the paper:
**Behavior Simulation. Give content, channel, time, predict behavior (High, Low). Reported values are 2-way classification accuracies**
(Brand Separated means that the train and test set don't have any overlap in terms of brands, Time Separated means that the test set starts after the last tweet in the train set)
| **Model** | **Behavior Training** | **Behavior Datasets Included In Training** | **Brand Separated** | **Time Separated** |
|---|---|---|---|---|
| GPT-3.5 (few-shot) | In-context-learning | | 58.84 | 64.19 |
| LCBM | Finetuning | Twitter (77.3k tweets) | 74.3 | **97.69** |
| LCBM | Finetuning | Twitter (77.3k tweets) + Youtube (40k) data | **76.87** | 92.19 |
**Content Simulation Experiment. Given behavior, channel, time, tweet media caption as prompt, predict content (tweet text)**
| Model | Behavior Training | Behavior Datasets Included In Training | Test-split | Bleu-1 | Bleu-2 | Bleu-3 | Bleu-4 | Rouge-l |
|---|---|---|---|---|---|---|---|---|
| GPT-3.5 (few-shot) | In-context-learning | | Brand Separated | 53.95 | 42.36 | 31.84 | 24.28 | 15.24 |
| | In-context-learning | | Time Separated | 57.69 | 45.11 | 33.67 | 25.52 | 15.27 |
| LCBM | Finetuning | Twitter (77.3k tweets) | Brand Separated | 62.29 | 46.59 | 33.98 | 25.64 | 14.44 |
| | Finetuning | Twitter (77.3k tweets) | Time Separated | 70 | 54.4 | 41.43 | 32.48 | 17.38 |
| LCBM | Finetuning | Twitter (77.3k tweets) + Youtube (40k) data | Brand Separated | **64.28** | **48.1** | **35.17** | **26.63** | **14.83** |
| | Finetuning | Twitter (77.3k tweets) + Youtube (40k) data | Time Separated | **70.23** | **54.54** | **41.52** | **32.54** | **17.45** |
The question about detailed information on how the visual encoder handles video input. We apologize for not having a clear enough coverage of this topic. We will make sure to cover this in the next version. The details are:
The pipeline uses the pre-trained EVA-ViT-G with the Global Multi-Head Relation Aggregator (GMHRA) which is a temporal modeling module. For token projection, we emply a pretrained QFormer with extra linear projection to incorporate additional query tokens to include video context for training the LLM. While training, the projected video encodings are used to represent video context in the training corpus.
(continued in the next comment)