UniAR: A Unified model for predicting human Attention and Responses on visual content

Summary

In this paper, the authors introduce a novel text-image framework designed to integrate various human-response tasks and multiple image domains. These tasks include attention map generation, scanpath prediction, and subjective preference evaluation, applied to images such as webpages, natural scenes, graphic designs, and mobile user interfaces. The framework utilizes a transformer architecture, accepting both text and image inputs. Text inputs specify the tasks, while the model itself comprises a transformer encoder and three predictors (heatmap, rating, and scanpath) to generate 3 types of outputs. Initially pretrained on foundational text-image tasks, the model is fine-tuned for diverse human-response tasks

Strengths

1. The question of unifying human response-related vision models is novel and holds significant potential, particularly if the advantages of integrating tasks are thoroughly explored. 2. The paper is well-organized and easy to understand.

Weaknesses

1. I have many concerns on the experimental results of this paper, where the authors try to demonstrate the superiority of their model performance by comparing with other methods. My concerns are as follows. a) The metrics in Table 3 are not a standard set of metrics for saliency prediction. Why don’t the authors report the complete of evaluation metrics in Table 3? It seems strange to drop the popular metrics such as sAUC and SIM in Table 3. It is a standard practice to report all standard metrics, ie., those commonly used in saliency predictions, as in the online benchmark (https://saliency.tuebingen.ai/evaluation.html). It seems odd to me the authors only report selected metrics in Table 3 but report all in Table 6 (which is in the appendix). b) Following a), why do the authors put both Table 3 and Table 6? Both of the tables occupy the same space. The authors can just put Table 6 in the main manuscript instead of Table 3. c) Why do the comparison methods vary greatly across different testing datasets? It will be helpful if the authors stick to several methods and compare their performance across the same testing datasets. Deep Gaze II and IIE are recognized for their strong performance on images of general scenes. Why did the authors choose to test these models only on the CAT2000 dataset and not on other general scene datasets such as Salicon and OSIE? d) In the SalGAN paper [60], it reports the metrics on Salicon dataset, why the authors do not report this in Table 3 or Table 6 for SalGAN on Salicon dataset, but instead, only report its performance on the webpage dataset? If the authors have simply copied results from other works, incorporating these additional benchmarks would require minimal effort and should have been included. If the authors have tested the models, why not test them on the popular benchmarks such as Salicon and OSIE? e) Can the authors explain why they don’t report SequenceScore for COCO-Search18? This column is denoted as "-" in Table 4. f) Furthermore, the reported results on COCO-Search18 of the comparison method FFM[79] in its own paper [79] are different from what what authors report in Table 4, both under target present conditions. The SemSS scores in [79] under target present conditions are consistenly above 0.53, and the method of Chen et al. [16] is 0.572. Can authors provide more details on how they obtained the current results in Table 4 for the comparison methods? g) Can authors explain why they didn’t compare scanpath prediction performance of ALOHA and baselines on COCO-FreeView in Table 4 even though they use COCO-FreeView in Sec. 4.4? h) It will be helpful if the authors can experiment with more popular benchmark datasets such as MIT300 and MIT1003, and compare with the performances published on the online benchmark (https://saliency.tuebingen.ai/results.html). i) Minor: some top performances are wrongly indicated, eg., on the webpage dataset (FiWI), the best performance on KLD should be Chen et al. [13]. In summary, excluding the SOTA models in common benchmark datasets significantly undermines the experimental evidence for ALOHA's competitiveness, especially considering that 7 out of the 11 datasets in the experimental section are for free-viewing saliency prediction (see Table 1). Based on all the above, I feel that current experiments cannot adequately supported the paper's claims.The paper needs more discussions and comparison results to support its claim that the model achieves SOTA performance. 2. Although the authors claim that the unified model could “act as a comprehensive reward model, predicting subjective ratings/preferences as rewards and providing additional insights from predicted human attention and behavior patterns,” the experiments do not clearly demonstrate the actual benefits of unifying the attention, rating, and scanpath tasks. The motiviation of this work is not well positionied. 3. Some technical details of the paper are not clear. E.g, a). the three predictors appear to operate independently, but it is unclear how they were chosen. b) Why text generation is chosen for scanpath generation. c) How is the model fine-tuned on various datasets.

Questions

1. How are the three predictors chosen for each task? Are they fixed assigned based on different input texts? 2. Are there any differences between the salience map and the importance map in terms of architecture or loss function, or do the differences lie solely in the training data? 3. In the "decoding during inference" section, the paper mentions, “If there is no fixation available in the predicted string, we mark the scanpath as invalid.” This invalid case (“no fixation”) only covers situations where all token pairs are non-numerical. If some tokens in the output sequence are in an unrecognizable format, rendering part of the results invalid, how should this be evaluated and handled? 4. Why is text generation used as a scanpath predictor? 5. How is the model fine-tuned on various datasets, which parts are freezed? 6. All my above questions regarding the numerical results (Table 3 and Table 6) in the Weakness section.

Rating

3

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The reported numerical results do not adequatly support the paper's claim. More experiments to demonstrate the model's task transferring ability are needed. For example, transferring from saliency map to rating, or from scanpath to saliency map.

Summary

Noticing the existing issues in human behavior modeling, such as isolating the study of implicit, early-stage perceptual behavior (like human attention) from explicit, later-stage behavior (like subjective preferences), specified visual content type; in this manuscript, the author(s) aimed to build an integrated human attention and preference behavior model for addressing multiple visual content types. Through empirical experiments, the author(s) demonstrated the effectiveness of the proposed model.

Strengths

In my opinion, the strengths of this manuscript are as follows: 1. Designed a unified approach to model human visual behavior: image+text=>human perceptual behaviors. 2. Extensive experiments are performed to validate the effectiveness of the proposed model.

Weaknesses

In my opinion, the weaknesses of this manuscript are as follows: 1. This manuscript seems not to discuss the model's limitations in real-world situations, such as how its performance is in dynamic environments or its adaptability to changes in user behavior over time. 2. As the author(s) described, ALOHA has 848 million parameters. Training such a large model requires significant computational resources, which may not be accessible to most researchers or practitioners.

Questions

I read the manuscript, and I have the following questions/comments. Thanks. 1. Are there any potential applications of ALOHA in real-world scenarios or in dynamic environments? How can it be used to optimize elements, such as user interfaces, graphic designs, and content creation? 2. Are there any potential biases in the current model's predictions? Tiny format issues in References: (1) Sometimes, the journal/conference name used an abbreviation; sometimes, not, sometimes, both, such as Ref.[25], Ref.[67]. (2) Keep the format of the references consistent, such as Ref. [6] vs. Ref.[15]. Please check carefully and correct the issues. Overall, ALOHA represents a significant advancement in modeling human visual behavior, offering a unified approach that spans from early-stage perceptual responses to later-stage decision-making. I think this is an interesting manuscript. I look forward to hearing from the author(s). Thanks.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

Yes.

Summary

This paper introduces ALOHA, a multimodal model that predicts human saliency, scanpath, and subjective rating of natural images, webpages, and graphic designs. ALOHA outperforms or performs similarly to baseline models across each of its prediction tasks while improving generalizability over task-specific models.

Strengths

*Generalizable model of human visual behavior and preferences.* The ALOHA model predicts multiple forms of human preference --- the saliency, scanpath, and rating --- of various types of image inputs, including web pages, natural images, and cartoons. This improves the generalizability of ALHOA over prior models that focus on a single prediction task or data input modality. *In-depth analysis of model performance compared to prior benchmarks.* The paper provides a thorough evaluation of ALOHA's task-performance across 13 metrics, 11 datasets, and 4 tasks. ALOHA outperforms or performs on par with pre-existing models across all evaluations. *Easy to read paper.* The paper is well motivated and clearly explained. Figure 1 is a particularly helpful overview.

Weaknesses

My biggest concern is the paper’s lack of engagement with the limitations and ethical considerations of modeling subjective human preferences. Currently there is a brief discussion of training data limitations in the supplement. I would like to see a thorough discussion of considerations included (or at least referenced) in the main text. Possible discussion points that come to mind include: * What are the ethical considerations of replacing humans with models trained to replicate their preferences? Humans have diverse and often opposing preferences, particularly for subjective notions of aesthetics or attention. It is possible that models trained to replicate human preferences could learn a more uniform or singular notion of preference. Could using these models limit the visual diversity in webpages and content creation? Or could these models actually help us generate more visually diverse content? * Whose preferences are included in the model and, more importantly, whose aren’t? Humans likely have diverse preferences when it comes to subjective notions like website attractiveness. I would suspect these preferences vary based on age, experience with technology, culture, etc. Building a model to replicate human perspectives may amplify a particular worldview that is not representative of all users. On the other hand, it may be easier to create a model trained on diverse human data than to get a diverse set of human feedback for every new webpage design. * The paper suggests using the model as a reward function to optimize content creation. Could this result in negative consequences, such as content that is optimized for a model’s preference function and may not be value aligned with what humans want? * How could human preference models, like ALOHA, be misused? For example, it seems like these models could be used to make ads more intrusive by optimizing their scanpath location or to make phising sites more convincing by optimizing the saliency of the credential input. * How should human preference models take into account blind or low-vision users? Scanpaths and saliency likely differ between users who rely on screenreaders and those who don’t. Optimizing the web for sighted preferences could further exclude blind and low-vision from online spaces. Given the [emphasis](https://www.ada.gov/resources/web-guidance/) on web accessibility, how can we use human preference models to amplify (not weaken) online accessibility? * How should models stay up to date with the changing human preferences? Human preferences, especially around aesthetics, are constantly changing. As a result, what are the implications for interpreting the evaluations used in this paper that span datasets from 2014–2023? The paper would also be strengthened by an audit of their model’s behavior beyond quantitative performance metrics, such as an analysis of the diversity of the humans included in the training dataset, a categorization of the types of mistakes the model makes, and a [Model Card](https://arxiv.org/abs/1810.03993) reflecting its design. I recognize that the paper is inheriting these ethical concerns from the preexisting datasets and tasks that it works on. I hope that by including a discussion of the models implications, the paper will progress research into these types of models and inspire work on new datasets, model analysis, and safe deployment techniques. I might also suggest that the authors should select a new name for their model. I am not an expert in Hawaiian culture, but I know that the term ‘aloha’ is culturally significant, and there have been [Hawaiian movements to reclaim it](https://www.usatoday.com/story/life/health-wellness/2023/01/13/stop-saying-aloha-out-of-context/10990192002/), such as the [‘Aloha Not For Sale’ protests](https://kawaiola.news/cover/aloha-not-for-sale-cultural-in-appropriation/). Using it out of context as a model name, may minimize the rich cultural meaning behind the term, and given it is an imperfect acronym anyway, should be an easy change.

Questions

Please see the Weaknesses section above.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Please see the Weaknesses section above.

Summary

In their paper "ALOHA: from Attention to Likes – a unified mOdel for understanding HumAn responses to diverse visual content" the authors describe a new unified model to predict human saliency(attention/importance), even more fine grained than that, scanpath, and ratings. After nicely introduced the motivation and an extensive outlook on related work, the authors describe the multimodal VLM-based encoder-decoder transformer architecture in detail, especially the three predictor head. Exploting the power of instruction tuned LLM, the authors introduce additional tokens to predict valid scanpath at inference time. The benefits of their model are evlauted based on many experiments, which are shortly discussed, followed by a short conclusion.

Strengths

The main strength of the paper are the experiments, which are extensive across several benchmarks and metrics. Integrating a three headed VLM for the tasks is intuitive and elegant. Their proposed ALOHA architecture is SOTA in sevaral settings (22/35) The writing is mostly easy to follow and there is a clear picture the authors want to paint.

Weaknesses

- The writing can be improved in several places, I'll note here a few: 1. Introduction: - In the introduction, it seems that the authors already have the conclusion at the end, switching from present to past tense for the main contributions of the paper. This makes it sounds as if the authors already published ALOHA previously. - In the same two points of the main contributions, the authors actually only have one contribution, their model ALOHA. The second "contribution" is only the evaluation of ALOHA. - regarding the previous comment -- the second contribution, even though from my point of view it is no contribution is the evaluation of ALOHA and not the training of ALOHA. 3. Unifying Human Visual Behavior Modeling from Attention to Likes: - The overall optimization criterion should be part of the main paper, not in the appendix. - Model training (section 4) from my pov should be part of section 3, it is not part of the experiments since you do not report multiple training strategies. Its unclear how hyperparameters were tuned. Code will not be made public, which is a major concern in the current reproducibility crisis in ML. Additionally the question in the questionnaire is wrongfully answered as N/A; the paper DOES INCLUDE experiments requiring code. (GUIDELINE: The answer NA means that paper does not include experiments requiring code.)

Questions

You mention that you pad images to 512x512 -- what happens with images larger than 512x512? How did you arrive at your optimization criterion, scaling for the learning rates and in general all hyperparameters? You did not report any hyperparameter tuning.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Thanks for the great feedback!

We would like to thank Reviewer Sv28 for the valuable feedback and suggestions. As mentioned in our response to Reviewer FgZM, we will significantly expand the relevant section to discuss the societal impact of this technique and propose strategies to mitigate any potential negative effects. **Positive side**: In this paper, we proposed a unified model to predict both early-stage perceptual behaviors, such as human attention, and later-stage behaviors, such as subjective preferences and likes. This model offers several avenues for positive societal impact, including: 1. Ensuring visual content is better aligned with human perception and preferences, thereby enhancing user experience. 2. Training the model with a diverse set of human feedback, particularly from underrepresented groups, to ensure that visual content quality is optimized for individuals from various backgrounds. Achieving this goal is significantly more challenging without a unified model, as it would require training multiple individual models and evaluating them on new content for each group, which is not scalable. 3. Assisting individuals with blindness or low vision by enabling them to parse visual content more effectively. The model can highlight the most important areas of a webpage via heatmap predictions, making these areas more accessible through screen readers. **Risk**: However, as with other research in the area of human behavior and preferences, we acknowledge that this technique could potentially have negative societal impacts, especially if misused, as highlighted by Reviewer FgZM. These potential risks include: 1. The possibility of making harmful websites more convincing. 2. Bias within the model could lead to unfair outcomes. 3. Over-reliance on the model, with the exclusion of human oversight, could result in issues such as reward hacking or other failures to align content with genuine human preferences. **Our actions**: As mentioned in our response to Reviewer FgZM, we will incorporate a discussion on societal impact and other relevant topics (e.g., model card, limitations of current models) into the paper. Additionally, we plan to conduct follow-up research on issues such as model bias and fairness, and adhere to established guidelines on the societal impact of human behavior and preference models. We will also enforce both internal and external guidelines for the use of this technique, such as preventing unauthorized access and misuse, and ensuring human oversight rather than relying solely on the model. These measures will help our technique realize its potential for positive societal impact while mitigating potential negative effects.