Summary
This paper presents LIIPA, a set of methods for measuring the implicit portrayals of characters along the dimensions of appearance, intellect, and power. As a test bed, authors create a dataset of 2000 stories containing characters portrayed with various levels of intellect/power/appearance. LIIPA methods prompt LLMs to either generate the portrayal classification directly, or generate intermediate word lists describing the character, and then classifying the portrayal. Results show that the story level method outperforms the sentence-level methods. Additionally, authors conduct investigations into the effect on classification performance of adding demographic attributes into the stories, finding that most demographic identities see a decrease in performance.
Weaknesses
Despite liking the overall goals of the paper, I found that the paper had several strong weaknesses that prevent the paper to be publication-ready in my eyes.
1. While I appreciate the focus on implicit portrayal with the creation of a synthetic dataset, there is severe issue of external validity of the experiments.
1. In real narratives, characters are both implicitly and explicitly portrayed; creating a dataset where only implicit portrayal present needs to be better motivated.
2. Furthermore, the fact that experiments are only done on the synthetic data call into question the usefulness or validity of methods on real datasets. How would the LIIPA method perform on real world datasets? I wish authors had done (even a small-scale) investigation of this method.
2. Relatedly, part of the issue is that the synthetic data is only loosely validated.
1. While authors mention a human validation in the appendix (this should really be mentioned in the main body of the paper), they do not state the results of that validation. How many stories did not adhere to the templates?
2. L148: Machine generation is notoriously biased against certain demographic groups, as evidenced in the Lucy & Bamman paper authors keep citing. This directly contradicts the claim that synthetically generated stories have less representational biases.
3. Furthermore, given the biases in performance for "negative" portrayals (e.g., low appearance, low intellect, etc.), it seems feasible that the stories generated for these negative portrayals aren't actually that negative?
3. The discussion of prior work is very limited (partly evidenced by the very small amount of papers cited in the related work section of the paper).
1. There have been many works examining implicit portrayals of characters and biases in the portrayal of characters (https://www.aclweb.org/anthology/D17-1247, https://www.frontiersin.org/articles/10.3389/frai.2020.00055/full, https://aclanthology.org/P19-1243/, https://aclanthology.org/P18-1043/, https://dl.acm.org/doi/abs/10.1145/3359190, etc. ). While some operate at the lexical level, their analyses are aggregated at the character level over full stories or documents, making them useful at larger contexts.
2. L136: Relatedly, this is a mischaracterization of prior work. While some prior work only examined protagonists, many of the references above examine many characters in stories.
4. I wish the authors expanded more on why they chose the three dimensions of intellect, appearance, and power.
1. The paper glosses over why these dimensions are relevant to character portrayals, simply citing three previous works to back up that choice. But there is a wealth of humanities and social science literature that motivates why certain aspects of character portrayal dimensions matter / are salient that authors should explore and cite.
2. Relatedly, what does high vs. low appearance mean here? Authors should provide more details.
5. There are issues around the dataset comparison:
1. It feels unfair to compare the newly generated dataset to other existing resources since some of these are explicitly not meant to be diverse (e.g., ROC stories is meant to be short commonsense stories only, TinyStories is machine-generated stories for children), and artificially inflates the qualities of the dataset... I wish authors did a literature survey of open-source narrative/story datasets to make fairer comparisons (e.g., TorontoBookCorpus, FanFic corpus, and others).
2. L221: Authors argue that their dataset has a better length distribution than other datasets they compared to, but I don't see why that is true? There is no reason to believe 5-30 sentences is the "right length" for stories, and in fact, I'd argue many stories are much longer (e.g., books).
3. L228: I don't agree that antagonists and victims are "long tail" characters. Side characters would be long-tail characters, but antagonists are famously very prominent in stories.
4. I wish authors had spent less time explaining why their new dataset is "the best" and instead focused on expanded discussion of previous literature, conceptual choices, etc. (see my other weaknesses).
6. The experimental setup, choice of baseline, and comparisons made:
1. L269: For the sentence-level word lists (COMET or LIIPA), authors do not specify how they aggregate the word lists over sentences towards a full-story label.
2. Furthermore, there is a conceptual issue here which is that sentence-level models might be accurate about predicting the portrayal at the sentence, but that does not necessarily reflect the portrayal of the character at the full story level. This issue is also related to the drop in performance for story-level character portrayals, which exhibit more of a drop with increased nb of sentences.
7. Other missing details and issues:
1. Table 4: how did authors come up with the titles and genres?
2. L186: Narrative quality is much more than diversity. Authors should either add more measures of quality (e.g., narrative flow, plot amount, etc.) or just say they measure diversity without mentioning quality.
3. In Figure 3, it is not specified how authors extract the character roles for stories, making it hard to interpret the results faithfully.
4. L261: COMET is a commonsense engine powered by an LLM (specifically, an LLM finetuned to perform commonsense inference). Saying that LLMs would outperform COMET doesn't make much sense if you don't specify which LLM powers COMET. I would rather say that a finetuned LLM for commonsense inference might not be as flexible as an out-of-the-box prompted LLM to extract these portrayals?
5. L312: Authors should re-highlight that accuracy is computed based on the character attributes that were selected during generation.
6. L322-328: I appreciate the discussion of result differences for different dimensions and labels. However, authors should also note that many modern LLMs' safeguarding might explain the refusal to categorize appearance, and their RLHF-induced over-agreeableness might induce their better accuracy for positive ("high") labels.
7. L353: Authors mention correlations between performance and nb of characters or sentences. Authors should actually run these correlations and significance tests, because the slopes do not look all that different to me.
I hope the authors understand that I really do like this task, but there are many things that they could improve to make this paper more publication ready. Furthermore, given the subject of this work, I would actually suggest submitting to ACL Rolling Review to publish at NLP conferences, which tend to have more work around narrative analyses and would find a more excited audience there.