Summary
The authors present an investigation of the phenomenon of ‘sound symbolism’ in a pair of cutting edge Vision-and-Language Models. Sound symbolism is an intriguing phenomenon whereby the meaning of a word can be in part traced back to the way the word sounds. It is an important phenomenon in psychology and linguistics because it challenges a strict view of the mapping between form and meaning as arbitrary (stated most famously by Saussure). The authors probe CLIP and Stable Diffusion by creating a small dataset of pseudowords that were constructed to reflect phonetic features that generally map onto “sharp” vs. “round” speech sound categories. They project the embedding vectors in 1024-dimensional CLIP space onto a one-dimensional semantic dimension of interest, defined by two sets of antonym shape adjectives (one set corresponding to synonyms of ‘round’ and the other corresponding to synonyms of ‘sharp’). They also define a phonetic score to measure phonetic/graphemic associations of the shape adjectives with the pseudowords. They analyze the models using these geometric measures, as well as with a human evaluation where participants are asked to select which of two Stable Diffusion-generated images is best described by a pseudoword. Across these evaluations, they find relatively strong evidence for the presence of sound symbolism in these models.
Strengths
One of the strengths of this paper is that it demonstrates how ideas from Cogntive Science can drive investigations into modern AI models. The phenomenon of sound symbolism is particularly interesting both for its intuitive appeal and its relation to foundational ideas in Cognitive Science. The paper was well written and largely a pleasure to read. The methods were overall well described and I found the evidence relatively convincing.
Weaknesses
The weakest part of the paper for me were the motivations, hypotheses, and implications. Yes, Vision&Language Models are increasingly powerful and increasingly deployed, but why are they interesting objects of study for Cognitive Science? We are told that “these methods could provide new insights into the classic questions of what aspects of sound are tied to meaning”, but not explained how this might happen or what this could look like. As well, the conclusion about “cultural universality” seems like a very difficult and non-obvious question that these methods would provide insight into.
And the flip-side, why is sound symbolism an important phenomenon to investigate in these powerful and deployable models? One specific weakness is that there were no hypotheses presented for the models, and you could imagine developing the hypotheses based on how these models were trained. I think there’s a lot to say about this, but all we got one sentence buried in the results section that mentioned that the models primarily saw valid English text during training and didn’t have access to the actual sound of the words.
The second major weakness for me was the description and analysis of the User Study. The authors write “Over two hundred volunteers participated in this study”. The use of the word “volunteers” here suggests that the participants were not paid, which I hope is not the case. Who are these participants? How were they recruited? Was there any ethical review of this study? How many participants exactly were recruited (this is in the supplement, but should be presented in the main text)?
On the analysis side, the use of a binomial test is not appropriate as the individual data points are not i.i.d. Participants gave multiple responses and, presumably, multiple items (pairs of images) were rated by multiple participants. Thus, something akin to a mixed-effects (multi-level / hierarchical) logistic regression model would be more appropriate to account for the participant- and item-wise variability, and provide a much more realistic analyses of the results (see Gelman & Hill, 2006 for background).
I found the Phonetic Scores in 3.2 the most difficult to understand and still am not completely sure I get it. From the math, I would have thought this score was something akin to a control measurement, and a way to contextualize the Geometric scores, but the authors seem to have a different perspective, which I didn’t fully get.
Finally, Figures 2 & 3 took up a lot of space for not that much information. They could easily be condensed to make room for some of the context in the response.
Questions
My questions are the same as the weaknesses. I’m not sure all of questions 1-3 need to be answered, but at least 2 of them do.
1. How can these results inform theories in Cognitive Science?
2. What hypotheses can be derived from these models a priori about sound symbolism, presumably by thinking about their training data?
3. Why is sound symbolism important for evaluating sophisticated, deployable AI systems?
4. Can you unpack the Phonetic Scores in a little more detail, and explain their relevance?
5. Please provide more details about the User Study and analyze the results in a more appropriate manner.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The limitations were okay but could be improved by thinking more about the societal impact of these (human) biases getting incorporated into deployable generative models. I would bet some of the impacts of sound symbolism have been discussing in the psychological literature, and it would be important to mention them and discuss them in the context of these things getting baked into generative AI.