Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge

Prior research has explored the ability of computational models to predict a\nword semantic fit with a given predicate. While much work has been devoted to\nmodeling the typicality relation between verbs and arguments in isolation, in\nthis paper we take a broader perspective by assessing whether and to what\nextent computational approaches have access to the information about the\ntypicality of entire events and situations described in language (Generalized\nEvent Knowledge). Given the recent success of Transformers Language Models\n(TLMs), we decided to test them on a benchmark for the \\textit{dynamic\nestimation of thematic fit}. The evaluation of these models was performed in\ncomparison with SDM, a framework specifically designed to integrate events in\nsentence meaning representations, and we conducted a detailed error analysis to\ninvestigate which factors affect their behavior. Our results show that TLMs can\nreach performances that are comparable to those achieved by SDM. However,\nadditional analysis consistently suggests that TLMs do not capture important\naspects of event knowledge, and their predictions often depend on surface\nlinguistic features, such as frequent words, collocations and syntactic\npatterns, thereby showing sub-optimal generalization abilities.\n

Paper

Similar papers

© 2026 NYSGPT2525 LLC