Multimodal neural networks better explain multivoxel patterns in the hippocampus

The human hippocampus possesses "concept cells", neurons that fire when\npresented with stimuli belonging to a specific concept, regardless of the\nmodality. Recently, similar concept cells were discovered in a multimodal\nnetwork called CLIP (Radford et at., 2021). Here, we ask whether CLIP can\nexplain the fMRI activity of the human hippocampus better than a purely visual\n(or linguistic) model. We extend our analysis to a range of publicly available\nuni- and multi-modal models. We demonstrate that "multimodality" stands out as\na key component when assessing the ability of a network to explain the\nmultivoxel activity in the hippocampus.\n