TouchAI: Exploring human-AI perceptual alignment in touch through language model representations

Aligning large language models (LLMs) behaviour with human intent is critical for future AI. An important yet often overlooked aspect of this alignment is the perceptual alignment. Perceptual modalities like touch are more multifaceted and nuanced compared to other sensory modalities such as vision. This study investigates how well LLMs can understand and interpret human touch experiences by focusing on their capacity to perceive the tactile qualities of everyday objects. For instance, it assesses whether LLMs can recognize that silk satin is softer and smoother than cotton denim. We developed a “Guess What Textile“ interaction using a custom AI system that enables participants to narrate their touch experiences in the “textile hand” task. Participants were given two textile samples–a target and a reference–to handle. Without seeing them, participants described the differences between them to the LLM. Using these descriptions, the LLM attempted to identify the target textile by assessing similarity within its high-dimensional embedding space, where its perceptual representations are encoded. Our results suggest that a degree of perceptual alignment exists; however, it varies significantly among different textile samples. For example, LLM predictions are well aligned for silk satin, but not for cotton denim. Moreover, participants felt that their textile experiences were not closely matched by the LLM predictions. This study is the first exploration into perceptual alignment around touch using LLM encoders, exemplified through textile hand task. We discuss possible sources of this alignment variance, and how better human-AI perceptual alignment can benefit future everyday tasks. • We address the gap in understanding perception alignment between human touch and AI. • First study on alignment between human touch experiences and LLMs in embeddings. • A novel interactive task probes LLMs’ learned representations for human alignment. • LLMs show perceptual biases, aligning better with certain textiles than others.