Math Education With Large Language Models: Peril or Promise?

The widespread availability of large language models (LLMs) has provoked both fear and excitement in education. On one hand, there is the concern that students will offload their coursework to LLMs, limiting what they themselves learn. On the other hand, there is the hope that LLMs might serve as scalable, personalized tutors. To investigate how LLM-based explanations affect learning, we conducted two large, pre-registered experiments with U.S. adults (N = 1,818). In both studies, participants first practiced math problems with different forms of assistance and then answered new but similar test questions without any help. Experiment 1 compared the effects of providing participants with only correct answers versus correct LLM explanations, revealing that LLM explanations improved learning, especially when participants first attempted problems on their own. Experiment 2 was similar, but included incorrect LLM explanations with arithmetic errors that produced incorrect final answers. Learning gains from these incorrect explanations were higher than from receiving only the correct answer but lower than from receiving correct explanations. These results suggest that LLM explanations can foster learning gains, although their effectiveness depends not only on the capabilities of the underlying models, but also on how the technology is designed and used to support productive educational experiences.