Large Language Models Need Consultants for Reasoning: Becoming an Expert in a Complex Human System Through Behavior Simulation

Large language models (LLMs), in conjunction with various reasoning reinforcement methodologies, have demonstrated remarkable capabilities comparable to humans in fields such as mathematics, law, coding, common sense, and world knowledge. In this paper, we improve the reasoning abilities of LLMs within complex human systems. We propose a novel reasoning framework, termed “Mosaic Expert Observation Wall” (MEOW) exploiting generative-agent-based simulation technique. In the MEOW framework, simulated data are utilized to train an expert model learning “experience” about a specific task in each independent time of simulation. It is the accumulated “experience” through the simulation that makes for an expert on a task in a complex human system. We conduct the experiments within two communication games that represent two kinds of complex human systems. The results indicate that our proposed methodology can cooperate with existing methodologies to enhance the reasoning abilities of LLMs in complex human systems.