Using Large Language Models to Automate and Expedite Reinforcement Learning with Reward Machine

We present LARL-RM (Large language model-generated Automaton for Reinforcement Learning with Reward Machine) algorithm to encode high-level knowledge into reinforcement learning using automaton to expedite the reinforcement learning. Our method uses large language models (LLM) to obtain high-level domain-specific knowledge using prompt engineering instead of providing the reinforcement learning (RL) algorithm directly with the high-level knowledge that requires an expert to encode the automaton. We use chain-of-thought and few-shot methods for prompt engineering and demonstrate that our method works using these approaches. Additionally, LARL-RM allows for fully closed-loop reinforcement learning without the need for an expert to guide and supervise the learning since LARL-RM can use the LLM directly to generate the required high-level knowledge for the task at hand. Moreover, we demonstrate LARM-RM robustness to LLM hallucination and show the theoretical guarantee of our algorithm to converge to an optimal policy. We show that LARL-RM speeds up the convergence by implementing our method in two case studies and compare it to other RL methods.