Relational-Grid-World: A Novel Relational Reasoning Environment and An Agent Model for Relational Information Extraction

Reinforcement learning (RL) agents are often designed specifically for a\nparticular problem and they generally have uninterpretable working processes.\nStatistical methods-based agent algorithms can be improved in terms of\ngeneralizability and interpretability using symbolic Artificial Intelligence\n(AI) tools such as logic programming. In this study, we present a model-free RL\narchitecture that is supported with explicit relational representations of the\nenvironmental objects. For the first time, we use the PrediNet network\narchitecture in a dynamic decision-making problem rather than image-based\ntasks, and Multi-Head Dot-Product Attention Network (MHDPA) as a baseline for\nperformance comparisons. We tested two networks in two environments ---i.e.,\nthe baseline Box-World environment and our novel environment,\nRelational-Grid-World (RGW). With the procedurally generated RGW environment,\nwhich is complex in terms of visual perceptions and combinatorial selections,\nit is easy to measure the relational representation performance of the RL\nagents. The experiments were carried out using different configurations of the\nenvironment so that the presented module and the environment were compared with\nthe baselines. We reached similar policy optimization performance results with\nthe PrediNet architecture and MHDPA; additionally, we achieved to extract the\npropositional representation explicitly ---which makes the agent's statistical\npolicy logic more interpretable and tractable. This flexibility in the agent's\npolicy provides convenience for designing non-task-specific agent\narchitectures. The main contributions of this study are two-fold ---an RL agent\nthat can explicitly perform relational reasoning, and a new environment that\nmeasures the relational reasoning capabilities of RL agents.\n