Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning

The goal of neural-symbolic computation is to integrate the connectionist and\nsymbolist paradigms. Prior methods learn the neural-symbolic models using\nreinforcement learning (RL) approaches, which ignore the error propagation in\nthe symbolic reasoning module and thus converge slowly with sparse rewards. In\nthis paper, we address these issues and close the loop of neural-symbolic\nlearning by (1) introducing the \\textbf{grammar} model as a \\textit{symbolic\nprior} to bridge neural perception and symbolic reasoning, and (2) proposing a\nnovel \\textbf{back-search} algorithm which mimics the top-down human-like\nlearning procedure to propagate the error through the symbolic reasoning module\nefficiently. We further interpret the proposed learning framework as maximum\nlikelihood estimation using Markov chain Monte Carlo sampling and the\nback-search algorithm as a Metropolis-Hastings sampler. The experiments are\nconducted on two weakly-supervised neural-symbolic tasks: (1) handwritten\nformula recognition on the newly introduced HWF dataset; (2) visual question\nanswering on the CLEVR dataset. The results show that our approach\nsignificantly outperforms the RL methods in terms of performance, converging\nspeed, and data efficiency. Our code and data are released at\n\\url{https://liqing-ustc.github.io/NGS}.\n