Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction

With the recent success and popularity of pre-trained language models (LMs)\nin natural language processing, there has been a rise in efforts to understand\ntheir inner workings. In line with such interest, we propose a novel method\nthat assists us in investigating the extent to which pre-trained LMs capture\nthe syntactic notion of constituency. Our method provides an effective way of\nextracting constituency trees from the pre-trained LMs without training. In\naddition, we report intriguing findings in the induced trees, including the\nfact that pre-trained LMs outperform other approaches in correctly demarcating\nadverb phrases in sentences.\n