Recursive processing is considered a hallmark of human linguistic abilities.\nA recent study evaluated recursive processing in recurrent neural language\nmodels (RNN-LMs) and showed that such models perform below chance level on\nembedded dependencies within nested constructions -- a prototypical example of\nrecursion in natural language. Here, we study if state-of-the-art Transformer\nLMs do any better. We test four different Transformer LMs on two different\ntypes of nested constructions, which differ in whether the embedded (inner)\ndependency is short or long range. We find that Transformers achieve\nnear-perfect performance on short-range embedded dependencies, significantly\nbetter than previous results reported for RNN-LMs and humans. However, on\nlong-range embedded dependencies, Transformers' performance sharply drops below\nchance level. Remarkably, the addition of only three words to the embedded\ndependency caused Transformers to fall from near-perfect to below-chance\nperformance. Taken together, our results reveal Transformers' shortcoming when\nit comes to recursive, structure-based, processing.\n