In this paper, we systematically assess the ability of standard recurrent networks to perform dynamic counting and to encode hierarchical representations. All the neural models in our experiments are designed to be smallsized networks both to prevent them from memorizing the training sets and to visualize and interpret their behaviour at test time.