Multi-Element Long Distance Dependencies: Using SPk Languages to Explore the Characteristics of Long-Distance Dependencies

In order to successfully model Long Distance Dependencies (LDDs) it is\nnecessary to understand the full-range of the characteristics of the LDDs\nexhibited in a target dataset. In this paper, we use Strictly k-Piecewise\nlanguages to generate datasets with various properties. We then compute the\ncharacteristics of the LDDs in these datasets using mutual information and\nanalyze the impact of factors such as (i) k, (ii) length of LDDs, (iii)\nvocabulary size, (iv) forbidden subsequences, and (v) dataset size. This\nanalysis reveal that the number of interacting elements in a dependency is an\nimportant characteristic of LDDs. This leads us to the challenge of modelling\nmulti-element long-distance dependencies. Our results suggest that attention\nmechanisms in neural networks may aide in modeling datasets with multi-element\nlong-distance dependencies. However, we conclude that there is a need to\ndevelop more efficient attention mechanisms to address this issue.\n