Neural operations as convolutions, self-attention, and vector aggregation are\nthe go-to choices for recognizing short-range actions. However, they have three\nlimitations in modeling long-range activities. This paper presents PIC,\nPermutation Invariant Convolution, a novel neural layer to model the temporal\nstructure of long-range activities. It has three desirable properties. i.\nUnlike standard convolution, PIC is invariant to the temporal permutations of\nfeatures within its receptive field, qualifying it to model the weak temporal\nstructures. ii. Different from vector aggregation, PIC respects local\nconnectivity, enabling it to learn long-range temporal abstractions using\ncascaded layers. iii. In contrast to self-attention, PIC uses shared weights,\nmaking it more capable of detecting the most discriminant visual evidence\nacross long and noisy videos. We study the three properties of PIC and\ndemonstrate its effectiveness in recognizing the long-range activities of\nCharades, Breakfast, and MultiThumos.\n