Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies

Self-supervised speech representations have been shown to be effective in a\nvariety of speech applications. However, existing representation learning\nmethods generally rely on the autoregressive model and/or observed global\ndependencies while generating the representation. In this work, we propose\nNon-Autoregressive Predictive Coding (NPC), a self-supervised method, to learn\na speech representation in a non-autoregressive manner by relying only on local\ndependencies of speech. NPC has a conceptually simple objective and can be\nimplemented easily with the introduced Masked Convolution Blocks. NPC offers a\nsignificant speedup for inference since it is parallelizable in time and has a\nfixed inference time for each time step regardless of the input sequence\nlength. We discuss and verify the effectiveness of NPC by theoretically and\nempirically comparing it with other methods. We show that the NPC\nrepresentation is comparable to other methods in speech experiments on phonetic\nand speaker classification while being more efficient.\n