Automatic detection of phoneme or word-like units is one of the core\nobjectives in zero-resource speech processing. Recent attempts employ\nself-supervised training methods, such as contrastive predictive coding (CPC),\nwhere the next frame is predicted given past context. However, CPC only looks\nat the audio signal's frame-level structure. We overcome this limitation with a\nsegmental contrastive predictive coding (SCPC) framework that can model the\nsignal structure at a higher level e.g. at the phoneme level. In this\nframework, a convolutional neural network learns frame-level representation\nfrom the raw waveform via noise-contrastive estimation (NCE). A differentiable\nboundary detector finds variable-length segments, which are then used to\noptimize a segment encoder via NCE to learn segment representations. The\ndifferentiable boundary detector allows us to train frame-level and\nsegment-level encoders jointly. Typically, phoneme and word segmentation are\ntreated as separate tasks. We unify them and experimentally show that our\nsingle model outperforms existing phoneme and word segmentation methods on\nTIMIT and Buckeye datasets. We analyze the impact of boundary threshold and\nwhen is the right time to include the segmental loss in the learning process.\n