Action segmentation refers to inferring boundaries of semantically consistent\nvisual concepts in videos and is an important requirement for many video\nunderstanding tasks. For this and other video understanding tasks, supervised\napproaches have achieved encouraging performance but require a high volume of\ndetailed frame-level annotations. We present a fully automatic and unsupervised\napproach for segmenting actions in a video that does not require any training.\nOur proposal is an effective temporally-weighted hierarchical clustering\nalgorithm that can group semantically consistent frames of the video. Our main\nfinding is that representing a video with a 1-nearest neighbor graph by taking\ninto account the time progression is sufficient to form semantically and\ntemporally consistent clusters of frames where each cluster may represent some\naction in the video. Additionally, we establish strong unsupervised baselines\nfor action segmentation and show significant performance improvements over\npublished unsupervised methods on five challenging action segmentation\ndatasets. Our code is available at\nhttps://github.com/ssarfraz/FINCH-Clustering/tree/master/TW-FINCH\n