We introduce a method to determine if two videos are of the same event even when they are taken from significantly different viewing directions. Based on this, a video can be retrieved from a database of events, and video collections can be clustered into sets of videos each viewing the same event. Our method works well even in cases where the videos are so different that appearance based methods (e.g. SIFT) are not sufficient. This viewpoint invariance is based on a new pixelbased feature, “motion barcode”, which records the existence/non-existence of motion as a function of time. While appearance, motion magnitude, and motion direction can vary between viewpoints, the existence of motion is viewpoint invariant. Based on the motion barcode, a similarity measure is developed for videos from very different viewpoints. This measure is robust to occlusions common under different viewpoints and can be computed efficiently We show the utility of motion barcodes using challenging sequences from stationary and hand held cameras.