The clustering of autonomous driving scenario data can substantially benefit\nthe autonomous driving validation and simulation systems by improving the\nsimulation tests' completeness and fidelity. This article proposes a\ncomprehensive data clustering framework for a large set of vehicle driving\ndata. Existing algorithms utilize handcrafted features whose quality relies on\nthe judgments of human experts. Additionally, the related feature compression\nmethods are not scalable for a large data-set. Our approach thoroughly\nconsiders the traffic elements, including both in-traffic agent objects and map\ninformation. Meanwhile, we proposed a self-supervised deep learning approach\nfor spatial and temporal feature extraction to avoid biased data\nrepresentation. With the newly designed driving data clustering evaluation\nmetrics based on data-augmentation, the accuracy assessment does not require a\nhuman-labeled data-set, which is subject to human bias. Via such unprejudiced\nevaluation metrics, we have shown our approach surpasses the existing methods\nthat rely on handcrafted feature extractions.\n