VisionTraj: A Noise-Robust Trajectory Recovery Framework Based on Large-Scale Camera Network

Trajectory recovery from snapshots captured by a city-wide multi-camera network facilitates urban mobility sensing and road network optimization. State-of-the-art solutions for such vision-based schemes typically rely on predefined rules or unsupervised iterative feedback, but they struggle with multiple challenges, such as the lack of open-source datasets for training the entire pipeline and the vulnerability to noise in visual inputs. In response to the dilemma, this paper proposes VisionTraj, the first learning-based model that reconstructs vehicle trajectories from snapshots recorded by road network cameras. Along with this, we present two well-designed vision-trajectory datasets that provide extensive trajectory data and corresponding visual snapshots, enabling the extraction of supervised vision-trajectory interactions. After the data creation, based on the results from the off-the-shelf multi-modal vehicle clustering, we first re-formulate the trajectory recovery problem as a generative task and introduce the canonical Transformer as the autoregressive backbone. Next, to identify clustering noise (i.e., false positives) based on the snapshots’ spatiotemporal dependencies, a graph convolutional neural network-based soft-denoising module is built upon the fine- and coarse-grained clusters. Additionally, we leverage strong semantic information extracted from the tracklet to provide detailed insights into the vehicle’s entry and exit behaviors during trajectory recovery. The denoising and tracklet components can also serve as plug-and-play modules to enhance baselines. Experimental results on the two hand-crafted datasets show that the proposed VisionTraj achieves a maximum improvement of +11.5% against the sub-best model. Furthermore, we explore potential downstream applications, and our model continues to outperform its peers. The code and data are available here <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/bonaldli/VisionTraj</uri>