Vision-Guided Forecasting -- Visual Context for Multi-Horizon Time Series Forecasting

Autonomous driving gained huge traction in recent years, due to its potential\nto change the way we commute. Much effort has been put into trying to estimate\nthe state of a vehicle. Meanwhile, learning to forecast the state of a vehicle\nahead introduces new capabilities, such as predicting dangerous situations.\nMoreover, forecasting brings new supervision opportunities by learning to\npredict richer a context, expressed by multiple horizons. Intuitively, a video\nstream originated from a front-facing camera is necessary because it encodes\ninformation about the upcoming road. Besides, historical traces of the\nvehicle's states give more context. In this paper, we tackle multi-horizon\nforecasting of vehicle states by fusing the two modalities. We design and\nexperiment with 3 end-to-end architectures that exploit 3D convolutions for\nvisual features extraction and 1D convolutions for features extraction from\nspeed and steering angle traces. To demonstrate the effectiveness of our\nmethod, we perform extensive experiments on two publicly available real-world\ndatasets, Comma2k19 and the Udacity challenge. We show that we are able to\nforecast a vehicle's state to various horizons, while outperforming the current\nstate-of-the-art results on the related task of driving state estimation. We\nexamine the contribution of vision features, and find that a model fed with\nvision features achieves an error that is 56.6% and 66.9% of the error of a\nmodel that doesn't use those features, on the Udacity and Comma2k19 datasets\nrespectively.\n

Paper

Similar papers

© 2026 NYSGPT2525 LLC