Learning to Drive Off Road on Smooth Terrain in Unstructured Environments Using an On-Board Camera and Sparse Aerial Images

We present a method for learning to drive on smooth terrain while\nsimultaneously avoiding collisions in challenging off-road and unstructured\noutdoor environments using only visual inputs. Our approach applies a hybrid\nmodel-based and model-free reinforcement learning method that is entirely\nself-supervised in labeling terrain roughness and collisions using on-board\nsensors. Notably, we provide both first-person and overhead aerial image inputs\nto our model. We find that the fusion of these complementary inputs improves\nplanning foresight and makes the model robust to visual obstructions. Our\nresults show the ability to generalize to environments with plentiful\nvegetation, various types of rock, and sandy trails. During evaluation, our\npolicy attained 90% smooth terrain traversal and reduced the proportion of\nrough terrain driven over by 6.1 times compared to a model using only\nfirst-person imagery.\n