Anomaly detection is a key goal of autonomous surveillance systems that\nshould be able to alert unusual observations. In this paper, we propose a\nholistic anomaly detection system using deep neural networks for surveillance\nof critical infrastructures (e.g., airports, harbors, warehouses) using an\nunmanned aerial vehicle (UAV). First, we present a heuristic method for the\nexplicit representation of spatial layouts of objects in bird-view images.\nThen, we propose a deep neural network architecture for unsupervised anomaly\ndetection (UAV-AdNet), which is trained on environment representations and GPS\nlabels of bird-view images jointly. Unlike studies in the literature, we\ncombine GPS and image data to predict abnormal observations. We evaluate our\nmodel against several baselines on our aerial surveillance dataset and show\nthat it performs better in scene reconstruction and several anomaly detection\ntasks. The codes, trained models, dataset, and video will be available at\nhttps://bozcani.github.io/uavadnet.\n