Visual data collected from Unmanned Aerial Vehicles (UAVs) has opened a new\nfrontier of computer vision that requires automated analysis of aerial\nimages/videos. However, the existing UAV datasets primarily focus on object\ndetection. An object detector does not differentiate between the moving and\nnon-moving objects. Given a real-time UAV video stream, how can we both\nlocalize and classify the moving objects, i.e. perform moving object\nrecognition (MOR)? The MOR is one of the essential tasks to support various UAV\nvision-based applications including aerial surveillance, search and rescue,\nevent recognition, urban and rural scene understanding.To the best of our\nknowledge, no labeled dataset is available for MOR evaluation in UAV videos.\nTherefore, in this paper, we introduce MOR-UAV, a large-scale video dataset for\nMOR in aerial videos. We achieve this by labeling axis-aligned bounding boxes\nfor moving objects which requires less computational resources than producing\npixel-level estimates. We annotate 89,783 moving object instances collected\nfrom 30 UAV videos, consisting of 10,948 frames in various scenarios such as\nweather conditions, occlusion, changing flying altitude and multiple camera\nviews. We assigned the labels for two categories of vehicles (car and heavy\nvehicle). Furthermore, we propose a deep unified framework MOR-UAVNet for MOR\nin UAV videos. Since, this is a first attempt for MOR in UAV videos, we present\n16 baseline results based on the proposed framework over the MOR-UAV dataset\nthrough quantitative and qualitative experiments. We also analyze the\nmotion-salient regions in the network through multiple layer visualizations.\nThe MOR-UAVNet works online at inference as it requires only few past frames.\nMoreover, it doesn't require predefined target initialization from user.\nExperiments also demonstrate that the MOR-UAV dataset is quite challenging.\n