Total autonomy will be 100% accident free by testing more than 8 Billion miles of sensor data*. However, testing the software in real driving situations on the road would be extremely time consuming, expensive, cumbersome. Numerous improvements of the algorithms are necessary to perfect the control software for autonomous driving. To train and test the algorithms we need sensor data. Here the challenge is to store and manage petabytes of sensor data across the global locations. This paper describes an end to end architecture and design of global data ingest, data processing and accessing sensor data effectively. It gives a scalable and reliable solution that is capable to handle hundreds of petabytes using hundreds of computer cores. Proposed design helps location-independent data Ingest, faster data availability, On-Demand usage of data effectively and recovery from accidental deletion of data. It helps to store exabytes of data in a tiered storage model in different categorizations. This architecture will reduce it to within 3 to 5 days for further processing of data.
Paper
Full text
Sensor data management for Autonomous Driving
Semantic Scholar · Engineering · 2019
Abstract
Total autonomy will be 100% accident free by testing more than 8 Billion miles of sensor data*. However, testing the software in real driving situations on the road would be extremely time consuming, expensive, cumbersome. Numerous improvements of the algorithms are necessary to perfect the control software for autonomous driving. To train and test the algorithms we need sensor data. Here the challenge is to store and manage petabytes of sensor data across the global locations. This paper describes an end to end architecture and design of global data ingest, data processing and accessing sensor data effectively. It gives a scalable and reliable solution that is capable to handle hundreds of petabytes using hundreds of computer cores. Proposed design helps location-independent data Ingest, faster data availability, On-Demand usage of data effectively and recovery from accidental deletion of data. It helps to store exabytes of data in a tiered storage model in different categorizations. This architecture will reduce it to within 3 to 5 days for further processing of data.