Integrating YOLO Object Detection with Augmented Reality for iOS Apps

Abstract

This paper proposes an implementation for detecting the identity and location of objects in real-time video preview and overlaying 3D graphics on them in iOS Apps. Object Detection is a technology that detects the identity (what) and location (where) of a variable number of objects in an Image. YOLO: You Only Look Once, is a real-time object detection system. TinyYOLO (YOLOv2 with a Darknet base network) is an implementation often used for mobile applications, which is much faster than YOLO, but less accurate [1]. An MLModel [2] was built using TinyYOLO that could detect the identity and location of the pre-trained objects in an image. Every frame of a video preview was passed as an input to the model. If the model detected an object, the 2D coordinates of the object in the image were recorded. The 2D coordinates were converted to 3D coordinates using the ARKit HitTest API [3]. After the 3D Coordinates were retrieved, ARKit was used to overlay a 3D graphics at the object’s location. Integrating Object Detection and Augmented Reality helped locate the 3D position of objects in the environment and overlay graphics on them.