GraspLook: a VR-based Telemanipulation System with R-CNN-driven Augmentation of Virtual Environment

The teleoperation of robotic systems in medical applications requires stable\nand convenient visual feedback for the operator. The most accessible approach\nto delivering visual information from the remote area is using cameras to\ntransmit a video stream from the environment. However, such systems are\nsensitive to the camera resolution, limited viewpoints, and cluttered\nenvironment bringing additional mental demands to the human operator. The paper\nproposes a novel system of teleoperation based on an augmented virtual\nenvironment (VE). The region-based convolutional neural network (R-CNN) is\napplied to detect the laboratory instrument and estimate its position in the\nremote environment to display further its digital twin in the VE, which is\nnecessary for dexterous telemanipulation. The experimental results revealed\nthat the developed system allows users to operate the robot smoother, which\nleads to a decrease in task execution time when manipulating test tubes. In\naddition, the participants evaluated the developed system as less mentally\ndemanding (by 11%) and requiring less effort (by 16%) to accomplish the task\nthan the camera-based teleoperation approach and highly assessed their\nperformance in the augmented VE. The proposed technology can be potentially\napplied for conducting laboratory tests in remote areas when operating with\ninfectious and poisonous reagents.\n