DEVICE AND METHOD TO LOCALIZE AND CONTROL A TOOL TIP WITH A ROBOT ARM

Described is a system for localizing and controlling a tool tip with a robotic arm. The system receives three-dimensional (3D) visual information of a scene having a tool with a tool tip proximate a target. A depth disparity map is generated based on the 3D visual information. The depth disparity map is filtered by depth continuity to generate boundaries of the tool. The boundaries of the tool are initially loose to ensure that the tool is contained within the boundaries. Thereafter, using level set segmentation, the boundaries of the tool are shrunk and extracted. Ends of the tool tip are identified based on the extracted boundaries, which are then used to generate a pose of the tool tip.