Where is my hand? Deep hand segmentation for visual self-recognition in humanoid robots

The ability to distinguish between the self and the background is of\nparamount importance for robotic tasks. The particular case of hands, as the\nend effectors of a robotic system that more often enter into contact with other\nelements of the environment, must be perceived and tracked with precision to\nexecute the intended tasks with dexterity and without colliding with obstacles.\nThey are fundamental for several applications, from Human-Robot Interaction\ntasks to object manipulation. Modern humanoid robots are characterized by high\nnumber of degrees of freedom which makes their forward kinematics models very\nsensitive to uncertainty. Thus, resorting to vision sensing can be the only\nsolution to endow these robots with a good perception of the self, being able\nto localize their body parts with precision. In this paper, we propose the use\nof a Convolution Neural Network (CNN) to segment the robot hand from an image\nin an egocentric view. It is known that CNNs require a huge amount of data to\nbe trained. To overcome the challenge of labeling real-world images, we propose\nthe use of simulated datasets exploiting domain randomization techniques. We\nfine-tuned the Mask-RCNN network for the specific task of segmenting the hand\nof the humanoid robot Vizzy. We focus our attention on developing a methodology\nthat requires low amounts of data to achieve reasonable performance while\ngiving detailed insight on how to properly generate variability in the training\ndataset. Moreover, we analyze the fine-tuning process within the complex model\nof Mask-RCNN, understanding which weights should be transferred to the new task\nof segmenting robot hands. Our final model was trained solely on synthetic\nimages and achieves an average IoU of 82% on synthetic validation data and\n56.3% on real test data. These results were achieved with only 1000 training\nimages and 3 hours of training time using a single GPU.\n

Paper

Similar papers

© 2026 NYSGPT2525 LLC