Human Action Detection Based on Multimodal Feature Fusion for Human-Robot Collaborative Assembly

With the development of human-robot collaboration technology, action detection, which can detect human actions during the assembly process and improve the fluency of collaboration, has a significant value in industrial assembly tasks. However, in practical application scenarios, the effect of action detection is affected by the low action difference of human-robot collaborative assembly tasks, single data modality, etc. In order to effectively solve this problem, a human-robot collaborative assembly framework based on action detection is proposed, in which we propose an action detection method Multi-Ad proposed with higher accuracy and generalization capability. Multi-Ad adopts the method of fusing multimodal features of RGB, optical flow, and skeleton sequences to enhance the extracted data information, which can improve the accuracy of action detection. Experimental results on the thumos14 dataset show that the proposed method is better than previous methods in terms of action detection accuracy.