Depth-Enhanced 3D Deep Learning for Strawberry Detection and Widest Region Identification in Polytunnels

Abstract

: This paper presents an investigation into the use of 3D Deep Learning models for enhanced strawberry detection in polytunnels. We focus on two main tasks: firstly, fruit detection, comparing the standard MaskRCNN and an adapted version that integrates depth information (MaskRCNN-D), both capable of classifying straw-berries based on their maturity (ripe, unripe) and health status (affected by disease or fungus); secondly, for the identification of the widest region of strawberries, we compare a contour-based algorithm with an enhanced version of the VGG-16 model. Our findings demonstrate that integrating depth data into the MaskRCNN-D results in up to a 13.7% improvement in mean Average Precision (mAP) from 0.81 to 0.92 across various strawberry test sets, including simulated ones, emphasizing the model’s effectiveness in both real-world and simulated agricultural scenarios. Furthermore, our end-to-end pipeline approach, which combines the fruit detection (MaskRCNN-D) and widest region identification models (enhanced VGG-16), shows a remarkably low localization error, achieving down to 11.3 pixels of Root Mean Square Error (RMSE) in a 224 × 224 strawberry cropped image. This pipeline integration, combining the strengths of both models, provides the most effective result, enabling their application in autonomous fruit monitoring systems.