Multi-modal On-Device Learning for Monocular Depth Estimation on Ultra-low-power MCUs

Monocular depth estimation (MDE) plays a crucial role in enabling spatially aware applications in ultralow-power (ULP) Internet of Things (IoT) platforms. However, the limited number of parameters of deep neural networks (DNNs) for the MDE task, designed for IoT nodes, results in severe accuracy drops when the sensor data observed in the field shifts significantly from the training dataset. To address this domain shift problem, we present a multimodal on-device learning (ODL) technique, deployed on an IoT device integrating a Greenwaves GAP9 microcontroller unit (MCU), an 80-mW monocular camera, and an <inline-formula> <tex-math notation="LaTeX">$8 \times 8$ </tex-math></inline-formula> pixel depth sensor, consuming <inline-formula> <tex-math notation="LaTeX">$\sim {\mathrm {300~\text {m}\text {W} }}$ </tex-math></inline-formula>. In its normal operation, this setup feeds a tiny <inline-formula> <tex-math notation="LaTeX">$\mathrm {107~\text {k}{} }$ </tex-math></inline-formula>-parameter <inline-formula> <tex-math notation="LaTeX">$\mu $ </tex-math></inline-formula>PyD-Net model with monocular images for inference. The depth sensor, usually deactivated to minimize energy consumption, is only activated alongside the camera to collect pseudo-labels when the system is placed in a new environment. Then, the fine-tuning task is performed entirely on the MCU, using the new data. To optimize our backpropagation-based on-device training, we introduce a novel memory-driven sparse update scheme, which minimizes the fine-tuning memory to 1.2 MB, <inline-formula> <tex-math notation="LaTeX">$2.2\times $ </tex-math></inline-formula> less than a full update, while preserving accuracy (i.e., only 2% and 1.5% drops on the KITTI and NYUv2 datasets). Our in-field tests demonstrate, for the first time, that ODL for MDE can be performed in 17.8 min on the IoT node, reducing the root-mean-squared error from 4.9 to 0.6 m with only 3k self-labeled samples, collected in a real-life deployment scenario.

Paper

Similar papers

© 2026 NYSGPT2525 LLC