The success of transformer networks in the natural language processing and 2D vision domains has encouraged the adaptation of transformers for 3D computer vision tasks. However, majority of the existing approaches employ standard back-propagation (SBP) - which requires the storage of model activations on a forward pass for use during the backward pass - making their memory complexity linearly proportional to model depth, hence inefficient. Furthermore, most 3D point transformers use the classic QK(V) matrix multiplication design which comes with a memory bottleneck. To address these issues, we propose a memory-efficient point transformer that makes use of reversible functions and linearized self-attention to minimize SBP and transformer memory complexities, respectively. Experimental results on benchmark datasets (Toronto3D and CSPC) from different sensor platforms (aerial, and mobile backpack) show that our approach uses less than half the number of model parameters (compared to its SBP counterpart), take more than twice the input sequence, and use less than half the memory compared to majority of the traditional approach. Overall, the proposed RPT attained competitive performance compared to the state-of-the-art.
Paper
Full text
A Reversible Transformer for LiDAR Point Cloud Semantic Segmentation
Semantic Scholar · Computer Science · 2023
Abstract
The success of transformer networks in the natural language processing and 2D vision domains has encouraged the adaptation of transformers for 3D computer vision tasks. However, majority of the existing approaches employ standard back-propagation (SBP) - which requires the storage of model activations on a forward pass for use during the backward pass - making their memory complexity linearly proportional to model depth, hence inefficient. Furthermore, most 3D point transformers use the classic QK(V) matrix multiplication design which comes with a memory bottleneck. To address these issues, we propose a memory-efficient point transformer that makes use of reversible functions and linearized self-attention to minimize SBP and transformer memory complexities, respectively. Experimental results on benchmark datasets (Toronto3D and CSPC) from different sensor platforms (aerial, and mobile backpack) show that our approach uses less than half the number of model parameters (compared to its SBP counterpart), take more than twice the input sequence, and use less than half the memory compared to majority of the traditional approach. Overall, the proposed RPT attained competitive performance compared to the state-of-the-art.