Medical image analysis using deep learning has recently been prevalent,\nshowing great performance for various downstream tasks including medical image\nsegmentation and its sibling, volumetric image segmentation. Particularly, a\ntypical volumetric segmentation network strongly relies on a voxel grid\nrepresentation which treats volumetric data as a stack of individual voxel\n`slices', which allows learning to segment a voxel grid to be as\nstraightforward as extending existing image-based segmentation networks to the\n3D domain. However, using a voxel grid representation requires a large memory\nfootprint, expensive test-time and limiting the scalability of the solutions.\nIn this paper, we propose Point-Unet, a novel method that incorporates the\nefficiency of deep learning with 3D point clouds into volumetric segmentation.\nOur key idea is to first predict the regions of interest in the volume by\nlearning an attentional probability map, which is then used for sampling the\nvolume into a sparse point cloud that is subsequently segmented using a\npoint-based neural network. We have conducted the experiments on the medical\nvolumetric segmentation task with both a small-scale dataset Pancreas and\nlarge-scale datasets BraTS18, BraTS19, and BraTS20 challenges. A comprehensive\nbenchmark on different metrics has shown that our context-aware Point-Unet\nrobustly outperforms the SOTA voxel-based networks at both accuracies, memory\nusage during training, and time consumption during testing. Our code is\navailable at https://github.com/VinAIResearch/Point-Unet.\n