It has been well recognized that fusing the complementary information from\ndepth-aware LiDAR point clouds and semantic-rich stereo images would benefit 3D\nobject detection. Nevertheless, it is not trivial to explore the inherently\nunnatural interaction between sparse 3D points and dense 2D pixels. To ease\nthis difficulty, the recent proposals generally project the 3D points onto the\n2D image plane to sample the image data and then aggregate the data at the\npoints. However, this approach often suffers from the mismatch between the\nresolution of point clouds and RGB images, leading to sub-optimal performance.\nSpecifically, taking the sparse points as the multi-modal data aggregation\nlocations causes severe information loss for high-resolution images, which in\nturn undermines the effectiveness of multi-sensor fusion. In this paper, we\npresent VPFNet -- a new architecture that cleverly aligns and aggregates the\npoint cloud and image data at the `virtual' points. Particularly, with their\ndensity lying between that of the 3D points and 2D pixels, the virtual points\ncan nicely bridge the resolution gap between the two sensors, and thus preserve\nmore information for processing. Moreover, we also investigate the data\naugmentation techniques that can be applied to both point clouds and RGB\nimages, as the data augmentation has made non-negligible contribution towards\n3D object detectors to date. We have conducted extensive experiments on KITTI\ndataset, and have observed good performance compared to the state-of-the-art\nmethods. Remarkably, our VPFNet achieves 83.21\\% moderate 3D AP and 91.86\\%\nmoderate BEV AP on the KITTI test set, ranking the 1st since May 21th, 2021.\nThe network design also takes computation efficiency into consideration -- we\ncan achieve a FPS of 15 on a single NVIDIA RTX 2080Ti GPU. The code will be\nmade available for reproduction and further investigation.\n