The reconstruction of the 3D permittivity map from ground-penetrating radar (GPR) data is of great importance for mapping subsurface environments and inspecting underground structural integrity. Traditional iterative 3D reconstruction algorithms suffer from strong nonlinearity, ill-posedness, and high computational costs. To tackle these issues, a 3D deep learning scheme, called 3DInvNet, is proposed to reconstruct 3D permittivity maps from GPR C-scans. The proposed scheme leverages a prior 3D convolutional neural network (CNN) with a feature attention mechanism to suppress the noise in the C-scans due to subsurface heterogeneous soil environments. Then a 3D U-shaped encoder–decoder network with multiscale feature aggregation (MSFA) modules is designed to establish the optimal inverse mapping from the denoised C-scans to 3D permittivity maps. Furthermore, a three-step separate learning strategy is employed to pretrain and fine-tune the networks. The proposed scheme is applied to numerical simulation as well as real measurement data. The quantitative and qualitative results show the networks’ capability, generalizability, and robustness in denoising GPR C-scans and reconstructing 3D permittivity maps of subsurface objects.