Despite the recent success of Graph Neural Networks (GNNs), training GNNs on\nlarge graphs remains challenging. The limited resource capacities of the\nexisting servers, the dependency between nodes in a graph, and the privacy\nconcern due to the centralized storage and model learning have spurred the need\nto design an effective distributed algorithm for GNN training. However,\nexisting distributed GNN training methods impose either excessive communication\ncosts or large memory overheads that hinders their scalability. To overcome\nthese issues, we propose a communication-efficient distributed GNN training\ntechnique named $\\text{{Learn Locally, Correct Globally}}$ (LLCG). To reduce\nthe communication and memory overhead, each local machine in LLCG first trains\na GNN on its local data by ignoring the dependency between nodes among\ndifferent machines, then sends the locally trained model to the server for\nperiodic model averaging. However, ignoring node dependency could result in\nsignificant performance degradation. To solve the performance degradation, we\npropose to apply $\\text{{Global Server Corrections}}$ on the server to refine\nthe locally learned models. We rigorously analyze the convergence of\ndistributed methods with periodic model averaging for training GNNs and show\nthat naively applying periodic model averaging but ignoring the dependency\nbetween nodes will suffer from an irreducible residual error. However, this\nresidual error can be eliminated by utilizing the proposed global corrections\nto entail fast convergence rate. Extensive experiments on real-world datasets\nshow that LLCG can significantly improve the efficiency without hurting the\nperformance.\n