In this paper, we conjecture that if the permutation invariance of neural\nnetworks is taken into account, SGD solutions will likely have no barrier in\nthe linear interpolation between them. Although it is a bold conjecture, we\nshow how extensive empirical attempts fall short of refuting it. We further\nprovide a preliminary theoretical result to support our conjecture. Our\nconjecture has implications for lottery ticket hypothesis, distributed\ntraining, and ensemble methods.\n