Autonomous agents must learn to collaborate. It is not scalable to develop a\nnew centralized agent every time a task's difficulty outpaces a single agent's\nabilities. While multi-agent collaboration research has flourished in\ngridworld-like environments, relatively little work has considered visually\nrich domains. Addressing this, we introduce the novel task FurnMove in which\nagents work together to move a piece of furniture through a living room to a\ngoal. Unlike existing tasks, FurnMove requires agents to coordinate at every\ntimestep. We identify two challenges when training agents to complete FurnMove:\nexisting decentralized action sampling procedures do not permit expressive\njoint action policies and, in tasks requiring close coordination, the number of\nfailed actions dominates successful actions. To confront these challenges we\nintroduce SYNC-policies (synchronize your actions coherently) and CORDIAL\n(coordination loss). Using SYNC-policies and CORDIAL, our agents achieve a 58%\ncompletion rate on FurnMove, an impressive absolute gain of 25 percentage\npoints over competitive decentralized baselines. Our dataset, code, and\npretrained models are available at https://unnat.github.io/cordial-sync .\n