Multi-agent reinforcement learning (MARL) has been increasingly explored to\nlearn the cooperative policy towards maximizing a certain global reward. Many\nexisting studies take advantage of graph neural networks (GNN) in MARL to\npropagate critical collaborative information over the interaction graph, built\nupon inter-connected agents. Nevertheless, the vanilla GNN approach yields\nsubstantial defects in dealing with complex real-world scenarios since the\ngeneric message passing mechanism is ineffective between heterogeneous vertices\nand, moreover, simple message aggregation functions are incapable of accurately\nmodeling the combinational interactions from multiple neighbors. While adopting\ncomplex GNN models with more informative message passing and aggregation\nmechanisms can obviously benefit heterogeneous vertex representations and\ncooperative policy learning, it could, on the other hand, increase the training\ndifficulty of MARL and demand more intense and direct reward signals compared\nto the original global reward. To address these challenges, we propose a new\ncooperative learning framework with pre-trained heterogeneous observation\nrepresentations. Particularly, we employ an encoder-decoder based graph\nattention to learn the intricate interactions and heterogeneous representations\nthat can be more easily leveraged by MARL. Moreover, we design a pre-training\nwith local actor-critic algorithm to ease the difficulty in cooperative policy\nlearning. Extensive experiments over real-world scenarios demonstrate that our\nnew approach can significantly outperform existing MARL baselines as well as\noperational research solutions that are widely-used in industry.\n