Convolutional neural networks have enabled major progresses in addressing\npixel-level prediction tasks such as semantic segmentation, depth estimation,\nsurface normal prediction and so on, benefiting from their powerful\ncapabilities in visual representation learning. Typically, state of the art\nmodels integrate attention mechanisms for improved deep feature\nrepresentations. Recently, some works have demonstrated the significance of\nlearning and combining both spatial- and channelwise attentions for deep\nfeature refinement. In this paper, weaim at effectively boosting previous\napproaches and propose a unified deep framework to jointly learn both spatial\nattention maps and channel attention vectors in a principled manner so as to\nstructure the resulting attention tensors and model interactions between these\ntwo types of attentions. Specifically, we integrate the estimation and the\ninteraction of the attentions within a probabilistic representation learning\nframework, leading to VarIational STructured Attention networks (VISTA-Net). We\nimplement the inference rules within the neural network, thus allowing for\nend-to-end learning of the probabilistic and the CNN frontend parameters. As\ndemonstrated by our extensive empirical evaluation on six large-scale datasets\nfor dense visual prediction, VISTA-Net outperforms the state-of-the-art in\nmultiple continuous and discrete prediction tasks, thus confirming the benefit\nof the proposed approach in joint structured spatial-channel attention\nestimation for deep representation learning. The code is available at\nhttps://github.com/ygjwd12345/VISTA-Net.\n