Emerging Relation Network and Task Embedding for Multi-Task Regression Problems

Multi-task learning (mtl) provides state-of-the-art results in many\napplications of computer vision and natural language processing. In contrast to\nsingle-task learning (stl), mtl allows for leveraging knowledge between related\ntasks improving prediction results on the main task (in contrast to an\nauxiliary task) or all tasks. However, there is a limited number of comparative\nstudies on applying mtl architectures for regression and time series problems\ntaking recent advances of mtl into account. An interesting, non-linear problem\nis the forecast of the expected power generation for renewable power plants.\nTherefore, this article provides a comparative study of the following recent\nand important mtl architectures: Hard parameter sharing, cross-stitch network,\nsluice network (sn). They are compared to a multi-layer perceptron model of\nsimilar size in an stl setting. Additionally, we provide a simple, yet\neffective approach to model task specific information through an embedding\nlayer in an multi-layer perceptron, referred to as task embedding. Further, we\nintroduce a new mtl architecture named emerging relation network (ern), which\ncan be considered as an extension of the sluice network. For a solar power\ndataset, the task embedding achieves the best mean improvement with 14.9%. The\nmean improvement of the ern and the sn on the solar dataset is of similar\nmagnitude with 14.7% and 14.8%. On a wind power dataset, only the ern achieves\na significant improvement of up to 7.7%. Results suggest that the ern is\nbeneficial when tasks are only loosely related and the prediction problem is\nmore non-linear. Contrary, the proposed task embedding is advantageous when\ntasks are strongly correlated. Further, the task embedding provides an\neffective approach with reduced computational effort compared to other mtl\narchitectures.\n