Summary
The authors propose an approach to loss re-shaping for a supervised prediction model with the goal of improving the performance of a larger system that performs a downstream task using the predictions of the prediction model. Such shaping is necessary because reducing loss on the prediction task may only roughly correspond to improved performance on the downstream task. The authors show that their approach mostly outperforms standard maximum likelihood estimation and achieves similar performance to existing baselines.
Strengths
TaskMet is an interesting method that enables injection of knowledge about downstream tasks without ignoring the potentially useful training data for the base prediction task.
Weaknesses
The clarity of the presentation could be significantly improved. For example, the distinction between "task learning" and "prediction" is not clear to readers who are not very familiar with the terminology; these are interchangeable synonyms in many fields. The descriptions of the benchmark tasks are also not very accessible to those not already familiar with them (what are "resources" in the Cubic problem?).
The experimental results in section 5.2 are relatively weak; TaskMet does not show improved performance in any of the three benchmark tasks. Although the average performance across tasks is slightly better than existing approaches, it's not clear if TaskMet outperforms the baselines if they are well-tuned (and the authors note that hyperparameter tuning for TaskMet is key).
In section 5.3 (the model-based RL experiments), the results are stronger. However, the setting is somewhat contrived, in that the authors study either a) a large number of artificial noise dimensions added to the state, which is likely to improve TaskMet's performance because it is specifically designed to rescale the loss of each dimension and b) when the network is unrealistically small (3 hidden units). Although the results in these settings are positive, they do not decisively show the benefit of the proposed method in real-world settings. Further, the baselines considered are either only using the prediction task loss (MSE) or only using the downstream loss (OMD). It's not clear if TaskMet outperforms other methods that utilize both objectives.
Some related work could be added, particularly work in meta-learning loss functions, but this is not critical.
The advantages of TaskMet re: interpretability are not actually validated experimentally, other than on a synthetic problem showing that TaskMet learns to ignore dimensions of the targets that are artificial pure noise.
Questions
Does TaskMet scale to more real-world settings, for example more difficult RL problems?
Does the learning metric produce useful information for real prediction problems, other than the artificial cartpole with noise setting?
Is TaskMet limited to learning a Mahalanobis norm? Why not a more complex/expressive learned loss function?
Does TaskMet require significantly more compute than MSE?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
Yes, although addressing the differences in computational requirements (if any) would be helpful.