Active Representation Learning for General Task Space with Applications in Robotics

Summary

The paper introduces an algorithm for active representation learning in multi-task settings. The algorithm generalizes to settings in which the “task space” is either discrete or continuous. The overall setting involves an (optionally) nonlinear map between an input space X and an intermediate feature space via $\psi_X$ and $\phi_X$, that then undergoes a final multiplication with “task features” obtained by applying $\psi_W$ and $B_W$ to task parameters $w$. The goal is to learn $\hat{\phi}_X$ and $B_W$ given the possibility of actively selecting tasks $w$ within a “source” domain, and evaluating on tasks in a “target” domain. The proposed approach can work for both “coarse exploration”, “fine target-agnostic exploration” and “fine target-aware exploration”. The authors provide a theoretical analysis for the “benign setting” in which the source task space satisfies specific assumptions. Moreover, they provide experimental results beyond this setting, and on some real-world datasets.

Strengths

The paper is heavily theoretical in nature, and seeks to formally introduce the problem with precise assumptions, also offering theoretical guarantees given a “benign” version of the problem setting. The experimental results show that the target-aware method indeed beats the target-agnostic method by a large margin, giving credence to the significance of the target-aware method.

Weaknesses

Overall, the paper is very hard to parse, and seems to take much background as granted. In the preliminaries, a very large number of assumptions are made on the problem setting. It is unclear given my limited expertise in the subfield whether all these assumptions undermine the significance of the approach or not. It would be useful to more thoroughly compare the assumptions and design choices with the literature on the topic, to better contextualize what the paper brings to the table and what the contribution is. The setting itself seems interesting, but not enough emphasis is put on why it is significant and worthwhile exploring. It seems that the approach effectively requires us to be able to at-will sample tasks within their “task parameter space”, which seems to be extremely hard to do for anything beyond simulation. The drone experiment seems to do away with this by defining the parameter space as a one-hot vector, but not enough emphasis is placed on the significance of such a choice. Furthermore, the results on the drone dataset experiment seem to show that the proposed method converges faster, but ultimately reaches the same accuracy as the passive strategy.

Questions

I would like the authors to address the points raised in “Weaknesses”, in order to better clarify my uncertainty related to the paper’s significance. Edit: increased the score after taking into account the authors' reply.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

1 poor

Contribution

2 fair

Limitations

The authors address in the appendix some limitations related to the theoretical analysis. It would however be helpful if they could list some limitations from the point of view of how related the setting is to real-world multi-task learning problems.

Summary

This work considers the multi-task learning problem and proposes an algorithm to select related source tasks to tackle a known target task in an active learning like setup. The analysis is primarily for the linear function classes and handles discrete and continuous task spaces. The proposed algorithm operates in 3 stages: (1) a coarse sampling of the tasks (2) More efficient target-agnostic sampling of diverse tasks (3) target-aware sampling from a sub-space of related tasks. The 2nd and 3rd stage in particular allow for tighter sample complexity bounds since it allows us to efficiently sample from the task space. As a result, the sample complexity bounds depend on $k*$ which is the relationship between the target space and the relevant source space. Finally, this algorithm is used in three scenarios and an active learning strategy is shown to outperform a passive one.

Strengths

The paper presents a theoretically motivated algorithm to sample source tasks that are relevant to the target task. While the theme of active sampling is not entirely new, few of them provide algorithms with sample complexity bounds. While there are many gaps to fill (linear function classes, isotropic Gaussian inputs), this paper takes a step in the right direction. There are two interesting ideas in this work. The first is to reduce the number of tasks that we consider by selecting a diverse set of tasks (subspace $V$) that spans the task space. This is the primary role of stage 2 or the task-agnostic exploration. The second idea (similar to Yifang et al.) is to perform a task-aware search of the task-space and only use source tasks that are relevant to the target task. This improves sample complexity and the benefit is stronger if a smaller subspace of source tasks is strongly related to the target task. The paper presents many results on 3 different experimental scenarios that primarily compare passive multi-task and active target-aware learning with clear sample complexity benefits for the latter.

Weaknesses

**Experimental setup**: To the best of my understanding, all the experiments in the manuscript (including appendix) compare passive to target-aware active learning strategies. How does the proposed method compare to a target-aware strategy that doesn't use stage 2 and only uses stage 3. I understand the theoretical benefit of stage 2 is clear and it allows us to work with fewer source tasks but is this reflected experimentally? **Improving the discussion on related work:** It would be nice to see the authors discuss some related work on multi-task learning and task-relatedness. This include some older work on passive multi-task learning [1,2], learning theory [3,4,5], task-distances [6,7,8], task-grouping [10,11] and weighted-training [6,12]. There are a lot more works that I haven't listed out that I think are very relevant to this work. 1. Caruana, Rich. "Learning many related tasks at the same time with backpropagation." Advances in neural information processing systems 7 (1994). 2. Baxter, Jonathan. "Learning internal representations." Proceedings of the eighth annual conference on Computational learning theory. 1995. 3. Ben-David, Shai, et al. "A theory of learning from different domains." Machine learning 79 (2010): 151-175. 4. Hanneke, Steve, and Samory Kpotufe. "On the value of target data in transfer learning." Advances in Neural Information Processing Systems 32 (2019). 5. Crammer, Koby, Michael Kearns, and Jennifer Wortman. "Learning from Multiple Sources." Journal of Machine Learning Research 9.8 (2008). 6. Thrun, Sebastian, and Joseph O'Sullivan. "Discovering structure in multiple learning tasks: The TC algorithm." ICML. Vol. 96. 1996. 7. Zamir, Amir R., et al. "Taskonomy: Disentangling task transfer learning." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018. 8. Achille, Alessandro, et al. "Task2vec: Task embedding for meta-learning." Proceedings of the IEEE/CVF international conference on computer vision. 2019. 9. Standley, Trevor, et al. "Which tasks should be learned together in multi-task learning?." International Conference on Machine Learning. PMLR, 2020. 10. Ramesh, Rahul, and Pratik Chaudhari. "Model Zoo: A Growing" Brain" That Learns Continually." arXiv preprint arXiv:2106.03027 (2021). 11. Fifty, Chris, et al. "Efficiently identifying task groupings for multi-task learning." Advances in Neural Information Processing Systems 34 (2021): 27503-27516. 12. Chen, Shuxiao, Koby Crammer, Hangfeng He, Dan Roth, and Weijie J. Su. "Weighted training for cross-task learning." arXiv preprint arXiv:2105.14095 (2021).

Questions

It seems like an important part of the algorithm is to use $g(f, A)$ from Equation 2, which estimates which estimates the most relevant source tasks. How would this change if were to use non-linear function classes. In general, I was wondering which parts of the algorithm rely on the function being linear. In practice, the sample complexity bounds may be too pessimistic (which isn't easy to get around). How do we decide the number of samples to select in each round of active sampling and what are some other key hyper-paramters? It feels like the additional complexity of the methods leads to a lot more hyper-parameters which seems like it will be harder to get it to work in practice. The paper has dense notation dense but I appreciate the authors effort to unpack all the results. While I was able to follow the paper, the notation can be hard to follow at times. It may help the reader if some parts are simplified in the main paper. For example the algorithm could be written without all the precise details needed to achieve the sample complexity bound.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The paper discusses some of the limitations in Appendix C in sufficient detail.

Summary

This paper proposes a framework for active representation learning that allows learners to optimally choose which source tasks to sample from. The framework covers both task-aware and task-agnostic settings and is compatible with deep representation learning practices. The paper provides several instantiations of the framework, including a bilinear model and a neural network model. The authors demonstrate the effectiveness of their approach on a variety of simulated robotic tasks, showing that their method outperforms existing methods in terms of sample efficiency and generalization to new tasks. Overall, the paper's contributions include a novel framework for active representation learning, several instantiations of the framework, and empirical results demonstrating the effectiveness of the approach.

Strengths

The paper proposes a novel framework for active representation learning that allows learners to optimally choose which source tasks to sample from. This framework covers both task-aware and task-agnostic settings and is compatible with deep representation learning practices. It considers a more general setting where tasks are parameterized in a vector space, allowing for more effective leveraging of similarities between tasks. The paper provides a thorough theoretical analysis of the proposed framework, including sample complexity bounds and convergence guarantees. The paper is well-written and organized, with clear explanations of the proposed framework and its instantiations. The proposed framework has broad applicability in robotics and other domains where there are multiple related tasks to be learned.

Weaknesses

The manuscript would be considerably enhanced by the inclusion of more comprehensive empirical assessments, specifically pertaining to real-world robotic assignments, as opposed to merely simulated tasks. Such addition would serve to confirm the applicability and efficacy of the suggested method in a broader, more realistic context. Further elucidation through an in-depth comparative study with established methodologies, notably those tailored towards active representation learning, would aid in comprehending the proposed method's advantages and shortcomings in relation to contemporary state-of-the-art techniques. Moreover, a thorough discourse on the constraints inherent in the proposed method is warranted, especially considering the presumptions regarding the task space and task distribution. An exposition of this nature would lend itself to an improved understanding of the circumstances wherein the approach proves most efficient, as well as areas of potential inapplicability.

Questions

Please see the Weakness.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please see the Weakness.

Summary

The paper studies active representation learning under the assumption that the source and target tasks are parameterized by a vector space (instead of a discrete set of tasks). The proposed algorithm works by first finding a subspace of the task space that spans the representation space and then iterates between refining the subspace and estimating for the target. The theoretical analysis shows that under certain assumptions, their active approach can achieve a better sample complexity than a passive one.

Strengths

- The paper explores an understudied problem statement in active learning and theoretically demonstrates its importance in comparison to less expressive active learning problem statements, which could have a large impact on future work on the applicability of active learning. - The paper is organized and the theory sections are well described.

Weaknesses

- The problem statement and motivation were not described precisely or concisely. E.g., it's not clear to me why robotics is the only target for this work. The methods section was robotics-agnostic, but the introduction uses robotics as the motivating example and the paper is titled as a robotics paper. - The experiment section lacks a comparison to baselines. E.g., a natural baseline would be to compare against an active learning approach that does assume a discrete source and task space. This would help show that the problem statement introduced is important empirically as well as theoretically. For this reason, I am recommending weak reject. - Nit: The methods section also appeared to be accomplishing a lot of things at once---demonstrating the necessity of the continuous task space, showing the improvement in sample complexity over the passive approach, and minimizing expected excess risk---and it's not clear to the readers how these contributions relate or should be prioritized in our minds. - Nit: the text on the LHS of figure 1 is cut off.

Questions

Why do the Pendulum metrics only include test loss and control error? I would expect to see a measure of success or some proxy for it (e.g., average reward).

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

4 excellent

Contribution

4 excellent

Limitations

N/A

Summary

The paper introduces and studies a theoretic and algorithmic framework for actively and optimally selecting the source tasks in multi-task learning, accommodating task-aware and task-agnostic settings. The authors present cohesive theoretical guarantees, particularly demonstrated through the pure bilinear setting with the task or sample complexity being improved. They further empirically validate their meta-algorithm with task-parameter-wise non-linearity settings, particularly with emphasis on applications in robotics, significantly outperforming baselines.

Strengths

The problem of active representation learning is interesting and important. The proposed framework is novel and effective, demonstrated with both theoretical and empirical evidence. The theoretical and experimental results are cohesive.

Weaknesses

- While the method and algorithms seem sound, their explanation could be better with more interpretation and transitions. - The experiment section (in the main manuscript) is not well-presented, e.g., lack of baselines/setting description as well as the interpretation and justifications of results. - To my best assessment, the experiments are not systematically and thoroughly conducted. **Minor:** - The paper misses the conclusion, thorough literature review, and limitation parts

Questions

See some questions in the *weakness* sections. * What is the trade-off and limitations of the proposed algorithm compared to previous methods? * When it comes to a single target task -- does the proposed method outperform the baseline? * Given that the model applies to multiple target tasks, how does it compare to previous works when several target tasks are negatively correlated or have detrimental effects on each other?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The paper does not explicitly describe the limitations of the approach. Please provide the limitations and the trade-off of the method.

Summary

Which source task to sample from is an important issue in multi-task learning. This paper proposes a general and versatile algorithm and framework for active representation learning. The proposed algorithm can be adapted to arbitrary target and source task space, and can cover task-aware and task-agnostic settings. The experiments demonstrate the efficacy of the proposed algorithm on various datasets.

Strengths

- Interesting and important topic.

Weaknesses

- Incomplete contents. - Poor writing and structure.

Questions

None.

Rating

1: Very Strong Reject: For instance, a paper with trivial results or unaddressed ethical considerations

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

1 poor

Contribution

2 fair

Limitations

No limitations are addressed.

Reply to Authors

I thank the authors for their reply. Overall, I still have concerns with respect to real world applicability of the method, specifically related to point 2) that has been raised in this reply (how to even sample from the task space at will in real world applications). However, due to their efforts in clarifying mine and other reviewers' points, I am willing to increase my score, albeit remaining at low confidence.

Thank you for the detailed response!

I thank the authors for the detailed response to many of my questions and for incorporating suggestions. There are some hurdles to overcome to make the algorithm more practical, but the manuscript has novel ideas that warrant further exploration.