Nonparametric Teaching for Multiple Learners

Summary

This paper extends nonparametric teaching from the setting of teaching each learner independently to teaching multiple ones simultaneously. The method is about teaching a vector-valued model, which improves over existing methods when multiple learners can communicate with each other. There are both theoretical and experimental results validating the effectiveness of this method.

Strengths

The paper seems to have solid theoretical analysis; it's hard for me to judge as a non-expert.

Weaknesses

Only one experiment on the RGB channels of images is shown in the paper. Another experiment in a different setting may help demonstrate the generalizability of the approach. As someone with no expertise in this area, I'll defer to others on the technical side of things.

Questions

Can you elaborate on what other possible applications can benefit from the proposed approach?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

The paper extends non-parametric machine teaching to the case of multiple learners. In particular, each learner learns one component of a vector-valued function. The authors consider the case where learners have no communication with each other and the case where there is "communication" via a matrix transformation of the outputs of each individual learner.

Strengths

Originality: I am not an expert in machine teaching so I can't say how original the contribution is relative to prior work. Quality: The paper appears to be of good quality, though I did not check the proofs carefully. Clarity: The paper was relatively clear, though with key exceptions referenced below Significance: The paper appears to me to be an incremental step beyond the non-parametric single-learner setting.

Weaknesses

I believe the paper needs more motivation and needs to contrast with the case where a single learner is learning a vector-valued function. Currently, the paper extends the scalar-valued single-learner setting to one where the teacher is teaching multiple learners---each outputting one component of a vector-valued output. In the introduction, the paper explains why teaching multiple learners at once would be better than teaching multiple learners separately. That seems quite clear, but then why not just teach one single learner that is learning a vector-valued function rather than multiple learners learning each component separately? The introduction seems to hint at computational issues but this should be made more explicit, and if computational issues are a critical motivation, then experiments showing the computational advantage should be included. Especially because the paper also includes a section on "communication" between the learners and shows that they achieve better performance when communication is allowed. (By the way, the communication, in this case, is just a matrix that the teacher gives each learner.. the learners aren't learning to communicate with each other, so it seems clear from the outset that this would lead to improvement) So if communication leads to higher performance, then why not full "communication", i.e., one learner that outputs all components?

Questions

I thought the experiments section could be made clearer (I also looked at the additional details in the appendix but was still confused). What is the domain and range of the functions being learned? What are the examples (x and y) that the teacher is giving for each settting? Are they pixels?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

I ask that the authors replace the Lenna image with any other image. There is no reason that this paper needs to use, of all images, the Lenna image, a cropped photo of a nude women from a Playboy centerfold. The continued unnecessary use of this image perpetuates an uninclusive environment. https://womenlovetech.com/losing-lena-why-we-need-to-remove-one-image-and-end-techs-original-sin/ https://www.washingtonpost.com/opinions/a-playboy-centerfold-does-not-belong-in-tj-classrooms/2015/04/24/76e87fa4-e47a-11e4-81ea-0649268f729e_story.html Lena herself has said that she no longer wants her image used: “I retired from modelling a long time ago,” said Lena in a new documentary film called Losing Lena. “It’s time I retired from tech, too. We can make a simple change today that creates a lasting change for tomorrow. Let’s commit to losing me.”

Summary

The paper studied nonparametric teaching in the presence of multiple learners. Following prior works on nonparametric teaching, the paper extended to a scenario where multiple learners simultaneously learn a separate component of the joint model. The paper first analyzed the performance of the Random Functional Teaching (RFT) and the Greedy FT (GFT) strategy and showed that both teachers can successfully guarantee reduction in the loss function when the learners are completely independently, and no communication happens. Secondly, the authors studied the effect of communication between learners and show that an affine transformation of the joint model does not increase the loss but can significantly enhance the loss reduction in the beginning of the training phase. Experiments validated the discoveries in this paper.

Strengths

(1). The problem of teaching multiple learners itself is interesting and underexplored in the machine teaching community. This paper pushes the frontier of machine teaching in this aspect. (2). The paper theoretically analyzed the loss reduction of RFT and GFT teacher, and also the benefit of communication. The results show that non-parameter teaching with communication can indeed help with loss reduction. (3). The paper performed extensive empirical study of the teaching strategy, and the results are convincing.

Weaknesses

(1). The assumption in the theoretical study is not clearly stated in the paper. For example, the theory seems to rely on the fact that the loss function must be convex, and this information should be made more prominent upfront. The authors may want to discuss the applicability of both the theory and the methodology developed in this paper. Does it apply with neural network model or only convex learners? (2). One thing missing from the discussion is that how does the teaching performance compare to vanilla learning. The non-parametric teacher only guarantees that the loss is always reduced. However, how does the loss reduction compare to standard learning process without teaching is an important topic to be discussed.

Questions

(1). What is the applicability of the theoretical results and the methodology developed in this paper? Is convexity a required property? (2). How does non-parametric teaching compare to vanilla learning without teaching. Does it help accelerate the learning process?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Yes

Summary

This paper investigates the iterative machine teaching problem under the non-parametric learner setting with vector-valued target models, also known as multi-learner nonparametric teaching (MINT). The authors consider two teaching strategies: Random Functional Teaching (RFT) and Greedy FT (GFT). The authors first theoretically analyze the convergence behavior induced by the teaching strategies in the vanilla MINT setting, where there is no communication between the learners. Then, they study the teaching strategies in the communicated MINT setting. Finally, they empirically compare the effectiveness of different teaching strategies.

Strengths

The paper is overall well-written, and the related work is extensively discussed. The theoretical results in this paper seem correct; I haven’t checked the details of the proofs.

Weaknesses

The availability of a powerful teacher with a vector-valued target model needs to be motivated with more realistic practical scenarios. The novelty of the contributions of this work in comparison to “Nonparametric Iterative Machine Teaching (NIMT)”: the problem formulation/setup, RFT/GFT teaching strategies are extensions from the NIMT paper. Thus, it is important to clearly discuss how non-trivial these extensions are and how different the proof techniques are from those of the NIMT paper.

Questions

The theoretical results in the paper are based on a synthetic example generation setting. If the teacher is restricted to choosing examples from a pool, that would result in different examples than the ideal one. Then, what would be the impact on the results? Given that the teacher can freely synthesize examples to guide the learner toward a target model, can the results discussed be extended to learners with non-convex loss functions?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The paper is of an algorithmic/theoretical nature and does not have any direct potential negative societal impact.