Minimax Forward and Backward Learning of Evolving Tasks with Performance Guarantees

Summary

This paper presents an incremental minimax risk classifier (imrc) to effectively utilize forward and reverse learning and explain evolutionary tasks. Furthermore, the performance improvements provided by forward and backward learning can also be described analytically based on the expected quadratic changes of the task and the number of tasks.

Strengths

In the article, the effects and limitations of IMRCs are verified by rich experiments, and using IMRCs can better use forward and reverse learning to explain the evolutionary tasks and improve the effective sample size.

Weaknesses

However, there are some deficiencies in the ablation aspect, and the experimental volume is relatively small.

Questions

1. The setting of the task in the article mainly starts from the high similarity continuous task, so whether the high similarity continuous learning has practical application, and whether this method can have a broad development prospect in the future technology development. 2. In the article, it is necessary to more clearly describe the task premise, between the similarity in the similarity continuous learning and the evolution in the evolutionary tasks, and then describe the clarity and rationality, why the similar continuous tasks are more consistent with the evolutionary principles. 3. In the article, I learned the role of positive learning and reverse learning in the experiment, but I want to know whether reverse learning alone is feasible, whether positive learning has an inhibitory effect on reverse learning? 4. In Figure 1 of the article, although the meaning of the expression can be roughly understood, the graph line is a little confused, especially the meaning of the black line is unclear. 5. In Figure 2 of the article, the meaning of j in the first red square and k in the second red square is not clear, and the symbols introduced in the text in Article 4.2 are inconsistent with the flow chart symbols in Figure 2, which is easy to confuse the readers' memory.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

see questions

Summary

IMRCs (Incremental Minimax Risk Classifiers) are a technique for incremental learning of a growing sequence of classification tasks. The key feature of IMRCs is their ability to exploit the similarity between consecutive tasks by leveraging both forward and backward learning. Forward learning utilizes information from preceding tasks to improve the classification performance of the current task. It recursively updates the mean and mean squared error (MSE) vectors by considering the difference between the current task and the preceding task. Backward learning, on the other hand, uses information from the current task to improve the classification performance of preceding tasks.

Strengths

(1) IMRCs leverage the evolving nature of tasks in a growing sequence. By utilizing both forward and backward learning, IMRCs effectively exploit the similarity between consecutive tasks. (2) By leveraging information from preceding and succeeding tasks, IMRCs can incorporate additional knowledge and enhance classification performance. (3) IMRCs minimize the worst-case error probabilities over uncertainty sets. This means that they provide robust classification performance, even in the face of uncertainty or variations in the underlying distributions of tasks.

Weaknesses

(1) The forward and backward learning processes, along with the uncertainty set updates, can require significant computational resources and time. How about the training time compared to other methods? (2) The selection of appropriate uncertainty sets can be challenging and may require prior knowledge or assumptions about the task distributions. (3) The authors assume that the distributional changes between tasks (p2-p1, p3-p2, ...) are independent and have a mean of 0. How can IMRCs be extended or modified to handle scenarios with non-zero-mean changes between consecutive tasks or explore novel tasks outside the existing sequence?

Questions

See weakness part.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This paper presents a novel method to tackle continual learning, called IMRCs which is able to exploit forward and backward learning to account for evolving tasks. The authors provide theoretical guarantees on the performance of IMRCs in terms of the tasks' expected quadratic change and the number of tasks. Authors compared the proposed method with prior work on multiple datasets, sample sizes, and number of tasks.

Strengths

+ The idea seems interesting and performance guarantee in the context of continual learning is a promising research direction. + A thorough theoretical analysis of IMRCs is provided.

Weaknesses

- My main concern about this paper is the experimental section. The diversity of the datasets and experiments with different sample sizes is a plus but at the same time it is different from the common continual learning setting in the literature and the baselines are outdated whereas the field of continual learning is rapidly growing and there has been numerous works developed between 2021 and now. - The paper does not discuss the computational complexity of IMRCs.

Questions

- Authors mention they used the original hyper parameters for the baseline. But given that some methods such as MER, GEM, EWC which were introduced on image data, cannot be taken as is and applied to tabular datasets without tuning hyper parameters. How did the authors make sure of a fair comparison? Note that results for some methods such as GEM and MER are somewhat equal to those from EWC (Table 1) but in vision applications this is not expected and EWC is always outperformed by every single memory-based method.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

No.

Summary

The papers studies a specific case of continual learning over a sequence of tasks. Authors extend the typical i.i.d. assumption of task meta-distribution to the case where only the differences between subsequent task distributions are assumed to be independent and zero-mean. Under this assumption, they devise two versions of a minimax risk classifier and prove performance guarantees that improve upon single-task bounds by relying on effective sample size instead of just the number of samples in a task. The effective sample size is no smaller than the single task size and can be larger if the quadratic change between consecutive tasks behaves in a favorable way, e.g. decreasing over time. The authors also present an experimental study of the proposed algorithms and show that it either improves the baseline performance or achieves similar one across 12 datasets.

Strengths

### Originality This is a novel approach that relies on independent differences between subsequent task distributions rather than on the typical i.i.d. assumption. ### Quality * Paper studies the presented algorithm theoretically and experimentally. * Illustrative theoretical guarantees that are always no worse than a single task with a clear influence of the task distribution properties in the form of effective sample size. * Experiments uses 12 different datasets and cover 7 baselines. * Authors provide insights into impact of number of tasks and number of samples on the performance of different variants of the algorithm. ### Clarity The paper is well written and easy to follow in general. ### Significance There are numerous ways of how i.i.d. assumption can be relaxed and paper studies one of them. The significance will rely on how close the new assumption is to practical scenarios. From experimental setup it seems to be a rather specific case that relies too much on exact task ordering, thus I would expect the paper to have average impact from theoretical point of view, but quite limited impact in practice.

Weaknesses

The paper doesn’t have major weaknesses. There are things that can be improved in the presentation and discussing limitations, which I cover in questions section.

Questions

The following things would help with the presentation: * The notations used are overloaded (e.g. by using double superscripts) and sometimes are hard to follow. The paper would benefit from a small table with all notations in one place. * Experiments present the result for samples size of 10 without any additional comments on exact value of 10. Appendix shows that the general conclusion is valid irrespective of the sample size, so it would be good to highlight that. * The main assumption on the task distribution needs to be made more precise and explicit in the theorems (introduce assumptions separately and then refer to it in theorems). A discussion on how realistic it is and providing more than one example would go a long way. * Theoretical analysis needs a discussion on different regimes where the ESS improvements would be realised and what they mean for the task distributions process. Authors do have a short discussion of that after Theorem 4, but it lacks the connection between different regimes of nd^2 and process properties. For example, $nd^2 \leq \frac{1}{j^2}$ translates into a rather fast convergence requirement of the task distribution process. * It would be great to discuss what happens to the algorithm when the main assumption breaks. Does it go back to single task performance? How would one figure this out in practice?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The limitation of the approach are not adequantly discussed. I’ve made a few suggestion in the questions section on how to address that.

Summary

Though class-incremental learning has been studied the most as a default setting for continual learning, task-incremental study could be crucial for other areas. This paper focuses particularly on the case where tasks being introduced over time are interrelated under specific conditions. Under the supervised learning framework, the task distributions $p_t$ (of instances and labels at any time t) are independent and identically distributed, which would be a strong and less-realistic condition. In contrast, this paper covers the case where the difference of two consecutive task distributions is independent, making the difference between two distant tasks have increasing variance over time. Then the authors propose Incremental Risk Minimization Classifier (IMRC) method that effectively leverage forward and backward learning algorithms, improving the performance against existing methods on various datasets, sample sizes, and the number of tasks.

Strengths

(1) The paper formally defines a non-trivial task-incremental learning setting whose assumption is more realistic than the earlier approach. (2) The paper analytically shows performance guarantees and effective sample sizes as well as forward and backward learning algorithm. (3) The paper addresses empirical improvement on multiple datasets against existing state-of-the-art algorithms.

Weaknesses

(1) Current draft is rarely self-contained. Readers must read the appendix frequently. Minimally, having explanations about methods and datasets in Table 1 would significantly increase the readability. (2) Run-time trade-off of considering the backward (possibly per different sample sizes) are missing. (3) Characteristics of each experimental dataset: how much they agree and disagree the independence assumption across consecutive task distributions are missing.

Questions

(1) Providing comprehensive details about different tasks and their distinctive characteristics helps readers better understanding the contributions of the paper beyond the theoretical excitement. Formalizing the time-evolving task with an independence assumption on consecutive task difference sounds exceptional. It will be great to know how much this assumption is agreed and violated on individual tasks on experiments. (2) In addition to (1), accessing real examples or qualitative explanations about hard cases (that were not successful in the previous approaches, but exceptionally successful with the current IMRC) will be useful. For example, it seems IMRC is distinguished notably from other algorithms on P. Supply, Usenet1, and Spam datasets. Understanding the rationale will be insightful. (3) Most importantly, run-time trade-off for doing backward process must be measured for making the provable guarantees practically effective. As the trade-off would change for different number of tasks and sample sizes, practitioners require better guidance.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

No specific limitations are described or probed.

Reviewer spbD

Thank you for the detailed feedback. I will keep my score which leans toward acceptance.

Thanks for your rebuttal.

Thanks the authors for their feedback on my comments and suggestions. I will keep my score based on their clarifiations.