Reinforcement Learning-Guided Semi-Supervised Learning

Summary

This paper introduces Reinforcement Learning-Guided Semi-Supervised Learning (RLGSSL), a novel method that combines reinforcement learning (RL) with semi-supervised learning (SSL). By formulating SSL as a one-armed bandit problem, the authors employ a RL-based loss function to guide the learning process. Additionally, the method incorporates a semi-supervised teacher-student framework to improve learning stability.

Strengths

The paper introduces a unique approach by framing semi-supervised learning as a one-armed bandit problem and integrating reinforcement learning to optimize the generation of pseudo-labels.

Weaknesses

The rationality of the method design is not entirely convincing. Firstly, in Equation (2), why is the non-differentiable MSE used instead of the differentiable cross-entropy loss? Why use RL when supervised learning could be applied? Additionally, in the explanation of Equation (3), how is the "meaningful weight" defined? Would this design still be applicable when there is a significant imbalance in the number of samples for each class in the dataset? Training with RL methods is challenging and often unstable, though RL has its own advantages. The authors are advised to add experiments demonstrating the benefits of the RL-based design compared to a supervised design. Some descriptions in the paper might be confusing for readers without background knowledge. For example, the terms "convex combination" in line 190 and "fluid decision boundaries" in line 191 might be unclear. According to the results in Table 3, the improvements brought by the proposed method are not significant.

Questions

1. In line 215, the authors mention the reward function that can enhance the model's robustness and generalization. How is this proven in the experiments? 2. Could the authors further explain the description from lines 228 to 231?

Rating

3

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

In addition to the limitations discussed by the authors in the appendix, please refer to my comments in the Weaknesses section for further details.

Summary

This paper presents a method called Reinforcement Learning Guided Semi-Supervised Learning (RLGSSL), which frames SSL as a one-armed bandit problem. The method features a reward function that measures the discrepancy between the model's predictions on mixed data and pseudo-labels, guiding the learning process. Additionally, it employs a teacher-student model to enhance stability and reduce noise in pseudo-labels. The proposed joint loss function combines RL loss, supervised loss, and consistency regularization loss. Experiments validated the performance of the proposed method and the indispensability of each component.

Strengths

1. This paper attempts to solve the problem of SSL from the perspective of the bandit problem, offering a fresh angle to the research. 2. The ablation study convincingly demonstrates the indispensability of each component of the proposed method. 3. The paper is written in a fluent and clear manner.

Weaknesses

1. While the paper attempts to solve the SSL problem from a bandit perspective, it refers to the loss function as RL loss. The key distinction between RL and bandit problems is the presence or absence of state transitions, and the authors seem to have conflated these concepts. 2. The paper employs bandit terminology to explain parts of the methodology where it might not be necessary. Forcing SSL into a bandit framework seems somewhat unnatural, despite the novel perspective. 3. The paper appears to combine previously existing methods -- Regularization-Based Methods, Teacher-Student-Based Methods, and Pseudo-Labeling Methods. This raises questions about whether the innovation is sufficient.

Questions

1. Why does Table 1 and Table 3 use different numbers of labeled samples for the same datasets, such as using 1000, 2000, 4000 for CIFAR-10 in Table 1, and 250, 1000, 4000 in Table 3? Additionally, why are the comparison methods different in these tables?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors have discussed limitations in their work.

Summary

The authors proposes a novel Reinforcement Learning (RL) Guided semi-supervised learning (SSL) method, RLGSSL, that formulates SSL as a one-armed bandit problem and deploys an innovative RL loss based on weighted reward to guide the learning process of the prediction model adaptively. The core idea is to use RL to guide the selection of informative unlabeled samples, thereby improving the learning efficiency and effectiveness of SSL models.

Strengths

The authors have introduced RLGSSL, a new approach based on Reinforcement Learning that effectively handles Semi-Supervised Learning (SSL). This method uses RL to learn effective strategies for generating pseudo labels and guiding the learning process. The authors have devised a reward function for assessing predictions that encourages the learning of accurate and reliable pseudo-labels while maintaining a balance between the usage of labeled and unlabeled data. They have also developed a new RL loss that allows the reward from pseudo-labels to be incorporated into SSL as a non-differentiable signal in a reinforced manner, promoting better generalization performance. Furthermore, the authors have investigated integration frameworks that combine the power of both RL loss and standard semi-supervised loss, providing a new approach that has the potential to lead to more accurate and robust SSL models. Extensive experiments have demonstrated that this proposed method outperforms state-of-the-art SSL approaches and validates the integration of RL strengths in SSL.

Weaknesses

The integration of RL introduces additional computational overhead, which may require substantial computational resources, especially for large-scale datasets. The current formulation assumes that the labeled and unlabeled data are drawn from the same distribution, which may not hold true in real-world scenarios. This limitation could affect the generalizability of the model.

Questions

1. How can the RLGSSL method be adapted to handle scenarios where the labeled and unlabeled data come from different distributions? 2. How can the hyperparameter tuning process be automated or simplified to make the RLGSSL method more user-friendly and less resource-intensive?

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

yes

Summary

One of the bottlenecks for Semi-supervised learning (SSL) is achieving high performance with limited labeled data, as the model is often complex and needs multiple loss functions. Recently RL has been increasingly used in fine-tuning complex models with non-differentiable reward functions. Thus with these observations, the authors proposed the Reinforcement Learning Guided Semi-Supervised Learning (RLGSSL) method, which uses RL to optimize the generation of pseudo-labels in SSL. More specifically, the pseudo-label predictor serves as the policy function and soft pseudo-labeling acts as the actions. Technically, the authors formulate SSL as a one-armed bandit problem with a continuous action space and deploy a novel RL loss to guide the SSL process based on a reward function specifically designed for semi-supervised data. Moreover, they further incorporate a semi-supervised teacher-student framework to augment the RL loss with a supervised loss and a prediction consistency regularization loss, aiming to enhance learning stability and efficacy. In this way, RL could provide exploration and manage non-differentiable operations.

Strengths

1. The idea of leveraging RL to optimize the pseudo-labels generator in Semi-Supervised Learning (SSL) is a good catch. 2. The motivation is reasonable and in the experiment section, the author conducted broad experiments to showcase the effectiveness of the proposed method.

Weaknesses

1. The comparison between the proposed method and other methods is not enough. E.g. the author could showcase the uniqueness of the proposed method in related work section, 2. The presentation of this paper is below average. E.g. there should be more figures and the writing should be more concise and logical. 3. The author did not mention the side effect of bringing RL into SSL. E.g. will the training process be more time-consuming?

Questions

1. The writing should be more concise, and the logic here is quite messy. E.g. in "We treat SSL as a special one-armed bandit problem with a continuous action space. One-armed bandit problem can be considered a single-step Markov Decision Process (MDP) [39]. In this problem, the agent takes a single action and receives a reward based on that action. The state of the environment is not affected by the action....". This content has been repeated a few times previously, but the reader still can not find the logic that connects these sentences. 2. The research question in this paper is not quite emphasized, it should be focused on, with deeper analysis. 3. Another thing is the authors may consider presenting the novelty more explicitly. E.g. do more comparisons between related work and the proposed methods in the introduction and related work section. 4. More related work. This is an extension of question 3, since the idea of RL to fine-tune complex models is prevailing, readers may expect the paper to show the related work of such an idea, e.g. in the paper, the authors said "Recently, RL has been applied to fine-tune complex models that typically fail to align with users’ preferences.", then the reader would expect more related work here. And if there exists a method that tries to combine SSL and RL in any sense, then it should appear in the baseline as well.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors did not mention the limitations of the proposed method. A possible limitation may fall on the training efficiency. E.g. according to Figure 1, there are multiple training loss and training objects.