FlatMatch: Bridging Labeled Data and Unlabeled Data with Cross-Sharpness for Semi-Supervised Learning
Semi-Supervised Learning (SSL) has been an effective way to leverage abundant unlabeled data with extremely scarce labeled data. However, most SSL methods are commonly based on instance-wise consistency between different data transformations. Therefore, the label guidance on labeled data is hard to be propagated to unlabeled data. Consequently, the learning process on labeled data is much faster than on unlabeled data which is likely to fall into a local minima that does not favor unlabeled data, leading to sub-optimal generalization performance. In this paper, we propose FlatMatch which minimizes a cross-sharpness measure to ensure consistent learning performance between the two datasets. Specifically, we increase the empirical risk on labeled data to obtain a worst-case model which is a failure case that needs to be enhanced. Then, by leveraging the richness of unlabeled data, we penalize the prediction difference (i.e., cross-sharpness) between the worst-case model and the original model so that the learning direction is beneficial to generalization on unlabeled data. Therefore, we can calibrate the learning process without being limited to insufficient label information. As a result, the mismatched learning performance can be mitigated, further enabling the effective exploitation of unlabeled data and improving SSL performance. Through comprehensive validation, we show FlatMatch achieves state-of-the-art results in many SSL settings.
Paper
Similar papers
Peer review
Summary
Inspired by Sharpness-aware minimization (SAM) technique, this paper proposes a new semi-supervised learning method FlatMatch. The main idea is in Eqn.(4), where FlatMatch picks both $\theta$ (current weight parameters) and $\tilde{\theta}$ (the parameters close to $\theta$, and maximize the loss on labeled dataset), and then try to optimize $\theta$, so that both $\theta$ and $\tilde{\theta}$ agree with the unlabeled data sets. The authors also propose an efficient version of FlatMatch called FlatMatch-e, that saves some computation. In experiments, the authors demonstrated that their method FlatMatch with fix label trick outperforms all the existing methods.
Strengths
I think it is an interesting idea to use SAM like techniques to solve semi-supervised learning methods. The presentation of this paper is fairly good, so I find the paper very easy to follow. The experiment section is fairly comprehensive. Originality: good, because it is my first time hearing using this technique to solve the problem. Quality: poor, as I will mention below. Significance: fair. If the authors can improve the experiments to validate the real improvement of their algorithm, I think it seems to be a nice contribution to the semi-supervised learning.
Weaknesses
I think the experimental setting, the FlatMatch (fix label) part, has serious issue. Specifically, I think the main motivation of semi-supervised learning, is to consider the case that the labeled dataset is fairly small, and the unlabeled dataset is huge. In that case, we hope to combine both the information of labeled data and unlabeled data to improve the performance of the model. However, in their setting, they mentioned that they pretrain a common SSL method, such as FixMatch for the dataset, and select high-confidence unlabeled data with pesudo labels, **convert them into labeled data. After that, they will use this "expanded labeled dataset" for their FlatMatch training**. Interestingly, although they claim they only "slightly augmenting the number of labels", they in fact added a huge set of labels: 500, 2500, 500, 5000 for CIFAR10, CIFAR100, SVHN, STL10, respectively. Without this "fix label" trick, their method is not as competitive as the existing methods. I personally feel that this is not a fair comparison. If "fix label" trick is indeed helpful, I think the authors should at least compare FlatMatch with the existing methods with fix label as well. If all the other methods does not have any improvement after using fix label trick, then FlatMatch (fix label) will indeed look promising.
Questions
I only have one question: can you improve Table 1 to include the cases for other methods with fix label trick?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
1 poor
Presentation
3 good
Contribution
2 fair
Limitations
I think limitation is not a main issue for this paper.
Summary
The paper focuses on the problem of semi-supervised learning (SSL). The authors first study the loss landscapes of labeled data and unlabeled data and find a generalization mismatch. Base on the findings, they propose FlatMatch to encourage consistent learning performance between labeled data and unlabeled data. Extensive experiments demonstrate the effectiveness of their methods.
Strengths
- To the best of my knowledge, this paper is the first to introduce sharpness-aware minimization into SSL, so the work is novel. - The experiments are wide-ranging and cover several different datasets. The experimental analysis is also relatively comprehensive. - The paper is well written and easy to follow.
Weaknesses
1.The analysis of loss landscape is somewhat unclear and contradictory. - The paper states in lines 42-43: The learning on scarce labeled data convergences faster with lower errors than on unlabeled data. However, in Figure 1, it can only be observed that the loss landscape of labeled data is sharper at both epoch 60 and epoch 150. If want to compare the convergence speed, maybe it is more intuitive to compare the change curve of loss in the time dimension. - The paper states in lines 44-46: the abundant unlabeled data…generalizing better than labeled data. However, the accuray of labeled data is better than that of unlabeled data in Figure 1 (right). - The paper states that using slight data augmentation to plot loss landscape is common in SSL. It is recommended to add relevant references to justify the drawing method. Based on the above questions, the reasonable of two critical flaws proposed in the paper needs further clarification. 2.The paper states in lines 8-9 that FlatMatch minimizes cross-sharpness measures to ensure consistent learning performance between the two datasets. But further verification is lacking. Does the cross-sharpness measure achieve consistent learning performance? If yes, why is it possible?
Questions
1. What is the function $l_d$ used in the experiment? 2. For "sharpness on $D^{l}(D^{u})$" in Table 2, is D^{u}(D^{l}) used? If yes, how to use it? If not, it is recommended to compare the performance of using SAM on the full dataset ($D^{l} \cup D^{u}$). 3. The significance of this work is somewhat limited, as the focus is only on visual datasets. The effectiveness and practicality of the method would be further improved if results from more types of datasets (such as text classification) can be provided.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors point out the limitations of their work.
Summary
This paper focuses on Semi-supervised Learning (SSL) where the training data consists of scarce labeled data and a massive amount of unlabeled data. The authors argue that the propagation of label guidance from labeled data to unlabeled data is challenging. This can cause the learning process on labeled data to progress much faster than on unlabeled data, leading to sub-optimal generalization performance. To alleviate this issue, the authors propose FlatMatch. This method seeks to increase the empirical risk on labeled data to obtain a worst-case model and then penalize the prediction difference (i.e., cross-sharpness) between the worst-case model and the original model, which can benefits the final model.
Strengths
1. This paper is well-written and easy to follow. The motivation is clear, and the unlabeled loss is indeed always larger than the labeled loss in the SSL setting. The idea is interesting. 2. The experiments on CIFAR, SVHN, and STL-10 demonstrate the effectiveness of FlatMatch. 3. The original FlatMatch algorithm is computationally complex, which makes it difficult to run FlatMatch on large-scale datasets like ImageNet. However, the authors propose an efficient version of FlatMatch to make the runtime more manageable.
Weaknesses
1. Although FlatMatch performs well on smaller datasets like CIFAR, SVHN, and STL-10, its performance on large-scale datasets remains unknown. Experiments on ImageNet are necessary to prove the generalization capabilities of the proposed method. 2. The authors train wide-resnets from scratch. However, many recent works have conducted SSL research using Vision Transformers[1,2], which can greatly improve the performance of SSL algorithms in many settings. The performance of FlatMatch when Vision Transformers are used as the backbone remains unknown. 3. While FlatMatch can achieve high performance, it remains computationally complex. The running time of the efficient version of FlatMatch is manageable, but its performance cannot surpass that of SOTA SSL algorithms. [1] Cai, Zhaowei, et al. "Semi-supervised vision transformers at scale." Advances in Neural Information Processing Systems 35 (2022): 25697-25710. [2] Wang, Yidong, et al. "Usb: A unified semi-supervised learning benchmark for classification." Advances in Neural Information Processing Systems 35 (2022): 3938-3961.
Questions
See weaknesses
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
The performance of FlatMatch on large-scale datasets remains unverified, and the computational complexity is high to achieve strong performance.
Further Discussion
Dear Reviewer yYJy: We want to express our appreciation for your valuable suggestions, which greatly helped us improve the quality of this paper. We are also glad that you agreed that our idea is novel and the performance is effective. We have taken our maximum effort to address your concerns on large-scale datasets and ViT models. Your further opinions are very important for evaluating our revised paper and we are hoping to hear from you. Thank you so much. Best, Authors.
Remaining Concerns
Dear Reviewer yYJy: We thank you again for your valuable time to review this paper, your constructive advice is really helpful. By carefully answering all your concerns, this paper has been improved on the scale of application, and validation on efficiency&effectiveness. We hope to know whether our rebuttal solves your concerns. Since the NeurIPS conference supports interactive discussion, we wish we can have to chance to make further efforts on polishing our work. Thanks again for your previous help, we wish to hear from you soon! Best, Authors.
Rebuttal Recap
Dear Reviewer yYJy: We again thank you for reviewing our paper. Here we provide a recap to help you read our rebuttal without feeling unfamiliar with this paper. Your initial concerns can be summarized into two points: * Lack of experiments on ViT model and ImageNet. * The effectiveness and efficiency are limited. Our rebuttal carefully addresses the concerns by: * Conducting experiments on ViT and ImageNet which re-validated the performance of FlatMatch. * Carefully discuss the effectiveness: FlatMatch achieves the best performance in most cases, and justifies the efficiency: Our FlatMatch-e achieves higher performance with lower computational costs. It is really important to us that you could kindly re-read our rebuttal and provide further questions if there are any. Thank you so much and hope you have a good day. Best, Authors.
Summary
The authors propose a semi-supervised learning (SSL) method that applies the sharpness-aware minimization (SAM) method for consistency regularization. Unlike FixMatch, it does not apply perturbation to the unlabeled training sample but evaluates the consistency between posterior distributions of predicted labels between neural networks with original and perturbed parameters. Experimental results show that the proposed method outperforms existing SSL methods with the help of initialization pre-processing to increase labeled samples when their numbers are small.
Strengths
The paper proposes a new SSL algorithm that shows consistently improved performance.
Weaknesses
The authors should show results of existing SSL methods that use SAM as the optimizer instead of SGD. The equations contain errors mixing the definition of objective function and their derivatives. For example, the second term in the first line of equation (2) looks for optimal \theta that minimizes the SAM's objective, but the third term is the derivative of the objective function.
Questions
Equation (4) explains the proposed method. Is min missing in the second term? Similarly, min is missing in the second and the third terms in equation (3).
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
3 good
Limitations
The authors explain the condition where the proposed method is weak when they use it alone.
Further Discussion
Dear Reviewer Y5aq: We really appreciate your efforts to help improve this paper. We have carefully addressed the mentioned concerns, such as experiments on SAM, and the formulation problems. Having further discussions really helps to achieve consensus and clarify misunderstandings, we are eager to know if are there any remaining problems. We will try our best to address them. Best, Authors.
Discussion for Remaining Concerns
Dear Reviewer Y5aq: We appreciate your opinions on our paper. We have tried with our maximum effort to address your concerns, but it has been a while since your initial comments. During our discussions with other reviewers, some misunderstandings have been clarified which helps to make a proper judgment of our paper. Since your opinion of our paper is still leaning negative, we assume there are still several concerns remaining. However, it would be unfortunate if some of the concerns are based on misunderstandings. We hope to avoid this situation and wish you could express your further opinions soon. Thanks again for your valuable time, wish you have a good day. Best, Authors.
Rebuttal Recap
Dear Reviewer Y5aq: We hope this letter finds you well. As there has been a while since you read our paper, therefore we **recap** your initial commons as well as our improvements to make sure that nothing is forgotten. Your major concerns can be summarized into two points: * Lack of comparisons to baseline methods with SAM optimizers. * Some equations are not rigorously formulated. To address these concerns, we have: * Validated our method through fair comparison with SAM optimizers. * Carefully reformulated all the equations. Since your opinions on our soundness and presentation are both ''fair'', and even think our contribution is **''good''**, we are really confused that why your rating is still so negative. To give a proper judgment of this paper, we sincerely hope you can provide further opinions soon. Thank you so much! Best, Authors.
Summary
This paper introduced FlatMatch, a new SSL method which encourages consistent learning performance between labeled and unlabeled data and aims to boost the performance of SSL methods without being limited by insufficient label information. The paper analyzes the loss landscapes of labeled and unlabeled data and proposes to use a worst-case model alongside the original SSL model to achieve agreement on unlabeled data through a novel cross-sharpness regularization method. The approach is tested on 4 computer vision datasets from image classification and obtains state-of-the-art results.
Strengths
- The presentation of the approach is clear and concise. Overall, this paper is very-well written. - First, I want to say that the idea is very interesting. SSL models usually obtain perfect accuracy on labeled data very fast and it is intuitive that at this point there is not much information gain. This paper solves this drawback in a very clever way. - The performance of FlatMatch is impressive, obtaining considerable improvements especially in “harder” datasets where there was room for improvement from prior approaches. - Additionally, extensive analyses are performed, which offer valuable insights and help further understand the approach.
Weaknesses
- A notable weakness is the absence of experiments on large-scale datasets (ImageNet?). It’s clear that FlatMatch performs well on “harder” datasets, however CIFAR-100 and STL are still low-scale. I believe ImageNet experiments are vital since it’s the closest to a real-world setup and would further solidify the effectiveness of your approach. At the same time, all recent proposed SSL methods verify performance on ImageNet (100/300K). - (Minor) The training efficiency is not that good. This is okay, but the graph in Figure 7 is built around FixMatch as the most efficient method. I think this is a bit misleading. FlexMatch has the same training time with a higher accuracy compared to FixMatch; so I think FlexMatch should be the reference. Moreover, it would be interesting to also see a convergence analysis as done in FlexMatch (plot accuracy/iteration) since I believe this would give a better sense of the training efficiency compared to how the plot is structured here.
Questions
For Figures 1 and 4 last column, how do the curves look at later iterations? The plots are at epochs 60 and 150 which is at the very beginning of training (e.g. CIFAR-100 trains for 9K epochs). Also, what is the dataset used for those plots? I haven’t seen it mentioned in text.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The paper states limitations. I believe the computational cost limitation should be explicitly stated here. I don't think this work has any negative societal impact.
Thank you for your rebuttal. 1. I understand pseudo label, but I think directly setting them as the "real label" during the training is a bit strange. 2. Even your expanded labels are less than 10% of the dataset, they are much more than the true labels. (e.g., 10, 40, 250 etc.) 3. I observe that your algorithm works worse in the case when the number of true labels is small. However, I personally feel that that is the setting where semi-supervised learning is more attractive. 4. Thank you for the additional experiments! The results look strange, in the sense that the performance of other algorithms is worse, when using this fix label trick, comparing this table with Table 1 in the paper. Therefore there are essentially two questions that I have in mind: 1. Your algorithm works worse when there are fewer labels. Is this a problem for semi-supervised learning algorithm? 2. You fix label trick is a bit strange, it seems that it brings worse performance to other algorithms. If you can have detailed discussion on this point, explain why this trick is in particular useful for your method, that will be great. Therefore, I will still maintain my score right now.
Further Comments
Dear Reviewer 9MiC: We appreciate your prompt reply, **1. The strange setting for the Fix label:** Here we would like to clarify that we do not leverage the expanded labels for risk minimization. Your strange feeling is due to the misunderstanding that we might have simply created “real labels”. In fact, we only use them to compute the gradients that are utilized to perturb the model, which can stabilize the sharpness minimization process. **2&3. Performance on barely-supervised learning:** Indeed, the capability of SSL can be manifested by extremely scare labels, but we respectively disagree that an SSL method should be only judged by the performance in an extremely-limited label setting. On the other hand, as many powerful large-scale models and pre-training techniques are emerging, we think the generalization performance is one promising direction to promote the development of SSL, therefore we leverage SAM optimization to assist SSL. **4. The performance drop of SSL on Fix label setting:** The results can testify to your strange feeling about creating “real labels” for SSL. We show that following your previous misunderstanding and directly using the “real labels” is harmful. Therefore, the results are actually reasonable. Thanks again for your discussion, we hope to hear your further opinions soon. Kind regards, Authors.
I think it is important to clarify this misunderstanding. In your paper, what you wrote is: "Then, we choose the most confidence, i.e. high softmax probability, unlabeled data to be pseudo-labeled as labeled data. Different from other unlabeled data that are also trained with pseudo labels, the selected unlabeled data are fixed with their labels and completely **act as labeled data** during the rest of the training". But you just said you did not create "real labels". I am a bit confused. Can you elaborate more on this? Moreover, if this trick is solely for SAM, then how can you run experiments for other methods like flexmatch and freematch with the same trick?
Further Discussion
Dear Reviewer 9MiC: We appreciate your responsive feedback, During our experiments, we found that sufficient examples with pseudo labels can help the computational stability of the first backward propagation from SAM. However, when the same amount of pseudo labels are incorporated into the second backward propagation to optimize the network parameters, the optimization could deviate, and the performance would drop. Therefore, the optimization of FlatMatch with Fix label is formulated as: $ \nabla_{\theta}\left[L_l(\theta)+R_{\textit{X-sharp}}(\theta+\epsilon^*(\theta))\right]\approx\nabla_{\theta}L_{l} + \nabla_{\theta}R_{\textit{X-sharp}}(\theta)|_{\theta+\rho\times\textit{Perturb-from-Fix-label}}, $ $ \textit{Perturb-from-Fix-label} := \textit{sign}(\nabla_{\theta}\hat{L}_l(\theta)),$ where $L$ denotes the risk computed from the original labeled set, and $\hat{L}$ denotes the risk computed from the expanded labeled set. Our optimization effectively avoids introducing too many noises that could harm the performance. We thank you for pointing out the unclear demonstrations, we will provide detailed explanations in future versions. As for Fix label on other SSL methods, we directly use the created labels as supervision for empirical risk minimization, and it is intuitive to get degraded performance. Best, Authors.
OK, so when you compute the worse case model with parameter $\tilde{\theta}$, you are solely using the expanded labeled set. Moreover, this is the only place that you use this expanded labeled set. Is it true? That means, essentially you have three types of datasets: 1. labeled set. 2. unlabeled set. 3. Fixed pseudo labeled set using FixMatch. Notice that this set of labels are not the same as pseudo labels in other algorithms, as the real pseudo labels will change over time, but this set will be fixed during training. This set is different from labeled set, because it will only be used in computing $\tilde{\theta}$, not other places. Therefore, it is only used for "approximated guidance for finding worse case model". Is it true?
OK. That makes more sense to me. I will raise my score because the performance of the paper is good, and the algorithm is reasonable. However, I am not sure whether you have addressed other reviewers' concerns, it seems that many of them are still negative on your paper.
Reply
Dear Reviewer 9MiC: We deeply appreciate your prompt reply! It has been our pleasure to address your concerns. We believe responsive discussion is the key to solving problems and misunderstandings. Since we have tried our best to fulfill the requirements of other reviewers, and we are assured that a consensus could be achieved if the discussion keeps going on. Best, Authors.
Thank you for your detailed rebuttal. After reading the rebuttal, most of my concern has been addressed. The figures provided in the rebuttal pdf are clearer and better illustrate the motivation than those provided in the main text, and I strongly recommend that the authors revise the paper. Furthermore, despite the empirical observation that flatness measures are consistently strongly correlated with generalization, why and under what circumstances flatness is correlated is still an open theoretical question, so I would not advise the authors to conflate the two (like Q2). Overall, I decide to raise my rating from 4 to 5.
Reply
Dear Reviewer Z9Pn: Thanks for your valuable advice and quick response, There would be three major modifications in our future versions: * Detailed discussions between generalization performance and learning accuracy will be added. We will further supplement our empirical justification of flatness consistency between labeled and unlabeled sets. * The motivation of our Fix label setting will be carefully demonstrated and how it is conducted will be summarized in an Algorithm. * The formulation of our equations and unclear descriptions will be carefully improved. It is really our pleasure to address all your questions and we sincerely appreciate your raising your score. Best, Authors.
Rebuttal Summary
Dear Reviewers, AC, and SAC: We deeply thank the hard work done by AC and SAC such as assigning reviewers, guiding the process, and further organizing the discussion. We also sincerely appreciate the reviewers for taking the time to read our paper, provide constructive opinions, and get involved in our discussion. Without your elaborative help, our paper could not have been improved. Here we summarize our rebuttal to present a macro perspective which could hopefully help grasp our contribution and modification quickly. Through interactive discussion, several consensuses have been achieved: * This paper is clearly formulated and **well-written**. * Most reviewers found the idea of our paper interesting and **attention-drawing**. * Although several reviewers questioned about the evaluation scale in their initial comments, we have carefully provided results on the large-scale ImageNet dataset, ViT model which demonstrates the **extendable effectiveness** of FlatMatch. * For a fair comparison, we have extended the experiments to comparison under Fix label setting and optimization using SAM, both of which show our **design is reasonable and effective**. Although two reviewers remained silent during the discussion phase, here we review potentially remaining concerns and further stress our contribution. * Formulations that missing the optimization sign ‘’min’’: We have carefully checked all our equations and made sure they are rigorously formulated. However, we want to emphasize that such an issue has **no influence on our ideology, methodology, technical implementation, and empirical validation**. * The efficiency of FlatMatch: The computational cost is mostly due to our implementation of sharpness using SAM. * At the methodology level, we can take advantage of the development from existing and future SAM-related approaches [1, 2, 3, 4] and improve the efficiency of our method. Therefore, **challenging the efficiency of FlatMatch is challenging the efficiency of the chosen sharpness method**, which is orthogonal to our major contributions. * Moreover, we have provided a feasible way to reduce the computation cost by using historical gradients. As the result in Fig. 1 from the rebuttal pdf, our computational cost is comparable to FreeMatch but with much **higher performance**, and we can achieve **higher performance with lower computation cost** than FlexMatch. (The two methods are very typical baseline methods.). * Though having solved during the discussion, here we again discuss our Fix label setting which is used to improve the performance when labels are extremely scarce: * The limited performance under barely supervised learning is caused by the nature of SAM. When the number of examples for computing sharpness is extremely limited, the gradient would not accurately estimate the true optimizing direction [5]. Therefore, the sharpness might not be effective. * Therefore, we only need to leverage extra labeled data to stabilize the gradient estimation from the first backpropagation, and our sharpness can be effectively improved. As for risk minimization from the second backpropagation for SSL, we can just use the original labeled and unlabeled set. [1] Du et al., Efficient sharpness-aware minimization for improved training of neural networks, in ICLR 2022. [2] Mi et al., Make sharpness-aware minimization stronger: A sparsified perturbation approach, in NeurIPS 2022. [3] Du et al., Sharpness-aware training for free, in NeurIPS 2022. [4] Zhong et al., Improving sharpness-aware minimization with fisher mask for better generalization on language models, in EMNLP 2022. [5] Zhang et al., Solving large scale linear prediction problems using stochastic gradient descent algorithms, in ICML 2004. In the past several months, we have tried our best to improve the quality of this paper and address each concern from all reviewers. We sincerely hope our effort could be contributed to the community and advance the development of SSL. Thanks again for your kind help and constructive opinions, we will keep trying our best to polish this paper in the future. Sincerely, Authors.
Decision
Accept (poster)