Weight Average (WA) is an active research topic due to its simplicity in ensembling deep networks and the effectiveness in promoting generalization. Existing weight average approaches, however, are often carried out along only one training trajectory in a post-hoc manner (i.e., the weights are averaged after the entire training process is finished), which significantly degrades the diversity between networks and thus impairs the effectiveness. In this paper, inspired by weight average, we propose Lookaround, a straightforward yet effective SGD-based optimizer leading to flatter minima with better generalization. Specifically, Lookaround iterates two steps during the whole training period: the around step and the average step. In each iteration, 1) the around step starts from a common point and trains multiple networks simultaneously, each on transformed data by a different data augmentation, and 2) the average step averages these trained networks to get the averaged network, which serves as the starting point for the next iteration. The around step improves the functionality diversity while the average step guarantees the weight locality of these networks during the whole training, which is essential for WA to work. We theoretically explain the superiority of Lookaround by convergence analysis, and make extensive experiments to evaluate Lookaround on popular benchmarks including CIFAR and ImageNet with both CNNs and ViTs, demonstrating clear superiority over state-of-the-arts. Our code is available at https://github.com/Ardcy/Lookaround.
Paper
References (52)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
The paper presents Lookaround, a novel optimizer for weight average ensembling (WA). Unlike existing approaches that perform weight averaging post-training, Lookaround adopts a two-step process throughout the training period. In each iteration, the "around" step trains multiple networks simultaneously on transformed data using different augmentations, while the "average" step combines these networks to obtain an averaged network as the starting point for the next iteration.The approach demonstrates clear superiority over state-of-the-art methods in extensive experiments on CIFAR and ImageNet datasets using both traditional CNNs and Vision Transformers. The paper provides theoretical justification and commits to open science by making the code publicly available.
Strengths
1. `Innovative Approach`: The introduction of Lookaround as a straightforward and effective optimizer for weight average ensembling brings a novel perspective to the field. The two-step process during training enhances network diversity and preserves weight locality, addressing the limitations of post-hoc weight averaging approaches. 2. `Theoretical Justification`: The paper offers strong theoretical support for the superiority of Lookaround through convergence analysis. Proposition 1 provides insights into the convergence variance of Lookaround in comparison to typical SGD and Lookahead optimizers. This rigorous theoretical analysis enhances the credibility and significance of the proposed method, demonstrating a solid foundation for its effectiveness and performance. 3. `Extensive Experimental Validation`: The authors conduct extensive experiments on popular benchmarks, including CIFAR and ImageNet, with both traditional CNNs and Vision Transformers (ViTs). The clear superiority of Lookaround over state-of-the-art methods on these datasets demonstrates its effectiveness and applicability. 4. `Commitment to Open Science`: The authors state their intention to make the code publicly available, fostering reproducibility and enabling further research in the field.
Weaknesses
1. `Computational Complexity and Comparison`: The paper lacks a comprehensive discussion of the computational complexity introduced by the Lookaround optimizer. As Lookaround incorporates an additional around step during training, it is crucial to assess its computational requirements and compare them to existing optimization methods. A detailed analysis of the computational trade-offs, including runtime and memory usage, would provide a more comprehensive understanding of Lookaround's practical applicability and scalability. 2. `Limited Experiments on Datasets Beyond Image Classification`: The paper primarily focuses on evaluating Lookaround on image classification tasks using datasets like CIFAR and ImageNet. However, to establish the versatility and effectiveness of Lookaround, it is important to explore its performance on datasets beyond image classification. Conducting experiments on diverse tasks such as language modeling or segmentation/detection tasks would demonstrate the generalizability of Lookaround across various domains and provide a more comprehensive evaluation of its performance.
Questions
1. With the Lookaround optimizer requiring k around steps and 1 average step, I am unclear about how the epoch and iteration are defined in this setting. Are they counted based on the average step or the around step? Could you provide clarification on how the epoch and iteration are structured in Lookaround? 2. Considering that each batch is augmented k times in Lookaround, I am curious to understand how this approach differs from simply applying `repeated augmentations`[A] to each batch. What distinguishes the effect of k augmentations and 1 step update in Lookaround versus performing each step update individually and then averaging all checkpoints? In other words, what advantages does Lookaround's approach provide in terms of diversity and ensemble performance compared to the alternative method of averaging checkpoints from individual updates? [A] Hoffer E, Ben-Nun T, Hubara I, et al. Augment your batch: Improving generalization through instance repetition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 8129-8138.
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
As discussed in the limitation section "additional cost of network searching using different data augmentation, resulting in a longer training time proportional to the the number of trained networks", the running time is a important concern.
Summary
This work provides a new optimizer, "Lookaround optimizer," which is built upon a previous proposed "Lookahead optimizer" [40]. By incorporating data augmentation, this work shows an improved convergence rate under low condition numbers. It also empirically shows some improvement in classification accuracy under several datasets.
Strengths
This work proposes a new optimizer and shows consistent (despite little) improvement in various settings. The supplementary material shows a solid derivation.
Weaknesses
1. The major concern lies in the limited improvement over the "Lookahead optimizer." The improvement in Figure 2 and Table 1-3 all seems to be subtle. Since the proposed methods see more augmented data in each epoch than others, it is questionable whether the improvement actually comes from these additionally augmented data. Also, the idea itself is almost identical to the Lookahead optimizer, so the novelty is rather limited. 2. Minor mistakes: In table 1, row CIFAR 10, column ResNext50 Top5 results, the best results happen in Lookahead (99.96), not the proposed method (99.95).
Questions
What is the d (number of data augmentation) used in the experiment? I cannot find it in the context.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
3 good
Contribution
2 fair
Limitations
The author addresses that the data augmentation process consumes extra time, which is likely to be the main limitation.
Summary
This paper introduces a new optimization algorithm named Lookaround, which draws inspiration from the recent achievements of weight averaging (WA) techniques in deep learning. The proposed Lookaround optimizer looks around nearby points by performing multiple gradient computations for a given training input using different data augmentations and averages them to obtain better generalizing solutions.
Strengths
The theoretical analysis in this paper largely relies on Zhang et al. (2019), including noisy quadratic analysis and deterministic quadratic convergence. While it does not introduce a novel form of analysis, it is significant as it establishes a theoretical foundation within the existing framework. For experiments, the baseline comprises commonly employed optimization techniques, such as SWA (Izmailov et al., 2018) and SAM (Foret et al., 2021), which are widely accepted as standard approaches in the field. In addition to convolutional neural networks of various scales, the authors also considered experiments on vision transformers.
Weaknesses
Experimental issues: * __There are no error bars.__ While the authors answered "yes" for "Error Bars", the paper only provide a single value across the tables and figures. It is unclear how many experiments were conducted to derive those values. It would be preferable if the authors also included averages accompanied by standard deviations to provide a more comprehensive representation of the experimental results. * __The outcomes obtained from the ImageNet experiments appear to be strange.__ Despite the authors' efforts to demonstrate the scalability of the proposed algorithm, the Top-1 accuracy of 72.27% reported in Table 2 appears to be comparatively low. After reviewing Appendix C.1.2, it seems that the experimental setup follows the PyTorch convention (https://github.com/pytorch/examples/tree/main/imagenet), except for some additional augmentations. It is widely recognized that the ResNet-50 model typically achieves an accuracy of approximately 76% on the ImageNet dataset using this standard setup, which significantly differs from the 72.27% accuracy reported by the authors. It would be beneficial to provide clarification on the reasons behind the considerable performance drop observed in the ImageNet results. Practical issues: * __Excessive training costs incurred by the proposed algorithm.__ The Lookaround algorithm, as proposed, demands $d$ times the number of forward and backward passes for each optimization step. This is considerably higher compared to SAM (Foret et al., 2021), which only requires twice the number of passes. One could argue that the training epoch is effectively enlarged by a factor of $d$ and this is the actual reason for the performance improvements. It would be valuable to present results akin to Table 2 in Foret et al. (2021), that is, exploring the impact of increasing the number of training epochs through an ablation study.
Questions
There are several issues in the experimental results of the current version of the paper as I mentioned above, and I would respectfully suggest the followings that could make the paper solid: * __The absence of SAM in Table 4.__ Is there a specific reason for excluding SAM from the results presented in Table 4? * __The disparity in results for ResNet50 on CIFAR100 between Tables 1 and 6.__ Table 1 presents Top-1/5 accuracies of 81.60/95.99, while these values are not reflected in Table 6. * __Top-5 accuracy does not provide a meaningful comparison in the main tables.__ It is suggested to exclude the Top-5 accuracy metric from the main results presented in Tables 1 and 4 for CIFAR10/100. Specifically, in the case of CIFAR-10, evaluation metrics where all techniques achieve 99.9% accuracy hold little significance. Instead, including the Top-5 accuracy in Table 2 would be more appropriate, as it provides a more meaningful performance evaluation for the ImageNet experiments. * __It would be nice to provide uncertainty metrics.__ While the Top-1 accuracy is the main evaluation metric for classification, it would be advantageous to incorporate additional metrics that can effectively demonstrate the superiority of the proposed algorithm. For instance, including the negative log-likelihood (NLL) as a measure of in-domain uncertainty would provide valuable insights into the algorithm's performance. * __It would be beneficial to offer an analysis of the training costs involved.__ Several algorithms, including the proposed one, impose additional computational burdens, which could impact their practicality (e.g., SAM requires performing double forward and backward passes; SWA requires an additional model copy in memory). Providing detailed information about the specific computational requirements of each algorithm would greatly benefit future researchers and engineers. The most straightforward analysis is measuring wall-clock time for training. It would be nice to replace the meaningless "#param." column with the training runtime in Table 4.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
1 poor
Presentation
2 fair
Contribution
2 fair
Limitations
The authors stated the additional training cost incurred by the proposed algorithm. --- __References:__ Zhang et al. (2019). _Lookahead optimizer: k steps forward, 1 step back_. Izmailov et al. (2018). _Averaging weights leads to wider optima and better generalization_. Foret et al. (2021). _Sharpness-aware minimization for efficiently improving generalization_.
Summary
This paper proposes Lookaround, a new optimization method that incorporates weight averaging into the optimization process. The algorithm consists of two steps: 1) the around step launches several parallel runs of gradient descent led by different data augmentations, 2) the average step does weight averaging of the networks obtained by these parallel runs. These two steps are repeated along the whole training process, with the result of the average step being a starting point for the next around step. The proposed method is similar to Stochastic Weight Averaging and Model Soups, though it does the averaging during training rather than at the end of training. Lookaround shows strong results compared to the existing optimization methods (SGD with momentum, AdamW, Lookahead, SAM).
Strengths
1. The idea of using weight averaging during optimization with various augmentations is novel and leads to a better generalization of neural networks. 2. This paper has a theoretical analysis of the quadratic noise setup which demonstrates faster convergence of Lookaround compared to SGD and Lookahead.
Weaknesses
Despite the fact that the paper proposes a novel and interesting method, in my opinion, it is not quite ready for publishing in its current form. I believe that addressing the following concerns could help to make it a much stronger submission. 1. The text of the paper is hard to follow, contains some vague parts and numerous typos. - The theoretical reasoning is heavily inspired by the Lookahead paper, which makes it impossible to understand the theory without reading the original paper (e.g., I could not understand Proposition 1 and the definition of $\alpha$ without reading the Lookahead paper). Moreover, there is no remark that equations 4 and 5 are derived in the Lookahead paper. - Line 5-6: weight averaging and ensembles are mixed up. - Line 313: it is not clear how gradient boosting is related to this setup. 2. I have found two contemporaneous works published on arxiv (https://arxiv.org/abs/2302.14685, https://arxiv.org/abs/2304.03094) that propose approaches very similar to Lookaround. I believe they need to be at least discussed in the related work section. 3. The experimental part of the paper raises some questions and is not convincing enough, in my opinion. - The main results (Table 1) lack standard deviations, even though they are claimed in the OpenReview form. - The baselines seem too weak, i.e., the Lookahead paper reports >75% ImageNet test accuracy on ResNet-50 for SGD baseline, while this paper shows 72.27%. - The augmentation policy during baseline training is not clear. If the same set of augmentations as for Lookaround is used, then this may be the reason for the bad quality of the baselines. - Despite the paper proposing a new optimization method, it lacks experiments on a wider choice of architectures. Table 1 illustrates only VGG-19 and four networks from the ResNet family. It would be beneficial to add experiments on more modern convolutional architectures and more extensive experiments on image transformers. - The paper lacks a fair comparison to logit ensembles. If I understood correctly, different models of the ensemble are trained with different Lookaround augmentations. This leads to the suboptimal quality of each model and, as a result, to a suboptimal ensemble. - The comparison to model soups in Appendix D is not fair as well, i. e., the original paper fine-tunes models with various hyperparameters and does a greedy search for the combination with the best validation accuracy. Near zero accuracy of the average of $\theta_1$ and $\theta_2$ may be due to improper fine-tuning hyperparameters. Moreover, this is an important baseline, and it would be better to move this comparison to the main part of the paper. - A proper ablation study of augmentations is required. Is it possible to train Lookaround without augmentations for the around step “branches” (e.g., the only source of randomness is batch ordering)? What if each branch utilizes the same set of augmentations? Why are the mentioned augmentations used (and not some other ones, e. g. vertical flip seems to be a strange augmentation)? Probably, some of these setups are covered in Section 4.4, but it is not clear from the text. - Top-5 accuracy is redundant and makes it difficult to read the tables. 4. Minor issues/typos: - Line 25: the sentence is about LMC, but the citation [7] leads to the paper about permutations (Entezari et al., 2022). - Line 112-113: we provide - Line 147-148: repetition - Line 156: we analyze - Line 162: it seems that it should be $\mathbb{E} [c_i^2]$ - Line 242: CIFAR
Questions
1. What is the augmentation policy during baseline training? 2. I do not understand the experimental setup in the ablation study (Section 4.4). Is the training without DA and WA similar to the regular training without augmentations? Is the training without DA but with WA similar to Lookaround, where the ordering of batches is the only source of randomness between branches? 3. Section 3.2.2, convergence on deterministic quadratic functions. How branches of Lookaround are different from each other if there is no randomness in gradient steps? 4. Could the authors elaborate on how the Lookaround method relates to the methods proposed in the contemporaneous works (https://arxiv.org/abs/2302.14685, https://arxiv.org/abs/2304.03094)?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
The main limitation of the proposed method is the increased training budget, which is highlighted by the authors. However, there is no fair comparison to networks trained for a larger number of epochs.
Dear reviewer, We sincerely appreciate the questions you posed during the initial rebuttal phase. In response to your first rebuttal, we have clarified that our experiments were conducted with strict fairness and will provide a more detailed explanation of the theory in the revised version, addressing your concerns with care. As we near the midpoint of the author-reviewer discussion stage, we look forward to receiving more feedback and engaging in productive conversations with you. Best regards, The authors of Lookaround
Thank you for the response and additional clarifications. Could you please elaborate more on the following points: 1. **ImageNet experiments.** Could you please provide results for the baselines other than SGDM for the ImageNet experiments without warmup? 2. **Model Soups.** Your results for model soups do not match the original model soups paper and the DiWA paper [1] (and also my own experience): uniform model soups should work much better than random. Could you please explain what are the reasons for such different behavior in your opinion? I suspect the reason is the poor choice of the fine-tuning procedure, which goes too far from the pre-trained checkpoint. Moreover, this procedure seems to lose the benefits of pre-training since it shows much worse results than the ones in the literature (84.5% in [2] and 86.4% in [3] v.s. yours 82%). Also, did you update batch norm parameters for model soups? 3. **Logit Ensembles.** Your results for logit ensembles are quite surprising, hence, I am not fully convinced by the experiments on one dataset. Did you make similar experiments on other datasets? Did you try to compare Lookaround to an ensemble of SWA models? [1] Ramé et. al. Diverse Weight Averaging for Out-of-Distribution Generalization. NeurIPS 2022. \ [2] Kornblith at. al. Do Better ImageNet Models Transfer Better? CVPR 2019. \ [3] Grill et. al. Bootstrap Your Own Latent A New Approach to Self-Supervised Learning. NeurIPS 2020.
Reply to the second comment [1/2]
We appreciate your feedback on our rebuttal. Please find our responses to your questions below. > Could you please provide results for the baselines other than SGDM for the ImageNet experiments without warmup? Thanks for your comment. We have conducted comprehensive experiments on ImageNet, comparing the performance of SWA, SGDM, Lookahead, and Lookaround. The results, as presented in Table S3, again demonstrate that Lookaround achieves notable performance improvements when no warm-up setting is employed. Table S3: Comparison between SGDM, SWA, Lookahead and Lookaround with ResNet50 on ImageNet. Results of SWA$^+$, Lookahead$^+$ are taken from the origin paper. | Method | Warm Up | Top-1(%) | Top-5(%) | | :--------: | :------: | :------: | :------: | |SGDM| ✗ |75.97 |92.89| |SWA| ✗ |76.78 |93.18| |SWA$^+$| ✗ |76.97 |-| |Lookahead| ✗ |76.52 |93.11| |Lookahead$^+$| ✗ |75.49 |-| |**Lookaround** | ✗ |**77.32**|**93.29**| > Your results for model soups do not match the original model soups paper and the DiWA paper [1] (and also my own experience): uniform model soups should work much better than random. Here we would like to remind the reviewer that uniform soups do not always improve model performance, which is also acknowledged in the original Model Soups paper (Please refer to Table J.1 in the original paper). It is mentioned in Appendix J.1 of Model Soups [1] that the learning rate lower than 1e-4 is easy to lead to the failure of weight average. This can be a reason for the failure of uniform soups under SGDM and the success of AdamW. However, as an optimizer that is sensitive to learning rates, SGDM's fine tuning performance is very poor at low learning rates. Thus we don't get a well-behaved uniform soups. > Could you please explain what are the reasons for such different behavior in your opinion? I suspect the reason is the poor choice of the fine-tuning procedure, which goes too far from the pre-trained checkpoint. The success of uniform soups in Model Soups heavily relies on stringent conditions. Specifically, the individual sub-models within uniform soups must remain within the same low-loss basin to ensure effective weight averaging and maintain high accuracy. However, when an excessive number of models is utilized, and in the presence of varying hyperparameter configurations, the use of data augmentation as a strong hyperparameter change can easily cause certain models to deviate and enter different low-loss basins, resulting in the failure of weight averaging. Consequently, achieving success with uniform soups requires careful adjustment of hyperparameter differences. This is precisely where the significance of our work lies. With the proposed Lookaround method, the weights are continuously averaged throughout the training process, ensuring their alignment within the same loss basin and convergence towards flatter minima. > Moreover, this procedure seems to lose the benefits of pre-training since it shows much worse results than the ones in the literature (84.5% in [2] and 86.4% in [3] v.s. yours 82%) Here we would like to, again, remind the reviewer that Lookaround is proposed for a quite different problem setting compared to Model Soups. Model Soups is designed for more efficient utilization of existing models, assuming that these models have already been pre-trained on specific datasets. Lookaround, on the other hand, serves as an optimizer for training deep models, devoid of any assumptions regarding the initialization of the model parameters. Regarding the difference between our ResNet50 results on CIFAR100 and the results in [2] and [3], the reason is that both [2] and [3] conducted about 50 grid searches on learning rate and weight decay for the experiment, while we didn't. > Also, did you update batch norm parameters for model soups? Following the setting in the original Model Soups paper and code, we did not make additional updates to the batch norm parameters.
Reply to the second comment [2/2]
> Your results for logit ensembles are quite surprising, hence, I am not fully convinced by the experiments on one dataset. Did you make similar experiments on other datasets? Did you try to compare Lookaround to an ensemble of SWA models? We conducted a full experimental validation of Logit Ensemble and SWA under a new dataset Stanford Cars, as shown in Table S4. Under the Stanford Cars dataset, Lookaround still performs better than Logit Ensemble. Additionally, adding SWA into the training process of Lookaround can further improve performance. Under the limit of single-model inference, Lookaround+SWA achieves a Top-1 accuracy of 91.04%, exceeding the optimal 88.11% (SWA with AutoAugment). Table S4: Comparison between different training strategies with ResNet18 on Stanford Cars dataset. Here "H" denotes random horizontal flip data augmentation. "R" denotes RandAugment data augmentation. "A" denotes AutoAugment data augmentation. "Logit Ensemble + SWA" denotes a logit ensemble approach that employs SWA models, which have been acquired through various data augmentation techniques. | Method | Data Augmentation Strategy | Top-1 Acc | Top-5 Acc | Training Budget | Inference Budget | | :----------------: | :------------------------: | :-------: | :-------: | :-------------: | :--------------: | | SGDM | "H" | 86.77 | 97.52 | 1x | 1x | | SWA | "H" | 87.60 | 98.21 | 1x | 1x | | SGDM | "R" | 87.02 | 98.01 | 1x | 1x | | SWA | "R" | 88.00 | 98.21 | 1x | 1x | | SGDM | "A" | 87.63 | 98.05 | 1x | 1x | | SWA | "A" | 88.11 | 98.15 | 1x | 1x | | Logit Ensemble | "H" + "R" | 89.24 | 98.44 | 2x | 2x | | Lookaround | "H" + "R" | 89.35 | 98.47 | 2x | 1x | | Logit Ensemble | "H" + "R" + "A" | 90.19 | 98.67 | 3x | 3x | | Lookaround | "H" + "R" + "A" | 90.76 | 98.78 | 3x | 1x | | Logit Ensemble+SWA | "H" + "R" + "A" | 90.39 | 98.80 | 3x | 3x | | **Lookaround+SWA** | **"H" + "R" + "A"** | **91.04** | **98.86** | **3x** | **1x** | [1] Wortsman M, Ilharco G, Gadre S Y, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. [2] Kornblith at. al. Do Better ImageNet Models Transfer Better? CVPR 2019. [3] Grill et. al. Bootstrap Your Own Latent A New Approach to Self-Supervised Learning. NeurIPS 2020.
Taking into account additional experimental results provided during the rebuttal, I've increased my score to reflect my increased confidence in the effectiveness of the proposed method. However, I still have a significant concern regarding the absence of adequate comparison with using the branching and averaging only at the end of training (as was done in model soups, for example). I think such a baseline is important since the proposed Lookaround algorithm is generally based on two ideas: 1) averaging during training helps (as was shown in Lookahead), and 2) averaging several independently trained branches works better than averaging several checkpoints from the same training trajectory (as was shown in model soups and DiWA). The experiments in the paper show that changing the averaging strategy helps (comparison with Lookahead) but do not show that using branching and averaging through the whole training process is better than just using it at the end of training. Ideally, I think three experiments are needed to show that: * Comparison with model soups in the transfer learning setup. I do not find current results convincing because of the low quality of baselines and random quality of uniform model soups. Appropriate training hyperparameters should be chosen for both SGDM and model soups baselines. Now both methods clearly underperform because of the sub-optimal fine-tuning procedure. Also, batch norm statistics (running mean and variance) should always be recomputed after model averaging (see SWA [1] and REPAIR [2]). The authors of the model soups did not do it because they used networks with layer norm, which does not have these statistics. I did some experiments with averaging several fine-tuned ResNet-50 on CIFAR-100 (standard PyTorch supervised pre-training on ImageNet and standard fine-tuning procedure from BYOL): even though uniform model soups do not always improve the results compared to one fine-tuned model, their predictions are much better than random after updating batch norm statistics (~84-84,5% both for individual models and uniform model soups of 5 models, which is higher than the results of Lookaround provided in the paper). * Ablation study where models are trained together at first, and the branching and averaging strategy is used only after some epoch N. If the averaging is only used once at the end of the training, that would be an analog of model soups for a non-transfer learning setup. * Comparison to DART [3]. Lookaround is extremely close to DART, which was published almost three months before the NeurIPS submission deadline. Hence, formally, the authors should provide an experimental comparison of the methods. Given the closeness of the DART release date to the two-month prior mark, I would not see that as a critical issue if the first two experiments were provided in the paper. However, in their absence, I can not be sure that the proposed idea of using branching and averaging through the whole training procedure works better than DART and hence have concerns regarding the novelty of the work. Generally, I still have concerns about the quality of the baselines (both ResNet-50 and VIT baselines in Table 4 significantly underperform, see, e.g., Table 10 in [4]), which I believe is an important issue for the paper proposing a new training technique. Also, I think batch norm statistics recomputation should be taken into account not only in the baselines but in the proposed method itself. The new baselines on ImageNet look much better! Regarding the ensemble experiments, I would recommend broadening this part of the paper and adding even more experiments since the current results look promising. [1] Garipov et.al. Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs. NeurIPS 2018.\ [2] Jordan et.al. REPAIR: REnormalizing Permuted Activations for Interpolation Repair. ICLR 2023.\ [3] Jain et.al. DART: Diversify-Aggregate-Repeat Training Improves Generalization of Neural Networks. CVPR 2023.\ [4] Chen et.al. When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations. ICLR 2022.
Reply to the third comment.
Thanks for your constructive comments. We sincerely hope our responses below fully address your questions. > Comparison with Model Soups in the transfer learning setup. Based on the pre-training settings given in Table-9 of reference paper [1], we re-conducted the Model Soups experiment, and the results are shown in Table S5. Specifically, we selected a more suitable learning rate and a Cosine learning rate scheduler. Thank you for introducing the "Update Batch Normalization Parameters" technique. We found it to be highly beneficial for Model Soups when combining multiple models. Furthermore, we conducted Lookaround experiments under new hyperparameters. We discovered that Lookaround with only three augmentation methods surpasses the accuracy of Uniform Soups under standard configurations, but still falls short of Uniform Soups$^+$ and Greedy Soups. Table S5: Comparisons between Lookaround, Model Soups on ResNet50. "M=18" represents 18 different hyperparameters. "M=3" represents that Lookaround is trained using 3 different data augmentation techniques. 18 different configurations come from the combinations of three data augmentations (RandomHorizontalFlip, RandAugment, AutoAugment), three initial learning rates: $\{0.01, 0.003, 0.001\}$ and label smooth or not. Unifrom Soups$^+$ represents that we use the top-10 accuracy models for uniform soups (the accuracy of these models ranges from 83.91% to 84.64%). || Update BN | Top-1 Acc | Top-5 Acc | | :---------------: | :-------: | :-------: | :-------: | |Unifrom Soups (M=18) |✗ |82.93| 97.44| |Unifrom Soups (M=18) |✓ |84.53| 98.10| |Unifrom Soups$^+$ (M=18) |✗|84.60|97.81| |Unifrom Soups$^+$ (M=18) |✓|85.57|98.04| |Greedy Soups (M=18) |✗|85.52|98.02| |Greedy Soups (M=18) |✓|86.09|98.11| | Lookaround (M=3)|-| 85.20|97.15| > Ablation study where models are trained together at first, and the branching and averaging strategy is used only after some epoch N. We conducted this experiment using ResNet50 on the CIFAR100 dataset. Unlike DART (where, during the actual training process of the Lookaround optimizer, different branches share the mean and variance of the Batch Normalizaition layer, differing from DART), we observed that the earlier the average step is executed, the better the final model's performance. Table S6: The impact of initiating the "average step" at different epochs on the performance of Lookaround with ResNet50 on CIFAR100. |Start Epoch|0| 40|80|120 | 160| | :---------:|:---:| :---:| :---:| :---: | :---: | |Top-1| 81.60 | 80.40 | 80.09 | 78.97 | 77.35 | |Top-5| 95.99 | 95.53 | 95.30 | 94.87 | 94.30 | > Comparison to DART. We compared Lookaround with DART with the same data augmentation, the results are shown in Table S7. And our results were better than DART, the reason behind this may come from the fact that we used more frequent "average steps" during training and started "average steps" earlier in the cycle. More frequent "average steps" are more conducive to the training of Lookaround, which is introduced and described in Figure 6 in our paper. Table S7: Comparison with the DART method with ResNet18 on CIFAR10 and CIFAR100. The results of DART$^+$ are taken from the original paper. | | CIFAR10 | CIFAR100 | |:--------:|:-----:|:------:| |DART$^+$|97.14|82.89| |Lookaround |97.22|83.16| > Quality concerns regarding the baselines (both ResNet-50 and ViT) in Table 4. We apologize that, in the early stages of the experiment, we focused too much on comparing different methods and overlooked the importance of establishing a good baseline for the pre-training process. Therefore, we restarted the pre-training experiment based on the settings in [1] and [2], the results are shown on Table S8. In our comparison experiments with the new baseline, we found that the results of ViT-B/16 under different optimizers were better than those reported in the original paper[2]. This is mainly due to our use of more data augmentation during the training process, aiming for a fair comparison with Lookaround. This result also indicates that Lookaround can still achieve better performance under improved baseline conditions. Table S8: The test set accuracy under the training procedure with pre-trained models using ImageNet-1k weights. The results of ViT-B/16$^+$ are taken from [2]. | Backbone | Method | CIFAR10 | CIFAR100 | | :------: | :--------: | :-----: | :------: | |ResNet50|SGDM|97.55|84.50| |ResNet50|SWA |97.48|84.81| |ResNet50|Lookahead|97.65 |84.78| |ResNet50|**Lookaround** |**97.82**|**85.20**| | ViT-B/16 | Adam | 98.34 | 91.55| | ViT-B/16$^+$ | Adam| 98.13 | 87.13 | | ViT-B/16 |SWA| 98.47 | 91.32| | ViT-B/16 | Lookahead |98.51| 91.76 | | ViT-B/16 | **Lookaround** |**98.71**| **92.21**| [1] When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations. [2] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.
Dear reviewer, We have conducted new ablation and finetuning experiments and compared our method with both Model Soups and DART. We look forward to your prompt reevaluation and response, as there remains but a scant half-day in the rebuttal stage. Thank you once again for your ongoing suggestions and concerns during the rebuttal phase, which will help make Lookaround more reliable. Best regards, The authors of Lookaround
Now most of my concerns are addressed, hence, I have decided to raise my score further. After the rebuttal and discussions, the paper will need to be seriously revised to make the theoretical part more clear and to include all the new experimental results, therefore, I keep the score at borderline acceptance.
Summary
This article introduces the Lookaround Optimizer, a novel optimization algorithm for deep neural networks. The Lookaround Optimizer is based on the idea of lookaround, which involves maintaining two sets of weights. The first set of weights is updated using the standard gradient descent algorithm, while the second set of weights is updated using the averaged gradients of the first weight. The Lookaround Optimizer has been shown to improve the generalization performance of deep neural networks, and it outperforms other state-of-the-art optimization algorithms on a variety of benchmark datasets. The authors also provide theoretical analysis of the Lookaround Optimizer. Overall, the Lookaround Optimizer is a promising approach for improving the training of deep neural networks.
Strengths
- Lookaound Optimizer performs better than other state-of-the-art optimization algorithms (Lookahead, SAM) on multiple benchmark datasets. - The proposed optimizer seems to be model-free and can be applied to various computer vision scenarios. - The paper is well-written and easy to understand.
Weaknesses
- While the authors have demonstrated fascinating performance on benchmarks, the technical innovation of this paper is limited. Similar ideas that utilize multiple models have already in proposed in varioius area, especially in meta learning. - With that being said, while the authors have mentioned the connections to these related topics, they did not compare with some of them. For example, I think at least the authors should compare the performance of Lookaround Optimizer with Model Soups (i.e., train multiple copies of models using the same initialization, the same order of data, but with different augmentation), and other model merging techniques like Model Ratatouille. - The training time will be a bottleneck for applying this method as the authors acknowledged. - Another limitation is that, the augmentation seems to be a must for this method. Therefore, it seems that this method is not applicable (at least not straightforward) when we try to apply it on other modality, such as language and speech.
Questions
See above
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The authors have addressed the limitation.
Dear reviewer, We are glad that the reviewer appreciates our attempt, and sincerely thank the reviewer for the constructive comments. As suggested, we have elucidated the distinctions between Lookaround and meta-learning approaches and have conducted an exhaustive comparison of Lookaround with other methods such as Model Soups and Model Ratatouille. Since two-thirds of the allocated time for reviewer-author discussion has already elapsed, we sincerely look forward to your reevaluation of our work and would very appreciate it if you could raise your score to boost our chance of more exposure to the community. Thank you very much! Best regards, The authors of Lookaround
Acknowledgement
Thanks for the detailed responses! I have looked in to the responses and I think that they have addressed my concerns. However, I have to admit that I might have overlooked some performance gap regarding the baselines. Therefore, I will increase my score and lower my confidence given this controversial situation.
Summary
Flatness-aware optimizers have gained significant attention in the field of research for training deep neural networks that are robust. Weight Averaging (WA) is a popular approach to finding solutions within the flat regions of the loss surface. However, previous WA methods have two limitations. First, when WA is performed within a single optimization trajectory after training convergence, resulting in limited functional diversity among the averaged members. Second, when WA is performed across diverse modes, the averaged weights may negatively impact performance. To address these challenges, this paper introduces a new method called Lookaround Optimizer, which aims to overcome these limitations.
Strengths
Originality - Proposed optimizer seems to be relatively simple and straightforward but still attractive. - This article presents a theoretical analysis regarding the variance of the steady state. Clarity - The paper is written effectively, ensuring high accessibility for readers. - Methods are simple and easy to follow.
Weaknesses
Experiments - Regarding the experiments conducted, the authors acknowledged in the Limitation section that their proposed method incurs additional training costs (I think the proposed optimizer requires nearly $d$ times more computational resources when averaging $d$ models). Therefore, for a fair comparison, it would be necessary for other baseline methods to undergo additional training epochs as well. However, it appears that the authors used the same number of training epochs for all the experiments. - The overall results of the experiments are rather counterintuitive. Particularly in Table 3, the reported accuracy of the Deep Ensemble method being lower than the accuracy of a single solution trained using the proposed method is quite unexpected. Similarly, in Table 1, the reported performance of flatness-aware optimizers such as SWA [1] and SAM [2] being lower than SGDM contradicts the knowledge and expectations of the research community. References [1] Pavel Izmailov, D. A. Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. Conference on Uncertainty in Artificial Intelligence, 2018. [2] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. International Conference on Learning Representations, 2021.
Questions
Experiments - The response regarding concerns addressed in the Weakness section will be critical to the final decision.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
3 good
Limitations
Limitations are adequately addressed in the Limitation section.
Thank you for the author's effort. > __Q2: The outcomes obtained from the ImageNet experiments appear to be strange.__ > > For the sake of fair comparisons in training data and computation cost, each method (including Lookaround and the competitors) was trained with the same data augmentations (random horizontal flip, random vertical flip, and RandAugment). With such a setup, we guarantee consistency in the data volume utilized by each method, thereby ensuring fair comparisons in terms of computation. The divergence in performance observed between Lookahead, as reported in the original paper and the current study, can be attributed to the differing data augmentation techniques employed. The checkpoint `ResNet50_Weights.IMAGENET1K_V1` from torchvision indicates a 76.13% accuracy using basic training receipt involving the SGD optimizer. However, the performance detailed in the current manuscript is significantly below expectations, despite using nearly identical settings (90 training epochs; MultiStepLR scheduler with decay steps at 30th and 60th epochs and a decay factor of 0.1; initial learning rate of 0.1 and a batch size of 256). This substantial difference in results might arise from flaws in the experimental configuration, raising concerns about the reliability of the experimental outcomes presented in the paper. Consequently, additional clarifications of why it happens are imperative to address this issue.
Dear Reviewer, we sincerely appreciate your prompt feedback on our rebuttal, and we deeply apologize for the delay in our response, which was due to the extensive clarification experiments conducted on Imagenet. We would now like to provide further clarification as follows. As we have previously mentioned in our rebuttal, in order to ensure fair comparisons in terms of computation, we have employed the same multi-augmentation techniques for all methods. Additionally, we have incorporated a one-epoch warm-up phase, as detailed in Appendix C.1.2, for evaluating all methods in our experiments. This warm-up phase has proven effective in all our CIFAR experiments, and thus, we maintained its inclusion in the ImageNet experiments. However, after conducting extensive and rigorous experiments, we discovered that the warm-up phase did not yield optimal results when combined with multi-augmentation on ImageNet. Detailed results can be found in Table S7. Consequently, by removing the warm-up phase, all our experiments on the competitors achieved performance comparable to the results published in prior works. Notably, the proposed Lookaround once again outperformed these baseline methods. We sincerely apologize for any confusion caused by our experimental results. We firmly believe that a superior optimizer should consistently yield improved results across various settings, rather than being limited to specific settings. The proposed Lookaround demonstrates superior performance in both our original settings (with warm-up) and the revised settings (without warm-up), further confirming the efficacy of our method. We will updata these results in the revised version. Thank you for your understanding and consideration. Best regards, The authors of Lookaround Table S7: Comparison between SGDM and Lookaround with ResNet50 on ImageNet. Here "H" denotes random horizontal flip data augmentation. "V" denotes random vertical flip data augmentation. "R" denotes RandAugment data augmentation. | Method | Data Augmentation Strategy | Warm Up | Top-1(%) | Top-5(%) | | :--------: | :------------------------: | :------: | :------: | :------: | |SGDM|"H" + "V" + "R"| ✓ |72.27 |90.99| |Lookaround |"H" + "V" + "R"| ✓ |75.11|92.43| |SGDM|"H" + "V" + "R"| ✗ |75.97 |92.89| |**Lookaround** |"H" + "V" + "R"| ✗ |**77.32**|**93.29**|
Dear reviewer, We sincerely thank you for your constructive comments. As suggested, we have conducted comprehensive experiments on ImageNet, comparing the performance of SWA, SGDM, Lookahead, and Lookaround. The results, as presented in Table S8, again demonstrate that Lookaround achieves notable performance improvements when no warm-up setting is employed. Table S8: Comparison between SGDM, SWA, Lookahead and Lookaround with ResNet50 on ImageNet. Results of SWA$^+$, Lookahead$^+$ are taken from the origin paper. | Method | Warm Up | Top-1(%) | Top-5(%) | | :--------: | :------: | :------: | :------: | |SGDM| ✗ |75.97 |92.89| |SWA| ✗ |76.78 |93.18| |SWA$^+$| ✗ |76.97 |-| |Lookahead| ✗ |76.52 |93.11| |Lookahead$^+$| ✗ |75.49 |-| |**Lookaround** | ✗ |**77.32**|**93.29**| The allocated time for reviewer-author discussion is nearing its end, we sincerely look forward to your reevaluation of our work and would greatly appreciate it if you could raise your score to boost our chances of gaining more exposure in the community. Thank you very much! Best regards, The authors of Lookaround
I appreciate all the efforts from the authors. The outcomes displayed in Table S7, which demonstrate a performance decrease in ResNet50 on ImageNet due to the warm-up phase, are somewhat puzzling in consideration of my own experiences or previous research findings (like He et al. 2019). Nonetheless, I will withhold my assessment of this aspect since the influence of RandAugment could have introduced some intricate factors. I believe newly attained ImageNet results that align with widely recognized numbers will make the paper solid and reliable. My main concerns (including previous issues with ImageNet performance) are that the results of the large-scale experiments presented in this paper show somewhat significant differences from previously reported values. I believe these new optimizer proposals should showcase overall benefits (if not all) in widely accepted experimental benchmarks, all the while being consistent with results from previous literature. For instance, Foret et al. (2021) clearly demonstrated the superiority of their proposed optimizer compared to previously reported state-of-the-art numbers (Table 3). However, the current manuscript does not seem to achieve this, and thus I would like to maintain my initial assessment. As a concrete example for ViT-B/16 experiments, the following table summarizes comparison with numbers from previous literature, where there are immoderate differences in performance: | Method | ViT-B/16 on CIFAR10 | ViT-B/16 on CIFAR100 | | :- | :- | :- | | Adam (pre-trained on ImageNet-21k; this work) | 92.91 | 74.50 | | SWA (pre-trained on ImageNet-21k; this work) | 93.27 | 76.05 | | Lookahead (pre-trained on ImageNet-21k; this work) | 94.21 | 77.57 | | Lookaround (pre-trained on ImageNet-21k; this work) | 95.23 | 78.38 | || | Adam (pre-trained on ImageNet; Dosovitskiy et al., 2021) | 98.13 | 87.13 | | Adam (pre-trained on ImageNet-21k; Dosovitskiy et al., 2021) | 98.95 | 91.67 | | Adam (pre-trained on JFT-300M; Dosovitskiy et al., 2021) | 99.00 | 91.87 | || | Adam (pre-trained on ImageNet; Chen et al., 2022) | 98.1 | 87.6 | --- (He et al., 2019) Bag of Tricks for Image Classification with Convolutional Neural Networks. (Foret et al., 2021) Sharpness-Aware Minimization for Efficiently Improving Generalization. (Dosovitskiy et al., 2021) An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. (Chen et al., 2022) When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations.
Thanks for your constructive comments. Before your comments, reviewer vFuf had also raised similar concerns. > Quality concerns regarding the baselines (both ResNet and ViT) in Table 4. We apologize that, in the early stages of the experiment, we focused too much on comparing different methods and overlooked the importance of establishing a good baseline for the pre-training process. Therefore, we restarted the pre-training experiments based on the settings in [1] and [2], and the results are shown in Table S9. One detail that needs to be pointed out is that the pretrained weights of the ResNet and ViT networks used in our paper were sourced from ImageNet-1k, not ImageNet-21k. This contradicts the description of our actual settings in your table. In our comparison experiments with the new baselines, we found that the results of ViT-B/16 under different optimizers were better than those reported in the original paper [2]. This is mainly due to our use of more data augmentation during the training process, aiming for a fair comparison with Lookaround. This result also indicates that Lookaround can still achieve better performance under improved baseline conditions. Thank you once again for your ongoing responses and concerns during the rebuttal phase. By comparing against the new baselines, we believe our Lookaround method will be more reliable. Table S9: The test set accuracy under the training procedure with pre-trained models using ImageNet-1k weights. The results of ViT-B/16$^+$ are taken from [2]. | Backbone | Method | CIFAR10 | CIFAR100 | | :------: | :--------: | :-----: | :------: | | ResNet50 | SGDM | 97.55 | 84.50 | | ResNet50 | SWA | 97.48 | 84.81 | | ResNet50 | Lookahead | 97.65 | 84.78 | | ResNet50 | **Lookaround** | **97.82** | **85.20** | | ViT-B/16 | Adam | 98.34 | 91.55 | | ViT-B/16$^+$ | Adam | 98.13 | 87.13 | | ViT-B/16 | SWA | 98.47 | 91.32 | | ViT-B/16 | Lookahead | 98.51 | 91.76 | | ViT-B/16 | **Lookaround** | **98.71** | **92.21** | [1] When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations. [2] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.
Apologies for the misunderstanding. I misinterpreted the configuration in the supplementary code initially. I have now verified that `B_16_imagenet1k` is being utilized. I believe that the experimental outcomes revised during the rebuttal period would enhance the reliability of the experimental results and thus make the paper solid. Aligning the results with other studies (not necessarily flawless, but to a certain extent) holds importance not just in establishing the paper's credibility but also in preventing confusion among future researchers. I trust the authors comprehend this, even though redoing the experiment might have been somewhat cumbersome. I appreciate the authors again for all their efforts. Expecting that the comments showcased during the rebuttal phase will be systematically arranged in the final manuscript, I raise my rating above the borderline.
Response to the author
I greatly appreciate your feedback and the clarification you've provided. I am content with the overall quality of the paper and would like to vote for a "Strong Accept" (8).
We express our gratitude to the reviewer for your positive feedback. We appreciate your valuable comments and we will carefully revise the paper based on your comments.
Dear reviewer, We extend our sincerest gratitude for the questions you raised during the initial rebuttal stage. We have taken great care in providing thorough answers to address your concerns. As we approach the halfway point of the author-reviewer discussion phase, we are eager to receive further feedback and engage in fruitful discussions with you. Best regards, The authors of Lookaround
I appreciate the thorough clarification and the inclusion of extra experiments. It appears that my concerns align with those expressed by reviewer vFuf and d83u. I will await the outcome of the discussion between the authors and vFuf, d83u before making my final evaluation.
Thanks for your reply!
We sincerely appreciate your feedback on our rebuttal. To address your concerns and those of other reviewers regarding Logit Ensemble and SWA, we have conducted more comprehensive experiments and provided clarifications. Through our experimental results, we have demonstrated the superiority of the Lookaround method over Logit Ensemble and SWA across different datasets. Moreover, we have further explained that our method can be combined with Logit Ensemble and SWA for even better performance. We sincerely hope you will carefully review the materials we have provided and re-evaluate our work. We understand that you may take into account the perspectives of others when making your assessment, which is perfectly reasonable. However, we also hope that you will make your judgment based on the new empirical data and analysis that we present, drawing on your expertise. We believe this will contribute to a fairer and more accurate review process. If you have any further questions or require additional clarification, please don't hesitate to contact us. We would be very grateful if you could provide a more impartial and objective evaluation of our work. Thank you again for your time and effort. We look forward to your response!
Dear reviewer, We have addressed the concerns raised by other reviewers about our paper and have received their positive responses. We are looking forward to your reevaluation of our manuscript, drawing upon your expertise, as there remains but a scant half-day in the rebuttal stage. We extend our gratitude to you once more. Best regards, The authors of Lookaround
I would like to express my gratitude to the authors for their dedication in conducting supplementary experiments during the review process. Due to their commendable efforts, I am inclined to raise my rating to borderline acceptance. However, considering the necessity for augmentation techniques and the somewhat inconclusive nature of the experimental outcomes, my confidence level slightly diminished.
Thanks to the author for the detailed responses that alleviate my concern on whether the improvement in accuracy comes from these additionally augmented data. However, for me, the novelty of the idea seems rather limited as it is almost identical to the Lookahead optimizer. Thus, I prefer to remain the rating as "borderline accept."
Thanks for your reply!
Thank you for your feedback and the positive rating given to our work. We would like to provide further clarification regarding the differences between Lookaround and Lookahead as follows. Despite their similar names, Lookaround and Lookahead are fundamentally distinct in their spirts and methodologies. The Lookahead optimizer combines fast and slow weights along the training trajectory to identify a lower loss basin. Intuitively, the algorithm chooses a search direction by looking ahead at the sequence of “fast weights" generated by another optimizer. However, Lookaround adopts an iterative weight average strategy throughout the whole training period to constantly make the diversity-locality compromise. The terms "foward, back" in Lookahead title and "around, average" in Lookaround title succinctly encapsulate their distinctions in both spirts and approaches. Furthermore, our work theoretically demonstrate that the proposed Lookaround can converge to a lower expected loss and a smaller steady-state variance, which means that the model is more stable and has a better ability to resist noise. At the same time, Lookaround also shows certain advantages in terms of convergence speed. These findings are further supported by extensive and comprehensive experiments, which validate the superior performance of Lookaround compared to the previous Lookahead method. At the same time, we observe that Lookaround has smaller oscillation and higher accuracy on the convergence curve of the test set, which further demonstrates our theory. We sincerely look forward to your reevaluation of our work and would greatly appreciate it if you could raise your score to boost our chance of greater exposure within the community. Thank you very much!
After viewing the author's response and discussion with other reviewers, I now agree that novelty is not the main issue here, so I raise my score to "week accept."
Decision
Accept (poster)