Summary
Summary: The authors' work undoubtedly stands as an exceptional and noteworthy contribution to the field, showcasing a plethora of remarkable achievements that have the potential to significantly influence and shape the landscape of optimization and extend their impact well beyond its boundaries. As I delve into the intricacies of their research, it becomes evident that their dedication and innovative approach have yielded results that are both impressive and promising.
However, amidst the numerous commendable aspects of their study, there also exist certain primary concerns that warrant attention and resolution. While I acknowledge the impressive strides made by the authors, it is essential to critically address certain aspects that could further enhance the credibility and applicability of their findings. In the following detailed comments, I will highlight specific areas that require closer scrutiny, offering constructive feedback and suggestions to fortify the integrity and comprehensiveness of their research.
Strengths
I am elated to express my joy and satisfaction upon the efficiency of the authors' method has transcended that of the renowned Adam optimizer, exhibiting its superiority across various tasks. This notable achievement undoubtedly highlights the ingenuity and prowess of the authors in devising such a remarkable approach.
The profound impact of their work is vividly demonstrated through the astonishing results presented in Table 1 and Figure 1, which undoubtedly leave one in awe. In particular, the Discovered optimizer, aptly named "Lion," showcased in Program 1, stands as a testament to the authors' ability to craft a solution that is both simplistic and elegant, evoking admiration from peers and fellow researchers alike.
Moreover, the decision made by the authors to simplify Program 4 and refine it into the more efficient Program 1 is not only logical but also reinforces the practicality of their approach. This thoughtful choice not only streamlines the methodology but also adds to the elegance and cohesiveness of their research.
In their diligent pursuit of showcasing the sparsity of the high-performance optimizer, the authors undertook a remarkably extensive and rigorous experimental endeavor, involving the execution of a staggering 2 million randomly selected programs. With meticulous precision and analytical acumen, they thoroughly assessed the performance of each program, leaving no stone unturned in their quest for a superior alternative to AdamW. Upon meticulously scrutinizing the results obtained from this monumental experiment, the authors arrived at a rather intriguing and significant revelation: none of the 2 million randomly selected programs exhibited a performance that surpassed the prowess of AdamW. This particular finding stands as a testament to the exceptional capabilities and efficiency of AdamW as a high-performance optimizer, solidifying its position as a prominent and leading choice in the optimization landscape. The sheer scale and comprehensiveness of this experimental undertaking are truly commendable, underscoring the authors' unwavering commitment to pursuing scientific rigor and thoroughness. Their methodical approach and the impressive volume of data analyzed serve as an invaluable resource for the scientific community, contributing to a deeper understanding of optimizer performance and fostering future advancements in the realm of optimization algorithms.
The authors' insightful use of meta-validation to select programs with superior generalization ability is indeed a reasonable and effective approach. Especially noteworthy is their ingenious technique of progressively increasing the complexity of the meta-validation task, aptly termed "Funnel selection." This strategic refinement undoubtedly enhances the efficiency of the selection process, allowing for the identification of programs with enhanced generalization capabilities.
Additionally, the thoughtful incorporation of sign updates and regularization is highly justified, especially when considering the context of adversarial attacks, where techniques like L_inf PGD attacks often rely on such sign update mechanisms. In fact, sign updates have also been previously employed as optimizers in prior works.
Undoubtedly, the analysis presented in Section 3.2 is both captivating and intellectually stimulating, offering valuable insights that resonate with the broader research community. The robustness and trustworthiness of these findings are indeed commendable and a testament to the authors' thoroughness in their investigations.
One fascinating observation that delights me is the correlation between model size and Lion's validation accuracy. The authors' remark on the positive relationship between increased model size and enhanced validation accuracy raises intriguing possibilities. The plausible explanation that the sign update technique increases uncertainty and ultimately improves the model's generalization ability opens exciting avenues for future research.
Weaknesses
While their research presents impressive insights and advancements, it is essential to address one concern that arises from the potential handcrafted nature of the simplification and derivation process for the programs. A more elegant solution would involve automating this process as an inherent aspect of defining the search space, further enhancing the overall methodology and augmenting its potential impact.
Sign updates have also been previously employed as optimizers in prior works.
It is worth kindly noting that, when the proposed method is combined with more potent data augmentations, the relative performance gains might appear less pronounced. This insightful observation highlights the need for further exploration and optimization to fully unlock the method's potential under varying experimental conditions. While considering this aspect, I must admit that I personally feel a bit disappointed.
I am genuinely interested in exploring the application of the Lion algorithm to the domain of Stable Diffusion. It is pertinent to acknowledge that training stable diffusion models can be an extremely resource-intensive endeavor, often incurring substantial costs (for instance, approximately 1 million dollars for stable diffusion v1). Given this context, the potential of the proposed Lion method to be utilized in conjunction with stable diffusion holds tremendous appeal and significance. If the Lion algorithm proves to be compatible with stable diffusion models, it would indeed be a highly advantageous and desirable outcome. The ability to leverage the proposed method's capabilities in optimizing stable diffusion models could potentially mitigate the significant expenses associated with their training, thereby opening up new avenues for research and applications in this area. The prospect of such synergy between Lion and Stable Diffusion sparks a sense of optimism and excitement, as it has the potential to drive transformative advancements in the field. As such, I eagerly look forward to any insights or findings that may shed light on the feasibility and practicality of applying the Lion algorithm to the challenging task of stable diffusion, as this would undoubtedly represent a valuable and impactful contribution to the scientific community.
One of the concerns in the field of AutoML revolves around the efficacy of the search process, causing a great deal of apprehension among researchers and practitioners. More precisely, a central aspect of this concern is rooted in the correlation between the performance exhibited by programs during the search phase, specifically on the small proxy, and their subsequent performance on the final task. This particular correlation is viewed with heightened attention, as it holds significant implications for the overall effectiveness of the AutoML approach. For instance, in the context of Neural Architecture Search (NAS), where the exploration of different architectural configurations plays a crucial role in achieving optimal results, there is a palpable sense of unease regarding architecture ranking. How well a particular architecture performs on the small proxy tasks might not always be indicative of its actual performance when applied to more extensive and state-of-the-art tasks, and thus, researchers seek to establish a robust link between the two domains. In attempts to address this pressing concern, the authors of the present paper have diligently employed a comprehensive set of measures. Their goal is to carefully select algorithms that can demonstrate a remarkable ability to generalize from small proxy tasks to larger and more complex real-world tasks, thereby enhancing the reliability and applicability of the AutoML approach. However, despite the earnest efforts made by the authors, there remains a notable omission in their work. They have not provided a quantitative analysis that establishes the explicit correlation between the performance achieved on the small proxy tasks and the corresponding performance on the final, real-world tasks. This apparent absence of a rigorous and quantitative examination is indeed regrettable, as such analytical insights are vital for instilling greater confidence in the effectiveness and reliability of the proposed AutoML techniques. In conclusion, while significant strides have been taken in addressing concerns regarding the search effectiveness in AutoML, the lack of a robust quantitative analysis linking the performance between the small proxy and the final tasks leaves a critical aspect unanswered. Efforts to bridge this analytical gap would undoubtedly enhance the overall understanding and trustworthiness of the AutoML methodologies employed.
Questions
See *Weaknesses.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Limitations
No. See *Weaknesses.