POMONAG: Pareto-Optimal Many-Objective Neural Architecture Generator

We are most grateful for the reviewer's thorough analysis and recognition of our work's quality. Regarding the Pareto Front filtering (lines 299-304), this occurs after architecture generation. For each architecture, we compute the parameters, MACs and inference latency, whilst using the predictor to estimate accuracy. From these Pareto fronts, we identify three configurations per secondary metric: the most efficient architecture (lowest metric), the most balanced (optimal accuracy/metric trade-off), and the most accurate (highest predicted accuracy). This approach provides practitioners with clear options suited to different deployment scenarios. The foundational works - MetaD2A (Lee et al., ICLR 2021), TNAS (Shala et al., ICLR 2023) and DiffusionNAG (An et al., ICLR 2024) - were all published at ICLR and establish MobileNetV3 and NASBench201 as standard benchmarks. Whilst POMONAG builds upon this established research trajectory, we have substantially expanded the validation across a broader range of datasets to demonstrate wider applicability. Our contribution extends well beyond enhancing DiffusionNAG. A primary innovation is the formulation of Many-Objective Reverse Diffusion Guidance, which elegantly balances four distinct gradients during generation. The optimisation of these gradients presented unique challenges: the scaling factors operate across vastly different scales, whilst maintaining convergence and architectural quality. We addressed this through a novel two-phase approach (lines 324-333) optimised via Hyperband pruning. The Performance Predictors underwent significant redesign, yielding marked improvements in Spearman correlation (from 0.687 to 0.855). The expanded Meta-Dataset properly supports multi-objective optimisation, whilst our Pareto-optimal filtering identifies three practical configurations (Acc/Bal/Eff) suited to different deployment contexts. The empirical results validate these contributions conclusively: POMONAG surpasses DiffusionNAG in both accuracy (+4.06% on NASBench201) and efficiency metrics, with remarkable reductions in parameters (90%) and MACs (93%). We might also note that POMONAG achieves these improvements whilst requiring only a single architecture to be trained per dataset, significantly reducing computational overhead compared to prior approaches. We trust these clarifications address the points raised and demonstrate the substantial nature of our contributions. We are grateful for the reviewer's careful consideration and hope these explanations enable a fuller appreciation of the work's merit.

Thank you to the authors for their detailed response. After reviewing the comments from other reviewers and the authors' replies, I have decided to maintain my original score. Upon reflection, my initial review remains generous given the collective feedback and responses. I appreciate the authors' thorough efforts in addressing the concerns and wish them success in their future revisions.

We are very grateful for the reviewer's contribution and dedication, as well as for his valuable comments and time. We would like to point out the first clear aspect of our work. The correlation between secondary metrics varies significantly depending on the type of architectures considered. For architectures with traditional layers such as Conv2D and Dense, observed in NASBench201, there is a natural correlation between metrics and MACs, as documented in Appendix A. However, this relationship becomes more complex in MobileNetV3 architectures, which employ advanced components specifically designed to reduce parameters while maintaining expressiveness. For example, 2D Depthwise Convolutions and Inverted Residual Bottlenecks are designed to drastically reduce the number of parameters by factoring convolutional operations. These blocks, although ‘light’ in terms of parameters, require a more complex sequence of operations: a Depthwise Convolution followed by point-wise convolutions and normalisations. Similarly, the Squeeze and Excitation blocks introduce an attention mechanism with few parameters but multiple global pooling operations and channel transformations. Inference time, in particular, has even more independent dynamics in both search spaces. Operations such as pooling or activations, while having minimal parameters and MACs, can have variable execution times depending on their hardware-specific implementations. This lack of direct proportionality between the metrics highlights the need for many-objective optimisation to effectively balance these aspects, which are complementary with accuracy, and not aligned with each other. The contributions of POMONAG extend significantly beyond the simple adaptation of DiffusionNAG. The Many-Objective Reverse Diffusion Guidance framework (lines 162-215) introduces an innovative approach that harmonises competitive gradients, ensuring stable convergence during both training and inference. This result is particularly relevant considering that the optimised metrics (accuracy, parameters, MACs and latency) not only conflict with each other in many cases, but also exhibit non-linear relationships that vary significantly depending on the type of architectural blocks used. Our two-stage optimisation strategy (lines 324-333) represents a key theoretical advancement through Pareto Front Stretching. This approach allows us to effectively explore regions of the solution space with naturally different scales, generating both highly efficient architectures and highly accurate models. The systematic exploration framework dynamically balances these conflicting goals, identifying Pareto-optimal solutions that would be difficult to achieve with traditional techniques. The innovations introduced produce substantial practical impacts: the Performance Predictors show a significant improvement (Spearman correlation from 0.687 to 0.855), while POMONAG achieves higher accuracy on both NASBench201 (+4.06%) and MobileNetV3 (+2.67%) with significantly improved efficiency (up to -90% parameters, -93% MACs). The validation on 15 different datasets with single training cycles demonstrates not only the effectiveness of the method, but also its computational practicality. The extensive empirical validation confirms that POMONAG introduces fundamental innovations in the field of neural architecture search, overcoming the limitations of previous approaches through many-objective optimisation integrated in the diffusion process. This opens up new directions for the generation of architectures that effectively balance accuracy and computational efficiency.

Optimising the scaling factors kϕ, kπ, kμ and kλ is a crucial aspect of POMONAG. Starting from the 10000 reference value used in DiffusionNAG for kϕ, we developed a systematic approach to calibrate these parameters that balance the different targets during the diffusion process. Given the inherently different nature of the targets, we defined two complementary search interval configurations. To generate accuracy-oriented architectures, we explored a wide range [10000, 50000] for kϕ, while maintaining smaller ranges [10, 50] for secondary metrics. Conversely, to prioritise efficiency, we reduced the range of accuracy to [1000, 5000] while increasing that of the other metrics to [100, 500], thus allowing for greater influence of efficiency goals during generation. Optimisation of these intervals was carried out through Tree-structured Parzen Estimation with Hyperband pruning, using Optuna for 100 evaluations on a representative set of datasets (CIFAR10, CIFAR100, Aircraft, Oxford III Pets). This process identified optimal values for both search spaces, as detailed in lines 324-333. The resulting scaling factors significantly influence the shape of the Pareto front by changing the sampling centroid during the diffusion process. The robustness of these values is empirically confirmed through extensive testing on different datasets, as documented in Appendix B. This systematic optimisation process ensures an effective balancing of potentially conflicting objectives, allowing POMONAG to generate architectures that satisfy different design priorities. Details on the computational costs of POMONAG are presented at the end of Section 4 (lines 469-479). We thank the reviewer for pointing out the need to also specify the training times of the score network and predictors, which we will include in the camera-ready. These times are strongly related to both the type of encoding used for the meta-dataset and its cardinality. On a single QUADRO RTX 6000, training requires: - Score Network: ~9h for NASBench201, ~19h for MobileNetV3 - Performance Predictors: ~8.5h for NASBench201, ~18h for MobileNetV3 These times can be significantly optimised through various strategies: - With a cluster of four QUADRO RTX 6000s, due to the increased batch size, score network training times are reduced to 1.5h for NASBench201 and 2.5h for MobileNetV3 - Higher-performance hardware such as the A100 can achieve similar or better results by utilising both higher TFLOPS and available memory - Optimised encoding of the meta-dataset makes it possible to maintain short times despite the increase in cardinality As far as generation is concerned, POMONAG requires: - 5:45 minutes on NASBench201 - 18:15 minutes on MobileNetV3 These times are particularly competitive when compared to other recent approaches: - MeCo [1]: 115 minutes on CIFAR10 (GPU not specified) - SWAP-NAS [2]: 6 minutes on CIFAR10/NASBench201 (on Tesla V100, superior hardware) - ZiCo [3]: 10 hours on NVIDIA 3090 (significantly slower despite superior hardware) It is important to emphasise that the pre-training of architectures is an integral part of the search space and available online, so it is not an additional overhead. Considering that the training time is a one-off and that POMONAG shows competitive performance even on non state-of-the-art hardware, we believe that these computational costs represent a significant advantage over alternatives in the literature. POMONAG's ability to achieve superior results with lower computational resources highlights the efficiency of our approach. A significant advantage of POMONAG lies in its ability to drastically reduce the number of architectures to be trained, thanks to the interaction between Performance Predictors and Pareto Front. The generated architectures are evaluated by estimating their accuracy using the predictors and directly calculating the other metrics (parameters, MACs, latency), allowing a complete Pareto Front to be constructed prior to any training phase. From this front, a single optimal architecture can be selected based on priorities: the most efficient, the one with the best accuracy-efficiency trade-off, or the one with the highest estimated accuracy. This approach represents a substantial advantage over alternative methods that, lacking accurate predictors and a view of the Pareto front, are forced to train a batch of candidate models to identify the optimal one.

We thank the reviewer for the valuable comments and insights on our paper. We recognise that the formulation of the Reverse Diffusion Guidance Process, being a major contribution, could be presented at the beginning. We have carefully considered this modification, which would make one of the key steps clearer. In the methods, we chose to briefly anticipate the DiffusionNAG formulation, focusing on single-target search, and then move the focus to many-objective inductively. The choice to place the related works before the method, and not at the bottom as is usual, stems from the need to provide the reader with a clear view of the neural architecture search context and the operation of DiffusionNAG, which may not be immediate. For this reason we have opted for this format, while recognising the validity of the alternative. We also find the recommended approach of using four separate diffusion processes to be subsequently balanced by many-objective optimisation extremely interesting. This would undoubtedly be a path to explore. However, the result would inevitably differ from that proposed in this paper. It would likely require longer generation times in proportion to the number of metrics to be considered. In any case, it remains an extremely valid route. We appreciate the constructive feedback on the need for a more in-depth analysis regarding the transition from single to multi-objective, despite the fact that it was described in two separate places in the paper (lines 161-215 and 302-323). In the revised version, we will include an analysis of convergence properties and the impact of weights on goal balancing. As far as the predictors included in this work are concerned, they have not been gone into in detail as they retain the DiffusionNAG architecture, the difference being that they have been adapted and dedicated to the prediction of secondary metrics. In the camera-ready we will specify this aspect in more detail, which will certainly be useful to the reader. Regarding the scaling factors, their search range was deliberately set wide ([1000,5000] for accuracy, [100,500] for the other objectives) to allow a complete exploration of the solution space through many-objective optimisation. This choice is motivated by two key considerations: - The different objectives (accuracy, parameters, MACs, latency) operate on naturally different scales, requiring proportionally different scaling factors to balance their contribution in the diffusion process. - The use of Tree-structured Parzen Estimation with Hyperband pruning allows to efficiently explore this space, quickly identifying promising regions and discarding sub-optimal ones. As demonstrated in our experiments (lines 324-333), this strategy led to convergence towards optimal values for both search spaces, generating Pareto-optimal architectures that effectively balance the different objectives. The empirical results in Appendix B confirm the robustness of these values across multiple datasets. As far as the version of POMONAG optimised only for accuracy is concerned, this achieves performance in the neighbourhood of the best models obtained with the many-objective version, but with more parameters, MACs and inference time. This is an expected result since the search spaces, although large, are constrained by the family of architectures they describe. It is therefore natural that there are solution sets with very close performance; the main challenge lies in balancing this performance with secondary metrics that tend to be in antithesis. The methodological differences between DiffusionNAG and POMONAG are discussed in section 3.1 (lines 161-215). The extension from single-objective to many-objective involves the introduction of new terms in the diffusion process for parameters, MACs and latency. Each additional term requires its own dedicated Performance Predictor to guide the deployment process towards optimal architectures with respect to that specific criterion. The main challenge, addressed through the Many-Objective Reverse Diffusion Guidance framework, was balancing these competitive gradients while maintaining process stability. We recognise that this analysis could be deepened. In the camera-ready version, we will add a more detailed discussion on how the interaction between the different gradients influences the diffusion process and the convergence towards Pareto-optimal solutions. In conclusion, we sincerely thank the reviewer for the valuable comments, and trust that the changes to the article and the clarifications may lead him to positively revise his assessment.

Thank the authors for the response. However, I still have concerns in the following aspects: 1, Maybe the authors have not fully understood my first question, which is [How to decide the scaling factors?]. I guess the authors just arbitrarily choose several valuse of scaling factors, but that sounds not sufficient. Indeed, such tradeoff between different objectives might be challenging, but still needs some methods. If I am wrong, please correct me. 2, The authors claimed that, the theoretical analysis would be added in camera ready version, but they do not provide the some clue for that analysis or proof. 3, The authors ignored my second question, [how about the performance of POMONAG if just considering adding one factor?]. In fact, I think extending the diffusion model from one factor to two factors might be more easy to provide the insight beneath this work. In summary, I agree that this idea is good and natural and important, but this work needs more verification. So I will keep my initial rating.

We thank the reviewer for investing his time and valuable advice. The advantages of using POMONAG over previous state-of-the-art work in generating architectures for image clustering lies in the gfeneration of more accurate architectures with less complexity in terms of parameters, macs and inference time. Furthermore, thanks to the selection on the Pareto front, it is possible to specify the willingness to take the best performing architecture in terms of secondary metrics, the one with the best balance between estimated accuracy and these metrics, or simply the architecture with the best estimated architecture. The Pareto front is generated from the 256 architectures produced at the end of the POMONAG generation phase. For these architectures, parameters, macs and inference times are calculated, while accuracy is estimated. From these values, Pareto fronts are created, on which the candidate architecture is then selected to be returned according to the user's requirements - whether they want the most efficient structure, the one with the best balance, or the one that simply performs best (lines 327-355). In general, the optimisation and generation process is totally many-objective, and it is a weighted sum of loss function components, as in practically all works that aggregate different optimisation constraints. So, to reiterate, generation and training are many-objective. As for the creation of Pareto fronts, these are then generated from the same architecture generation. Parameters and macs tend to be fairly consistent with them, while they deviate from inference time in terms of architecture ordering. However, it is possible to identify clusters in the sense that the architectures with fewer parameters and operations tend to be the fastest at inference. For this reason, we preferred not to implement a non-dominated sorting process, typical in NSGA algorithm families. In any case, we thank you for the very interesting and certainly timely food for thought. In the camera-ready we will make all necessary changes to make this clear. The methods suggested for further comparisons were carefully considered, but they operate on different search spaces that would make the comparison meaningless: SWAP-NAS uses NASBench-101/201/301 and TransNAS-Bench-101. On NASBench201, in common with our work, Spearman correlation coefficients are reported - with similar performance to ours, albeit in a different setup - but not test accuracy. ZiCo was evaluated on NASBench101, NATSBench-SSS/TSS and TransNASBench-101, while MeCo on NAS-Bench-101, NATS-Bench-TSS, ATS-Bench-SSS, NAS-Bench-301 and Transbench-101. We had already considered including these works in our comparative analysis, but felt that the differences in the search spaces did not allow for a methodologically correct comparison. POMONAG belongs to the TransferNAS family which is substantially different from zero-proxy methods. TransferNAS methods leverage pre-trained model knowledge to guide architecture search, identifying promising patterns and reducing exploration of suboptimal designs. While generally faster, zero-proxy methods evaluate candidates without training using metrics like FLOPs and parameter counts, potentially missing important performance characteristics that only emerge during actual training. POMONAG is distinguished also inside the TransferNAS family by the use of a diffusion process that allows for a denser stochastic exploration of the generation space due to its energy-based nature. With regard to time complexity, we have reported the generation times for the search spaces studied and the training times for the generated models, which vary according to the priority of efficiency or performance. A direct comparison with the suggested methods would be inaccurate due to the different search spaces and hardware used. However, analysing the available data: - POMONAG needs 5:45 minutes on NASBench201 and 18:15 minutes on MobileNetV3 - MeCo takes 0.08 GPU days (115 minutes) on CIFAR10 (GPU not specified) - SWAP-NAS, on Tesla V100, takes 6 minutes on CIFAR10/NASBench201 (+4.35% compared to POMONAG, on higher hardware) - ZiCo takes 0.4 GPU days (10 hours) on NVIDIA 3090, being significantly slower despite superior hardware These comparisons, although approximate and estimated to the disadvantage of POMONAG, highlight the superior efficiency of our approach. We are grateful for this opportunity for analysis, which allowed us to highlight the advantages of POMONAG even over other NAS techniques not initially included in the study. The diffusion process is carried out for 10000 steps, as in the reference work, and this corresponds to the achievement of convergence in a very expansive environment characterised by the steadiness of the oslution. Reducing the number of steps too much would lead to even shorter generation times, and we will certainly investigate this further. Increasing them showed no improvement in performance.

Thank you for your detailed comments. We apologise for having to split the response into two, but the issues included by the reviewer were numerous, and the answers must be comprehensive. We address the issues raised point by point: - Links will be added after acceptance, as their earlier inclusion would have compromised anonymity. We thank for the valuable comments on the figure and confirm that we are already working on its improvement for camera-ready. - We have clarified that ‘noisy architecture’ means architecture that has not yet been cleaned of noise, i.e. the matrix representation of the architecture during the diffusion process. We have added an explanatory line for this definition. - The equations have been numbered as required and we have added the necessary definitions, correcting the reported typo. Given the equivalence of concepts, we have renamed everything as ‘Reverse Diffusion Guidance Process’ to avoid confusion. At Line 183 we present the process equation proposed by An et al., from which POMONAG's formulation is extended - we have added a clarifying sentence. We have also specified that s_θ(A_t,t) represents the diffusion step applied to architecture A at time t. - The decision not to use predictors for parameters, MACS and inference time in denoised architectures is motivated by greater efficiency and accuracy in direct calculation. Estimates are only used during generation, when a concrete architecture does not yet exist. The only metric necessarily estimated post-generation is accuracy, which is essential for the Pareto front and the selection of the architecture to be trained. - The choice of ViT-B-16 is amply justified in the appendix (lines 972-1010), demonstrating an improvement in the Spearman correlation of predictors. - As explained, this work fits into a well-defined and structured strand of literature on image classification (MetaD2A, TNAS, DiffusionNAG - all ICLR). The suggested metrics, while interesting, apply to different tasks and research spaces. We appreciate the suggestion for future developments. - We regret reading the final considerations, especially considering that the article clearly demonstrates how POMONAG represents the state of the art in accuracy in the research strand under consideration, requiring the training of a single architecture and being faster even in the generation phase than the models cited by the reviewer. The three metrics analysed serve to investigate a crucial dimensionality of the dissemination process: they capture the percentage of valid architectures generated (validity), their heterogeneity to prevent collapses (uniqueness) and diversity with respect to the training set to assess generalisation (novelty). Sub-optimal values in these metrics would compromise the entire generation process, while the positive results obtained confirm their validity. We are deeply grateful for the reviews and the many valuable suggestions received. We believe we have clarified many potentially ambiguous points and supplemented the required information to make the camera-ready even more meaningful. With sincere gratitude, we hope that the reviewer will positively reconsider his assessment, in light of the provided demonstrations on the scientific validity, superior performance, computational efficiency and innovativeness of POMONAG.

We thank the reviewer for his time and valuable comments. Our contributions extend significantly beyond adapting DiffusionNAG. The Many-Objective Reverse Diffusion Guidance (lines 162-215) introduces a novel framework that harmonises competing gradients, ensuring stable convergence during both training and inference - a non-trivial achievement given the complexity of balancing multiple conflicting objectives simultaneously. Our two-phase scaling optimisation approach (lines 324-333) represents a key theoretical advancement, introducing Pareto Front Stretching for effectively navigating solution spaces with disparate scales. This systematic exploration framework enables discovery of truly Pareto-optimal solutions. These innovations yield substantial practical impacts: Performance Predictors show marked improvement (Spearman correlation from 0.687 to 0.855), whilst POMONAG achieves superior accuracy on both NASBench201 (+4.06%) and MobileNetV3 (+2.67%) with significantly enhanced efficiency (up to -90% parameters, -93% MACs). These gains, validated across 15 datasets with single training cycles, demonstrate meaningful progress towards deployable architectures. POMONAG builds upon established Transferable NAS research for image classification, following seminal ICLR works (MetaD2A, TNAS, DiffusionNAG) that established MobileNetV3 and NASBench201 as standard benchmarks. Our validation extends significantly across 15 diverse datasets, providing thorough comparison with previous approaches and state-of-the-art DiffusionNAG. Exploring transformer spaces like HW-GPT-Bench presents an intriguing future direction. However, our current focus remains on advancing image classification NAS, where we demonstrate substantial improvements within this well-defined research trajectory. The ablation study in Appendix C rigorously demonstrates each component's independent contribution - from enhanced Performance Predictors to Meta-Datasets - validating POMONAG's innovations beyond DiffusionNAG's structure. Due to space constraints, we present comprehensive performance metrics in Table 5, demonstrating POMONAG's advantages across accuracy, parameters, MACs and latency. The complete Pareto front visualisations are available in Appendix B. We appreciate your feedback on the paper's structure. While positively received by reviewers w9TN, s4FH and hqyM, we acknowledge room for improvement. The structure - related work, contributions, method, experiments, results and discussion - follows a format we've found effective, though we understand preferences vary. Our frequent DiffusionNAG references aimed to highlight POMONAG's innovations rather than repeat established foundations. As the latest TransferNAS advancement, POMONAG naturally builds upon previous work while introducing substantial novel contributions. We are particularly grateful for noting the font size error caused by an uncommented /footnotesize from page 4. This technical oversight has been promptly corrected, ensuring full ICLR compliance within the 10-page limit. The camera-ready version will reflect this correction. Regarding scaling factors: These values balance multiple objectives during diffusion. Starting from the reference value 10000, we systematically explored and calibrated ranges for secondary metrics (lines 342-346), optimising for expected accuracy on the Pareto front (lines 327-355). The impact is significant: scaling factors critically influence the Pareto front shape by shifting the sampling centroid. Given their importance, we shall provide detailed analysis in the camera-ready version. Regarding sampling: POMONAG generates 256 Gaussian noise tensors, guided by predictors estimating multiple metrics during diffusion. After filtering inconsistent configurations, remaining architectures are evaluated for parameters, MACs and latency, forming a Pareto front that enables selection based on efficiency, accuracy/metric ratio, or peak accuracy (lines 309-325). We use original dataset splits where available, otherwise creating stratified validation sets (seed 42). Multi-objective optimisation employs Tree-structured Parzen Estimation with Hyperband pruning via Optuna's default configuration. We sincerely appreciate your thorough review and valuable external perspective. The points highlighted have been comprehensively incorporated into the manuscript, enhancing its depth and clarity. We are particularly grateful for noting the font size error, which was immediately rectified. We trust that this technical oversight significantly influenced the initial assessment. Given our thorough clarifications and responses, alongside the extensive experimental validation conducted over years of research - which aligns with and extends the standards established in previous editions of this conference - we hope you might reconsider your evaluation. Thank you for your detailed guidance in strengthening our submission.