Summary
This paper introduces three benchmark datasets created by training CNN, ResNet, and ViT architectures using a set of activation functions generated from a three-node computation graph that combines unary and binary operations.
The benchmarks serve to showcase the efficacy of utilizing the 2D UMAP of the Fisher information matrix (FIM) spectrum and/or activation outputs as a cost-effective surrogate for predicting activation performance. Leveraging the 2D feature representation, an efficient activation optimization method, AQuaSurF, is developed by employing regression techniques to model activation accuracy across the 2D feature space, requiring only 100 function evaluations. The benchmark results further demonstrate the effectiveness and statistical reliability of this approach.
The proposed method is successfully applied to various vision tasks, where the discovered activation functions consistently outperform existing baseline activations. Moreover, the top activations identified through this search exhibit successful transferability to a new vision task.
Strengths
The paper is well-written and easy to follow.
The approach of utilizing the UMAP embedding of the FIM spectrum with activation outputs to assess activation performance is novel and interesting.
In contrast to previous methodologies that relied mostly on evolutionary algorithms and required thousands of function evaluations, the method proposed in this work demonstrates efficiency by outperforming baselines with just 100 function evaluations.
Furthermore, the benchmark datasets introduced in this work, may potentially help accelerate research on activation optimization.
Overall, this paper offers valuable insights for assessing activation performance and also introduces a more efficient methodology for activation optimization.
Weaknesses
In Section 6, the authors apply their proposed method to more challenging datasets and a larger activation search space, compared to those used to create the benchmarks. To further evaluate the effectiveness of the approach it would be beneficial to apply the method (KNR on UMAP embeddings) to vision tasks involving new network architectures as well.
While the chosen baseline activations in Table.1 already include ReLU and Swish, used in the original three architectures studied in the paper, in order to further strengthen the results it would still be advantageous, and perhaps straightforward, to extend the list of baselines at least to those used in PANGAEA, including GELU, LeakyReLU etc.
Questions
1- In the first paragraph of page 3 the authors observe, based on the scatter plots in Fig 2, that "best results come from discovering functions specialized to individual tasks". However, upon comparing the upper-left and lower-right corners of the plots with the upper-right region it appears that the best functions on one task also transfer effectively to, and are potentially among the best on, the other task. Is this interpretation correct?
2- In the middle row of Fig 4. The UMAP depends only on the activations and not the model. However, there appears to be differences in the distribution of points in the 3 plots (and also compared to Fig.3). Is this because of filtering out failed activations and possible rescalings / reflections of the space? A brief comment on this would enhance clarity for readers.
3- On lines 228-229 of the manuscript "Thus, activation functions are embedded close to each other in this space if they have similar shapes, if they induce similar FIM eigenvalues, or both", considering that the metric on the union of the representations is the sum of the metrics on the individual representations, then shouldn't the activations be close to each other only if they have both similar shapes and similar FIM eigenvalues?
4- How does AQuaSurF compare with PANGAEA in terms of performance? given the partial similarity of the search spaces, is it possible to make a direct comparison between the two methods (e.g. by limiting the space to non-parametric functions)?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
Limitations are partly addressed in the Future Work section in the appendix. There are no concerns regarding negative societal impact.