1. The parameter count in KANs and MLPs are not directly comparable:
We acknowledge that the parameters of KANs and MLPs are not directly comparable since they are different architectures. We have also added a remark that the number of parameters is not necessarily the most useful measure of complexity of a model. However, the total number of parameters is generally indicative of the efficiency of a model and as a first step in our analysis, we have taken this measure of complexity. We remark that future work would entail comparing KANs and MLPs using more fine-grained measures of complexity. On the other hand, we note that if one focuses on the running time; in the $3d$ GRF examples using $50000$ samples, KANs are almost ($\approx$74mins) as expensive as MLPs ($\approx$56mins) on a gpu, while in practice one can also make KANs even more efficient by optimizing the implementation. These additional remarks have been added in the appendix.
2. Implication of Thm 4.1 to spectral bias or high-frequency performance:
Thanks for the suggestion. We have added more discussion about how the eigenvalues of the Hessian relate to the spectral bias, or performance of the model on high-frequency components. Specifically, Thm 4.1, compared with its MLP counterpart established in Hong et al. (2022) and Zhang et al. (2023), implies that for shallow models, KANs are able to fit high-frequency components of the target function more efficiently than MLPs when trained using gradient descent.
3. Restriction of the analysis to shallow KANs:
We acknowledge that our analysis of the spectral bias is fairly restrictive (this is in contrast to our analysis of the expressiveness and approximation theory, which holds for general deep MLPs and KANs). We have added more clear remarks on the limitations of our analysis before section 4.2. Nonetheless, we argue that despite its limitations, our analysis provides a theoretical explanation for observation that KANs fit high-frequency data better than MLPs. Finally, we remark that in addition to the theoretical analysis, we also provide empirical evidence supporting this claim. We leave a detailed analysis of the spectral bias of deep KANs, which we expect to be quite challenging, to future work.
4. Statistical significance of the experiment:
Thank you for pointing this out. We agree that in order to conclusively demonstrate a difference in the spectral bias of KANs and MLPs, one would require a large body of experiments covering a diverse set of practical applications., which is beyond the scope of a single paper. In our work, we have done a preliminary set of experiments that indicate a difference in the spectral bias of KANs vs MLPs. To the best of our knowledge, we have followed best practices and performed the relevant ablation studies. For instance, in the first example, we average over numerous random seeds. In the latter two experiments, we used a fixed random seed to make the experiments reproducible, but we have also observed a similar pattern using other random seeds. We have also added additional experiments to the appendix to strengthen our conclusions. In conclusion, we argue that our preliminary experiments are significant, and agree that future empirical work is necessary to conclusively validate our findings.
5. Theoretical Expansion: How does the depth of Kolmogorov-Arnold Networks (KANs) influence their spectral bias and approximation capabilities:
Thanks for this question, which we believe is too complicated to include in this paper, and which we intend to leave for future work. For example, one could compute the eigenvalues of the NTK matrices of deeper KANs empirically to investigate their spectral bias from a theoretical perspective, see for example Wang et al. (2024). Deeper KANs have better approximation capabilities and rigorously studying their spectral bias properties is an important future research direction.
6. Limitation of KANs, including computational cost, overfitting, scalability, practical implementation challenges, and real-world applications and comparison with SOTA models:
This is a very good point, and in the paper, we remark on this limitation of our analysis. Indeed, we only take into account the approximation, representation power (in terms of the number of parameters), and spectral bias of KANs. Our analysis of the spectral bias of KANs is relevant to overfitting, and indeed our results indicate that KANs may be more prone to overfitting since they are more sensitive to higher-frequency noise. Other important practical considerations including those mentioned by the reviewer are not discussed in this work. We remark that some of these aspects are discussed in the contemporary literature, including some that we cite in the Prior Works section. Right now our observation is that KANs are able to address large-scale problems very well once we scale up the sizes of KANs, made possible by the more efficient implementations of KANs now.