Complex-valued Neurons Can Learn More but Slower than Real-valued Neurons via Gradient Descent

Summary

The authors presents theoretical results on learning real-valued and complex-valued neurons with gradient descent. The key takeaways are that: * complex-valued neuron learns real-values neurons and complex-valued neurons with convergence rate $O(t^{-3})$ and $O(t^{-1})$ respectively * two-layer real-valued neural network cannot learn a single complex-valued neuron * complex-valued neuron learns slower than real-valued neurons when learning real-value neurons

Strengths

* The paper is well-written and easy to understand even for non-experts (myself) * This appears to be a significant advance in the theory of complex-valued neuron learning, with previous work largely focusing on real-valued neuron learning. However, as someone outside this field, I defer to other reviewers' for their assessment of significance.

Weaknesses

* While the main results in this paper are theoretical proofs, the paper could be furthered strengthed with empirical experiments that validate the proofs. * table 2 appears to containa typo? the result for theorem 2 should be $O(t^{-1})$?

Questions

* Do the authors expect the convergence rates to be further improved with other learning rate schedules/learning algorithms (other than projected gradient descent)?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

yes

Summary

The goal is to explore the novel approach of complex valued neural networks using gradient descent. The paper investigates when and to what extent CVNNs outperform RVNNs using gradient descent for learning tasks. The researchers prove that a single complex-valued neuron can efficiently learn functions expressed by one real-valued neuron and those expressed by one complex-valued neuron, while a two-layer RVNN with finite width cannot learn a single complex-valued neuron. However, they also show that complex-valued neurons learn more slowly than real-valued neurons, with a lower bound of Ω(t−3) for learning functions expressed by one real-valued neuron using a complex-valued neuron. Theoretical comparisons between RVNNs and CVNNs provide insights into the success of CVNNs and the potential slow convergence of CVNNs learning.

Strengths

The authors appear to have thoroughly analysed a complex valued neural network and developed what appears to be rigorous theoretical results calculating the convergence rates of these complex valued neural networks. The ideas appear to be sound

Weaknesses

- While the notation appears to be self-consistent (in the main body - I did not read the appendix, which is very long and technical), the notation is very hard to parse through. If there were some tables to layout the important notation it could help lay the groundwork earlier on. - I am not familiar with these types of reformulations, but it is not clear to me why the authors want to try to use a two layer network to model a single neuron. The authors write "Although learning one neuron is the simplest case of learning neural networks, it is sufficient to embody the learning capability and efficiency of neural networks.", which I ask the authors to motivate more. Is it not obvious a two layer network with more parameters would be able to model a single neuron of just the weighted inputs? - I am not sure I fully understand this paper but it seems the authors only develop the proofs and theorem 1 and theorem 2 for 1d input such that they calculate the convergence rates for a two layer network to model a linear function wx where w is a vector length n and x is just a scaler number. If I understand this correctly, I am not sure how much we learn from this? Does this theorem apply for as real neuron where d is not one? -The ideas appear to be sound but they are not very palatable to a reader.

Questions

- Motivation of application - I’m surprised the authors do not mention more applications to systems in the world of physics. Fluid dynamics, beam physics, particle physics, where phase information is ubiquitously relevant to quantum materials, dynamics, and -States that the CVNNs have shown to have some desirable properties - universal approximation - boundedness, stability → why is this desirable? Could the authors elaborate on this? - It is not clear to me, in Theorem 1 and theorem 2 when d=1, does this mean that x is now no longer a linear summation of inputs but truly just an input and the weight parameter? I did not work through the proof, but does the theorem generalize to inputs with more than one dimension?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

It often feels like one cannot truly understand what they are discussing in the paper without referencing the very dense and long appendix.

Summary

The authors contrast the learning and convergence properties of real-valued neural networks and complex-valued neural networks. Specifically, they study the problem of learning the function implemented by a single neuron using a 2-layer finite width network. Notably, they show that a complex valued neural network can learn functions expressed by a real-valued neuron as well as a complex-valued neuron. This result indicates the superior capacity of complex-valued neural networks as compared to their real-valued counterparts. Furthermore, they also study the convergence properties of learning a function expressed by a real-valued neuron with a complex-valued neuron and prove that its slower compared to learning with a real-valued neural network. Taken together, complex-valued neural networks have a better learning capacity at the cost of slower learning properties compared to real-valued neural networks. Overall, I think this paper makes a strong theoretical contribution to understanding the efficiency and capacity of complex-values neural networks in practice.

Strengths

1. The paper has strong theoretical foundations and presents arguments through rigorous proofs. 2. Despite the involved mathematical machinery, the authors present a simplified view of the underlying learning dynamics and present the existence of phases in the learning process. 3. Despite the assumption of the MSE loss, I feel the authors present a strong theoretical framework to analyze learning in complex-valued neural networks and contrast them with real-valued neural networks.

Weaknesses

1. The authors present strong theoretical results, and use specific assumptions to prove their results. However, it is unclear the extent to which these assumptions would hold in practice. 2. In addition to their (very impressive) theoretical results, I feel the current version of the paper would benefit from some empirical validation. Having some toy experiments would also be sufficient to significantly improve the readability of the paper, while also increasing the reader pool. 3. The manuscript is generally well-written, but there are certain typos or use of abbreviations/notations which make it confusing for the reader. E.g. it seems that there is a typo in Table 2, wherein the second row third column (complex-valued Neuron --> complex-valued neuron) should be $\mathcal{O}(t^{-1})$. Also, abbreviations CVNN and RVNN or the notation $t$ is not introduced in the abstract.

Questions

1. The caption of Fig. 2 is currently not very descriptive. Could you please update it such that the takeaway is clear to the reader without reading in detail the text? 2. Is it possible to add a toy regression example where you can empirically demonstrate the different phases during the learning process?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

The paper develops theoretical results for the convergence rates of complex and real valued neural networks on complex-valued problems. It develops analytical models for the training regimes of complex valued NNs.

Strengths

The results would be useful for those attempting to apply complex-valued networks to problem settings where they are not normally used. An intriguing result is that complex valued neurons have slower convergence rates for the same problems as real valued neurons. The theorems are thorough, and the diagrams provide excellent illustration to the proofs. They also provide a proof that real valued networks cannot learn complex valued functions, which is not surprising, but nice to have proved.

Weaknesses

A fundamental flaw of the paper is that it does not have empirical verification of the theoretical results. The analysis makes many assumptions about different training regimes, but it is not obvious if complex valued neural networks actually behave like this. The complex-valued models in the theorems should be implemented and the convergence losses should be plotted alongside the theoretical predictions. Without supporting empirical evidence, the assumptions of the proof seem too strong. Fortunately, the problem settings should be simple to code up in any package, and the results would be easy to include in the author's presentation, given Figures 1 and 3. Page 4, line 156: “whereas a complex-valued neuron may only activate a small part as controlled by the parameter psi” This line makes it sound like the authors conclude that the primary benefit is the extra parameter in zReLU, not the complex arithmetic. The use of activation functions with trainable parameters is also a major distinction between the CVNN and the RVNN. Maybe all of the identified differences can come from a network with real weights, but an activation function with a learnable parameter? zReLU can be encoded in real-valued weights, if you interpret two real weights and use a learnable ReLU. Two variations are required to ablate this confounder: 1. complex weights with a zReLU with a fixed psi (Would this be easier to study theoretically than the existing proofs? It would also be easy to demonstrate empirically.) 2. real weights with a pseudo-zReLU with a learnable psi (Possibly hard to prove, but easy to empirically study.) I think considering case (1) is necessary to support the claim made by the paper that complex weights are the key differentiator.

Questions

- In Section 4.3, could you clarify how the outputs of the RVNN are matched to the outputs of the complex valued target function? - In Section 5, does the trainable zReLU make the network effectively deeper? Does the convergence rate change if psi is not trainable? - Define the schema of $L_{rr}, L_{cr},$ etc., and remind the reader every time you use them. By the time I got to Lemma 4 and Lemma 5, I did not know which result was which. - $t4 was never defined. - Define rvnn and cvnn in the abstract - Add equation numbers.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

4 excellent

Contribution

3 good

Limitations

A limitations section is missing from the paper.

Summary

The theoretical paper compares the learnability of real-valued neurons and complex-valued neurons via gradient descent. In a nutshell, the paper proves that a complex-valued neuron has a lower convergence rate than its real-valued counterpart, but learn exponentially slower. The theoretical discoveries help explain the success of CVNNs and point out the potential slow convergence of CVNNs learning.

Strengths

1. The paper makes the first attempt ever to comparing the learnability of real-valued and complex-valued neurons on a theoretical level. 2. Even though the theoretical investigations are carried out on a single neuron, the findings still cast light on the success and failure of CVNNs. 3. The discoveries are presented as Theorems and proved with rigor in the appendix, in which I did not find any technical errors. 4. The paper is very well written. The implications of each Theorem are clearly explained and summarized in tables.

Weaknesses

1. The analysis is carried out on a simple type of complex neuron network, and it is hard to generalize the achieved findings to practical RVNNs and CVNNs. 2. The discoveries could be more convincing by comparing the learning curves of RVNN and CVNN on toy examples, e.g. to show that a real neuron cannot well learn a complex neuron. 3. The research questions should be reformulated. Currently they are quite ambiguous and do not directly reflect the research focus.

Questions

1. It is argued in the paper that learning one neuron can embody the learning of neural networks. However, unlike RVNNs, multi-layered CVNNs may have intermediate layers with complex-valued input and outputs, a different setup from this study. Is it possible to employ a similar approach to this paper for studying these layers? 2. The bias term is removed for technical reasons. Could you please elaborate on how the presence of bias term will influence the analysis, and whether similar findings can still be obtained?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have adequately addressed the limitations of the work.

Reply to the authors' rebuttal

I would like to thank the authors for their response and effort during the rebuttal process. > The settings and assumptions used in this paper are common ones [1,2,3,4] to make the analysis tractable. Noted. I must admit that I am not very familiar with this line of research and therefore, wasn't sure how reliable these assumptions are. But I understand that the assumptions are fairly standard in the area. Thank you for pointing it out. > We provide toy experiments in the global response, where we verify our findings in more general settings. This is great! It would be great if you could add it to the paper, along with the details of the experiment and hyperparameters (in Appendix, if space constrained). > For empirically demonstrating learning phases, it should be emphasized that these phases only work for proof but may not exist in practice. It is however, striking to see the similarity between Fig. 2 and the empirical plot Fig 1 (that you added in the general response). I guess the correspondence is weaker when the dimension conditions are softened and bias is added (Fig. 2 of your general response). Overall, I am convinced that this is an interesting theoretical contribution and would be interesting to the NeurIPS community. I have therefore, increased my score. I wish the authors best with their endeavours and would like to congratulate them again on their excellent work.