Summary
The authors introduce a new feedforward spiking delay neural network (DeNN) that can be trained with backprop to competitive task accuracy on MNIST, CIFAR-10, N-MNIST, DVS-Gesture, and GSC. Critically, DeNN does not have weights; instead DeNN only has synaptic delays. In contrast, existing SNN works generally have weights and synaptic delays.
The spiking neurons of DeNN only spike once, and this spike time is a function of the arrival times of presynaptic spikes. The authors choose the spike time function to be a summation of presynaptic spike times weighted by a temporal kernel (later spikes contribute less than earlier spikes). The authors represent each synaptic delay via a Gaussian function whose argument is a learned parameter. The Gaussian function always produces positive input, as delays are positive, and the sign of the learned parameter indicates that postsynaptic connectivity is inhibitory (negative) or excitatory (positive). The authors then introduce a standardization procedure, which could possible be used to have the spike time function ignore late spikes.
The authors introduce an input processing methodology for converting event-based datasets into coarser-grained frames with equal numbers of events in each frame. Importantly, this preprocessing step generates frames such that the timing information of the events within each frame is compressed in some fashion so that some timing information is still available to the DeNN.
The authors define a convolutional architecture for the tasks listed above and add a learnable long-term timing memory input preprocessing stage for the GSC task. The authors achieve competitive performance across all tasks and with relatively modest compute/memory cost compared to other baseline DNNs.
Weaknesses
In this section, I describe the two key weaknesses that cause my rating to be “marginally below acceptance threshold.” Given substantial improvement in the following two weakness areas, I would consider improving my rating; however, I am unsure whether the necessary changes would be too substantial to be completed in the short rebuttal period.
((1)) The key weakness of this paper is lack of clear exposition. Here are several specific points that illustrate some opportunities to improve the clarity of the exposition.
Line 107, what is an “analog” neuron? Analog hardware? Real-valued?
Line 144, first the authors say that they use delays instead of weights, and then they compare large weights to large delays. The reader is left thinking “I thought we were not using weights, so what am I comparing large delays to here, if weights don’t exist?”
Line 154, do the neurons in DeNN only have a single spike time? (Only spike once?) Is this implied here by saying “the” spike time? Also, is time discrete or continuous here? Would be beneficial to clearly state DeNN is a single-spike network.
Line 170, how does the -\kappa (t_i+1) term represent an incompressible delay?
Line 170, where is the discontinuity at d_ij = 1?
Line 182, I know what a ReLU is, but what is a ReLU-like function? Or stated differently, how is what you are describing in this paragraph achieving a ReLU-like function?
Line 183, what is “some value”?
Line 185, what’s the significance of “almost equivalent”? Does it matter that it’s not exactly equivalent?
Line 185, is q dynamically computed or specified before inference?
Eq 5, what is the definition of the “std” function? I inferred it is likely subtracting mean and dividing by standard deviation, but I do not know because std is not clearly defined.
Line 231/Figure 2, I struggle to understand several aspects of the preprocessing stage, given the current exposition. For instance, why 2rN? Do I have two images, one for positive and negative cells? Do events arrive in continuous time, discretized time? In Figure 2, is “N” the same “N” as in “N total pixels in the image”? What is \delta t? Are all \delta ts the same? What is “T”?
Figure 1, do neurons in the next layer have to wait until all neurons in the previous layer have spiked (in the q=1 case)? This seems like a major limitation of the DeNN methodology that should be clearly stated. A further improvement could be addressing the following - if all time indeed needs to be processed in a previous layer before the next layer can start its processing, how is a delay-only network substantively different than a single time-step weight/activation network? Spoken alternatively, it would be very valuable to understand the connection between conventional weight/activation DNNs if I consider delays as weights and spike times as activations. E.g., where would the aforementioned ReLU-like nature come in?
Line 301, rN is not 2rN?
Eq 10, it would be helpful to give more context/verbal exposition regarding the high-level functioning of Eq 10.
Figure 5, “Bottom line…” I understand that this figure shows differences between frames, but what is the purpose of showing this in the context of the paper’s narrative? When I was reading, I did not get the “bottom line” when the “bottom line” was highlighted, and I was confused as to what the bottom line is.
Table 1, what are the units of computation? FLOPS? Such a unit would be more informative than “# comp.”
Figure 3, what is the correct class in this case?
Line 421, what does it mean for the activity of neurons to accelerate? I understand, e.g., that objects accelerate in physics, but I don’t understand the correspondence with neural activity acceleration. Is there, for instance, a second derivative in the neural activity?
Line 470, “train cars”; do the authors mean training for automated driving?
Figure 6, I would recommend labels for left/right panels in the figure caption.
((2)) The secondary weakness of this paper technical incorrectness. Specifically, the authors make several statements that are too strong and/or not true. Highlighting a few examples to illustrate this general weakness:
For example, Line 30, “these networks [DNNs] are [not] energy efficient” – I can understand why one would want to improve the efficiency of DNNs or aspire to reach the energy efficiency of the brain, but to say DNNs are not efficient as a whole class ignores a lot of work on efficient DNNs out there. I’d suggest a less strong statement than “these networks are not energy efficient”; something like “DNNs can use a lot of energy”.
Another example is Line 50, “temporal plasticity is essential to treat temporal information” is not true. For example, one can treat temporal information with RNNs that do not have temporal plasticity.
Another example is Eq 8: later in the paper, Adam is used as an optimizer, and Eq 8 is not the equation for Adam.
Another example is Line 207, “event-based datasets consist of images from event-based cameras…” Not all event-based datasets are image datasets. And Line 2010, “the number of images and events is huge” – the total number of events does not change. The number of images may get huge, but the number of events does not get huge.
Questions
The questions that follow are more minor compared to the weaknesses communicated in the above “Weaknesses” section.
How do delays compare to synaptic time constants? For example, in LIF neurons models, often post-synaptic input ramps up over time with some time constant and then ramps down. If the ramp-up time is large, this acts similarly to a large delay. Could synaptic time constants act as substitutes to delays?
What are the advantages and disadvantages of SSNs in which neurons spike only once, versus SNNs in which neurons spike multiple times?
Is there anything that can be said about the correspondence between weight pruning (e.g., by magnitude) and your q-value (which can be used to ignore synaptic connections with long delays)?
Overall, thank you for this fascinating work. I am enthusiastic about your exploration here in delay networks and the fascinating technical details, such as the Gaussian on learnable delays to make the delays real, and your input preprocessing methodology. I find your contribution valuable, and I hope you publish an improved version of this work regardless of the ICLR review process.