The rapidly increasing size of deep-learning models has caused renewed and growing interest in alternatives to digital computers to dramatically reduce the energy cost of running state-of-the-art neural networks. Optical matrix-vector multipliers are best suited to performing computations with very large operands, which suggests that large Transformer models could be a good target for optical computing. To test this idea, we performed small-scale optical experiments with a prototype accelerator to demonstrate that Transformer operations can run on optical hardware despite noise and errors. Using simulations, validated by our experiments, we then explored the energy efficiency of optical implementations of Transformers and identified scaling laws for model performance with respect to optical energy usage. We found that the optical energy per multiply-accumulate (MAC) scales as $\frac{1}{d}$ where $d$ is the Transformer width, an asymptotic advantage over digital systems. We conclude that with well-engineered, large-scale optical hardware, it may be possible to achieve a $100 \times$ energy-efficiency advantage for running some of the largest current Transformer models, and that if both the models and the optical hardware are scaled to the quadrillion-parameter regime, optical computers could have a $>8,000\times$ energy-efficiency advantage over state-of-the-art digital-electronic processors that achieve 300 fJ/MAC. We analyzed how these results motivate and inform the construction of future optical accelerators along with optics-amenable deep-learning approaches. With assumptions about future improvements to electronics and Transformer quantization techniques (5$\times$ cheaper memory access, double the digital--analog conversion efficiency, and 4-bit precision), we estimated that optical computers' advantage against current 300-fJ/MAC digital processors could grow to $>100,000\times$.
Paper
Similar papers
Peer review
Summary
The authors analyze the performance, efficiency, and robustness of free-space optical dot-product engines for Transformer accelerations. Measurement results on an SLM-based optical system are demonstrated on some layers in a GPT-like model. System performance/efficiency are estimated and compared to digital computers. Scaling of optical processors are discussed to show the scalability of optical computing platforms.
Strengths
1. Experimental results on SLM-based free-space optical system has been demonstrated for matrix multiplication. 2. Scalability with future technologies are discussed to show the benefit of optical computing in the future.
Weaknesses
1. The novelty of the paper raises some concerns as no new hardware design or algorithm innovations have been shown. The SLM-based system and its experimental demonstration are not new. No customized hardware is shown for Transformer. The claimed optical hardware is designed for CNNs/MLPs. On the algorithm part, device quantization, the LUT-based training method, noise analysis, and 4-pass multiplication are standard methods for analog computing. NeurIPS community usually requires certain machine learning contributions. What is the main ML contribution? Probably other venues in the optics community are more suitable for this paper. 2. The demonstrated system is weight-in-place which needs a large number of parallel MVM to amortize the weight programming/encoding cost. However, the dynamic attention operations in Transformer and fully-connected layers usually have low arithmetic intensity, especially GPT-like architecture with KV cache, which cannot provide enough batch dimension to amortize such cost for weight-in-place systems. More justification for the usage of the weight-in-place system needs further discussion. A weight streaming system might be the suitable architecture for Transformer. 3. In Fig. 2, only a small part of layers in the Transformer block are implemented by optics, while other operations are on all digital computers. In this hybrid case, how large are the efficiency/performance benefits, or is it worthwhile to use optics? 4. For the noise analysis, only shot noise is emphasized, which is much smaller compared to other variations in the system both on the electrical and optics sides. A simple Gaussian added to the output results might be oversimplified as system error modeling. 5. In Line 291, the system assumes a 10 GHz light modulator array. If I understand correctly, the spatial light modulator typically has high resolution but very low switching frequency. This 10 GHz modulation speed needs further justification. How fast is the switching freq for weights and input feature maps? The modulation energy cost is based on thin-film lithium niobate modulators, which are fairly large. How many such large modulators are required to modulate a million pixels? 6. Also, the light source/TIA/ADC power consumption in the camera will be very large if working at such a high frequency. The incoming data fetched from memory will also be a bottleneck, which might not be able to fast enough to feed the 10 GHz optical core. In Line 297, the memory part is completely ignored when compared with digital computers, which might not be a fair comparison even in the near future. Only multiplications are done in optics, the partial product summation is done digitally, especially when it requires 4-pass, which raises concerns about the benefit of this SLM system for Transformer acceleration. More discussion on the system performance/efficiency is recommended. 7. The paper title optical Transformer is very broad, however the current paper only focuses on SLM-based free-space optics, which suffer from bulky optical setups (low compactness) and noise/alignment/sensitivity issues in practical deployment. More discussion and comparison on other integrated photonic/diffractive hardware is important and necessary. Otherwise, the scope/title of the paper is more suitable to be narrowed down to an SLM-based Transformer acceleration platform.
Questions
My questions are listed in the above weaknesses.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
1 poor
Limitations
Yes.
Summary
This paper propose a photonic hardware accelerator to process the inferences of large language models, i.e., transformers, using optical multiply-accumulate (MAC) operations. Optical MACs are suitable for computations with large operands, thereby leading to asymptotic energy advantages over the digital hardware accelerators.
Strengths
1. The paper is well-organized. 2. The paper works on an important problem.
Weaknesses
1. The paper does NOT consider the energy consumption of analog-to-digital converters, digital-to-analog converters, and various memories such as on-chip SRAM and off-chip DRAM. I totally agree with the cornstone of this paper, which is optical MACs or matrix multiplications are super energy-efficient. However, gaining this energy advantage is not easy. Reading 530B parameters of a transformer, converting these many digital parameters to analog optical signials, and converting the analog optical result signials back to digital values may donimate the energy consumption of an inference. As a result the energy efficiency improvement may not be very large. Please check the comparison in this paper: W. Liu, W. Liu, Y. Ye, Q. Lou, Y. Xie and L. Jiang, "HolyLight: A Nanophotonic Accelerator for Deep Learning in Data Centers," 2019 Design, Automation & Test in Europe Conference & Exhibition (DATE), Florence, Italy, 2019, pp. 1483-1488, doi: 10.23919/DATE.2019.8715195.
Questions
Please comment on the energy consumption of analog-to-digital converters, digital-to-analog converters, and various memories such as on-chip SRAM and off-chip DRAM during the otpical transformer inferences.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
No potential negative societal impact.
Summary
This paper explores the feasibility and benefits of employing optical computing techniques for machine learning and specifically focusing on large language models (LLMs). The paper builds upon earlier work on optical neural networks, primarily [61] (Wang et al., Nature, 2022), which experimentally demonstrated the feasibility of performing dot products optically in a two layer neural network applied to MNIST while achieving around 90% accuracy at about one photon per multiplication optical energy. As LLM computations involved a large and rapidly growing number of multiply accumulate operations, the objective of the paper is to explore whether optical techniques can yield benefits over existing CMOS-based accelerators (GPUs, TPUs). The paper tackles this by employing a simulation based methodology where the simulator attempts to model the various noise sources (systemic, shot noise) along with limited precision of a potential electro-optical system. The evaluation shows that as LLMs continue to scale such optical systems may potentially yield many order of magnitude benefits in terms of energy efficiency over current approaches.
Strengths
Makes a reasonably strong case for further exploration of optical computing for large language models.
Weaknesses
Not enough discussion of the remaining challenges that need to be overcome to make such systems competitive in reality. Somewhat limited contributions in-so-far as earlier works have already explored using optical for ML. Unclear how accurate the simulation methodology is. Some aspects unclear.
Questions
How far are current optical systems from the 10GHz operating frequency assumed in this paper (Line 291) and what would Figure 5 look like and to what extent would the main conclusions about the benefits of optical be undermined if more modest (or perhaps realistic) rates were assumed? How much improvement is required in this direction to achieve the hoped for benefits? What frequency do current modulators operate at? How do you separate out shot noise from systematic noise in the measurements in Figure 3? Does the y-axis in Figure 4 of the main paper include the electrical energy overheads like those in Figure 3 and 4 in the supplemental? Figure 3 in the supplemental seems to show optical energy is entirely negligible so in this case the accuracy of Figure 4 in the main paper would be entirely dependent upon a full accounting of all other energy sources. What assurances can you give that all significant sources of non-optical energy are properly accounted for? The assumption of weights-in-place seems quite unrealistic and while the supplemental discusses (Section G) a "chunking" technique to scale up to larger numbers of parameters it was unclear whether the assumptions used to plot Figure 6 in the supplemental section are optimistic or conservative and to what extent. Please comment. Transformers are having a lot of impact, which makes the focus of this paper on them make sense, but can you comment on how much optical may other network architectures? Accelerators like GPUs and TPUs try to be relatively general purpose and so were able to be quickly retargeted to transformer based models without needing much in the way of hardware changes. Can your simulation based study approach yield insights about whether optical will help networks other than transformers and/or under which circumstances? How accurate is the simulation methodology in Figure 2? Can you attempt to compare your simulators predictions versus your actual hardware for a small network like the one studied in [61]? I did not follow what the lookup-table (LUT) in the simulator does. What values do the LUTs in the simulator contain and what quantity is used to index into the LUT?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
There is some discussion of limitations. If the paper is meant to rally others to work on optical techniques for machine learning it would be helpful if the authors could more systematically highlight the remaining challenges to making such systems practical.
Summary
The authors perform experimental analysis with a spatial light modulator to optically perform the computations of the linear components of the Transformer architecture. These measurements allow them to create a noise-model that is then used to simulate a GPT-2 like model and measure performance (validation perplexity) in function of model parameters (system noise, not including optical shot noise). These optical systems are physically constrained by "optical shot noise" that dictates the minimum number of photons to achieve a target precision. This optical shot noise scales favorably with larger models, and the authors then extrapolate their observations to existing large Transformer models (like PaLM), and further to even larger hypothetical future models, showing a substantial advantage of a large scale hypothetical optical system over existing electrical systems (the energy advantage scaling with the width of the model).
Strengths
1. While there have been previous works that examine smaller scale optical neural networks and the optical shot noise scaling behavior seems to be well established (I'm inferring this from the works cited in the submission, I'm not familiar with the area), the authors highlight the importance of the scaling behavior in context of the Transformar architectures that are typically used with large language models. I think the combination of innovative research in the field of optical computation with the computation requirements of applied machine learning models at the largest scales is particularly interesting. 2. The authors used small scale experiments to establish realistic system properties and then simulated an entire Transformer to check how the physical properties would influence overall (validation) performance of the model. They did this at a scale (GPT-2 like quantized model with 15M-416M parameters) that seems large enough to gather realistic predictions. 3. The interpolation of the simulated data to much larger models is necessarily based on many assumptions, both in scaling from the simulation to larger models, and with respect to the hypothetical hardware that would run very large models on optical hardware. The appendix goes into some detail on the different assumptions that led to the conclusions summarized in the main part of the article. 4. The overall presentation of the work seems adequate for a public that is knowledgeable in machine learning, but probably has much less experience with physical properties of optical computational hardware, which is no easy task given the relatively large gap between these domains.
Weaknesses
1. All the discussion of the optical vs. electrical implementation of the computation is in the lens of energy consumption. After reading the paper I'm not sure how the proposed architecture would fare with respect to latency/speed, or other constraints (e.g. if there is a theoretical limitation to the size – or cost – of different components that scales very differently between the currently used IC technology of purely digital microchips vs. optical components). Latency is mentioned in Appendix G, but there is no mention of these other dimensions in the main text. 2. The authors highlight the scaling behavior of optical shot noise, but other sources of noise (called "systematic errors" in the paper) are simply measured for the system at hand, and no information is given how these other sources of noise would scale when the system is scaled. For example, I assume that any realistic scaling would require miniaturization of the optical components, and I would assume that this miniaturization also causes the systematic errors to change in relative magnitude. It would be interesting to at least briefly discuss the scaling behavior of these other sources of noise.
Questions
Small things I noticed while reading the paper: a) 155-157: not clear what this means: “We collected lookup tables (LUTs)—mappings of the available discrete levels in both the display and SLM devices—and used them to train a “LUT-aware” optical Transformer model to run on the setup.” why are the LUTs needed? what does “collected” refer to? (measurements used for downstream analysis? used for the computation itself?) … maybe referencing what is exactly meant with LUTs (is it a hardware component?) would also help here b) 171, 179: the abbreviation QAT is used without being introduced (it’s spelled out on line 194) c) Table 1 is a bit hard to read because the “Setting” column spans multiple rows, but the text neatly aligns (Hardware->QAT, Simulation->Eval, etc) with the individual rows (one could e.g. add more horizontal lines in columns 2+ to make this clearer) d) 196: when saying “int8” is it actually “uint8” ? e) Figure 5 I find it surprising to see TPU int8 use 800fJ/MAC vs. NVIDIA A1000 fp16 300 fJ/MAC – I could not readily find these numbers in [47] – how exactly were they computed? f) 329: either “study” or “illustrate” (but not “our studies illustrates”) g) Appendix, Figure 4 (left): “Digital Ops” are not visible – why? If they are zero (e.g. “not doing any compute”) then that should be mentioned. h) Appendix, Figure 4 (right): the colors are hard to tell apart (especially it would be interesting to clearly see optical vs. electrical). It’s also not very clear what are “Ele FF” and “Ele Attn FF” (maybe ReLU6, LayerNorm, and Add from Figure 2?)
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
I don't see a negative societal impact of the presented work, other than making Transformers more energy efficient would allow to scale them further than otherwise possible, and accordingly accentuate any potential opportunities and risks of models such as LLMs. I don't think it's required to point these out explicitly in the paper. There seem to be a number of technical limitations with respect to the predictions in the sense that there is a lot of uncertainty whether the hardware to run optical neural networks could be scaled up as much as state of the art digital circuits. But the authors make it clear already in the abstract that their main interest is in establishing a scaling law and not concrete predictions about the future of the implementation of large scale optical neural networks.
Thanks for the rebuttal. I increase my score.
I would like to thank the authors for their excellent rebuttal, both in the common part, as well as in the individual follow-ups. This has reinforced the confidence in my rating, and it will be interesting to see if other reviewers adapt their scores accordingly. Some follow-up comments from my side: 1. I'm looking forward to the inclusion of additional optical systems in the paper ("Comparison to Other Experimental Platforms" from the common rebuttal). In particular, this would give a good opportunity to include some of the discussion above about speed/latency, errors from other systems and how they likely scale with miniaturization. I'm somewhat skeptical of the authors' claim "One could also imagine scaling up a system by fanning out data to multiple copies of the same system", as this only allows for very limited scaling. 2. Looking at Figure 4 in (Reuther, 2022) still leaves me wondering why Figure 5 in the submission has the efficiency ordering ASIC (100 fJ/MAC) > A100 (300 fJ/MAC) > TPUv1 (800 fJ/MAC), since that same Figure 4 in (Reuther, 2022) shows (approximately) 400W for 3e5 GOps/sec (which would be 1333 fJ/Op, and not 300 fJ/Op, and reverse the ordering A100/TPUv1). 3. I found the additional explanations as to the specific use of LUTs very insightful. I would appreciate it if the authors updated the main text in the submission to make this point clearer.
We thank the reviewer again for their additional feedback. Regarding the GPU estimate, a general approach we took throughout this work was in cases of ambiguity we preferred to overestimate the GPU's efficiency and underestimate the ONN's advantages. We therefore aimed to provide as generous an estimate as possible for the NVIDIA A100 because finding consistent information about its real world, FP16, inference-mode performance was hard. For example the value reported by Reuther et. al appeared to be for training, which is more data-bottlenecked than inference. Also, the 400W GPU appears to be the SXM variant of the A100 which consumes an additional 100W of power. So to build an estimate we divided the rated power consumption of a typical GPU by the peak reported performance on NVIDIA's datasheet for the card; we assumed typical GPU power consumption of 200W for ~624 TFLOPs of compute. In the best case (200W, but a more typical power consumption for A100 would be ~250-300W), this yields roughly 300 fJ/MAC (depends on how MACs/FLOPs are counted, however). This way, despite ambiguous information about the A100's real-world characteristics, we could be reasonably confident that our reported ONN energy advantages were not overestimated. We agree with the reviewer that in practice the cost could be higher than estimated for GPU. We also agree that the work could benefit from additional discussion about related ONN works/hardware, miniaturization/scaling, and the LUTs. We hope to add these to a revised version if possible.
Further Comments
Thanks for the responses. 1. LLM Inference Caching, Weights-in-Place vs Streaming Weights Even though the author claims a 10GHz SLM, for most weight-in-place integrated photonic accelerators to have low power, lowloss and small area, the switching speed is much lower than the computing speed (1 us vs 100 ps). It is not justified which workload, even for the referred Transformer that performs the full attention/MLP computations, can amortize such a high reprogramming cost. I don't think the batch/token for any vision/language task can merge this gap. So the claimed advantage of weight-in-place design over streaming one is not convincing. 2. Shot Noise vs Other System Variations. The claim is for free-space optics (some prior work indeed claims low light, like sub-photon per MAC), but not for current integrated photonics. Given the current PD sensitivity, responsivity, and OSNR limit, it is not convincing why shot-noise will dominate. More discussion is needed if this paper is for general optical architectures. 3. Data Access Cost (line 297 main text) Similar to the above comments, reusing itself is hard to amortize the memory loading and device programming latency for most optical hardware, even for large batches. Some special computer architecture designs are necessary to hide the latency, which should be pointed out. This is the fundamental bottleneck for most analog hardware accelerators. It definitely requires cross-layer solutions to mitigate it, not just reusing.. 4. Cost of Summation The four-pass method will introduce ~4x the latency and energy cost compared to one-shot computing. I still don't quite understand why this is insignificant in the overall efficiency/performance. 5. Discussion on Integrated Photonic Platforms This is the critical limitation of this paper, which claims "optical transformer" but lacks systematic analysis/comparison across different optical hardware platforms, and it requires significant modification/major revision of the paper.
Re: Further Comments
We thank the reviewer again for the additional feedback. **Amortized Loading Costs** For a fully weights-in-place system, the weights do not need to be switched. In other cases, Transformers' weight reuse can be sufficient to amortize loading costs. Typically, recent LLMs have sequence length $L \sim 10^4$. Without batching, at 10-GHz input, this only requires ~MHz-regime switching of weights, ~1us latency. For energy, we discussed this scenario in appendix G. For example, with 10G weights memory, ONNs enjoy a > 100x energy-efficiency advantage for the largest models, with models smaller than FUTURE-129T retaining nearly their full advantage. For attention weights-in-place may not be worthwhile as frequent switching is necessary. In all of our calculations we already assume that attention operations are computed with a streaming-weights approach. In this case, the weights are streamed by a light modulator in a similar fashion as the input data. We also acknowledge this at the end of section 4 (line 313). Consequently, implementing attention efficiently is hard, but MLP is the majority of Transformers' compute; ONNs still achieve an advantage in total MACs versus total energy (Appendix E). **Clarification About Noise/Error Analysis** We did not wish to claim that shot noise may dominate in most ONN systems, but rather to point out that a certain number of photons is necessary for a particular SNR requirement. Testing this requires examining the shot-noise-dominant regime. There also have been ONNs using low light, <100 photons for each photodetection [2, 4]. This translates to ~10 SNR, which is greater than hardware error. **Latency** Proposing a detailed specialized computer architecture is beyond the scope of our work. We acknowledge that the design of a suitably fast and efficient ONN system is still an open question. We hope that future ONNs continue to mature in this direction. **Cost of Summation** Altering energy costs by small constant factors (<10x) does not affect the main conclusions of this work: - **At the large scale, ONNs could achieve orders-of-magnitude energy-efficiency advantages over GPU**. As long as the overhead cost is also not orders-of-magnitude more expensive, it will not affect this claim - **The ONN advantage scales with the model size**. An increase in data-access cost only shifts the still asymptotically different energy curve - **The energy calculations are estimates**. We intended for our energy calculations to offer a discussion about ONNs' efficiency, not to predict specific numbers exactly; The discussions about scale and energy use of Transformers would not change Also, we remark that there would be *less than a 4x penalty* --- there are other considerations that would be unaffected but were a significant portion of the costs of digital ops. The presence of activations like ReLU and softmax means only two passes are needed in certain scenarios. Coherent-light ONNs can natively process real numbers (see systems comparison in Author Rebuttal). **Comparison to Other Platforms** Our work concludes that Transformers *can* run and are worth running in the presence of common analog-optical-system behaviors and pathologies. This suggests that in general, the use of optics to accelerate these large-scale models is worth pursuing. These central claims stand regardless of our limited discussion about the specifics of other ONN systems. In pursuit of these higher-level claims, we considered aspects that are fairly general: - Optical fan-out/in and related energy advantages are common to many ONNs - Other ONNs do have similar error profiles to ours [1, 2] - The scale-relative behavior (rescaling of operands) is common in analog computing - We emulated imprecision via the use of real LUTs - We tested Transformers at different precision levels, not just those in experiment (Appendix B.4, main text Fig. 3) - We assumed the use of memory, DAC, etc., which ONN accelerators require - We deliberately used commonplace techniques to make conclusions that do *not* rely on the details of a bespoke software or hardware approach. - Our energy estimations and assumptions are similar to the approach used for other ONN architectures in the literature [3, 4] These encompass much of the designs of ONNs, sufficient to paint the general picture of ONNs and Transformers. Nevertheless, we agree that this discussion would be helpful to include in the article. [1] Feldmann et al. Parallel convolution processing using an integrated photonic tensor core. arXiv:2002.00281. (2020) [2] Sludds et al. Delocalized photonic deep learning on the internet's edge. Science (Vol. 378, Issue 6617, pp. 270–276). (2022) [3] Hamerly et al. Large-scale optical neural networks based on photoelectric multiplication. Physical Review X, 9(2):021032. (2019) [4] Wang et al. An optical neural network using less than 1 photon per multiplication. Nature Communications, 13 (1). (2022)
Decision
Reject