MIMONets: Multiple-Input-Multiple-Output Neural Networks Exploiting Computation in Superposition

Summary

This study introduces Multiple-Input-Multiple-Output Neural Networks (MIMONets) that can process multiple inputs simultaneously, reducing computational cost. Two types of MIMONets, MIMOConv for CNNs and MIMOFormer for Transformers, are presented. MIMOConv can handle multiple image inputs with minimal accuracy loss, while MIMOFormer effectively calculates attention scores for two concurrent inputs. These models offer a dynamic balance between accuracy and processing speed, using a fixed set of parameters.

Strengths

MIMONets significantly improve the processing speed by handling multiple inputs simultaneously, reducing the computational cost per input. hey can be applied to various neural network architectures, including CNNs and Transformers.

Weaknesses

1. The concept of superposition can lead to interference between inputs, which may affect the model's accuracy. 2. The integration of variable binding mechanisms and transformations for holistic processing may increase the complexity of the model, potentially making it harder to implement and understand. 3. While the speed of processing is improved, there is a noted drop in accuracy when handling multiple inputs, which may not be suitable for applications requiring very high precision.

Questions

1. Can the MIMONets approach be generalized to other types of neural network architectures beyond CNNs and Transformers?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Not applicable.

Summary

The main content of the article is about MIMONets, which are multiple-input-multiple-output neural networks that exploit computation superposition. By using fixed-width distributed representations in vector-symbolic architectures, MIMONets can represent a variable number of inputs in a data structure and process them holistically with nonlinear neural transformations. This leads to a significant reduction in computational burden per input and offers a dynamic trade-off between accuracy and throughput. The article presents two instances of MIMONets (MIMOConv and MIMOFormer), which apply the concept of computation in superposition to convolutional neural network (CNN) and Transformer architectures, respectively. Empirical evaluations show that MIMONets achieve significant speedups while maintaining high accuracy.

Strengths

Strengths: 1. The method cleverly combines multiple inputs into a single sample for inference, reducing the computational cost. This is highly meaningful for practical applications. 2. The method achieves promising results in both CNN-based and Transformer-based structures, indicating its versatility and applicability.

Weaknesses

Weaknesses: 1. The experimental datasets and network applications in the study are relatively limited. It would be beneficial to apply the method to more datasets, such as ImageNet, and explore its effectiveness on a wider range of networks. 2. When the number of stacked samples is 4, a noticeable performance drop is observed.

Questions

1. In lines 86 and 87, why is it stated that n and $x^{(1)}$ are orthogonal? 2. I don't fully understand how Dynamic Inference is performed. 3. Are unbinding keys shared among all data or does each sample have a corresponding unbinding key?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I believe the limitations of the method mainly manifest in two aspects: 1. Performance significantly decreases when a large number of samples are stacked. 2. It remains uncertain whether similar performance can be maintained when applying the method to a wider range of structures.

Summary

This paper proposes a novel method named multiple-input-multiple-output (MIMO) neural networks, which aims to achieve simultaneous inference for several inputs together by mixing them into one input. To this end, the authors invent one method to first encode the inputs, whose output can be decoded to give separate outputs. The proposed method is evaluated on both convolutional and attention models on CIFAR datasets and LRA framework.

Strengths

The proposed method to mix several inputs is novel and interesting. Also, the authors conducted simple analysis and several experiments.

Weaknesses

The method is only verified on some small datasets like CIFAR. Also, the results in Table 1 seem to give a significant accuracy drop for multiple input cases, compared to using a single input. Also, it would be better if the author could provide more results on other tasks, like object detection or segmentation.

Questions

Could the author provide more explanation on how to apply the proposed method in practice?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors did not provide limitations. I think the practical application of the proposed method can be limited.

Summary

UPDATE: scores updated based on rebuttal. This paper proposes a method (MIMONets) for multiplexing multiple independent samples in superposition in such a way that one can train neural networks to simultaneously process those samples in training and inference. The method is adapted both for CNNs and Transformers and there is empirical evaluation for both that show the viability of the method.

Strengths

The paper is well written and easily understandable, and the method is relatively well grounded in foundations of previous works and new analysis.

Weaknesses

The authors claim that this is the first time this has been done, but they fail to mention some related work such as Murahari, Vishvak, et al. "DataMUX: Data multiplexing for neural networks." Advances in Neural Information Processing Systems 35 (2022): 17515-17527. Please review that work and discuss similarities and differences, and if possible conduct experimental comparison. The DataMUX paper claims an impressive 40 samples multiplexing for the transformers, while MIMONet uses just 4. At least, the CNN part of the MIMONet work seems to give better empirical results, though. One of the strongest motivators for the MIMONets is the ability to “dynamically” scale computation vs. accuracy at inference run-time using the same trained weights by changing the way input and output is processed, and by creating in-network ensembles if more resources are available. The authors should center the motivation more in this area unless they can have a strong argument to the “static” case. Regarding the static case, there is less convincing results. For example, since there is a drop in accuracy with multiplexing 2-4 samples compared to 1, the comparison should be done with a smaller baseline with the same accuracy, and the authors should include full FLOPS figures for the models, including the bind/unbind operation. Also, for the static case, some comparison methods include pruning, quantization aware training, compression. It would be needed to compare these in terms of intOPS/FLOPS and power consumption estimates using models of the same accuracy. MIMOConv For the MIMOConv model, the authors should include the mathematical description of bind/unbind including all dimensions and indexes. Table 1 does not clearly state the FLOPS / sample for each of the rows. Table 1 does not discuss what would be the FLOPS size of a baseline model that reaches the same accuracy as e.g., N=4 MIMOConv models. MIMOFormer The authors should add some more information on the Transformer model, for example, in the Figure 3 it seems every layer does bind/unbind. This seems different from the CNN models, so that authors should describe this more in detail and give more background to these choices. What is the complexity addition in FLOPS from these bind/unbind operations? Table 2 (MIMOFormer results) is not fully clear what is the computational complexity (per sample) for each of the rows. For example, do the MIMOFormer models have higher complexity in FLOPS because of the bind/unbind operations? I assume that MIMOFormer N=4 is almost 4x more efficient pers sample than N=1, but it would be great to report full model FLOPS/sample for each of the rows. Table 2 seems to hint that the +MLP version performs worse (especially for N=4). But do I understand correctly that it also has less FLOPS since the -MLP version repeats the same MLP for each of the superimposed samples? This should be described. Would the +MLP version perform better if MLP was bigger? Table 2 is missing comparison to a non-MIMO Performer baseline that would have the same FLOPS complexity per sample. In Table 2 there seems to be some tasks, such as retrieval, where N=4 seems to make the performance drop dramatically. Could the authors discuss these?

Questions

See above

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Comparison for DataMUX missing, some apples-to-oranges comparisons in the experimental section.

Comment

Thank you for your rebuttal! One further comment. You say "Furthermore, our approach is orthogonal to pruning, quantization, compression, and low-rank matrix decompositions. Each of them could be applied on top of our MIMO approach." I think this is too strong statement without proof. Can you explain why you believe the gains from MIMO and e.g., quantization and compression are fully additive? Intuitively it feels like quantization, compression would make it harder to do the MIMO approach with good performance. Or put in another way, there might be high dependency between these approaches and combining them might not give fully additive gains.

Comment

Thank you. You write "Those are issues with training stability, as is apparent in the large standard deviation reported in this configuration. We could opt to discard outliers or report the median instead.". It would be much better to understand the source of unstability and fix that instead. Any thoughts?

reply to author's feedback

Thanks Reply. I recognize your work, but I still have some doubts and suggestions. 1. Thank you very much for explaining dynamic inference in detail. I want to confirm again: Does Dynamic Inference refer to a single model that can infer 1 to N images? 2. It is better to verify the results on a more convincing dataset. I can understand that this article may be more analytical and theoretical. But if there is a lack of more convincing verification, it is like a lack of soul.

Thanks for the authors' response

Dear Authors, Thanks for the response and discussion. I would prefer to keep my score based on it. Best.