Library

Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts

The softmax gating function is arguably the most popular choice in mixture of experts modeling. Despite its widespread use in practice, the softmax gating may lead to unnecessary competition among experts, potentially causing the undesirable phenomenon of representation collapse due to its inherent structure. In response, the sigmoid gating function has been recently proposed as an alternative and has been demonstrated empirically to achieve superior performance. However, a rigorous examination of the sigmoid gating function is lacking in current literature. In this paper, we verify theoretically that the sigmoid gating, in fact, enjoys a higher sample efficiency than the softmax gating for the statistical task of expert estimation. Towards that goal, we consider a regression framework in which the unknown regression function is modeled as a mixture of experts, and study the rates of convergence of the least squares estimator under the over-specified case in which the number of fitted experts is larger than the true value. We show that two gating regimes naturally arise and, in each of them, we formulate an identifiability condition for the expert functions and derive the corresponding convergence rates. In both cases, we find that experts formulated as feed-forward networks with commonly used activation such as ReLU and GELU enjoy faster convergence rates under the sigmoid gating than those under softmax gating. Furthermore, given the same choice of experts, we demonstrate that the sigmoid gating function requires a smaller sample size than its softmax counterpart to attain the same error of expert estimation and, therefore, is more sample efficient.

stat.MLcs.LGComputer ScienceMathematics
View graphSource

Paper

References (39)

01Understanding Expert Structures on Minimax Parameter Estimation in Contaminated Mixture of ExpertsFanqi Yan, Huy Nguyen, Dung Le et al.2024 · arXiv.org · 7 citations In Library
02Mixture of A Million ExpertsX. He2024 · arXiv.org · 67 citations In Library
03Statistical Advantages of Perturbing Cosine Router in Mixture of ExpertsHuy Nguyen, Pedram Akbarian, Trang Pham et al.2024 · International Conference on Learning Representations · 16 citations In Library
04Mixtures of Experts Unlock Parameter Scaling for Deep RLJohan Obando-Ceron, Ghada Sokar, Timon Willi et al.2024 · International Conference on Machine Learning · 87 citations In Library
05On Least Square Estimation in Softmax Gating Mixture of ExpertsHuy Nguyen, Nhat Ho, Alessandro Rinaldo2024 · International Conference on Machine Learning · 25 citations In Library
06FuseMoE: Mixture-of-Experts Transformers for Fleximodal FusionXing Han, Huy Nguyen, Carl Harris et al.2024 · Neural Information Processing Systems · 108 citations In Library
07CompeteSMoE - Effective Training of Sparse Mixture of Experts via CompetitionQuang Pham, Giang Do, Huy Nguyen et al.2024 · arXiv.org · 25 citations In Library
08Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?Huy Nguyen, Pedram Akbarian, Nhat Ho2024 · International Conference on Machine Learning · 20 citations In Library
09DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language ModelsDamai Dai, C. Deng, Chenggang Zhao et al.2024 · Annual Meeting of the Association for Computational Linguistics · 990 citations In Library
10Mixtral of ExpertsAlbert Q. Jiang, Alexandre Sablayrolles, Antoine Roux et al.2024 · arXiv.org · 2.0k citations In Library
11A General Theory for Softmax Gating Multinomial Logistic Mixture of ExpertsHuy Nguyen, Pedram Akbarian, TrungTin Nguyen et al.2023 · International Conference on Machine Learning · 29 citations In Library
12Approximating Two-Layer Feedforward Networks for Efficient TransformersR'obert Csord'as, K. Irie, J. Schmidhuber2023 · Conference on Empirical Methods in Natural Language Processing · 29 citations In Library

Scroll for more · 27 remaining

Similar papers

Convergence Rates for Softmax Gating Mixture of ExpertsHuy Nguyen, Nhat Ho, Alessandro Rinaldo2025 · IEEE Transactions on Information Theory · 3 citations In Library
Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating FunctionT. Pham, Thinh Cao, Viet Nguyen et al.2026 · arXiv.org · 0 citations In Library
Demystifying Softmax Gating Function in Gaussian Mixture of ExpertsHuy Nguyen, TrungTin Nguyen, Nhat Ho2023 · Neural Information Processing Systems · 41 citations In Library
A General Theory for Softmax Gating Multinomial Logistic Mixture of ExpertsHuy Nguyen, Pedram Akbarian, TrungTin Nguyen et al.2023 · International Conference on Machine Learning · 29 citations In Library
On Bayesian Softmax-Gated Mixture-of-Experts ModelsNicola Bariletto, Huy-Duc Nguyen, Nhat Ho et al.2026 · arXiv.org · 0 citations In Library
On Least Square Estimation in Softmax Gating Mixture of ExpertsHuy Hoang Nguyen, Nhat Ho, Alessandro Rinaldo2024 · arXiv (Cornell University) · 26 citations In Library
Statistical Perspective of Top-K Sparse Softmax Gating Mixture of ExpertsHuy Nguyen, Pedram Akbarian, Fanqi Yan et al.2023 · arXiv (Cornell University) · 31 citations In Library
Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?Huy Nguyen, Pedram Akbarian, Nhat Ho2024 · arXiv (Cornell University) · 20 citations In Library
On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating FunctionsHuy Nguyen, Xing Han, Carl Harris et al.2024 · arXiv.org · 15 citations In Library
Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model SweepsDo Tien Hai, Trung Nguyen Mai, T. Nguyen et al.2025 · arXiv.org · 3 citations In Library

Peer review

Accept (poster)NeurIPS 2024 · via OpenReview
Reviewer 4kwq6/10 · confidence 3/52024-07-11

Summary

This paper provides a theoretical analysis of the convergence rate of the least squares estimator for learning MoE with sigmoid gating. Based on the results, the authors conclude that sigmoid gating enjoys a faster convergence rate and requires a smaller sample size to achieve the same error compared to softmax gating.

Strengths

1. The topic is very interesting. It is intriguing to see how the selection of gating functions can affect sample efficiency in MoE. 2. This problem is particularly relevant to recent important topics regarding language models, given that MoE has been applied in popular models like Mixtral.

Weaknesses

1. The biggest concern to me is that the results from this paper do not fully support the main claim that “sigmoid gating is more sample efficient than softmax gating”. If I understand correctly (correct me if I'm wrong), the comparison between sigmoid and softmax is not under the same scenario. The convergence rate for softmax in Table 1 is from the analysis in [21], where the ground truth is an MoE model with softmax gating, while this paper considers the setting where the ground truth is an MoE model with sigmoid gating. These are two different setups and do not necessarily imply that, given the same ground truth function, sigmoid is necessarily more sample efficient than softmax. 2. The only existing empirical work discussing the potential superiority of sigmoid gating that the authors have mentioned is [3], where the provided intuition is that softmax introduces competition between experts, while sigmoid does not have this issue. This is very straightforward, although not yet formally and theoretically justified. On the other hand, this paper explains the advantage from the perspective of sample efficiency. I wonder how these two perspectives can be connected. Is there anything regarding the competition view that can be reflected in this paper’s analysis? I believe this is important for understanding the effect of sigmoid gating. 3. Following point 2, it would be helpful to provide empirical results to further support the main conclusion that sigmoid gating is more sample efficient. Even experiments on toy data can help. Otherwise, the statement regarding sample efficiency lacks empirical evidence. Additionally, even "sigmoid gating is better than softmax gating" alone does not seem to be well-supported by empirical observations from the literature.

Questions

See my questions raised in Weaknesses.

Rating

6

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors have discussed certain limitations in Section 4.

Authorsrebuttal2024-08-13

Response to Reviewer 4kwq

Dear Reviewer 4kwq, We would like to thank you for raising your concerns. **We hope that our following response will address those concerns, and eventually convince you to increase your rating**. Per your suggestion, we conduct both theoretical and empirical sample efficiency comparisons between the sigmoid gating and the softmax gating under the setting where the data are generated from the same source. Below are the expert estimation rates when using the softmax gating MoE and the sigmoid gating MoE to fit the data, respectively. **Same data generation setting:** The data $(X_1,Y_1),\ldots,(X_n,Y_n)$ are generated from a regression framework $$Y_i=g(X_i)+\epsilon_i,$$ where the features $X_1,\ldots,X_n$ are sampled from a probability distribution $\mu$, and $\epsilon_1,\ldots,\epsilon_n$ are independent Gaussian noise variables such that $\mathbb{E}[\epsilon_i|X_i]=0$ and $Var(\epsilon_i|X_i)=\nu$ for all $1\leq i\leq n$. The unknown regression function $g(x)$ is not necessarily a mixture of experts (MoE). Then, we use either the sigmoid gating MoE or the softmax gating MoE to fit the data. In particular, the least square estimator $\widehat{G}_n$ in Eq.(3) now converges to a mixing measure $\widetilde{G}\in\mathcal{G}_k(\Theta)$ where $$\widetilde{G}\in\arg\min_{G}||f_{G}-g||_2,$$ in which $f_{G}$ is the regression taking the form of the MoE associated with either the sigmoid gating or the softmax gating. Below are the expert estimation rates resulted from our convergence analysis: **When $f_{G}$ is the softmax gating MoE:** the rates for estimating feed-forward expert networks with ReLU activation are of order $n^{-1/4}$, while those for polynomial experts are slower than any polynomial rates, and could be as slow as $1/\log(n)$; **When $f_{G}$ is the sigmoid gating MoE:** the estimation rates for feed-forward expert networks with ReLU activation and polynomial experts share the same order of $n^{-1/2}$, which are significantly faster than those when using the softmax gating MoE. **Empirical validation:** To justify the above theoretical results, we conduct a simulation study in the General Response section where we take polynomial experts into account. The empirical result totally matches its theoretical rates. As a consequence, **we can conclude that the sigmoid gating is more sample efficient than the softmax gating even when fitting the same data**. Please feel free to let us know if you have any further concerns regarding the paper. We are more than happy to address all of them. Best, The Authors

Reviewer 4kwq2024-08-13

Thank the authors for their response. I’m curious about how the new theoretical results presented relate to those in the original paper. Are they completely independent, an extension of the original findings, or already implied by them? If they are a more generalized version of the original results, do we need to introduce any new assumptions here, particularly any assumptions on $g(x)$?

Authorsrebuttal2024-08-13

Dear Reviewer 4kwq, Thanks for your question. We would like to confirm that the analysis of the sigmoid gating (resp. softmax gating) MoE under the same data generation setting is implied by the analysis in our paper (resp. in [a]). **Assumption on the regression function $g$.** Firstly, let us recall a result on the universal approximation of the sigmoid function in [b]. In particular, let $g:\mathcal{X}\to\mathbb{R}$ be a function such that there is a Fourier representation of the form $$g(x)=\int_{\mathcal{X}}e^{i\omega\cdot x}\tilde{g}(\omega)d\omega,$$ for some complex-valued function $\tilde{g}(x)$ for which $\omega\tilde{g}(\omega)$ is integrable and the term $$C_g=\int_{\mathcal{X}}||\omega||_2|\tilde{g}(\omega)|d\omega$$ is finite. Then, there exists a linear combination of $\tilde{k}$ sigmoidal functions $f_{\widetilde{G}}(x)$ such that $$\int_{\mathcal{X}}[f_{\widetilde{G}}(x)-g(x)]^2d\mu(x)\leq\frac{(2C_g)^2}{\tilde{k}}.$$ **Analysis for the sigmoid gating MoE.** Subsequently, we combine the above result with our current analysis in the paper. More specifically, by treating the mixing measure $\widetilde{G}$ as the mixing measure $G_*$ in the paper, we are able to design a Voronoi loss $$\mathcal{D}(G,\widetilde{G})=\sum_{j=1}^{k}\sum_{i\in\mathcal{A}_j}$$ $$\Big[|\beta_{0i}-\tilde{\beta}_{0j}|$$ $$+||\beta_{1i}-\tilde{\beta}_{1j}||$$ $$+||\eta_{i}-\tilde{\eta}_{j}||\Big],$$ and show that $$\mathcal{D}(\widehat{G}_n,\widetilde{G})=O_P(\sqrt{\log(n)/n}).$$ From this bound, we deduce that the expert estimation rates are of order $O_P(n^{-1/2})$. **Analysis for the softmax gating MoE.** Similarly, we also leverage the universal approximation of the softmax function in [c], and the analysis in [a] to derive the expert estimation rates under the same data generation setting. In particular, we find out that the rates for estimating feed-forward expert networks with ReLU activation are of order $n^{-1/4}$, while those for polynomial experts are slower than any polynomial rates, and could be as slow as $1/\log(n)$. **As the discussion period deadline is approaching, please let us know if you have any further concerns. We are more than happy to answer your questions. Additionally, if you find that our response sufficiently addresses your concerns, we hope that you will re-evaluate the paper and increase the rating. Thank you again!** **References** [a] H. Nguyen. On least squares estimation in softmax gating mixture of experts. In ICML, 2024. [b] Andrew R. Barron. Universal Approximation Bounds for Superpositions of a Sigmoidal Function. IEEE Transactions on Information Theory, 1993. [c] Assaf J. Zeevi. Error Bounds for Functional Approximation and Estimation Using Mixtures of Experts. IEEE Transactions on Information Theory, 1998.

Reviewer 4kwq2024-08-13

Thank the authors for addressing my concerns. I believe the step of connecting the two data generation processes with the universal approximation properties of the sigmoid and softmax functions is essential in clarifying and legitimizing the comparison. It would be beneficial to include this in the revised version. I have adjusted my score accordingly.

Authorsrebuttal2024-08-13

Thank You!

Dear Reviewer 4kwq, We would like to thank you for increasing your rating to 6, we really appreciate that. We will include the connection of the universal approximation properties of the sigmoid and softmax functions and the derivation of expert estimation rates under the same data generation setting in the revision of our paper. Best, The Authors

Reviewer tkzC6/10 · confidence 3/52024-07-16

Summary

The paper argues that the sigmoid gating function is more sample efficient than the softmax gating function in mixture of experts (MoE) modeling. It removes competition and estimates the contribution each expert independently. Empirical studies show that sigmoid gating achieves superior performance, but the paper aims to provide theoretical backing for this claim. They consider a regression framework and analyze the rates of convergence of the least squares estimator in over-specified cases. The convergence rates for expert estimation are derived under two regimes: Regime 1 (all over-specified parameters are 0) and Regime 2 (at least one over-specified parameter is not 0).

Strengths

The paper demonstrates that sigmoid gating is more sample efficient than softmax gating, requiring fewer samples to achieve the same level of accuracy in expert estimation. experts with feed-forward networks and commonly used activations (ReLU, GELU) have faster convergence rates under sigmoid gating than softmax gating. The sigmoid gating mechanism is compatible with a broader class of expert functions compared to softmax gating. (ReLU and GELU, as well as polynomial activations)

Weaknesses

The results are heavily dependent on specific assumptions, such as the distinctness of expert parameters and the boundedness of the input space. If these assumptions are violated in practical scenarios, the theoretical guarantees may not hold While the derivations are very appropriate and i appreciate the citations, it would be interesting to see even toy experiments byt the authorsthat show similar conversions to the theoretical ones.

Questions

the paper lacks justification for the choice of the regimes. We also need to deep dig to understand the reasons for strong identifiability conditions (for experts to be distinct enough, if i understand correctly), bracketing entropy (model is not overly complex) and concentration inequalities. it is quite difficult to read, so some more intuition and explanation would be helpful. Minor: 124 in can seen

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

As mentioned by the authors, The paper assumes that the ground-truth parameters are independent of the sample size, leading to point-wise rather than uniform estimation rates. The assumption that the true regression function belongs to the parametric class of MoE models under sigmoid gating is restrictive. This assumption is likely to be violated in real-world settings. I still find these assumptions reasonable.

Reviewer PmAq5/10 · confidence 3/52024-07-18

Summary

This paper presents a theoretical analysis of expert estimation in MoE models using sigmoid gating, in contrast to the more widely used softmax gating. The authors show that sigmoid gating leads to better sample efficiency compared to softmax gating for estimating expert parameters. In particular, the paper analyzes convergence rates for least squares estimation under two regimes: * Regime 1) when all over-specified gating parameters are zero * Regime 2) when at least one of the over-specified gating parameters is non-zero One of the key findings is that experts formulated as NNs with common activations like ReLU and GELU obtain faster convergence rates under sigmoid gating compared to softmax gating. This work provides theoretical justification for the potential empirical benefits of sigmoid gating, showing it requires smaller sample sizes to achieve the same estimation error as softmax gating.

Strengths

- The paper provides rigorous proofs and establishes convergence rates under different conditions. - Sigmoid gating function has been less explored in the MoE field and analyzing its behaviour for expert estimation problem is interesting.

Weaknesses

1) The structure of the paper is very confusing and could be much improved. There is way too much repetition in the paper. For example, the section Technical challenges in the introduction which defines Regime 1 and 2 (Lines 83-92) is largely copy pasted to lines 155-165 and is redundant. The various sections in the introductions particularly the contributions should be kept at a more high level and not just copy paste the method section. The authors should provide a more concise overview of the problem and main contributions. Remove technical details that are repeated in later sections. 2) The recent insights into training Sparse MoEs demonstrate that "the common practice of setting the size of experts in MoE to mirror the feed-forward layer (of a base dense network) is not optimal at almost any computational budget" (Jakub Krajewski et al. 2024). If in practice, top-k combination of small fine-grained experts is proven to be more effective than a single large expert, are we dealing in a regime where all true experts are *over-specified* in practice? If this is the case what are the implications for Regime 1? Does it mean that none of the experts are input-dependent as all the over-specified parameters $β^*_{1i}$ are equal to $0_d$? Does Regime 1 hold in practice? 3) It would be best if the authors could find some practical implications of their findings. The theoretical work presented in the paper does not establish any relationship to any of the SoTA MoE models in the literature. It is unclear to me how impactful the convergence analysis with the considered regimes is in practical MoE research works. 4) At present, the content and organization of the paper is very close to the paper "On Least Square Estimation in Softmax Gating Mixture of Experts" (Nguyen et al. 2024). The sections are so similar that may cause copyright issues. References: 1. Krajewski, Jakub, et al. "Scaling laws for fine-grained mixture of experts." arXiv preprint arXiv:2402.07871 (2024). 2. Nguyen, Huy, Nhat Ho, and Alessandro Rinaldo. "On least squares estimation in softmax gating mixture of experts." ICML (2024).

Questions

Minor point: - misspecified is misspelled in line 175.dq

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

Two limitations concerning the assumptions used in the analyses are discussed in the limitations section.

Reviewer tkzC2024-08-11

Thank you for addressing the questions. As I am leaning towards acceptance, I raised my score to 6.

Authorsrebuttal2024-08-12

Thank You

Dear Reviewer tkzC, We want to thank you for increasing your score to 6. Please let us know if you still have any concerns about the paper. Best, The Authors

Reviewer PmAq2024-08-12

Thanks for your detailed response. Most of my concerns are addressed. I would encourage the authors to improve the conciseness of the presentation in the revised version as discussed in the comments. I raise my rating to 5

Authorsrebuttal2024-08-12

Thank you!

Dear Reviewer PmAq, We would like to thank you for increasing the rating to 5. We will definitely incorporate the modifications into the revision of our paper as discussed. Please feel free to let us know if you have any further concerns. Best, The Authors

Reviewer 4kwq2024-08-12

Thank you to the authors for their responses. However, I am not entirely sure how the authors' response addresses my first question. My question was: this paper and Nguyen et al. [21] consider different settings. This paper considers the case where the data are generated with sigmoid gating, and a model with sigmoid gating is trained to fit the data; Nguyen et al. [21] consider the case where the data are generated with softmax gating, and a model with softmax gating is trained to fit the data. Since the data in these two cases are not generated in the same way, we cannot claim that sigmoid gating is more sample efficient than softmax if we use them to fit the same data. In other words, if the paper had shown that, given the same data generated in a certain way, using sigmoid to fit the data is more sample efficient than using softmax, then I would find the conclusion well justified. The new empirical results seem to be done in the desired setting, where the data are always generated with softmax gating, whether using the sigmoid gating model or the softmax gating model to fit the data.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC