Bridging Discrete and Backpropagation: Straight-Through and Beyond

Backpropagation, the cornerstone of deep learning, is limited to computing gradients for continuous variables. This limitation poses challenges for problems involving discrete latent variables. To address this issue, we propose a novel approach to approximate the gradient of parameters involved in generating discrete latent variables. First, we examine the widely used Straight-Through (ST) heuristic and demonstrate that it works as a first-order approximation of the gradient. Guided by our findings, we propose ReinMax, which achieves second-order accuracy by integrating Heun's method, a second-order numerical method for solving ODEs. ReinMax does not require Hessian or other second-order derivatives, thus having negligible computation overheads. Extensive experimental results on various tasks demonstrate the superiority of ReinMax over the state of the art. Implementations are released at https://github.com/microsoft/ReinMax.

Paper

References (41)

Scroll for more · 29 remaining

Similar papers

Peer review

Reviewer A6oc7/10 · confidence 3/52023-07-07

Summary

The presented paper tackles the optimization of the parameter of the distribution of a discrete random variable through stochastic gradient descent by addressing the problem of gradient estimation given samples from the discrete distribution. The authors consider three popular estimators as a starting point for their investigation, namely, the reinforce estimator, the straight-through estimator and the straight-through gumbel-softmax estimator. The authors first establish that the straight-through estimator is a first-order approximation of the true gradient, since the difference $f(I_i) - f(I_j)$ in the true gradient is replaced by $\frac{\partial f(I_j)}{\partial I_j}(I_i - I_j)$. The author thus replace the $\frac{\partial f(I_j)}{\partial I_j}(I_i - I_j)$, which gives a first-order approximation of $f(I_i) - f(I_j)$, by $\frac{1}{2} (\frac{\partial f(I_j)}{\partial I_j} + \frac{\partial f(I_i)}{\partial I_i})(I_i - I_j)$, which yield a second order approximation. The authors then empirically evaluates both the baselines and the proposed estimator on a synthetic task (polynomial programming), unsupervised modeling (unsupervised sequence parsing), and structured output prediction (generative modeling).

Strengths

- The theoretical analysis is well written and intuitively explained. The interpretation of the straight-through estimator as a first-order approximation of the true gradient allows the author to use numerical analysis results to improve the theoretical behavior of their proposed solution. - The experiment section is decent, with qualitatively different setups addressing different application domains. - The choice of the baseline for the baseline subtraction method as well as the tuning of the temperature parameter is appropriately discussed.

Weaknesses

- Equation (6) does not seem to be valid. I would expect $\sum_i (f(I_i) - E[f(D)]) \frac{d \pi_i}{d\theta} = \sum_i \sum_j \pi_j(f(I_i) - f(I_j)) \frac{d \pi_i}{d\theta}$, thus I don't understand why the term $\sum_i E[f(D)] \frac{d \pi_i}{d \theta}$ vanishes. - The derivation of remark 4.1 as provided in appendix C does not seem to be valid. I don't understand how $\sum_i \sum_j \pi_j \frac{\partial f(I_j)}{\partial I_j}(I_i - I_j)) \frac{d \pi_i}{d\theta} = \sum_i \frac{\phi_D}{\pi_D} \pi_D \frac{\partial f(I_j)}{\partial I_j} \sum_i I_i \frac{d \pi_i}{d\theta}$. Furthermore, there should be a typo at line 155 since it should be $\pi_D$ who is the output of the softmax and could take very small values. This would be consistent with the fact that it is the denominator of the fraction $\frac{\phi_D}{\pi_D}$. - I have no idea how equation (8) has been derived.

Questions

It is said that the derivation of equation (8) leverages the derivative of the softmax function, thus there may be a handy formula about these derivative that I am not aware of and which justify all the equations that do not seem valid to me. Could the authors elaborate on my concern about the validity of their equations ? I highly doubt that the equations are wrong, given that they coincide with previous work on the topic and the consistent improvements provided in the experiment section. I would be willing to increase my score if the author correctly addresses my concerns.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

There is a decent discussion about the limitation of the proposed method througout the paper. Albeit, given that high variance is the main limiting factor of the REINFORCE estimator, and given that the other estimators are biased, a focus on bias variance tradeoff would have been welcome.

Reviewer fihA7/10 · confidence 4/52023-07-09

Summary

This paper works on gradient approximation for discrete latent variables. The problem is challenging because the discreteness hinders direct backpropagation through the neural networks; hence, an approximation is needed. The authors address the issues from the perspective of a second-order approximation of the gradient. Specifically, the authors proposed ReinMax and showed that ReinMax achieved second-order accuracy with negligible computation overheads. The experiments on Polynomial Programming, MNIST-VAE, and ListOps demonstrated the superiority of ReinMax over the prior methods such as Straight-Through Gumbel-Softmax, Gumbel-Rao Monte Carlo, and Gapped Straight-Through.

Strengths

The problem has been challenging for a long time. Most prior work addresses the issue from the perspective of Straight-Through Gumbel Softmax (STGS). To the reader's best knowledge, the perspective of second-order approximation is novel in this field and the performance improvement is believable. Below highlights the strengths of the paper. a) The use of second-order approximation is novel and the theoretical derivation is believable. b) The performance gain is clear with similar computational overhead as in the prior work. c) The experimental analysis covers some relevant topics such as temperature scaling and the choice of baseline. The conclusion is clear from the analysis.

Weaknesses

a) The design choice of ReinMax is unclear. Although ReinMax is well-motivated by the second-order approximation, the reader might be confused by the form of ReinMax at Eq. (7). For example, why mixing $\pi$ and $D$ by introducing $\pi_D=(\pi+D)/2$? Why substracts by $\nabla_{ST}$? Where are the coefficients (2 and 1/2) from? It would be nice if the authors can elaborate more on these. b) The implication of the experiments remains unclear. For a fair comparison with the prior work, the authors follow the literature and run ReinMax on Polynomial Programming, MNIST-VAE, and ListOps. However, these are toy problems with limited implications for the applications. For example, does the good performance of ReinMax on Polynomial Program/MNIST-VAE/ListOps imply any possible applications? Conversely, if we have a reinforcement learning problem with discrete actions or a VQ-VAE model, can ReinMax be helpful? It would be nice if the authors can evaluate more on the application side so that the other practitioners can follow.

Questions

Why the proposed method is called ReinMax? For the other questions, see the weakness section.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No potential negative social impact.

Reviewer txhR7/10 · confidence 4/52023-07-09

Summary

The paper proves that the straight-through (ST) gradient estimator for the categorical distribution can be viewed as a first-order approaximation of the true gradients. Based on this point of view, a new gradient estimator, ReinMax is proposed based on Heun's method which is a second-order method. Experiments are conducted which show that ReinMax improves upon the state of the art methods.

Strengths

- **The paper has good writing quality.** The background is introduced in a well-organized manner and contributions are stated clearly. Mathematical formulae are also well laided out and are easy to understand. Relationship with existing work (ST, REINFORCE, baseline subtraction) are also expanded upon in good detail and nuance. - **Solid contribution with a comprehensive set of experiments.** The proposed new method has solid theoretical underpinnings (Heun's method being more accurate than Euler's method), is still very efficient (not requiring the Hessian), and is proven to be effective under multiple different settings (polynomial programming, MNIST, etc.). The new method is compared against many commonly used methods (STGS, DisARM, ...) and there is also discussion of using a different baseline for subtraction. Extensive hyperparameter tuning is performed and advantage of the proposed method is demonstrated well.

Weaknesses

My main concern with this paper is that it **oversells its theoretical contribution**. The view of the straight-through estimator as a first-order Taylor expansion of the unbiased gradients is not news to people who study gradient estimators. The authors are fair in pointing out that previous work (Tokui & Sato 2017) only dealt with the Bernoulli case, but one might argue that The categorical distribution is just a natural extension of the Bernoulli case. Regardless, I think it is **potentially misleading to say that "ST as a first-order approximation to the gradient" is a novel perspective**. Despite this, I still consider the proposed approach to be novel enough that the contribution of the paper as a whole is still significant. Another concern is the lack of comparison with RODEO (Shi 2022) which as far as I know is the state of the art for REINFORCE methods with variance reduction. The paper only reported numbers on RLOO and DisARM in table 3 but these methods are known to be weaker than RODEO, which is reported to outperform DisARM significantly in terms of gradient variance.

Questions

- What is the effect of temperature scaling on the proposed method? How does it influence the relative performance of ST-type methods when compared with unbiased REINFORCE-based methods? - Is there a complexity-efficiency trade-off here? For example, would using 4th-order Runge-Kutta instead of Heun's method yield much better gradient estimations and improve the overall result?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The authors do not explicitly discuss the limitations of their work.

Reviewer ZCX17/10 · confidence 2/52023-07-12

Summary

This work develops a new method, ReinMax, to compute gradients of parameters used to generate discrete random variables; specifically, the parameters of a multinomial's softmax parameterization. It provides a new perspective onto the existing straight-through (ST) gradient estimator as a first-order approximation. The new method is a second-order approximation and, in contrast to ST, relies on gradient differences rather than a single gradient. The ReinMax gradient estimator is compared to other approaches on polynomial programming toy problems, unsupervised parsing on ListOps, and ELBO training of a VAE on MNIST.

Strengths

- (S1) The paper provides a new perspective onto the Straight-Through (ST) technique, which simply replaces the backpropagation through a non-differentiable operation by applying the identity. This adds a new theoretical justification why such heuristics work. Framing ST as first-order approximation also naturally suggests an extension to second-order, as presented by the paper. - (S2) The new method does not seem to add significant overhead to existing approaches. The analysis includes a sensitivity analysis over different problem hyper-parameters which suggests that the estimation works consistently.

Weaknesses

- (W1) Limited to softmax parameterization of multinomial: The main text focuses on computing gradients for the parameters of a multinomial parameterized by a softmax. The authors should comment on the (im)possibility to treat other parameterizations or distributions. - (W2) Experiments: In the presented experiments, ReinMax seems to consistently work well and perform comparably or more favourably than the competitors. However, I do not have the expertise to judge whether the experiments represent state-of-the-art tasks. I am willing to adapt my score and confidence during the discussion phase with the authors and other reviewers. - (W3) Clarity: Some steps could be easier to follow by providing the required mathematical properties. The figures and tables should be moved closer to where they are referenced in the text. Here is a list of actionable suggestions which I think would improve clarity: - In Eq. (4), explicitly add the term $\partial D / \partial \pi$ that is set to the identity in ST - In Eq. (6), mention that $\sum_i \partial \pi_i / \partial \theta = \partial \sum_i \pi_i / \partial \theta = \partial 1 / \partial \theta = 0$. - I think it would be helpful to move parts of appendix E to the main text to have a more formal description of first- and second-order approximation in the specific context. - Fix '2rd-order' into '2nd-order' in various places in the main text and the appendix. - Mention the derivative of a softmax around L171, that is $\partial \pi_i / \partial \theta_k = \pi_k (\delta_{ik} - \pi_i)$ - Clarify the notation $\delta_{\mathbf{D} k}$ in the appendix - Minor suggestions: Use consistent symbol $\mathcal{R}$ in L51, 'softmax' in caption of Figure 1, remove 'on' in L84, 'computational efficiency' rather than 'computation efficiency', 'approximation' in L185, use bold symbol for $\theta_i$ in L210 and add $\in \mathcal{R}^2$, 'phenomena' rather than 'phenomenon' in L218), add 'that' between 'one' and 'uses' in L264

Questions

- Why is the method called ReinMax? - ReinMax is a second-order extension of ST. Can this be extended to even higher orders? Would this be beneficial or are there diminishing returns in including higher orders? - What is $\phi_i$ in L150 and what are its constraints?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See (W1)

Reviewer WPn88/10 · confidence 4/52023-07-20

Summary

This paper is about designing a new approach to approximating the gradient of parameters in generating discrete random variables. The work first starts with discovering the connection of Straight-Through (ST) estimator and the first-order approximation of the true gradient. From the insight, the authors aim to improve the ST estimator, by applying second-order approximation of the true gradient. To apply second-order approximation without actually calculating second-order derivatives, the authors use Heun's Method, which approximates second-order derivatives using two first-order derivatives; thus no expensive calculation is needed. The proposed estimator is coined ReinMax. Through mathematical analyses, the effectiveness of using the expected value of function outputs is proven. From evaluations, the authors empirically prove ReinMax method outperforms other estimators, along with presenting other properties and insights e.g. sensitivity to the number of dimensions, batch size, convergence speed, memory usage, and running time.

Strengths

- The paper is well-organized and easy to follow - Mathematical background is given thoroughly - The method can be applied with minimal change of code

Weaknesses

- Equation 6 seems to be one of core findings from the paper; the derivation process may be written more comprehensibly (like in Appendix A) - More evaluations on benchmark datasets close to real-world distribution would be beneficial. For instance, how much improvement will be made when we apply the ReinMax estimator into other models for NLI or sentiment analysis instead of ListOps?

Questions

- In experiments using the most basic REINFORCE algorithm, not RLOO or DisARM-Tree, is there any baseline subtraction used? If not can the $E[f(I_i)]$ baseline be used also in REINFORCE?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

As mentioned in the checklist, I suppose no potential negative societal impact will arise by this work. Experiments are done quite fairly, including standard deviations of scores and implementation code.

Reviewer GD4h7/10 · confidence 3/52023-07-20

Summary

The paper shows that the straight-through estimator is a first-order approximation of the gradient. The authors then propose a method, called ReinMax, which provides a second-order approximation with negligible computational overhead. Experiments are performed in several settings involving discrete variables (polynomial programming, structured output prediction, and discrete latent variable generative models), showing that ReinMax is more accurate and stable.

Strengths

**Originality** The originality of the paper is fairly strong. As I see it, the main contributions of the paper are in 1) identifying the straight-through estimator as an instance of the forward Euler method, which is a first-order approximation of the gradient, and 2) using Heun’s method to derive a second-order gradient approximation. To someone mostly outside of the field of gradient approximation, these are non-trivial insights that, to the best of my knowledge, are unique to this paper. **Quality** The paper is high quality. I particularly appreciated the effort that the authors put into the empirical evaluation, performing hyperparameter sweeps to demonstrate stability, as well as empirically verifying that the ReinMax gradient estimator provides improved estimates of the true gradient. Overall, the result is a paper that provides compelling evidence of 1) improved understanding of straight-through and 2) an improved gradient estimator. **Clarity** Overall, the paper is quite clear. The authors do an excellent job of presenting mathematical notation to help the reader see the similarities across gradient derivations and estimators. Theoretical results are presented in a clear, logical order. The empirical results are also generally presented well, with clear labeling of tables and plots. **Significance** While many papers have mentioned that the straight-through estimator is an approximation of the gradient, it appears that none of these papers have formally shown it. If, indeed, this paper is the first to do so, then it is a significant contribution to our theoretical understanding. The proposed improved estimator, ReinMax, also appears to offer some performance improvement over previously proposed (first-order) estimators. Considering that ReinMax has similar computational overhead as straight-through, then this could serve as a drop-in replacement for the straight-through estimator throughout various applications. However, it is unclear to me whether this serves as a drop-in replacement for all existing instances of the straight-through estimator, or merely those that operate on Multinomial distributions.

Weaknesses

I see two relatively minor weaknesses in the paper as-is: larger-scale empirical evaluation and slight improvements to the presentation. These are discussed below. The current empirical evaluation involves three settings: quadratic programming, structured output prediction, and latent variable generative modeling with discrete latent variables. While the authors demonstrate ReinMax in all three settings, much of the empirical evaluation revolves around the final setting (categorical VAE on MNIST). These results are a useful indicator of the benefits of ReinMax, however, the empirical setting itself is rather toy-ish compared with modern settings. Given that ReinMax is a drop-in replacement for the straight-through estimator, I would imagine that it should be trivial to replace the ST estimator in an existing scaled-up setting, e.g., vector-quantized VAEs or Hafner et al.’s discrete world models. The authors could alternatively / additionally explore categorical VAEs on more complex data. This would allow the authors to more definitively claim that their proposed gradient estimator leads to tangible empirical improvements. Additionally, several minor aspects of the presentation could be improved. * The plots in Figure 1 are presented with fairly minimal context in the surrounding text. The caption states “Details are elaborated in Section 6.” Then it may make sense to place this figure closer to Section 6. * As far as I can tell, the name “ReinMax” is never actually explained. * The labels for the baseline methods, e.g., in Figure 1, 4, etc. are difficult to read; they are various dashed lines. Colors and/or larger lines would make this clearer. * The shaded regions in Figure 5 (right) are rather jagged — perhaps there’s an issue with the evaluation interval or the plotting setup. * In various tables, e.g., Tables 1, 2, …, the results for ReinMax are bolded, despite falling within the error bounds of the baseline methods. I find this to be somewhat misleading. * The citation in the second sentence of the introduction is incorrect.

Questions

Could you please elaborate on the applicability of the ReinMax estimator? Does this estimator (or the insights developed in the theoretical sections of the paper) alleviate the “laborious and time-consuming” work of developing “different ST variants for different applications in a trial-and-error manner”?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

As mentioned above, I would appreciate a clearer discussion of which existing forms of straight-through estimator the ReinMax estimator can replace.

Reviewer ZCX12023-08-14

Rebuttal follow-up

Dear authors, thanks for the rebuttal. I am satisfied with your discussion of ReinMax's applicability/limitations (W1) and the possible extension to higher-order estimators. **Please make sure to include them in the draft**. You should also include a footnote that explains the algorithm's name, as this was brought up by multiple reviewers. Based on the overall positive feedback on your experiments (W2) from the other reviewers, as well as the additional results you provided in the rebuttal, I have decided to raise my score.

Authorsrebuttal2023-08-14

Author response

Thanks again for your timely response and detailed suggestions! We will include these discussions and additional results in the revision.

Reviewer GD4h2023-08-15

Response to the Authors

I have read the other reviews and the authors' rebuttals and have decided to maintain my score. A majority of the reviewers feel that this paper should be accepted; the main contention is over the degree of significance. I hope that the authors include the new experiments, as well as the reviewers' suggestions, in the revised paper.

Authorsrebuttal2023-08-16

Thanks for your comments

Thank you for acknowledging our contribution. To highlight the significant potential of our proposed method, we provide additional experimental results (as elaborated in our general rebuttal). We will incorporate these discussions in our revised version.

Reviewer A6oc2023-08-16

I thank the authors for taking the time to carefully right each step of their derivation to me. As other reviewers suggested, some details needed for the full derivation should be mentioned in the main text, or I would alternatively suggest to include the derivations in the appendix if the main body run out of space. Since the proposed contribution consistently outperforms alternative methods in the experiments provided in the paper as well as the additional experiments provided in the global rebuttal, I am increasing my score accordingly.

Authorsrebuttal2023-08-17

Author Response

Thank you for your feedback. We appreciate your acknowledgment of our work's performance in the experiments, and we will include the discussions and clarifications here.

Reviewer txhR2023-08-16

I thank the authors for their response and for providing further insights on the theoretical aspects of their work. The authors added comparison with the state of the art method for REINFORCE variance reduction and provided insight on the strengths of each method which I found to be helpful for my understanding of their work. I am also happy to know that the authors have acknowledge my point on the novelty of the theoretical perspective and are willing to adjust the writing. With the above concerns addressed, I am bumping up my score to a 7.

Authorsrebuttal2023-08-17

Author Response

Thank you for your constructive feedback. We're glad our clarifications were helpful. we will include the discussions and clarifications in the revision.

Authorsrebuttal2023-08-21

Thanks all reviewers for their valuable time, efforts, and constructive suggestions

We would like to express our gratitude to the reviewers for their valuable time, efforts, and constructive suggestions. To briefly recap our primary contributions: 1. We formally establish that Straight-Through works as a first-order approximation in the general multinomial case. 2. We propose a novel and sound gradient estimation method ReinMax that achieves second-order accuracy without requiring any second-order derivatives. ReinMax is shown to improve state-of-the-art methods in extensive experiments. In our submission, we adhered to the experiment design of the existing study, focusing on small-scale problems for controllability and resource efficiency. In the discussion, we presented additional experiment results across diverse settings, including a real-world application, to further demonstrate the potential of ReinMax. We also wish to note that we will utilize the extra space available in the final version to enhance our paper's clarity and presentation.

Program Chairsdecision2023-09-21

Decision

Accept (oral)

© 2026 NYSGPT2525 LLC