Summary
The authors build on previous autoencoder-based neural image compression methods. Earlier works have shown that the quantized latent representations output by an end-to-end trained encoder for any given image are suboptimal for a decoder. Concretely, for a fixed set of decoder weights, one can usually find latent representations that yield a better rate-distortion than the representations output by the encoder. Hence, previous works have proposed optimizing the rate-distortion loss of an image using gradient descent for a fixed set of decoder weights. Since the latent representations we wish to optimize are discrete, previous works have developed "soft-quantization" approaches based on the Gumbel-Softmax trick, which introduces a temperature parameter $t$, such that for $t > 0$ the optimization problem becomes continuous, while $t \to 0$ recovers hard quantization. One starts with a temperature $t \gg 0$, which is then annealed towards $0$ during the optimization.
Specifically, the authors build on stochastic Gumbel annealing (SGA, Yang et al., 2020), which uses the function $f_\tau(x) = -\tanh^{-1}(x) / \tau$ to compute the soft quantization log-probabilities for a given temperature $\tau$.
In the present paper, the authors propose three alternatives for $f_\tau$ and conduct experiments analogous to the ones in Yang et al. (2020).
## References
Yang, Y., Bamler, R., & Mandt, S. (2020). Improving inference for neural image compression. Advances in Neural Information Processing Systems, 33, 573-584.
Strengths
Unfortunately, I couldn't find any particular strengths worth highlighting.
Weaknesses
Unless I have completely misunderstood the paper, the authors' motivation for the paper is incorrect. Concretely, for the function $\exp(-\tanh^{-1}(x))$ used by SGA, the authors claim on Line 155 that "The problem is that the gradients tend to infinity when the function approaches the limits of $0$ and $1$." While this is true for the limit point $1$, this does not hold for $0$. This is a crucial point, as the derivative tending to $\infty$ at $1$ is desirable, as it prevents rounding to the wrong value, while if this occurred $0$, it would cause optimization difficulties. However, it can be easily verified that the function I mentioned above doesn't have an infinite derivative at $0$ by plotting it. In fact, I am not sure what function the authors have plotted in Figure 1a under the name "atanh", as the graph of $\tanh^{-1}$ is not centrally symmetric.
In fact, the training instability of SGA arises because the derivative at $0$ becomes infinite as the temperature $\tau$ is annealed to $0$. All of the authors' proposed alternatives have the same issue, meaning they do not tackle the very problem they set out to solve.
The experiments are also not insightful and do not demonstrate that any of the proposed modifications lead to a significant improvement.
Furthermore, the paper's writing should be significantly improved. The authors review many unnecessary, basic details (such as the first two paragraphs of the introduction or the societal impact section, which should be cut altogether). The notation should also be cleared up; for example, the authors introduce duplicate or equivalent notations for the same objects: $\pi$ in Eq (3), $p(y)$ in Eq (4) for the soft quantization probabilities, $v_L$ and $w$ for the fractional part of $v$, and $n$ for the inverse temperature $\tau$. Finally, the font sizes should be increased in each figure as they are currently unreadable without zooming in significantly.