> Regarding the variance of growth rate: I understand the test's consistency is irrelevant to the variance of the growth rate. However, I was referring to the testing power, which is a finite-sample result. That is related to the variance of the growth rate.
Variance of the log wealth is not a metric that has ever been considered before in any paper on betting from the original (non-testing) works in information theory like Kelly (1954), Breiman (1963), Cover (1970s), or in any paper on using betting for statistical hypothesis testing (i.e., papers cited in our work). It is the variance of the payoffs that is important, not the variance of the growth rate (i.e., not the variance of the log payoffs): in particular, this is the reason why we have two terms in our bounds on the growth rate and why under the ''low-variance'' regime, we get a faster growth rate (the expected margin is not squared). We have finite sample guarantees on the rate of growth of wealth (due to the use of nonasymptotic regret bounds in our proofs), but we have stated the asymptotic growth rate because it is more interpretable.
Regarding additional finite-sample results, we now also have bounds for the stopping time $\tau$, namely $P(\tau>t)$ and $\mathbb{E}[\tau]$, where an upper bound on the latter easily follows from the former bound (since $\mathbb{E}[\tau] = \sum_{t=0}^\infty P(\tau>t)$) for the case when the 2ST null is false. For simplicity, assume that the same classifier $g$ (e.g., the Bayes-optimal classifier) is utilized for betting using the payoff: $f(Z_t,W_t)=W\cdot g(Z_t)$. We show that:
$\mathbb{E}[\tau] \leq O\Big( \Big( \frac{1}{\mathbb{E}[W \cdot g(Z)]} + \frac{1}{\sup_{s\in[0,1]}(1-R_s(sg))} \Big)\cdot \log(1/\alpha) \Big)$,
meaning that the expected stopping time is upper bounded by the sum of the reciprocals of the expected margin of a classifier and the deviation of its (optimized) squared risk from the worst-case value. We get the squared risk term exactly because we account for the variance of the payoffs (otherwise, the upper bound is proportional only to the reciprocal of the *squared* expected margin of a classifier, which is always worse). We have added this result with the proof to the Appendix, and we are more than happy to provide a short proof sketch!
> As for the comparison with [1], did you use CNN for your method and use a kernel to construct the sequential MMD test for [1]?
This is correct.
> I re-read the paper, especially the experiments and conclusion. If I understand that correctly, the gain of testing power is attributed to the use of a predictive model over the kernel instead of the form of constructed statistic (e.g., with the payoff $W_tg(Z_t)$)? In fact, I found the gap between the proposed work and [1] smaller than I thought. The sequential MMD test, which is used as an example for the test by betting in [1], can also be simply constructed with a predictive model by replacing it with a predictive model; although might not be contained in the RKHS anymore. Hence, the superiority of the proposed method over the ''kernel'' test counterparts seems to be questionable to me.
While the two betting-based tests are closely related (which we highlighted at the end of the Introduction), our paper actually constructs a different betting game from that in [1]. Both papers are indeed similar from the standpoint of using a chosen distance measure between distributions (e.g., kernel-MMD in [1], or the squared risk of a classifier in our work) as a starting point for designing effective betting strategies. However, our paper is not just about constructing a valid test, but characterizing its quality: we explicitly relate the growth rate and consistency to interpretable and intuitive metrics that are associated with classifiers, namely their risks and margin.
When we use the squared risk as a measure of distance, the resulting bets indeed happen to take a similar form as in the sequential kernel-MMD test from [1]. However, our theoretical results provide practical suggestions for designing powerful predictive 2STs, e.g., we show that training a classifier via direct margin maximization / minimizing the hinge loss (which may be hinted at by the distance measure in [1]) will hurt the power of the resulting 2ST, i.e., the growth rate of the wealth. Further, we also relate our test to earlier sequential predictive 2STs and show how we address their limitations.
Last, we never argued that either of the approaches: kernel- or classifier-based, is superior to the other and confirmed that across different experimental setups. Our motivation was not to develop a predictive test that should replace a kernel one but rather to complement the existing kernel approach with a new method that may be better suited for high-dimensional or structured data (like images).