Weaknesses
My major concern is the contribution of the algorithm and the theoretical analysis listed as below
- In section 4.1, the delivered message, specifically $w_i / (w_i + w_j) = h_{ij} / (h_{ij} + h_{ji})$ is somewhat trivial. The rationale behind highlighting this equivalence is unclear, as it does not seem to directly address any core technical challenges. A stronger connection to the problem statement or contribution would help clarify the authors’ intent here.
- In section 4.2, The proposed method employs the square loss to approximate the binary distribution, defined as $L = E[(p - p^)^2]$. The authors do not provide a clear reason for selecting this loss function over the more conventional KL divergence, $L = E(\mathrm{KL}(p \parallel p^))$, which is typically a “soft” version of cross-entropy loss. It would be beneficial for the authors to clarify why KL divergence was deemed unsuitable for this application and how the square loss better serves the purpose of approximating the distribution $p$.
- In addition, the algorithm requires repeated evaluations of $h$ for each individual preference pair $(i, j)$. The authors should clarify why this $h$-step data cannot be directly used within DPO but instead must be aggregated into a squared loss function. For fair comparison, given that $h = 5, 9, 15, 21$ were chosen in the appendix, the iterative DPO dataset should ideally leverage these same values ($h = 5, 9, 15, 21$) rather than a single annotator in vanilla iterative DPO. A clarification on this point is necessary.
- In section 4.3, the theoretical justification here requires further detail. For example, Assumption 4.1 cannot hold universally for any $P_h$. The authors may want to define the releasability on the ground truth $P^$, indicating that $P_h \to P^$ as $h \to \infty$ (per Assumption 4.2). However, this assumption may be also impractical, as it requires a significantly large $h$. Moreover, the theoretical analysis as presented is insufficient. For example, equation (18) essentially states that if $L = (p - q)^2 = 0$, then $D_{TV}(p, q) = 0$, which is self-evident. Similarly, equation (19) mirrors the DPO analysis. Furthermore, the role of $C_t$ in Theorem 4.5 is not well-explained, especially within the online iterative framework. A more thorough explanation of these theoretical points would strengthen the analysis.
- The authors should explicit show the selection of $h$ in the main paper.
### Regarding the proof in the appendix, here're a few issues I'm suggesting to revise.
- When applying the mean-value theorem, the correct expression is $|\sigma(x_1) - \sigma(x_2)| = \Big|\sigma’(x_0)\Big| \times |x_1 - x_2|$ (with absolute value of $|\sigma'(x_0)|$ rather than simply $\sigma’(x_0)|x_1 - x_2|$, though this does not affect the correctness of the proof.
- In line 810, this is a concentration inequality rather than the law of large numbers. The scaling factor of $O(1 / \sqrt{n})$ relies on certain conditions, which should be explicitly verified by the authors (e.g. Hoeffding's inequality with R-sub-Gaussian). The law of large numbers only guarantees convergence to zero but does not provide scale guarantees.