Enhancing Preference-based Linear Bandits via Human Response Time

Summary

The paper explores the interactive preference learning problem. Traditional binary choice feedback is limited in conveying preference strength. The authors propose leveraging human response time, which inversely correlates with preference strength, as additional information. They adopt the difference-based Drift-Diffusion Model with linear human utility functions and propose a computationally efficient linear utility estimator that incorporates human response times. Discussions about the proposed estimator and traditional estimators relying on the logistic regression are provided. The authors also integrate the proposed estimator into the Generalized Successive Elimination(GME) algorithm for the fixed-budget best arm identification problem. The proposed method is evaluated on both synthetic and real-world datasets, demonstrating the effectiveness of incorporating human response times for essay queries.

Strengths

- The idea of incorporating human response times is novel and interesting. The authors provide a clear motivation to incorporate human response times, give illustrative examples to illustrate the benefits and explain their methods in detail. - The paper is well-written and easy to follow. The background, methods, and experiments are well-organized and well-explained. - The authors clearly discuss the effects of parameters $x^T\theta^{*}$ and $a$, and compare the estimator with and without response times, which provides a good understanding of the proposed method. - The authors conduct extensive experiments on both synthetic and real-world datasets, and provide a detailed analysis of the results about the estimation performance and identification performance. The methods to pre-process the datasets and determine the parameters are also well explained.

Weaknesses

- Although the authors provide the asymptotic analysis for their estimators, they do not provide a theoretical analysis for the proposed algorithm. - As mentioned in the paper, when the barrier $a$ is small, incorporating response times may not improve estimation performance. Since the parameter $a$ is also unknown, like $\theta^{*}$, learners may not know whether to incorporate response times. It would be beneficial if the authors could have a discussion about the practical choice of the estimator under these conditions.

Questions

Please refer to the weaknesses mentioned above.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are adequately discussed in the paper.

Summary

The paper studies linear bandit preference learning, where binary preference data have been augmented with response times. A joint model for choice and response time falls from setting the linear preference model as the drift parameter in a drift-diffusion model. Experiments and theoretical analysis show that including response time in the model increases the value of "easy" responses, and thus improves bandit performance.

Strengths

The paper is well-written. Including response time in a linear bandit preference model is novel to my knowledge, and certainly useful. The development of the method is clear, and the method itself is well-motivated and seems computationally reasonable and useful. The theoretical analysis is helpful, and I really appreciated how the paper uses that analysis to draw insights into the source of improvement from including response times in the model. The experiments were well-designed, and included the necessary baseline of choice-only. The analysis of the experiments provides helpful guidance for the situations in which the method does not outperform baselines.

Weaknesses

The paper has one significant weakness: all of the experiments use simulated response times, simulated from the model developed in the paper. We thus don't get a sense for the "real-world" performance of the method, where response times will not adhere precisely to the DDM model. I agree with the paper that response time is easy to collect alongside binary preferences. Surely there are some datasets available that include actual human response times? Real human response times are the big missing piece of the experimental evaluation.

Questions

On the topic of real human response time data, should we expect the model to be robust to lapses, or would some sort of data pre-processing be necessary to remove those?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Addressed.

Summary

The submission proposes to use response times to obtain additional information from participants in preference learning settings. They apply a variant of the Drift-Diffusion Model, a popular model of human decision making from psychology, and combine it with an algorithm applicable to linear bandits. In asymptotic analysis and application to real data, they demonstrate the benefits of taking advantage of response times, especially for queries with large value differences.

Strengths

I think this is a very sensible approach to improving preference learning, considering the fact that response times can be available "for free" from existing preference learning paradigms. I also think the application of a simplified variant of the DDM (EZ-Diffusion) as a way to get an analytic handle on things is also a good idea, and I think the results are compelling. It's a good paper, and I enjoyed reading it.

Weaknesses

* I think the paper slightly misrepresents the actual model it's using. Unless I'm missing something, the DDM is usually defined as the first passage time of a two-boundary Wiener process, and the likelihood of a RT under that model involves solving an SDE which has no closed-form solution. However, moments of the DDM RT distribution are available, which is what enables approaches that use them to approximately solve for parameters (this includes the E-Z Diffusion model of Wagenmakers et al., the current contribution (as far as I can tell), as well as the related work of Shvartsman et al. 2023 (arxiv:2306.06296). I think using E-Z diffusion type approaches is great, I just think the paper shouldn't represent them as the full DDM. Also note that the E-Z diffusion line of work provides for closed-form estimation of nondecision time (by estimating drift (i.e. value difference) based on response time variance, and then backing into the response time mean from there -- this might help address the issue of unknown nondecision time identified by the authors. * Related to the above, moving away from the simplifying assumption of E-Z diffusion would introduce the concern of other unknown DDM parameters such as drift variability, nonsymmetrical initial conditions, etc.

Questions

* Why should choice-only would ever do better asymptotically. L186 makes this claim w.r.t. section 3.2 but I don't see it in the figure -- the orange lines (dashed or solid) seem always above the gray lines. * For figure 2, if possible it should be moved below section 3.2., so that it can be interpreted after the asymptotic min-weight of both estimators has been discussed. This is also the part of the submission which is most confusing -- mental calisthenics are needed to map from asymptotic min-weight to the variance of theta estimates and therefore to the influence of observations at various value differences. Another editing / clarification pass could be helpful. Additional notes: * All figure axis tick labels are very tiny, should ideally be larger. In addition, adding color legends in fig 4 (outside the caption) would help with readability.

Rating

8

Confidence

5

Soundness

4

Presentation

3

Contribution

4

Limitations

Discussed and addressed, even with additional analyses (and as noted above, the limitation regarding nondecision time might be possible to work around, though I could be missing something there).

Summary

This paper studies whether leveraging human response times can lead to better performance in bandit learning from preference feedback. More specifically, the paper integrates the drift-diffusion model (DDM) from psychology into the best-arm identification problem in linear bandits. Given a fixed interaction time, the goal is to utilize response times alongside binary choices so as to maximize the probability of recommending the optimal arm. The paper introduces an estimator of the preference/reward vector using both binary responses and response times via linear regression. This estimator can be incorporated into bandit algorithms. Asymptotic normality results and three simulations indicate that this new estimator leveraging response times can make easier queries more useful, in comparison with traditional estimators that only use binary responses.

Strengths

- The studied problem seems novel, interesting, and relevant to the community. To my knowledge, DDMs have not been (widely) explored in bandits and reinforcement learning, although I am not very familiar with the DDM literature in psychology and neuroeconomics. This model may be of particular interests to researchers studying dueling bandits and RLHF. - The model leads to a simple and clean estimator of human preferences $\theta^*$, which uses both binary preference feedback and response times. To my understanding, this estimator can be integrated into various bandit algorithms, not limited to ones for best-arm identification. - The paper presents both theoretical and empirical evidence on the role of response times, and discusses the intuition behind when/why they can be useful. - The paper is well-organized and well-written. It is a joy to read.

Weaknesses

- It would be helpful to see more discussion on related work, especially on DDMs and race models. For example, what evidence has been given in the psychology literature that DDMs can explain the human decision-making process? What are the typical objectives in papers that study DDMs, and how are they different from the best-arm identification problem in this work? Have DDMs been considered in bandits and reinforcement learning? - On the theoretical side, there are no non-asymptotic results regarding the performance of the bandit algorithm, such as bounds on the error probability. - On the empirical side, only one dataset contains the response times of the participants. The last two experiments simulate response times according to the DDM -- it is reasonable to expect that they are then useful to estimating $\theta^*$. The algorithm also has a hyperparameter that requires tuning.

Questions

See above for questions on related work. I am also curious -- the estimator that uses only binary responses essentially assumes the Bradley-Terry model, if my understanding is correct. Looking at the logistic sigmoid function, I can see that the when $x^\top \theta^*$ is away from 0, the curve becomes flat and therefore not much information can be gained here. Have you considered other noise models that do not use a link function like the logistic sigmoid function? Would you expect similar behavior?

Rating

7

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

Limitations have been discussed.