Summary
This paper gives an algorithm for differentially private reinforcement learning in two-player zero-sum games. The paper considers a standard model for differential privacy already established for single-agent RL. In this model, in each episode a unique user follows a policy $\pi$ recommended by the RL agent i.e. the user encounters states $s$, takes actions $a$ sampled from $\pi(s)$, and receives rewards $r$. The goal is for the RL agent to learn optimal policy recommendations, without revealing the private information of each user consisting of the trajectory of states, actions, and rewards. The paper considers two standard models for privacy in RL, joint differential privacy (JDP) and local differential privacy (LDP). The proposed algorithm achieves nearly optimal (in certain parameter regimes) regret in both of these privacy constrained settings.
Weaknesses
- The algorithm proposed is a straightforward combination of prior work [1] achieving privacy for single-agent RL and [2] achieving low-regret learning for self-play in zero-sum games. It is not clear from the paper what new ideas, if any, are needed, beyond directly applying the private counts and bonuses from [1] to compute the upper and lower confidence bounds used in [2].
- While privacy for multi-agent RL seems quite relevant and interesting, privacy for two-player zero-sum games seems much less well-motivated. For example, in the autonomous driving case, general multi-agent RL corresponds to learning in a setting where there are many autonomous vehicles on the road and one wants to keep the information of each one private. Two-player zero-sum markov games instead correspond to the setting where there are exactly two autonomous vehicles on the road in each episode, and somehow they are in direct zero-sum competition (e.g. a one-on-one race). In fact, the only reason differential privacy makes sense in this setting is that the paper assumes that a different pair of users competes in each episode, and it is privacy across these different pairs that is preserved. In general, it really seems to me that the most important questions regarding privacy in RL relate to a large number of interacting agents, and that the setting of this paper was chosen specifically so that the techniques of [1] and [2] could be directly applied, rather than because the problem itself seemed important to solve.
Specific Issues:
- The text in Table 1 is too small to read.
[1] Qiao, Dan, and Yu-Xiang Wang. "Near-optimal differentially private reinforcement learning." International Conference on Artificial Intelligence and Statistics. PMLR, 2023.
[2] Liu, Qinghua, et al. "A sharp analysis of model-based reinforcement learning with self-play." International Conference on Machine Learning. PMLR, 2021.
Questions
1. Are there any technical challenges to overcome when combining the known algorithm for private single-agent RL with the algorithm for self-play in zero-sum games?
2. Why is privacy across episodes of two-player zero-sum games an interesting problem? Is there some natural approach to generalize to mutli-agent RL? Are there natural examples where one would want to preserve privacy in a sequence of direct competitions between two players?