Summary
The work proposed a new algorithm for federated policy optimization. The proposed work uses prima-dual update to replace the primal update of FedNPG, so that the communication cost reduces from $d^2$ to $d$. The proposed work enjoys same rate of convergence as FedNPG under certain assumptions, and numerical experiments were proposed to show the efficacy of the algorithm.
Strengths
The paper is well-rounded and the presentation is clear. The proposed method clearly enjoys the supremacy in terms of communication error. The numerical experiments were proposed to show the efficacy of the algorithm.
Weaknesses
(Please reply to the Questions section directly) The motivation of the proposed algorithm seems a bit less significant. Moreover, the convergence theory requires policy gradient bounded, which seems quite strong. In the numerical experiments, more results could be included to further show the efficiency of the proposed algorithm.
Questions
Here are the my questions and suggestions:
1. Concerns on the motivations: the idea of combining reinforcement learning and federated learning is interesting, which is also largely un-explored in the literature. It is good to see the exploration on this direction. An intriguing question is that can we consider existing works on reformulating the TRPO, such as the KL-penalitzed objective in [1], to federated setting? Since if we get rid of the constraint, we don’t need to communicate the Hessian in the first place (and FedAvg-type algorithm may also enjoys similar rate of convergence in theory). In summary, the authors propose to study the policy optimization in federated setting, but eventually ended up at a less significant improvement over FedNPG (and this improvement comes from essentially the constrained federated learning, not from RL or policy optimization), which makes me feel less confident about the contribution in the RL field;
2. In terms of the assumptions for convergence analysis, the authors proposed bounded policy gradient and Lipschitz continuous (Assumption 4.1) and claimed that they are fairly standard in the existing literature. I’m actually not very sure about the boundedness of the policy gradient. Could the authors just include the specific assumptions in the existing literatures? Same question for Assumption 4.2.
3. The penalty parameter $\rho$ is the penalty to make sure that each local $y_i$ consensus with the global $y$. In the experiment $\rho$ is simply a small number as 0.1 or 0.01. My concern is that during the update, is $y_i - y\rightarrow 0$ true or not? The authors could also plot the consensus errors.
4. In the experiments, for example Figure 2 and 3, FedNPG-ADMM seems not better than FedNPG in terms of the rewards. It would be interesting to see why this is the case. I know that the main claim of FedNPG-ADMM is its commutation efficiency, and I think the authors could also plot the curves wrt the CPU times.
References:
[1] Schulman, John, et al. "Proximal policy optimization algorithms." arXiv preprint arXiv:1707.06347 (2017).
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The limitation is well stated in weakness and question sections. The authors also include the discussions in limitations at the end of the work.
I’m not aware of any potential negative social impact of this work.