Summary
The paper studies value-based control agents for Weakly Coupled MDPs (WCMDPs), which comprise independent subproblems linked by an action constraint. The authors generalize an existing Lagrangian relaxation approach to action-value methods learned by stochastic approximation to obtain Weakly Coupled Q-Learning (WCQL). Essentially, a Lagrange multiplier is used to create an overestimate of $Q^*$ called $Q^{\lambda*}$; taking the minimum of these two value functions can lead to faster convergence without changing the fixed point. A deep-learning extension of this method relying on a soft constraint added to the DQN loss yields the titular method Weakly Coupled Deep Q-Network (WCDQN). The proposed methods are tested against several well-known baselines (such as DQN and Double DQN) and compare favorably on three WCMDPs: electric vehicle charging (tabular only), multi-product inventory control, and ad matching problems.
Strengths
- Self-contained descriptions of Weakly Coupled MDPs and Lagrangian relaxation, as well as a complete related works discussion, make the paper accessible to people (like me) who do not normally work in this setting. The explanations are clear and well written.
- Interesting theoretical results in the tabular setting. The convergence results look correct and intuitive to me. I checked the proofs in the appendix.
- Interesting, realistic experiment domains and strong empirical results.
Weaknesses
- WCQL relies on discretizing $\lambda \geq 0$ into a finite set of values $\Lambda$ to approximate the upper bound $\min_{\lambda} Q^{\lambda}$. Although this does not affect the ultimate convergence to $Q^*$ in the tabular case, I would imagine that this choice has a significant impact on empirical performance, yet the authors do not discuss this or how to choose the set (that I was able to find at least).
- The computational requirements of the proposed methods are downplayed. The authors state, “We note that, especially in settings where the limiting factor is the ability to collect enough experience, these bounds come nearly for free in that they only require computing a few additional quantities and importantly, they do not require additional experience.” I think this is a strong claim; compared to Q-Learning, at least $N$ additional Q-functions are needed to form the upper bound $Q^{\lambda*}$. In the deep RL setting, only one extra network appears to be needed, in order to generalize over these $N$ functions, but that still doubles the overall computation. Thus, I do not think it is fair to say that the method comes for free; however, the strong empirical results do justify the extra expense.
- Although the tabular theory is sound, the implications of porting it to deep function approximation are unclear and not fully appreciated by the paper in my opinion. The authors say, “the [loss] penalty softly ensures that learned network satisfies the bounds obtained from the Lagrangian relaxation.” However, because the constraint is soft, it does not ensure anything. If the approximated upper bound ever dips below Q*, then the wrong value will be bootstrapped by the network and the learned returns will be suboptimal. Of course, this is inevitable with function approximation (a hard constraint cannot be enforced), but it is worthwhile to clarify in the paper.
Questions
- Why is $\lambda$ called the penalty vector? Is it not a scalar, i.e., the Lagrange multiplier?
- How long (in timesteps) were the agents trained in each of the experiments? Since the figure x-axes use episodes, did agents receive unequal training time?
- How was the penalty coefficient $\kappa_U=10$ chosen?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
- Need to choose the set $\Lambda$
- Added computational expense
- Upper bound potentially violated with function approximation and soft constraint