Summary
This submission uses RL to address the pump-scheduling problem in water distribution networks (WDNs). First, the authors demonstrate the performance of agents trained using PPO on a WDN environment using a particular WDN's historical operational data, and with reward functions designed to optimize constraints and costs; these agents show substantial improvement in ensuring constraint violations, in particular. The authors further test out a hybrid approach that switches between the previously-used baseline model (a query model) and the RL model, with the aim of engendering trust; they evaluate a few different strategies for when the RL model's recommendations should be injected.
Strengths
This paper addresses an important task of real-world significance: The management of water distribution networks. This problem will only become increasingly important as, e.g., climate change affects both water supply and electricity prices (given that water distribution networks rely on electricity to operate).
The use of RL is well-motivated, and empirical results on a simulation environment show substantial performance improvements compared to the currently-used query-based model.
The introduction of a hybrid RL methodology switching between the previously-used model and the new RL model is interesting, as this paradigm could indeed be helpful in the integration of RL into systems where operators may be hesitant to rapidly adopt new technologies.
The paper is well-written and easy to follow.
Weaknesses
While there is nothing significantly wrong with the paper itself, I unfortunately do not think it is a fit for ICLR. In particular, to be presented at ICLR, a paper must present either (a) a strong methodological contribution in ML or (b) a strong applied contribution that provides further insight and directions to researchers in the ICLR community.
In terms of ML methodological contribution: The introduction of the hybrid approach is genuinely interesting, but would need to be further developed and tested out to truly constitute a strong ML methodological contribution. In particular, the different metrics for switching between the query model and RL actions seem somewhat ad hoc, which is potentially very reasonable in the context of the application at hand, but would need to be developed in a more principled/structured way to constitute a strong methodological contribution in ML.
In terms of applied contributions providing insight to the ICLR community: My evaluation here is along the lines of two questions:
* *Does the paper provide useful insight on considerations associated with using ML in real-world settings?* - While the paper does discuss considerations like the need to satisfy constraints, these are not described in sufficient detail. Such detail and accompanying discussion would be necessary to shed light on ways that RL methods could be fundamentally improved to, e.g., better satisfy the kinds of constraints at hand in real-world settings.
* *Is the end-to-end real world implementation convincing from the application perspective - e.g., is it actually deployed on a real-world system, and does it appropriately fulfill application-specific criteria?* - While the paper's evaluation uses real-world data from a WDN, it is not clear to me how realistic the simulation environment is or how close to deployment the methods are.
With all that said, I really do commend the authors' work. My comments above are truly only about fit for this venue, and not a statement on the objective quality of this work (notably, complex ML methods are not always needed!).
Questions
* Can the authors shed additional light on the kinds of operational constraints that agents in this application domain must satisfy?
* Can the authors shed additional light on the realism of the simulation environment, and the maturity/closeness-to-deployment of the RL methods presented?
Rating
3: reject, not good enough
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.