Summary
This paper presents a method for using evaluator feedback as safety constraints in a constrained RL framework. The novelty comes from the nature of the feedback and the sampling scheme. It is motivated by the fact that previous approaches have various limitations: designing cost, especially comprehensively, is expensive and even seemingly impossible a priori. Offline feedback can help, but existing approaches often don't scale, and/or are constrained to receive feedback at the state level, which is also expensive. Giving feedback at the trajectory level is an option, but also a difficult state-credit assignment problem; and even getting feedback for every trajectory is expensive.
This paper presents RLSF, a method in which binary feedback is given on segments or entire trajectories and a novel surrogate loss is presented to make this a useful learning signal, then a new "novelty-based sampling mechanism" is used to train efficiently.
The paper takes us through the methods, including the feedback collection process (human or procedural evaluators), the policy improvement step, the nature of the feedback (1 if every state in a given segment is safe, 0 otherwise), and how the cost function is inferred from the collected data. It also presents a surrogate loss function for turning segment-level feedback into a dense signal.
The paper then goes through experiments and results in the Safety Gym and Driver simulation environments. The tasks have various constraints in position and velocity. Experiments compare to three general baselines - self-imitation safe RL, safe distribution matching, and PPO where the true cost function is known (considered an upper bound). The methods are tested directly on the tasks, then learned cost functions are transferred to a new agent. The paper then goes through ablations removing the novelty-based sampling method. The results show that this RLSF algorithm shows significant improvement over baselines.
Strengths
### Originality
- I am not aware of any methods that use exactly this surrogate loss or sampling method. RLSF does borrow heavily from prior work, but the loss function, sampling method, and otherwise novel (to my knowledge) mix of existing elements does constitute an original work.
### Quality
- Experiments section, particularly ablation of the sampling section and cost transfer, is creative and shows lots of important aspects of the study. The cost transfer results are particularly promising.
- Results speak for themselves! They show a system that works in simpler environmetns and is ready to be scaled up and tested.
### Clarity
- Well written and clear. There is a fair amount of technical content in terms of defining problems and setting up notation, and the paper handles it all well.
- The novelty-based sampling mechanism is particularly well-explained.
### Significance
Seems good. It's good to see new ways of integrating feedback as human interaction with robots comes nearer and nearer.
Weaknesses
### Quality
- "We posit that with a sufficient number of samples, the number of correct labels will outweigh the number of incorrect labels for each state" - doesn't this require sparsity of safety violations and small segments (meaning a hefty feedback load)? Questions like this would require answers to judge if this work is useful.
- It would be good to see more error modes - for example, the paper cites novel states as a hotbed for potential errors, but it's only one. What are the others?
- In the experiments section, it would be good to see cost transfer compared to other cost function learning methods, not just PPO vs. RLSF vs. PPOLag.
### Clarity
- Tables should have bolded numbers, even if all the bolding will be in one column. The tables are incredibly dense right now and tiring to read.
- Explanation of surrogate loss derivation would benefit from more justification. E.g. I see that it's true that the product term substitution makes it a state-level term instead of a segment-level term, but explain that. more clearly and why it's okay to do and still expect the optimization to work out.
Questions
- The paper cites sparse cost violation as a motivation for collecting feedback on shorter segments. Why is that? Is the idea that when cost violations are dense, it's hard to give more specialized feedback, but when they're more sparse then the evaluator can just go for it? Or is it purely a horizon issue (shorter horizons have fewer cost violations), in which case mentioning sparse cost violations at all doesn't make sense?