From Text to Trajectory: Exploring Complex Constraint Representation and Decomposition in Safe Reinforcement Learning

Summary

This paper introduces a type of universal natural language constraint to model diverse real-world safety requirements. Also, to avoid using the specific human-designed cost function, this paper introduces a Unified Trajectory-Level Textual Constraints Translator (U3T) for aligning text with trajectories and assigning costs via attention mechanisms. By conducting experiments on two environments, this paper reaches the conclusion that U3T can accurately predict whether a given trajectory violates the natural language constraint, and the policy trained with predicted cost can behave more safely.

Strengths

1. Broad applicability: U3T introduces trajectory-level textual constraints, capable of modeling diverse constraint requirements in real-world scenarios, making it more widely applicable across different types of constraints and complex environments compared to traditional methods. 2. Novelty: Addressing the natural language constrained reinforcement learning challenge through a novel approach involving multimodal alignment and novelly integrating a credit assignment component within the framework. 3. Automated constraint handling: Through its text-trajectory alignment and attention-based cost assignment components, U3T automates constraint handling and cost allocation, reducing the need for manual design of cost functions and enhancing system flexibility and generality. 4. Empirical results demonstrate that policies trained with U3T achieve lower violation rates compared to standard cost functions.

Weaknesses

1. Complexity and computational cost: Implementing U3T involves complex multimodal learning and attention mechanisms, which may require significant computational resources and time costs, especially when dealing with large-scale datasets or real-time applications. 2. The authors do not specify exactly what model of text encoder was used in the experiments. 3. Data Dependency: Effective implementation of U3T heavily relies on high-quality and well-annotated textual and trajectory data, which may be challenging to acquire and curate in some practical applications.

Questions

1. Is it feasible for the authors to experimentally analyze the inference speed of the proposed framework? 2. Could the authors specify the model of the text encoder used in the experiments and provide an in-depth exploration of its selection rationale? 3. Leveraging large language models for text understanding shows promise. Have the authors considered strategies for integrating these models to augment the comprehension and deconstruction of intricate textual constraints?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

The authors have discussed the limitations in the paper.

Summary

* The paper proposes U3T, a new system for more robust safe RL under general text constraints. * The key innovation of the paper is to generalize text constraints such that the constraints don’t refer to a specific entity/state and addressing cost sparsity where constraints are only violated at the final time step. * The proposed system consists of a trajectory encoder and text encoder that are trained jointly using a contrastive loss defined by the KL-divergence. * The authors solve cost sparsity by identifying actions/states that contribute to the violation of a constraint in an attention mechanism.

Strengths

* The paper solves an interesting and intuitive problem. There is a clear application of generalizing text constraints from single entity to general entities. * The formulation is intuitive and well-explained. The components introduced build on a simple and effective design * Ablation studies seem to demonstrate the benefit of the cost assignment portion of their system design, a portion that could be extremely expensive cost-wise. * The general cost (Figure 2/Figure 3) of their method seems to reduce in comparison to their baselines that use ground truth.

Weaknesses

* I found some details missing in the text. I discuss some portions that could use clarification here. * Which text encoder was used in this experiment? I couldn’t find this in the paper and from your appendix section A.4, it seems untrained. You refer to finetuning with LORA though so I was confused. A small nit, could be interesting to connect the text encoder performance to results. I realize space is tight but if these could be incorporated into the main paper, that would clarify and make the experiments easier to interpret. * I’m unsure how to interpret the average reward results in comparison to the cost results. See questions. My intuition was that rewards should be larger given the pareto frontier results. * I’m having trouble differentiating this paper from the Lou et. al. paper. I took a quick look at Lou et. al. -- I can understand that entities needed to be modeled in the paper but from what I can tell, that paper could also be applied in the settings you considered given the use of the pretrained LLM. From Figure 1, I couldn’t see the specific entity problem you were mentioning in the paper. Why wasn’t this added as a baseline? Could you provide more discussion on why these two are different? I think the approach in this paper has considerable novelty so I won’t say that this is entirely negative but it does diminish the contribution slightly.

Questions

* I noticed that generally, while the cost of the method would decrease, the reward wouldn’t change significantly (Figure 2). I actually found this a bit unintuitive. I figured that lower cost would imply a longer trajectory and therefore, longer windows to collect reward over, so average reward could increase. I found this discussion missing in the text. Could this be elaborated? * How long were the trajectories used in the paper? (What was the average size of T?) Could there be a context bottleneck?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

* See question about context bottleneck. Could this be a limitation to discuss?

Summary

- This work proposes an approach to training RL agents with constraints. The proposed approach learns language embeddings of constraints and embeddings of trajectories. During training, a similarity score is used to align the space of constraint embeddings and the space of trajectory embeddings. - Moreover, the language embedding of the constraint is also used to do fine-grained credit assignment across the actions in a trajectory. The authors find that this improves performance.

Strengths

- The approach is novel in that the technique of credit assignment is applied to the constraint rather than the reward. Or at least this is the claim, and I was not able to find otherwise. - The approach is shown to be effective. With an ablation study, the authors show that their credit assignment approach gives better performance - I have questions about the setup and novelty (see weaknesses), but overall this work seems to represent a positive contribution: the authors show that it is possible to encode trajectory-level constraints with text and to train an RL agent to respect those constraints.

Weaknesses

- I have a hard time understanding where this work stands in relation to prior works. The main contribution seems centered around the approach of encoding trajectory level constraints using natural language. But both trajectory level constraints and natural language constraints have been explored independently. And the combination of these two aspects seems straightforward: encode the trajectory level constraint using natural language. I may be overlooking something that makes this less straightforward. I don't quite follow the explanation for novelty on lines 101-104. The Related Works section mentions prior work which learns language representations of constraints, but says that these works do not consider trajectory-level constraints. But because the constraints are given in natural language, it doesn't seem that anything in principle prevents these prior approaches from doing so? What is the technical innovation of the proposed work that allows for trajectory level constraints? - A few things seem missing from the setup (see questions) - I have concerns about the soundness of the described approach. Where does the ground truth come from when training the text-trajectory alignment component? As far as I can tell, there is notion of a textual constraint but not a notion of a ground truth verification of a constraint. From what I understand of appendix 1, the textual constraint is determined to be violated by a discriminator model (see questions). In any case, the authors should explain how constraints are determined to be violated. This matters because it means that the discriminator model and the agent could both make the same systematic linguistic errors. - I have concerns about the soundness of the approach in general. Using natural language to describe constraints seems at odds with the enterprise of making RL more "safe." Natural language can be ambiguous, and machine learned representations of language can be imperfect, and prone to making semantic errors. In contrast to other approaches which provide formal guarantees on behavior [1], this seems much less safe. However, I am conscious that others in the field think differently, given the prior work. ## minor - Figure 3: Labels are missing in the legend - Line 297: "Table" --> "Figure" ## references [1] Fulton, Nathan, and André Platzer. "Safe reinforcement learning via formal methods: Toward safe control through proof and learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.

Questions

- Line 174: What is a "negative pair"? Is it a constraint and a trajectory that violates that constraint? - Line 175: Given that the mapping from trajectories to constraints is many-to-many, doesn't $q$ need to be normalized so that the probabilities sum to 1? That is, there are many positive pairs that contain the same constraint, so they should all receive an equal share of positive probability. - Fig 5: I think I'm missing something basic: how are is the Pareto front determined? I could assume that per cost, the best reward possible can be determined by some procedure? Is this what was done? - Section 6: Where do the textual constraints come from? Are they human annotated? Appendix 1 seems to suggest not. - Is it guaranteed that the discriminator can tell when a textual constraint is violated? - Line 289: What is the difference between ground-truth mode and cost prediction mode? Is it the cost assignment?

Rating

5

Confidence

3

Soundness

1

Presentation

2

Contribution

2

Limitations

Limitations are mentioned in the appendix.

Official Comment by Reviewer T5E9

Thanks for the authors' response. I have no further concern and will lean to keep the score.

Response to authors

I thank the authors for their response. I will keep my score. I think I understand the argument for novelty that the authors are making a little better. But there are still some gaps for me. The rebuttal says that "previous methods fail to offer a unified framework for trajectory-level textual constraints," but I don't really understand what "unified framework" was lacking in the prior work. As long as a constraint can be specified in natural language, can it not be considered part of a "unified framework"?