Offline Reinforcement Learning for Mixture-of-Expert Dialogue Management

Summary

This paper tackles issues with using RL for dialogue management related to covariate shift when using offline RL and the requirement for many online human-bot interactions when using online RL, and both suffering from a large action space. Current NLP models lack the ability to plan dialogue interactions beyond the next interaction, which is a crucial aspect of successful conversation. To address these points, the authors propose a mixture-of-expert language model (MoE-LM) and design several offline RL algorithms with different benefits. The structure of the MoE-LM is a hierarchical one, where there are a number of experts each optimised for a different intent (things like empathy, rage, etc.) and a dialogue management model that chooses one of the expert utterances conditioned on the conversation history. This latter aspect should help with non-myopic objectives that are naturally part of conversations. The authors compare their MoE-LMs trained with several different algorithms to SotA offline RL algorithms as well as behavior cloning and a bandits method that greedily optimises the next conversation turn (just reward maximisation). On two datasets, they show their method outperforms the offline RL baselines in terms of return in a simulated conversation (DialoGPT). They also show that the best performing RL algorithm for MoE-LMs which they propose has a more uniform selection of experts than the worst performing one.

Strengths

Very clear and detailed explanation of the algorithms, well-motivated and important problem. Strong results of proposed method compared to baselines on two different dialogue tasks.

Weaknesses

Some of the claims seem to not be substantiated by the results of the experiments: - It seems likely that the MoE-LMs generate more diverse conversations, but strictly you can't assume that based on a higher return in terms of sentiment; can you supplement the results with metrics that evaluate diversity? - It's not clear to me from experiment 2 that the MoE method is able to do long-term planning. Again, it seems likely given the performance increase over the offline methods and especially the Bandit, but simply based on the higher return in sentiment this can't be claimed. Can you isolate / evaluate the effect of long-term planning? How do we know from these results that the method is actually better at long-term planning and the performance increase is not due to other aspects that result in a better sentiment? - I might have missed it but there does not seem to be an explicit treatment of the sample-efficiency of your method over others; can you quantify this? It's pretty hard to interpret these results based solely on sentiment calculated over conversations with a simulated user. A baseline that would help interpret this to some extent is a simple prompted LLM to hold these conversations; what kind of sentiment would it achieve? Additionally, can you do a small human eval comparing your method to bandits and the best-performing offline RL method? Perhaps this can be combined with the questions above about diversity and long-term goal achieving / planning, asking humans to rate the conversations along these axes.

Questions

Answers to the questions written in Weaknesses would address my main concerns with this paper. Some small other questions/suggestions that are not related to my score: - I don't really follow contribution 2; how does leveraging pre-trained LMs and prior regularization result in high-level dialogue management? - I would rewrite line 58-62, very hard to parse with usage of "-- --" twice - Line 102 should be textual citation (citet not citep) - Table 1 and 2 need more comprehensive captions. Without reading the main text a reader should understand what the numbers and error bars refer to. What are your methods, what are the takeaways from these numbers, etc.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

There's no separate limitations section or discussion of the limitations of your proposed method; that would be a welcome addition.

Summary

The authors of the paper proposed a suite of off-line reinforcement learning methods utilizing Mixture-of-Expert Language Models (MoE LMs) to train dialogue management agents. Moreover, they experimented their RL methods on two open-domain dialogue datasets and showed better overall performance of their methods (MoE specific offline RL) over SOTA offline RL methods.

Strengths

S1: In their experimental section, it is shown that their MoE specific RL methods outperformed the SOTA offline RL methods. S2: Human evaluation is also done by recruiting 80 workers. S3: It is shown that those agents which have better performance utilize all the knowledge of different experts in a balanced way.

Weaknesses

W1: In the conclusion section of the paper, it is stated that their specialized offline RL methods have better sample efficiency, however, I did not see any experimental proof for it. W2: In the introduction section (page 2), they described their first component of their methods two times: “Our methods consist of three main components: 1) a primitive LM which, using a probabilistic encoder and decoder, is capable of generating diverse semantic intents 1) a primitive LM that uses a probabilistic encoder-decoder pair to generate sentences with diverse semantics and intents”. W3: In the section 6 (page 7) before experiment 1, the second appendix is not referenced properly: “More details and results can be found in Appendix E and ??” W4: The implementation is not provided so that one cannot reproduce their results.

Questions

See weakness section.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

NA

Summary

This paper introduces multiple reinforcement learning algorithms for dialogue management, in particular when combined with mixture-of-expert language models. Generally, a primitive (general) language model, as well as expert language models which have a specific intent or personality, generate candidate utterances. The dialogue management module then learns to choose among them.

Strengths

The paper introduces multiple novel RL algorithms tailored for dialogue management. Using RL-based dialogue management over mixture-of-expert LMs is well-motivated. The language models may generate fluent and diverse outputs, while the limited size of the action space allows for efficient learning. Coordinating multiple language models (or especially a single language model with different adaptors/parameter-efficient fine-tuning modules) could be impactful by improving generation quality and diversity given a fixed number of parameters.

Weaknesses

The description of the algorithms is quite dense. As multiple approaches are introduced, a more thorough discussion of which one(s) to prefer under certain conditions would be helpful. The number of turns is fixed in the experiments. Conversations could be much richer if the conversation length was more flexible. Without sharing the code, some of the experiments may be difficult to reproduce. The results in tables 1 and 2 are difficult to interpret.

Questions

Could you clarify "Our experiments demonstrate that model-based evaluation can significantly improve dialogue management over the model-free counterpart [...]"? Does the evaluation method change the DM policy? Could you describe the evaluation approaches in more detail? What do the reported numbers exactly represent? Could the approach be adapted to work without a primitive LM (i.e. only expert LMs with a specific personality)? Do you have examples of conversations generated with different approaches? [L49/50] Component #1 is repeated.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Some limitations are discussed in the appendix.

Summary

The authors address offline RL training of dialogue models. They note that gathering trajectories with humans-in-the-loop is potentially expensive (and perhaps even dangerous). Their method is a MoE involving a general LM and several specialized intent-accounting-for LMs, which generate candidates for a dialogue manager to select from: this process reduces the action space significantly (because selecting the next utterance becomes a task at the utterance level, rather than at the word level). The reward is a roberta sentiment classifier applied to simulated dialogue responses. Experiments compared to several single-policy/MoE methods show the proposed method is better able to elicit positive sentiment responses in simulation.

Strengths

The authors study an interesting problem --- offline RL for dialogue, and frame the task well --- DialoGPT offers a nice simulation environment, and I understand the motivation for the reward function the authors optimize. I commend the authors for working in a multidisciplinary area, mixing RL and dialogue modeling contributions. The experiments cover a broad range of RL algorithms, as well as 2 dialogue corpora.

Weaknesses

- I would have liked to have seen some human evaluations of response quality/elicited sentiment (I think the question that operationalizes the reward may be: "Which model's response would cause you to write a more positive response?"). Current evaluations are limited to reward optimization, which makes sense from an RL perspective. But, at the very least (if a small-scale human evaluation were prohibitive), it was odd to see a paper about dialogue with no dialogues shown. - The reward seems a bit reductive --- 1) What is the RoBERTa classifier trained on? Unfortunately, Liao et al. 2021 is locked behind a paywall so I can't check. I know sentiment models are often trained on specific domains (e.g., movie reviews or yelp reviews), so it might not generalize well (I am fearful of reward hacking); and 2) should it really be the case that the optimized reward be entirely based on sentiment? This seems like it might risk just pushing the dialogue manager to be a sycophantic "yes-man". I would have appreciated some justification for this choice. (in fact, few potential limitations of the work are discussed --- this type of reflection would have been appreciated). - The model itself seems a bit convoluted --- the idea of compressing the primitive encoder's output into a single vector (and then sampling from a mixture of normal distributions conditioned on that encoding as the MoE), felt a bit roundabout. Why not, e.g., not pool, and train a separate decoder for each expert, ala T5? Latent variable modeling is cool, but it felt superfluous, even to the core message of this paper (which, for me, focuses on MoE for state space reduction) --- I at least would have liked to have seen the ablation of simpler methods, e.g., over-generation with differently prompted LMs in a zero-shot way. UPDATE: the authors do have human eval (and will surface from the appendix) and made a few more clarifications in response). I have raised my score.

Questions

- What is the reward model from Liao et al. 2021 trained on? - Is it possible to run human evaluation or include discussion of the outputs to make sure reward hacking isn't occurring? - Why the latent variable model instead of, e.g., just encoder/decoder? It seems a bit orthogonal, and some simpler baselines suggested by this encoder/decoder choice are missing. - Can better discussion be added for potential limitations of this particular reward in this particular setup?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

No, see above.

Thanks for the rebuttal

**Diversity** Though I understand that your method promotes diversity, that doesn't mean you shouldn't also evaluate diversity if you want to claim it actually generates diverse dialogue. Just saying it probably will generate diverse conversations based on the method without evaluating it explicitly does not suffice. **Long-term planning** I still don't see how this experiment shows that your method is better at planning than just any non-bandit RL method. Again, the higher return might be due to other aspects. **Baseline comparison** I understand that you won't be able to surpass SotA LLMs without also using those as a base, of course, however my comment is directed at a baseline that helps in interpreting sentiment scores. **Human eval** Since you claim in the main text that your methods outperform offline RL baselinee, and since the human eval in Appendix E seems to show 2/3 of your methods do worse than baselines in terms of fluency and sentiment, actually showing the method is more sample-efficient seems important for the current work. I would like to see the human eval section brought to the main text. All in all, I remain with my points that the claims about the diversity and sample-efficiency as well as overall performance (shown by the human eval) are overstated, and will keep my rating.

Thanks!

Thanks for your response. Here are some response-responses : - Human eval: thank you, this is great! I did not see these and would definitely recommend moving them into the main body. - Thanks for sharing these! I think, ideally, in addition to showing a few examples, some commentary could be added about the relative strengths of the predictions of each approach. Even better would be an error analysis. But --- thanks for this, it helps! - Sentiment: thanks for the clarification - MoE vs. a simpler baseline. I do appreciate the authors points about the advantages of MoE. But I do feel that training a separate decoder for each intent/expert is a similarly simple-to-train baseline that would have been nice to see. I will raise my score in light of these updates.