Regularized Conditional Diffusion Model for Multi-Task Preference Alignment

Summary

This paper presents a novel approach using a regularized conditional diffusion model to align with preferences in multi-task reinforcement learning (RL) scenarios. The proposed method generates preference representations that effectively distinguish between various task trajectories and returns. By maximizing mutual information between conditions and generated trajectories, the model regularizes existing diffusion frameworks, enabling it to produce optimal task-specific trajectories aligned with given preferences.

Strengths

- The motivation is clear - generating preference representations for various tasks to differentiate them and taking them as the conditions in the diffusion model so that we could consider preference learning in the multi-task scenario. - The method and all the designs in this method are reasonable to me.

Weaknesses

My main concern is about the results. It seems that CAMP doesn't perform better than existing methods. For details, please refer to the Questions part. Thanks.

Questions

- Could you provide an additional ablation study on the influence of the number of tasks? Specifically, does the representation for task A differ when learning tasks {A, B} together versus tasks {A, B, C} together? Additionally, does the number of tasks impact the accuracy of task A? - Could you highlight the best results in each row of Table 1? It appears that CAMP's performance is not consistently better than existing methods across these environments, and its average result is not particularly strong. Does this imply that CAMP struggles to learn a good task representation? Alternatively, do you have any other explanations for these observations? - Regarding the multi-task dataset case, Tables 4 and 5 indicate that CAMP performs well on some tasks but poorly on others. Could you explain why this is the case? Is it due to an imbalance in the training of the multi-task scenario?

Rating

5

Confidence

2

Soundness

2

Presentation

3

Contribution

3

Limitations

N/A

Summary

The paper proposes a novel method called the Regularized Conditional Diffusion Model for multi-task preference alignment, addressing the challenge of generating trajectories that align with human preferences in multi-task settings. This method introduces preference-based representations and a mutual information regularization technique to improve the alignment between generated trajectories and preferences.

Strengths

1. The method effectively captures and utilizes versatile preference representations, allowing for better alignment with human intents across multiple tasks. 2. The introduction of a mutual information regularization objective enhances the consistency between conditions and generated trajectories, improving performance. 3. The diffusion model guided by learned preference representations generates trajectories that align closely with desired preferences.

Weaknesses

1. The authors’ proposed method shares a similar motivation with OPPO. In the appendix, they explain the differences between their method and other approaches like OPPO and MTDiff. Their method improves upon OPPO and MTDiff by incorporating Diffusion and other techniques such as regularization for multi-task preferences. It is recommended that the authors emphasize these distinctions in the main text and update the contribution list to highlight their unique contributions. 2. From the comparative experimental results presented in the figures and tables, it is evident that the authors’ method does not achieve the best performance across all metrics. It is suggested that the authors provide a corresponding discussion instead of merely describing the contents of the figures and tables. 3. The authors’ method involves computationally intensive modules such as mutual information and diffusion models. It is advised to include a discussion on the computational costs of the model, such as space complexity and time complexity, to enable a comprehensive comparison with related models. 4. Given that the authors’ model addresses multi-task preferences, it would be beneficial to discuss the impact of the quality of the provided preference data on the results.

Questions

Please see the weaknesses

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

n/a

Summary

This paper proposes a regularized conditional diffusion model which aligns with preferences for multi-task scenarios. To achieve this, learnable representations for preferences are aligned with preference labels, which can be adopted as condition inputs to guide the generation process of diffusion models. Meanwhile, in order to keep consistency between conditions and generated trajectories, a mutual information regularization method is utilized to improve their alignment with preferences. To verify its effectiveness, extensive experiments have been conducted on two datasets D4RL & Meta-World which cover both single- and multiple-task scenarios, and the results demonstrate the superior performance and generalization ability of the proposed method in aligning preferences across different tasks.

Strengths

1. This paper is well-written and detailed. The authors provide sufficient technical principles and experimental details for readers to understand and be able to reproduce their work. 2. This paper gives a novelty idea in human preferences alignment and combines it with diffusion models in multi-task scenarios, which can be conducive to further foundation models finetuning. 3. Conducting extensive experiments on public datasets and comparing them with different state-of-the-art methods makes the proposed method more practical and convincing.

Weaknesses

1. From Table 1, we can see a large variation in performance for different tasks with the same implementation method. 2. The proposed method seems to have high computational complexity. 3. From Table 2, although the generalization ability for unseen tasks is somewhat improved by this approach compared to the baseline methods, the average performance is still not very good.

Questions

1. I wonder what factors contribute to this phenomenon and how we choose the optimal approach for a given task. 2. I would like to know whether the method proposed in the paper receives a greater impact on efficiency as the type and number of tasks increase. 3. Is the generalization ability of this approach feasible in practical applications?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Response to the author

Thanks for the response. Q1: Can I understand that the non-significant superiority of CAMP in Table is because the baseline IQL is too strong? If so, why do we just use IQL? Q2: Do we have any solution for the different difficulties of the tasks in MTL? It's indeed the imbalance problem in MTL and should be solved carefully.