Summary
The paper introduces EgoPlan, an egocentric vision-language planning framework that leverages large multi-modal models (LMMs) and diffusion models to handle long-horizon tasks in household scenarios. EgoPlan employs a diffusion model to simulate state-action dynamics and integrates computer vision techniques like style transfer and optical flow to enhance spatial modeling and generalization across different environments. The LMM serves as a planner, decomposing instructions into sub-goals and selecting actions aligned with these sub-goals. Experiments demonstrate that EgoPlan improves task success rates compared to baselines in egocentric views.
Strengths
1. Innovative Integration of LMMs and Diffusion Models: The paper presents a novel approach by combining LMMs with diffusion models for planning and action prediction in egocentric embodied environments.
2. Incorporation of Computer Vision Techniques: The use of style transfer and optical flow enhances the model’s ability to generalize across different scenes and adapt to spatial changes, which is crucial for embodied agents.
3. Dataset Contribution: The authors have collected the VH-1.5M dataset on VirtualHome, providing egocentric observations, fine-grained action information, and visualizations like optical flow, depth maps, and semantic segmentation, which can benefit future research in navigation and manipulation tasks.
4. Improved Long-Horizon Task Performance: Experimental results indicate that EgoPlan outperforms baselines in long-horizon tasks from an egocentric perspective, showcasing the effectiveness of the proposed framework.
Weaknesses
1. Lack of Planning Instructions and Time Details: The paper does not provide specific planning instructions for the high-level goal decomposition shown in Fig. 2, nor does it mention the duration of the planning process. This omission makes it difficult to evaluate the efficiency and effectiveness of your planning method.
2. Insufficient Details on Diffusion Model Training: There is a lack of detailed information on how the diffusion models (particularly the World Model and the Image Subgoal Generator) were trained. Without these details, assessing the validity and reproducibility of your results is challenging.
3. Dataset Limitations and Overfitting Concerns: Relying solely on the VH-1.5M dataset may be inadequate. Additionally, there is a risk of overfitting without information on out-of-distribution (OOD) evaluations and how the training and testing data are partitioned.
4. Limited Generalizability: The applicability of the model to other scenarios, such as outdoor environments, has not been demonstrated. This raises questions about the generalization capabilities of the embodied agent in diverse environments.
5. No Discussion on Time Efficiency and System Stability: The paper lacks details on inference time efficiency and system stability, especially considering the multiple estimation components involved. Understanding potential bottlenecks is crucial for evaluating the feasibility of your method.
6. Lack of Detailed Reasoning Process: The reasoning process of the Large Language Model (LLM) is critical for evaluating and explaining outcomes, but the paper does not sufficiently discuss this aspect. It seems the dataset contains only subgoal text without detailed reasoning steps.
7. Confusion in Fig. 7 Results: In Fig. 7, the results without LoRA fine-tuning sometimes appear better than those with LoRA. This raises concerns about potential overfitting and the model’s ability to handle significant environmental changes.
Questions
1. What Is the Duration of the Planning Process?
Could you provide more details on the time taken for the high-level planning process in Figure 2 and the specific planning instructions used?
2. Training Method for the Image Subgoal Generator:
How did you train the diffusion model used for subgoal prediction? Is it conditioned only on the final goal? When there is a significant difference between the subgoal and the initial screen, how do you ensure accurate outputs without environmental context? If the output quality is consistently high, is there a risk of overfitting?
3. Details on Dataset Partitioning and Evaluation:
How did you partition the dataset for training and evaluating the two diffusion models? How do you maintain output quality when the predicted image differs greatly from the input? Did you perform out-of-distribution (OOD) evaluations to address potential overfitting issues?
4. Assessment of Generalizability:
Have you tested your model in other scenarios, such as outdoor environments, to evaluate its generalizability and potential applicability?
5. Inference Time Efficiency and System Bottlenecks:
Could you provide more information on the system's inference time efficiency? Which components might be potential bottlenecks? Have you conducted ablation studies to assess the system’s stability?
6. Inclusion of LLM Reasoning Process in the Dataset:
Does your dataset include detailed LLM reasoning processes, or only subgoal texts? Could you provide some comparative reasoning cases with multi-modal input to offer deeper insights?
7. Clarification on Fig. 7 Results:
Could you explain why, in Fig. 7, the results without LoRA fine-tuning sometimes appear better than those with LoRA? Does this indicate potential overfitting or limitations in the model’s ability to handle significant environmental changes?