Summary
This paper offers a novel perspective on understanding a 4D environment via the introduction of 4D Panoptic Scene Graph (PSG-4D). PSG-4D transforms 4D sensory data into nodes, representing entities with precise location and status data, and edges that capture temporal relationships. The authors have developed a high-quality PSG-4D dataset, which includes 3K RGB-D videos with a total of one million frames. Each frame comes with 4D panoptic segmentation masks and detailed, dynamic scene graphs. In order to solve PSG-4D, they propose PSG4DFormer, a model capable of predicting panoptic segmentation masks, tracking masks across the time axis, and generating corresponding scene graphs via a relation component. Extensive experiments demonstrate that their method forms a robust baseline for PSG-4D. Furthermore, the paper provides a real-world application example demonstrating how dynamic scene understanding can be achieved by incorporating a large language model into the PSG-4D system.
Strengths
1. This paper is significant for several reasons: it formulates a new problem, annotates a novel dataset, and provides simple baselines. These aspects could advance future research and encourage applications in robotic systems. Even though the research appears to be a logical progression from panoptic scene graph generation and panoptic video scene graph generation, the introduction of a new data modality provides potential for a broader range of applications. SGG in 4D space is an area that has not been extensively explored, to my knowledge.
2. The dataset selection and construction pipeline is detailed and appears to be reasonable.
3. The paper is well articulated and easy to comprehend. The incorporation of the large language model (LLM) in the proposed application seems to be an intriguing and viable addition.
Weaknesses
1. I have a significant concern with the Experimental section of the paper. The task of Scene Graph Generation is intricate, particularly when extended to a 4D scenario. In building a pipeline to solve this problem, it is reasonable to use various components each serving a unique role. However, it remains unclear which component holds the most promise for future research focus. While the ablation study in Table 2 gives us insights into the depth map data and naïve temporal modeling, it would be valuable for the authors to provide more detailed analysis and insights.
2. In Line 112, there is a minor formatting error: 'M' should be formatted as an equation rather than plain text.
Questions
Please refer to the weaknesses.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The authors mentioned potential challenges and societal impact.