Summary
This paper introduces GarmentLab, a new set of garment manipulation environments, assets, annotations and tasks based on IssacSim. The differences compared to existing benchmarks is the support of a broader range of tasks (such as tasks involving interaction with other objects such as hanger, fluid, and human avatar), more robot types, more assets, and other features such as motion planning, teleoperation, and sim2real transfer. A real-world garment manipulation benchmark is also proposed with the aim for evaluating real-world performances. Several existing algorithms are tested on the proposed tasks, and their performances are analyzed.
Strengths
- The overall system looks very comprehensive and cover most of the needs for garment manipulation, including a broad range of tasks, assets, and robot control methods.
- The sim2real analysis and support for real-world data collection are nice features.
- Overall the whole system looks impressive and can be a good contribution to the deformable object manipulation community.
Weaknesses
I mainly have the following questions:
- Regarding the real-world benchmarks: I appreciate the authors provide the 3d scans of the real objects and provide semantic annotations. However, I think the proposed real-world benchmark still lacks details such that every lab can easily replicate the set up in the real world. For example, it would be ideal to include purchase links for the chose objects. The protocols presented in F.3 seems to be just describing the task in a very high-level way, without details such as how the robot / objects should exactly be initialized, e.g., their relative position, and how to exactly measure the success / performance. Without such details it would not be possible for other labs to perfectly replicate this real-world setups and to form a fair comparison for different algorithms. I would encourage the authors to provide more details on such protocols to ensure that this can be well replicated in any lab, otherwise this proposed real-world benchmark would seem infeasible.
- For some readers not familiar with the context, it is hard to understand the right part of figure 6. How can one interpret this figure to understand that the performance becomes better after applying the sim2real method such as point cloud align? E.g., what does the color scheme mean, and why is the color scheme after better than before? Also what is the figure in the dashed bounding box supposed to show? I would suggest to revise this figure to make these more clear.
- Since this is a benchmarking and dataset paper, it is a little bit hard to access its quality and potential contribution to the community without actually using/seeing the code. For example, numerous robotics benchmarking has been proposed in the past, but only a few of them will be widely adopted by the community, where factor such as easy-to-use APIs, intuitive UI design, informative debugging tools, are all critical for the survival of the benchmarking. As the code of this paper is not released, it's hard to access the above qualities. I would encourage the author to release the code as soon as possible, and take into consideration the above factors, especially given the proposed system seems complicated and involve many different components.
- An error in table 1: PyBullet actually support garments. See Antonova et al., Dynamic Environments with Deformable Objects, NeurIPS Datasets and Benchmarks Track, 2021. This paper is also highly relevant and should probably be discussed in related work.
- There are some typos in the paper, e.g., line 6 in the abstract: "which exhibit offer limited diversity" -> "which offer limited diversity". Please have a thorough check over the whole paper for typos.