AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animation

# Response to reviewer `#bzXT` (1/2) We appreciate the reviewer's inquiry regarding the comparison with SMPL-based 4D HOI generation methods. We acknowledge the valuable contributions made by existing methods in advancing human motion generation. Our approach builds upon the strengths and robustness exhibited by these methods, aiming to enhance the realism and usability of 4D HOI generation. With a better 4D HOI method, our approach will also show improved performance. We hope for this paper to not only showcase the potential of 4D HOI but also to potentially advance and elevate our field. > a substantial body of related work in HOI generation, particularly in 4D HOI generation methods. These methods have contributed significantly to the field by generating SMPL and SMPL-X poses for HOI scenarios. > How does AvatarGo compare to SMPL-based 4D HOI generation methods? We concur with the reviewer's observation that Programmable Motion Generation, Chain-of-Contacts, CG-HOI, and HSI have significantly advanced the field. However, these advancements primarily focus on generating motion sequences for SMPL or SMPL-X models, which lack intricate clothing details. Additionally, these methods rely on training datasets, limiting their adaptability in practical scenarios. While InterDreamer recently introduced a zero-shot framework for generating appearance within 4D HOI, their models still closely resemble the original SMPL design, resulting in minimal clothing details in their outputs. In contrast, our approach offers a novel avenue for zero-shot text-guided 4D HOI generation, producing both realistic appearance and geometry enhancements. > While the paper focuses on generating realistic appearances using Gaussian splatting, which differs from the approaches of these methods, it is essential for the authors to analyze and position their work in the context of existing studies in 4D HOI generation. Similar to the above discussion, our approach presents a novel opportunity for zero-shot text-guided 4D HOI generation, emphasizing the generation of realistic appearance and geometry. We will include more discussion with these 4D HOI generation methods and highlight the importance of the SMPL-based 4D HOI methods within this research domain. > Another weakness of the proposed method, as mentioned by the authors in the limitation section, is that it assumes a consistent geometric relationship between the interacting body part and the object. We agree with the reviewer on this point. However, We believe our approach to handling rigid-body interactions and continuous contact in 4D human animation represents a significant advancement. Meanwhile, this scenario, where the generated avatar interacts with rigid objects, is common in real-world applications, yet existing methods struggle with it. In other words, our proposed method outperforms all existing related approaches. Moreover, similar applications in robotics, such as embodied AI, primarily focus on resolving interactions between rigid structures. Similarly, SAGA[1] and other works focus only on a small range of settings, such as approaching and grasping an object. As an early study on generating 4D HOI scenes, we believe our method is valuable to the research community and will inspire future studies addressing more challenging cases. We will also aim to solve this problem in the future works. > Aside from the apparent appearance advantage, does AvatarGo achieve a similar interaction quality to the SMPL-based methods? Our methodology is centered around continuous interactions between humans and objects, making direct comparisons with SAGA[1] and similar works that concentrate on object grasping somewhat unfair. When comparing with SMPL-based approaches operating within a similar context, we would like to direct the reviewer to the webpage of InterDreamer[2] (their code is not publicly released for comparison). It is noticeable that this method encounters challenges in maintaining realistic human-object interactions, often leading to insufficient contact or significant penetration. These observations demonstrate that our method achieves comparable or superior performance when compared to existing SMPL-based techniques.

# Response to reviewer `#E4gh` Thank you very much for recognizing our paper! We appreciate your efforts during the stages of review and rebuttal. > My only question concerns the diversity of objects included in the text prompts used in the experiments. Additionally, how does your method perform across different types of objects? In our experiments, we incorporated over 15 randomly selected types of objects, and our methods demonstrated strong performance across this diverse set of objects.

# Response to reviewer `#1bKd` > The reviewer wonders if the human motion is given as input or is generated by the proposed method. Our approach has the capability to accept either a human motion sequence directly as input or utilize a text prompt as input to generate a human motion sequence through existing techniques. > The reviewer requests clear explanations of what are the inputs and outputs of the proposed pipeline. Our method is capable of accepting either text prompts alone or a combination of text prompts and motion sequences as input to generate 4D dynamic human-object interactions. When only provided with text prompts, our approach initiates by generating the SMPL-based motion sequence via existing techniques before proceeding through our pipeline for further processing and generation. > The objective function of the proposed generative model is spatial-aware SDS (L276), that is introduced by ComboVerse (Chen et al., 2024b), not newly proposed by the authors. We definitely agree that the spatial-aware SDS (SSDS) was first introduced by ComboVerse, and we do not attribute SSDS as our contribution. Unfortunately, simply applying SSDS for 3D compositional generation may not lead to good results in human-object interactions. In Section 3.2, our primary contribution then lies in leveraging a large language model to extract contact area details, thereby improving the robustness of 3D human-object interaction generation. > This process operates with a heuristically pre-defined threshold alpha, and the reviewer wonders if this heuristic threshold alpha applies to various contact body parts, e.g., fingers, feet, head, and legs. In our method, we apply the same threshold alpha to various contact body parts, such as fingers, feet, head, and legs. > Then, if the 3D objects and humans are generated in poor quality, how can the authors mitigate the error propagation in the proposed method? We appreciate the question raised by the reviewer. Firstly, it's important to emphasize that our methodology is primarily designed to address interactions between humans and objects. Furthermore, existing techniques demonstrate a strong ability to generate high-quality 3D models for both objects and humans, despite some occasional instances of failure. In addition, our method still optimizes the attributes of 3D Gaussians when composing the human and object. This optimization leads to a slight improvement in quality and aids in mitigating common issues like "hole artifacts," which are often encountered in 3D Gaussian-based generation processes. > The reviewer requests more visualization results of diverse objects and body parts, such as the head, feet, and legs. In Fig. 1, we have showcased results depicting interactions involving the head and feet, as seen in the examples "Harry Potter - bear head hat" and "Naruto in Naruto Series - football." While we aim to present additional cases involving interactions with various body parts, we recognize that interactions between humans and objects predominantly occur around the hands, feet, and head. > In L310, the reviewer wonders how the authors define the threshold alpha We follow GaussianEditor[1] to define the threshold alpha as 0.5. [1] GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting. CVPR 2024

# Response to reviewer `#D2nt` > The overall appearance quality of generated humans and objects is still limited. The results are often blurry and lack realism. There is also still human-object penetration (see “Naruto in Naruto Series stepping on a football under his foot” on the webpage). While we acknowledge the reviewer's feedback regarding the quality of the generated humans and objects, it's important to emphasize that our work represents the pioneering effort in zero-shot 4D human-object interaction generation. For the "Naruto-football" case, we want to highlight that the penetration is minor, and achieving fully realistic results without such minor overlaps can be challenging. Rest assured, we are dedicated to further enhancing the quality of our work in forthcoming endeavors. Similarly to TC4D, our method also can deliver higher quality results with extended training durations. However, it is notable that TC4D necessitates approximately 15 hours for training, which is nearly 20 times longer than the training duration required by our method. > The notations in object animation and correspondence-aware motion optimization are poorly explained. What is G_c and G_o? G_c and G_o represent the affine deformations utilized to transform SMPL vertices from the canonical space to the observation space. > How are they computed? > The paper mentioned they are derived from x_ci and x_oi, but it’s unclear. We determine G for each SMPL vertex by applying Equation (6). In our experiments, we derive G_{c_i} by finding the nearest vertex in the canonical SMPL model to x_{c_i}. Similarly, G_{o_i} is obtained by locating the closest vertex in the observed (posed) SMPL model to x_{o_i}. In the upcoming revised version, we will enhance the clarity of this explanation for better understanding. > It seems the method requires training for each human-object pair, which can be quite inefficient for large-scale generation. Is there anyway for the method to train the same character interacting with multiple objects? Certainly, similar to other zero-shot text-guided 3D/4D generation techniques, our method necessitates training for each human-object pairing. However, once we have generated the 3D composite human-object interactions, our approach can then be trained to animate these interactions into various poses. Training a model capable of enabling the same character to interact with multiple objects is challenging without the requisite training datasets. Each type of object presents unique interaction dynamics; for instance, the interactions between a human and a book would differ from those between a human and a chair. In light of this, we plan to explore training a model tailored for category-based human-object interactions in the future. This approach will allow us to train a single character to engage with diverse objects effectively in future iterations of our work. > Is each HOI trained with one motion only? Can it generalize to multiple motions? Each HOI can be trained to exhibit various motions. Nonetheless, achieving this would require training a correspondence-aware motion field specific to each motion sequence. > Fig. 4 only shows one frame of animation, which is not animation but reposing. Multiple frames need to be shown. Thanks for the suggestion. In the revised edition, we will update Figure 4 to display multiple frames.