Cinematographers adeptly capture the essence of the world, crafting compelling visual narratives through intricate camera movements. Witnessing the strides made by large language models in perceiving and interacting with the 3D world, this study explores their capability to control cameras with human language guidance. We introduce ChatCam, a system that navigates camera movements through conversations with users, mimicking a professional cinematographer's workflow. To achieve this, we propose CineGPT, a GPT-based autoregressive model for text-conditioned camera trajectory generation. We also develop an Anchor Determinator to ensure precise camera trajectory placement. ChatCam understands user requests and employs our proposed tools to generate trajectories, which can be used to render high-quality video footage on radiance field representations. Our experiments, including comparisons to state-of-the-art approaches and user studies, demonstrate our approach's ability to interpret and execute complex instructions for camera operation, showing promising applications in real-world production settings.
Paper
Similar papers
Peer review
Summary
This paper introduces ChatCam, a system that enables camera operation via natural language interactions. This system has two key components. CineGPT is proposed for text-conditioned trajectory generation and an Anchor Determinator for precise camera trajectory placement. Experimental results illustrate ChatCam’s effectiveness in text-conditioned trajectory generation and show the potential to simplify camera movements and lower technical barriers for creators.
Strengths
1. ChatCam is a novel system to generate camera trajectory via natural language. 2. The paper is well-written and easy to follow. 3. The proposed GineGPT and Anchor Determinator are technically solid.
Weaknesses
The system is only verified on several limited 3D static scenes, and the generated trajectory is relatively short. Some components are not fully verified. For example, what’s the effect of the text prompt “smooth panning speed” in Fig. 3? How does this text prompt affect the trajectory? Since the method was only evaluated on a small dataset, and the trajectory seems short, it's not clear whether the generations overfit to a small set of text prompts. Some experimental details are unclear. 1. How about the training dataset? L114 suggests there are 1000 trajectories to train the trajectory generation. I would suggest introducing the training dataset in detail in Sec 4. 2. What’s the time cost of Anchor Determination in inference?
Questions
See limitations. 1. Reference information inaccurate: [32,34], NeurIPS 2023 paper, not NeurIPS 2024 2. For the same text prompt, ChatCam is expected to generate different trajectories for different scenes. A verification is interesting to illustrate the effectiveness of an Anchor Determinator. 3. How long is the trajectory sequence? 4. It’s not clear whether the approach works for dynamic scenes, but it seems the Anchor Determinator does not model the temporal relations.
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
Yes
Summary
This paper propose ChatCam, a pipeline that utilizes GPT to translate the natural language into professional camera trajectory, which enhances the video production process for common users.
Strengths
1. This paper divide the task into three steps: observation, reasoning and planning. The first two steps is completed by GPT and get the instruction of every subtasks. In the third step, the author proposes Anchor Determinator and CineGPT to generate the initialization point and camera trajectory. 2. The qualitative and qualitative results and video demo proves the efficiency of this method. 3. The paper is well written and easy to follow.
Weaknesses
The design of this paper is clever and makes good use of the advantages of LLM. But I still has some concerns. If the authors can address my concerns, I would change my rate. 1. In the planning part of Figure 2, how can you ensure there is no collision with 3D scenes in every sub-step, such as “an S-shaped path, smooth panning speed”. If the structure of the room is complicated, it is easy to collide with the scenes. 2. Is natural language really the best way for users to interact？Maybe user can import the 3D GS of scenes into Unreal Engine, where Luma AI provide this plugin (https://www.unrealengine.com/marketplace/en-US/product/luma-ai), and generate the camera trajectory by adding camera key points in the 3D space by UI interface, with position, rotation, camera intrinsics. The frames between key points can be obtained by using bilinear interpolation. You can preview whether collision has occurred in real time, and you can also adjust camera parameters by drag. (https://dev.epicgames.com/documentation/en-us/unreal-engine/creating-camera-cuts-using-sequencer-in-unreal-engine) 3. Can you show more complicated camera trajectories, such as dolly. The “straight forward and roll”, “S-shaped path”, “from left to right”, which just need coarse control, seems can be easily obtained by the item 2 I propose above (UE).
Questions
If the generated trajectory is not satisfactory or contains collision, how can user interact with the ChatCam? Whether the pipeline can modify the generated trajectory?
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
The collision in the 3D scenes is not explicitly and hardly constrained in this pipeline.
Summary
The paper proposes a method for generating camera trajectories for rendering a 3D scene, conditioned on natural language. This problem statement is novel and original, very useful, and the paper demonstrates convincing results. The method uses two components: a language-conditioned camera trajectory generator, and a language-conditioned camera anchor generator. An LLM takes a high-level language query as input and creates a plan based on these two components.
Strengths
The problem statement is novel and seems to be very useful with high potential impact. The method is creative and novel. Results are excellent. They demonstrate wide range of trajectories on complex scenes with several interesting elements.
Weaknesses
The paper fails to demonstrate the planning output of the LLM. Is Fig. 2 an actual output of the method, or just an illustration? I would expect to see many more results of the actual plan used to generate the trajectories. It is very difficult to understand the workings of the method without these. The method seems far from reproducible. Sec 3.3 talks about several important components of the method, but only at a very high-level that would not enable anyone to reproduce the results. The details of the finetuning proposed in L142-146 are also unclear. The quantitative evaluations rely on ground truth trajectories. Shouldn't there be many possible GT trajectories consistent with a language query? How would this diversity impact the evaluations? The same holds true for output of the system. Can the method generate diverse outputs? If so, the paper should include such results, and ideally, should also update the metrics to reflect this. In the output in Fig. 2, there does not seem to be an end anchor point. How would the method know the extent or volume of space the S-shaped path should cover? Are there many cases where the volume of the space is undetermined based on the plan? Comparisons to baselines should also be included as videos, especially as the paper talks about their trajectories.
Questions
What are the limitations of the approach? Does the LLM fail to produce reasonable plans at times? When does that happen? Is the scale of the trajectories always fully determined? Can you comment more on the failure cases?
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
4
Limitations
Limitations are not mentioned
Summary
The paper proposes a method to generate camera trajectories from user prompts. The idea is to pass on the prompt to an LLM, find a starting anchor location (searched and refined using initial images used to construct the radiance field) and then use CineGPT (a cross-model transformer) trained for the next token prediction on quantized camera trajectories. Compared with some minimal baselines, the proposed approach attains favourable results in terms of MSE errors (on translation and rotation) and user-reviewed visual quality and alignment metrics.
Strengths
- The paper is well written and well presented. I was able to understand the parts "explained in the paper" quickly. The figures are well made, and the videos in the supplementary section were easy to consume. - The idea is well-motivated and I felt the work is in a good direction. - I liked the idea of next token prediction on quantized camera trajectories and it appears novel to me.
Weaknesses
- Baselines could be better; I feel the comparisons were extremely weak - Many details in the paper are missing, making it difficult to comprehend the approach fully. The major misses are the details on the dataset construction, the proper explanation of baselines, and the LLM prompts. I give more detailed questions on the same in the next section. - related work can be strengthened as well. They can cut a bit on the literature of radiance fields and 3D scene understanding and probably add more on trajectory learning/optimization
Questions
- It appears a path prediction network might do well in this case. What is eventually needed is to select a set of anchor locations on the path and then interpolate between them. It is not clear which interpolation algorithm was used for the baselines in the paper, which I believe is a crucial detail. Also, please bring more insights on which aspect of the baselines actually fails, is it the keypoint/keylocation selection or the interpolation. - I would argue for replacing CineGPT with a rule-based optimizer or an interpolation algorithm. Since it does not observe visual modality, why is it not possible to do it through rule-based system (quantizing key kinds of camera movements). - It is not clear to me, how to select the extent of camera movement. For example, in Figure 2, after determining the initial anchor (outside the window), CineGPT is called to pan straight forward. What aspect of the algorithm decides how long this pan should go? What stops it from bumping into walls, if the visual modality is not seen or explicit start-end points are not given? - It is not clear how the dataset was constructed. Was it done in scenes with objects/people present in the CG? Who collected the dataset, and what instructions were given to the collector? How the collection was planned? What kind of camera movements were included? All these details are extremely crucial. - There are open-source blender movies (e.g. https://spring-benchmark.org/) which are utilized in several benchmarks. Would it be a good idea to exploit these in the dataset construction? - I would argue for an ordered keypoint prediction network and then spline interpolation between them. It would be a well designed meaningful baseline in this case. - Not sure if clip can efficiently perform the task of location selection, would like to hear more from the users on the same - Not fully clear, why would the method generalize to out-of-domain objects/places (for example, an opera house in this case. was it included in the train set?). That reduces the confidence on the final presented output. - It would be useful to share the used prompt into the author's response, which would help the reviewers to try it out themselves and observe its limitations - Discussion on some works on camera trajectories would be useful. Professional camera paths are often composed of constant, linear and parabolic segments [A][B]? The static trajectories also play a key role in this? Did the authors think about the importance of static trajectories and did they include it in the dataset? - The final trajectory composition step appears to be non trivial to me and not fully clear at this stage. It looks like a hack right now. [A] Grundmann et al. Auto-Directed Video Stabilization with Robust L1 Optimal Camera Paths. CVPR 2011 [B] Gandhi et al. Multi-Clip Video Editing from a Single Viewpoint. CVMP 2014
Rating
4
Confidence
5
Soundness
3
Presentation
3
Contribution
3
Limitations
The authors did not discuss the limitations properly. I do not see an explicit limitations section.
Trajectory Composition
The algorithm for trajectory composition works as follows: 1. It takes as input a list of trajectories and anchor points. If adjacent anchor points are encountered in the input, the algorithm will report this as illegal input. 2. Merge adjacent trajectories until there are no adjacent trajectories left in the list. During the merging process, calculate a Euclidean transformation (6 DoF) and apply it to the latter trajectory, ensuring its starting point coincides with the position and rotation of the endpoint of the previous trajectory. 3. For all trajectories, if a trajectory has two adjacent anchor points, find a similarity transformation (7 DoF) to make the position of its starting point and endpoint coincide with the two anchor points. If it has one adjacent anchor point, calculate a Euclidean transformation (6 DoF) to make the position and rotation of its starting or ending point coincide with this anchor point. We will include pseudo-code for this algorithm in future versions to enhance clarity.
On rule based trajectories
- As you created the trajectories in the dataset, in similar way you could create a dictionary of camera movements and give them a name (S-shaped, U-turn). Add few parameters to control and then use them across anchor points. Seems doable to me. Optionally, apply some optimization to post-process (example: L1 norm optimization or smoothing) - There are too many stochastic components and the reliability of the final result is doubtful I am still not convinced and I am staying with my original rating.
Thanks for your response
Thank you for your feedback! Your suggestion of "creating a dictionary of camera movements and giving them names (S-shaped, U-turn)" aligns with our understanding and implementation of the rule-based baseline during the rebuttal. However, as shown in the attached PDF and discussed with Reviewer **Shd5**, the results from this approach do not perform as well as our proposed method. This rule-based approach relies on a predefined dictionary of camera movements, which is inherently **finite**, whereas our GPT-based text-to-trajectory translation model can handle theoretically infinite input prompts. Additionally, we want to remind the reviewer that the rule-based approach cannot determine **anchor points** independently. The accurate determination of anchor points is a key part of our technical contribution. Moreover, the rule-based method lacks the ability to interact with users to modify or improve results, which is another significant advantage of our proposed approach. Regarding your concerns about the reliability of our final results, we respectfully disagree with the assertion that there are "too many stochastic components" in our approach. We welcome the reviewers to point out specific stochastic elements in our method so we can engage in further discussion. Before receiving your feedback shortly before the discussion period ends, we had already provided the LLM prompt as per your suggestion, allowing the conversational process in our method to be fully reproducible. **We encourage reviewers to try it out.** Additionally, any aspects of the final results that the user finds unsatisfactory can be adjusted through further interaction. Therefore, we believe that our results are reliable, especially when compared to rule-based approaches.
LLM Prompt （1/2）
As suggested by reviewer **kHgE**, here we post our designed prompt to instruct the LLM agent. You are a dialog agent that helps users to operate cameras in 3D scenes using dialog. The user starts the conversation with a 3D scene represented by NeRF or 3DGS. The user will describe the camera trajectory in his mind in words, and you help him generate the camera trajectory. You have two useful tools, the first is CineGPT, which can help you translate text into trajectory. The second is Anchor Determinator, which can find anchor objects to correctly place the trajectory in the 3D scene. Please act according to the following instructions: INSTRUCTIONS: 1. The user-provided description includes (a) descriptions of the trajectories, including the camera’s translation (i.e., “pan forward”), rotation (i.e., “turn left”), and camera parameters (i.e., “increasing focal length”) in the trajectory or the trajectories’ features like shape or speed, and (b) some specific descriptions of the scene, which we call anchor points, such as “starting with the close-up of the car” or “a bird’s-eye view of the temple”. 2. For (a) descriptions of the trajectories, invoke the API of the CineGPT to translate human text into trajectory. When calling CineGPT, try to use the description of the trajectory itself without involving any specific scene information. i. To summon CineGPT, the command is termed "infer_cinegpt". Its arguments are: "traj_description": "<traj_description>". ii. This API returns a JSON containing the camera trajectory consisting of camera pose and camera intrinsics for each frame. 3. For (b) some specific descriptions of the scene, invoke the API of the Anchor Determinator to get anchors to place the trajectory in the 3D scene. You need to find a description of an object or an image from the user's words. Anchor Determinator will find the picture that best matches your input and return its camera pose as the anchor. i. To call Anchor Determinator, the command is termed "get_anchor". Its arguments are: "anchor_description": "<anchor_description>". ii. This API returns a JSON containing the anchor camera pose and camera intrinsics. 4. When the user's description contains multiple stages, you need to learn to split it into units of trajectory and anchor points, and call CineGPT and Anchor Determinator accordingly. In this case, you would interleave calls to CineGPT and Anchor Determinator. 5. Invoke the API of trajectory composition to combine the obtained sub-trajectoires and anchor points. i. To call trajectory composition, the command is termed "traj_compose". Its arguments are: " compose ": "<list_of_traj_anchor>". ii. This API returns a JSON containing the composed camera trajectory consisting of camera pose and camera intrinsics for each frame. iii. When encountering illegal input, this API will raise an error. 6. Your generated plan should follow these steps. i. Call CineGPT with trajectory descriptions (one or more times). ii. Call Anchor Determinator with anchor descriptions (one or more times). iii. Connect output trajectories ensuring they pass through the determined anchor points. iv. Ask the user for further clarification if needed. v. Render a video using the generated camera trajectory.
LLM Prompt (2/2)
You should only respond in JSON format following the template below: RESPONSE TEMPLATE: { "observation": "observation", "reasoning": "reasoning", "plan": "a numbered list of steps to take that conveys the long-term plan" } In your response JSON, it is generated in the order of "observation", "reasoning", and "plan". Put any string in one line, do NOT include any new line character in observation, reasoning, plan, self-critique or speak. Here are some examples you can refer to: EXAMPLE 1: User Input: " Starting with a close-up of the man in the red T-shirt, the camera pans right to the woman in the white dress, and then looks around." Your Output: { "observation”: “The user describes a camera trajectory starting with a close-up of a man in a red T-shirt, panning right to a woman in a white dress, and then looking around.", "reasoning": " I need to find the anchor points for the close-up of the man in the red T-shirt and the woman in the white dress. Then, I will generate the trajectory for panning right and looking around.", "plan": " 1. Call Anchor Determinator with 'close-up of the man in the red T-shirt'. 2. Call Anchor Determinator with 'the woman in the white dress'. 3. Call CineGPT with 'pan right'. 4. Call CineGPT with 'look around'. 5. Connect the output trajectories, ensuring they pass through the anchor points. 6. Ask the user for further clarification if needed. 7. Render a video using the generated camera trajectory." } EXAMPLE 2: User Input: "Take a dolly zoom shot of the woman's face." Your Output: { "observation": "The user requests a dolly zoom shot of the woman's face.", "reasoning": "I need to identify the woman's face as an anchor point and generate a detailed trajectory for the dolly zoom shot, which means moving forward while decreasing focal length to maintain the size of the woman's face in the frame.", "plan": "1. Call Anchor Determinator with 'the woman's face'. 2. Call CineGPT with 'move forward while decreasing focal length to maintain the size of the woman's face in the frame'. 3. Connect the output trajectories, ensuring they pass through the anchor points. 4. Ask the user for further clarification if needed. 5. Render a video using the generated camera trajectory." } EXAMPLE 3: User Input: " Start from the iron gate and move forward to the computer monitor, then u-turn and walk out of the iron gate." Your Output: { "observation": "The user describes a camera trajectory starting from an iron gate, moving forward to a computer monitor, then making a U-turn and walking out of the iron gate.", "reasoning": "I need to find the anchor points for the iron gate and the computer monitor, then generate a trajectory that moves forward to the monitor, makes a U-turn, and moves back out of the iron gate.", "plan": "1. Call Anchor Determinator with 'the iron gate'. 2. Call Anchor Determinator with 'the computer monitor'. 3. Call CineGPT with 'move forward'. 4. Call CineGPT with 'U-turn and move backward'. 5. Connect the output trajectories, ensuring they pass through the anchor points. 6. Ask the user for further clarification if needed. 7. Render a video using the generated camera trajectory." }
Conversation Example
As suggested by reviewer **dvT5**, here we post the complete conversation between the user and ChatCam for the new result. Please refer to Figures A & B of the attached PDF for the visualization of trajectories and rendered video frames. **User:** From the top of the stairs, pan downwards to the entrance of the green tunnel. Move forward to go through the tunnel. Make a u-turn and move forward to the yellow bulldozer on the table. **ChatCam:** { "observation": "The user describes a camera trajectory starting from the top of the stairs, panning downwards to the entrance of a green tunnel, moving forward through the tunnel, making a U-turn, and moving forward to a yellow bulldozer on a table.", "reasoning": "I need to find anchor points for the top of the stairs, the entrance of the green tunnel, and the yellow bulldozer. Then, I'll generate trajectories for panning downwards, moving forward through the tunnel, making a U-turn, and moving forward to the bulldozer.", "plan": "1. Call Anchor Determinator with 'top of the stairs'. 2. Call Anchor Determinator with 'entrance of the green tunnel'. 3. Call Anchor Determinator with 'yellow bulldozer on the table'. 4. Call CineGPT with 'pan downwards'. 5. Call CineGPT with 'move forward through the tunnel'. 6. Call CineGPT with 'U-turn'. 7. Call CineGPT with 'move forward to the bulldozer'. 8. Connect the output trajectories, ensuring they pass through the anchor points. 9. Ask the user for further clarification if needed. 10. Render a video using the generated camera trajectory." } **User:** This trajectory hits a wall! Starts all over again. But after you make a u-turn, try going to a colorful bulldozer, then to a white table with books on it, and finally to a yellow bulldozer. **ChatCam:** {"observation": "The user indicates that the initial trajectory hits a wall and needs revision. The revised trajectory starts from the top of the stairs, pans downwards to the entrance of the green tunnel, moves forward through the tunnel, makes a U-turn, then moves to a colorful bulldozer, proceeds to a white table with books, and finally ends at a yellow bulldozer.","reasoning": "I need to find anchor points for the colorful bulldozer and the white table with books, in addition to the previously identified points. I'll regenerate the trajectory to include these new elements and ensure the path avoids any obstacles like walls.","plan": "1. Call Anchor Determinator with 'top of the stairs'. 2. Call Anchor Determinator with 'entrance of the green tunnel'. 3. Call Anchor Determinator with 'colorful bulldozer'. 4. Call Anchor Determinator with 'white table with books'. 5. Call Anchor Determinator with 'yellow bulldozer on the table'. 6. Call CineGPT with 'pan downwards to the entrance of the tunnel'. 7. Call CineGPT with 'move forward through the tunnel'. 8. Call CineGPT with 'U-turn'. 9. Call CineGPT with 'move to the colorful bulldozer'. 10. Call CineGPT with 'move to the white table with books'. 11. Call CineGPT with 'move to the yellow bulldozer'. 12. Connect the output trajectories, ensuring they pass through the anchor points and do not hit any walls. 13. Ask the user for further clarification if needed. 14. Render a video using the generated camera trajectory."} **User:** Very good! After this, I hope to add a dolly zoom for the yellow bulldozer. Give me the final plan. **ChatCam:** { "observation": "The user wishes to add a dolly zoom effect targeting the yellow bulldozer at the end of the revised camera trajectory.", "reasoning": "To incorporate the dolly zoom effect, I'll need to adjust the camera movement and focal length as it approaches the yellow bulldozer, ensuring that the size of the bulldozer remains consistent in the frame while creating a dramatic spatial effect.", "plan": "1. Call Anchor Determinator with 'top of the stairs'. 2. Call Anchor Determinator with 'entrance of the green tunnel'. 3. Call Anchor Determinator with 'colorful bulldozer'. 4. Call Anchor Determinator with 'white table with books'. 5. Call Anchor Determinator with 'yellow bulldozer on the table'. 6. Call CineGPT with 'pan downwards to the entrance of the tunnel'. 7. Call CineGPT with 'move forward through the tunnel'. 8. Call CineGPT with 'U-turn'. 9. Call CineGPT with 'move to the colorful bulldozer'. 10. Call CineGPT with 'move to the white table with books'. 11. Call CineGPT with 'move to the yellow bulldozer'. 12. Call CineGPT with 'dolly zoom on the yellow bulldozer'. 13. Connect the output trajectories, ensuring they pass through the anchor points, do not hit any walls, and incorporate the dolly zoom effect. 14. Ask the user for further clarification if needed. 15. Render a video using the generated camera trajectory." }
Confusion about the Figure C.
In my opinion, you replacing CineGPT with a rule-based algorithm in the Figure C. So the anchor points should be same for these two methods. Figure C does not show the anchor 'entrance of the green tunnel'. CineGPT generate camera trajectories only based on text description without spatial information. How can you avoid collision when calling CineGPT with 'move forward through the tunnel'? I think if the anchor point is accurate, the collision will not occur with a high probability. In the same time, this will also be easy for rule-based method too.
Thanks for your response
Thank you for reading our rebuttal and for your question. In our implementation, we replaced the entire system (CineGPT + Anchor Determinator + LLM agent) with a rule-based algorithm, rather than just replacing CineGPT. Specifically, we used a 1-to-1 rule-based mapping to generate two sub-trajectories from the instructions "pan downwards" and "move forward." We then assembled these sub-trajectories and positioned them at the estimated first-frame key location ("top of the stairs"). This key location was estimated using LERF instead of our proposed Anchor Determinator. We made this choice as the rule-based system, without the LLM agent, lacks the capability to extract necessary inputs for anchor determination, such as "the entrance of the tunnel." While the "move forward" instruction can be executed correctly if the anchor is accurate, this assumption does not hold for the rule-based baseline. In fact, the rule-based approach cannot understand specific object-level prompts unless those objects are covered by the predefined rule set. In other words, this rule-based baseline can only generate a trajectory based on the input but lacks the ability to accurately place it within the scene. In this case, human assistance was required to determine the first key location, whereas our approach is capable of doing this autonomously. Additionally, we would like to note that different prompts were used for Figures A/B and Figure C. This was necessary because the input for Figures A/B would have led to meaningless results with the rule-based baseline. The rule-based baseline can only handle simple inputs like "pan forward," while our approach can manage much more complex inputs without human assistance to specify key locations. It can also collaborate with users to refine the results through conversation. We hope this clears up your confusion. We welcome you or other reviewers to raise any further concerns so that we can address them in the discussion period.
I appreciate the efficiency of your whole pipeline (user input -> LLM agent -> Anchor Determinator -> CineGPT). But while comparing the CineGPT with the rule-based/interpolation method, I think you should use the anchors generated from LLM agent and Anchor Determinator as the input of the rule-based and interpolation method, which can directly show the efficiency of the CineGPT. When the anchors are accurate, I am curious about the differences between CineGPT and the rule-based/interpolation method. For "move forward", I think it is easy to generate a straight trajectory based on rules and interpolation and adjust the scale to fit the start and end anchors. I am not sure if the specific object-level prompts are useful for CineGPT, according to "Since our dataset is not attached to any objects or places, suggested movies cannot be directly used in the dataset construction. ". Is there difference between the prompt "move forward through the tunnel" and "move forward along the road". I think the key of success is the spatial accuracy of the anchors.
Thanks for your response
Thank you for your detailed feedback! Based on your comments, we believe you agree that the entire ChatCam pipeline is efficient. This reinforces the significance of our study on conservation AI in camera control. We appreciate your suggestion to compare CineGPT with a rule-based approach while keeping the anchors the same, as this directly demonstrates the effectiveness of CineGPT. In the revised version of our paper, we will include both qualitative and quantitative results from this experiment. Since we are unable to modify the PDF at this stage, we will verbally describe the results of the rule-based/interpolation method in this scenario: with accurate anchor points, the rule-based approach successfully goes through the tunnel with the "move forward" prompt. However, it is also important to emphasize that the accurate determination of anchor points is not trivial and is a significant part of our contribution. Furthermore, the performance of the rule-based approach depends on the rule set, while the number of possible rules is theoretically infinite. If the user input includes more complex instructions beyond the predefined rule set, the rule-based method cannot perform effectively. We hope this can address your remaining concern regarding whether CineGPT could be replaced by a rule-based method. We would be happy to engage in further discussion on this or any other aspect.
Thanks for your response. My concerns have been addressed. It's good to add the ablation of the CineGPT and rule-based/interpolation method in the final version to show the key design of your pipeline.
Facilitating the author-reviewer discussion
Dear Reviewer of the submission 684, Thank you for your time and effort so far in reviewing this paper. The author submitted the rebuttal to your review, and the author-review discussion period will end tomorrow (13th August). It would be great if you could check if the rebuttal addresses your concern or not. Essentially, we have mixed reviews on this paper and need your input. Thank you in advance. Best regards, Your AC of submission 684
Thanks for the response. It addresses my concerns. The revised paper should clearly mention the issues with defining the scale of the trajectories and no collision handling, and show corresponding results.
Thanks for your response
Thank you for your response and for appreciating our research! We will mention these issues and include the experimental results in the revised paper. We will also incorporate the reviewers' other suggestions to further strengthen our work.
After reading other reviewers' comments and authors' response, most of my concerns are addressed. Conversation Example shows how to handle the collision by language in an iterative way, which is user-friendly. In the response to kHgE, they prove the efficiency of CineGPT compared to the interpolation and rule-based methods, which means this method requires fewer anchor points than existing tools. I hope authors could add these experiments and discussions in the final version. I would change my rate to borderline accept.
Thanks for your response
Thank you for your response! We greatly appreciate the valuable discussion and will revise our work based on the reviewers' suggestions.
Thanks for the response. Most of my concerns have been solved.
Thank you for your response
Thank you for the response and we will revise the submission according to the reviewers' concerns.
Decision
Accept (poster)