We cast real-world humanoid control as a next token prediction problem, akin to predicting the next word in language. Our model is a causal transformer trained via autoregressive prediction of sensorimotor trajectories. To account for the multi-modal nature of the data, we perform prediction in a modality-aligned way, and for each input token predict the next token from the same modality. This general formulation enables us to leverage data with missing modalities, like video trajectories without actions. We train our model on a collection of simulated trajectories coming from prior neural network policies, model-based controllers, motion capture data, and YouTube videos of humans. We show that our model enables a full-sized humanoid to walk in San Francisco zero-shot. Our model can transfer to the real world even when trained on only 27 hours of walking data, and can generalize to commands not seen during training like walking backward. These findings suggest a promising path toward learning challenging real-world control tasks by generative modeling of sensorimotor trajectories.
Paper
Similar papers
Peer review
Summary
This paper proposes to use next-token-prediction as a learning objective and train a casual transformer for humanoid locomotion. Compared to previous RL-based methods, the advantage of this method is to fuse data from different sources, including mocap data, videos, RL controller, and MPC controller. The authors successfully show that their model has lower tracking errors when trained with unlabeled data. The trained model is deployed on a Digit humanoid robot and shows robust locomotion across diverse real-world areas.
Strengths
1. This paper introduces the next-token-prediction framework into humanoid robots. The intuition is simple yet very interesting. 2. The usage of different data sources, such as the RL controller and Internet videos, is novel. 3. The real-world experiments are very solid. It is exciting to see such a framework would bring benefits on real-world robot learning.
Weaknesses
1. As shown in Figure 8, the prediction error is correlated to the tracking error. I am curious about how much data the authors need to make these two errors correlated, since from my knowledge, the real world is much more complex and it is usually hard to have such a correlation, 2. Recently, there have been numerous impressive advancements in humanoid robotics. While these developments are exciting, it is unfortunate that most of these works do not open-source their code. I understand the reasons behind keeping the code proprietary, but open-sourcing it would provide significant benefits to the broader community. I believe there are a lot of engineering details behind the simple framework, both in the algorithm and in the real world deployment, which however might not be shown in the paper.
Questions
See weaknesses.
Rating
8
Confidence
4
Soundness
4
Presentation
4
Contribution
3
Limitations
The authors have mentioned the limitations.
Summary
Rather than training an RL model to learn how to walk, this paper focuses on an SSL based approach towards humanoid locomotion. Using autoregressive prediction of actions and sensor data, they pre-train and deploy zero shot to the real world. The results are strong, performing better than RL at times.
Strengths
* The idea of using a masked token for unknown modalities is smart and works well * Generalization to walking backwards is impressive * The approach is scalable and outperforms RL in many cases
Weaknesses
* There was a lack of details/comprehensiveness on some of the experiments. For example, it was not clear to me how many trials were used for Figure 5. The authors also did not compare to existing MPC and only compared to RL based approaches. Limited details for hparams were given. * There are more ablations that could be done to further confirm design decisions (see questions).
Questions
* Did the authors do experiments with quantization/VQ? If so it would be nice to see the results for that. * Ablations for masked token? * I did not understand this comment: "Rather than predicting the next token in a modality-agnostic way, we make predictions in a modality aligned way. Namely, for each input token we predict the next token of the same modality." * Ablations on which way to pre-train--noisy and incomplete or joint? * Not much information on the modalities themselves/examples modalities. If there are tons of motors/sensory information used at any given point are all of these predicted at once, with the same model? In that case, how is the causal mask constructed? * What is the prediction error in Table 1c? * Was predicting both \hat{o}_{i+1} and \hat{a}_{i+1} at the same time tried?
Rating
8
Confidence
4
Soundness
4
Presentation
4
Contribution
4
Limitations
Yes
Response by Authors
Thank you for your response. We are glad to hear that we addressed all of your initial comments. We would like to offer some clarifications regarding the related work discussion. We believe that reviewer FS4P’s suggestion about an additional reference in robotic manipulation was a minor point, which we agreed to include. Reviewer FS4P acknowledged that this concern was fully addressed in their response and also noted the novelty of our work in their review. We want to emphasize that we do not claim "Autoregressive Pre-training for Robotics" as our primary contribution. In fact, we already have the "Transformers for Robotics" section in the related work that discusses a number of papers on autoregressive pre-training in robotics. We are happy to expand this section further. The novelty of our work lies in casting real-world humanoid locomotion as a joint distribution learning problem with transformers. We further show that this formulation enables training with noisy or missing data (e.g., derived from videos) and achieves excellent results in the real world. While our work can have excellent impact on multiple areas, the paper and title focus on humanoid locomotion. We chose this problem as a test bed because it is particularly challenging due to its highly dynamic, complex nature. The success of our method in this domain is particularly noteworthy (few prior approaches used learning and the ones that did rely on RL). Given these clarifications and the original review, we believe the original score more accurately reflects our contribution and that it would be unfair to reduce it. We respectfully ask you to reconsider maintaining your initial score.
I appreciate the response. Perhaps, this was a mistake on my end, as while reading the paper I believed that the idea of autoregressive pre-training for robotics was a primary contribution. Adding to this, I think that the lack of inclusion of autoregressive pre-training approaches in the "Transformer for Robotics" section is of slight concern, as *autoregressive pre-training approaches for robotics are the most similar to this work.* (I say lack of inclusion because all approaches listed were either not pre-training or not autoregressive as far as I could tell). Together, these enhanced my initial beliefs regarding the merit of this work. I acknowledge the contributions discussed and the excellent results. Hence, I think a score of 8 is fitting. This updated score is based off of the planned inclusion of more autoregressive pre-training approaches in the related works.
Response by Authors
Thank you for your response. We appreciate your acknowledgement of our contributions and results. We would like to clarify a few points below: - Autoregressive pre-training is not a primary contribution of our work. While our approach can be used with pre-training, our method does not involve pre-training. We are "just" training a model from scratch and deploying it zero shot, without multi-stage training or fine-tuning. - We are more than willing to expand the “Transformers for Robotics” section with work on autoregressive pre-training in robotics to provide a more comprehensive context. - Our primary contribution lies in formulating real-world humanoid locomotion as a joint distribution learning problem, enabling training with noisy or missing data and achieving very strong results in a particularly challenging domain. Thank you for your consideration.
Summary
This paper trained an autoregressive transformer for humanoid robot walking control using four types of data. The data sources included trajectories rolled out by RL Policy and Scripted Method, existing datasets, and human poses from YouTube videos. This work successfully enabled the humanoid robot to walk in urban environments without falling. The study explored how to train using data with missing actions, specifically by utilizing observation data and observation-action data and demonstrated the effectiveness of training with missing data. The robot in this work does not have visual capabilities.
Strengths
1. The research problem in this work is very novel, and it has been verified on real humanoid. The effects shown in the video are excellent. 2. This work is dedicated to applying autoregressive Transformers to humanoid locomotion, which is a promising direction. 3. This work explores the usefulness of training with action-free data.
Weaknesses
1. Claiming that training with video might be overclaiming; in fact, it only uses human poses from the video, which is quite different from training with video. 2. Pre-training with action-free data is not a novel contribution. 3. Based on the model architecture and training data, it appears that the robot can only walk randomly without controlling speed or turning. However, in Figure 5, it seems to control the robot’s walking. How is this achieved? What exactly is the observation? It doesn't seem to be the observed images. How is the observation transformed into R^m? 4. For the four types of data collected—Neural Net Controller, Model-based Controller, MoCap, Internet Videos—the authors did not conduct ablation experiments to verify the usefulness of each type of data. Especially the MoCap and Internet Videos from inverse dynamics are highly questionable in their usefulness. Considering that the authors did not prove the usefulness of data from inverse dynamics, the paper would degrade to merely fitting some existing policies with a transformer, significantly limiting its practical value. 5. The paper does not specify the control frequency or how the control frequency changes with model size, nor does it explain how to deploy the robot in the field. Where is the GPU located? 6. The pre-training method used in GR1[1] is very similar to this work. The only difference is that GR1 uses two-stage training while this paper uses joint training. Both output observation-action, but this paper does not discuss this. [1] UNLEASHING LARGE-SCALE VIDEO GENERATIVE PRE-TRAINING FOR VISUAL ROBOT MANIPULATION Writing suggestions: 1. Figure 3 is currently unclear. I suggest adding sequence numbers to the tokens to distinguish their order and using superscripts to differentiate predicted tokens from given tokens. 2. Equations 5-10 are not very necessary and could be condensed into 1-2 lines. More space could be used to explain the training data for the transformer, which is more important. 3. Using subplots to redraw Figures 6 and 7 would be clearer.
Questions
1. How to control the robot's walking? 2. What exactly is Observation, and how is it transformed into R^m? 3. From my understanding, the Neural Net Controller is the RL policy, the Model-based Controller is a scripted policy (non-learning method), and MoCap is a robot dataset, but not for the used Agility Robotics. Is my understanding correct? 4. Is it possible to conduct ablation experiments on each type of data? 5. What is the control frequency, what GPU and model size are used during deployment, and is the GPU placed on the robot or on a nearby PC?
Rating
7
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
1. One key limitation of the article is the lack of visual input, which is not discussed in the paper. 2. Another limitation of the article is that it does not show how the real humanoid turns; it appears that the robot in the video only walks randomly. 3. Is it compliant to use YouTube videos, especially those featuring humans as the main subject?
Response by Authors
Thank you for your response and for raising the score. Please find our responses below: > The project page doesn't include videos of the robot turning; can it actually turn in the real world or walk for an extended period of time? However, the project page seems to be inaccessible now. It would be interesting to see its obstacle avoidance capabilities under control. Our policy can turn in all directions and walk for extended periods of time, up to 2 hours on a single battery charge. We have extensively tested these capabilities during a week-long deployment in San Francisco. Thanks to the responsive and accurate command following, our robot can avoid obstacles and navigate busy city streets with pedestrians. As an additional data point, our omnidirectional walking capabilities are comparable to the state-of-the-art RL approach in Figure 3 of [33], with improved command following as demonstrated in Figures 5 and Figure 6 (left) of our manuscript. We apologize for the inaccessible project page. It seems that our GitHub-hosted anonymous website was suspended by GitHub, which we are working to resolve with GitHub. Per the rebuttal instructions, it seems that we are not allowed to provide a new link to external pages in the rebuttal. We welcome any suggestions on how we might share the videos demonstrating these capabilities within the constraints of the rebuttal process and commit to include them in the release. > Additionally, if the robot needs to go uphill or downhill slightly, would it fail? And it will be fine to analyze its robustness. Our policy can handle gentle slopes, and we have tested it in the real world on inclines of up to 8.7% (5 degrees). For example, Figure 1 (row 3, column 1) shows the robot walking up a 7% slope in San Francisco. Regarding robustness, our approach trained purely on offline trajectories without any domain randomization has shown surprising effectiveness in walking in the real world. However, as we acknowledge in the limitations section, our policies may be less robust to sudden, large disturbances compared to RL policies. In follow-up experiments, we found that pre-training policies with our proposed framework and fine-tuning them with a small amount of RL leads to excellent results, enabling rapid acquisition of new capabilities like walking on steeper slopes and robustness to large disturbances. > If this method were extended to enable humanoid robots to go up and down stairs, what challenges would arise? To extend our method to enable the robot to go up and down stairs, we would need to incorporate vision to guide the foot placement. Our method can be readily extended to achieve this by incorporating images (potentially pre-processed using a vision encoder) as part of observations in addition to proprioception. One challenge would be in collecting training trajectories that contain visual inputs. This would be relatively straightforward for trajectories collected using the same robot body (e.g., via prior RL or MPC controllers). To incorporate human trajectories from mocap or videos, we would need to include ego vision inputs where available (e.g., first-person videos) and use our strategy for training with missing data for trajectories without ego vision (e.g., third-person videos). > Some of the suggestions may go beyond the scope of this paper Thank you for raising these questions. We will include the discussion of capabilities, limitations, and future work in the revised version of the manuscript with an extended discussion section.
Added videos of the robot turning to the project page
We have resolved the hosting issue with GitHub and our anonymous project page is accessible again via the link provided in the original manuscript. As requested, we have included videos of the robot turning in the real world under the “Turning in the Real World” section at the bottom of the page. Please let us know if all your concerns are now fully addressed. We would be happy to provide any additional information.
Thank you for your response. I appreciate your excellent experiments; it's fascinating to see how they can be applied to real-world robotics. I agree with qt3e's point—a more detailed autoregressive model for the robotics section would better reflect the contributions of this paper.
Response by Authors
Thank you for your response. We are glad to hear that we have addressed your remaining concerns regarding robot capabilities and appreciate your acknowledgement of our results. We are more than willing to include a more detailed autoregressive models for robotics section. We will incorporate the suggestion in the revised manuscript. We believe that we have addressed all of your concerns from the initial review and subsequent discussions. Given these and in light of the strengths you and all other reviewers have highlighted, we respectfully ask you to consider increasing the overall score to better reflect our contributions.
I believe the score I have given is appropriate. This paper represents yet another success of Next Token Prediction in a different domain, a success that has been replicated many times. There are few new techniques introduced. Essentially, papers in this paradigm primarily involve preparing data, training, and testing, without much technical complexity. I also took into account that this paper does not achieve significant breakthroughs; previous methods could also achieve tasks such as walking and turning, just not using next token prediction. A notable drawback is that the scenarios considered are particularly simple, with only 100 dimensions of the robot's body state being considered as observations. The challenge might lie in processing and unifying large amounts of data. The primary reason this paper achieves such a high score maybe is that it effectively addresses the problem of studying humanoid locomotion using AI methods and works well in the real world. If the same approach were applied to robotic manipulation, it might not score as highly. However, the manipulation problem involves more complex considerations than humanoid locomotion, such as language instructions, visual perception, and the robot's state. Despite some limitations, I still believe this work is valuable because it demonstrates that next token prediction works well in such high degrees of freedom, given the appropriate data. Considering the author's commitment to open-sourcing their work, which could significantly advance the progress of humanoid locomotion—a field that has been relatively unexplored—I believe a score of 7 is reasonable.
Summary
This work view the robot locomotion control problem as an next token prediction problem. A causal transformer is trained autoregressively on various sources of data. The performance on a full-sized humanoid robot's locomotion indicates that this formulation can be a promising path for complex robotic control problems to incorporate diverse sources of data.
Strengths
1. This work presents a promising way to scale robotic learning in terms of data via generative modeling. 2. Real-world experiments, and some metrics indicate the comparable performance of the proposed method to RL-based / model-based controller.
Weaknesses
1. In Sec. 3.6 Model inference, the first step might not follow the statement. > At inference time, our transformer model will always have access to observation-action pairs. 2. It lacks description on the inference frequency, considering the large scale pretraining with real-world locomotion task. 3. It might be beneficial to ablate on how different sources of the data affect the performance.
Questions
1. In Table 1(c), why no Pred. Err. for staged training? 2. In Sec. 4.3, can the authors detail the inverse kinematics problem they solve? > In order to use these trajectories for training a robot, we solve an inverse kinematics problem to 183 find the corresponding robot poses. 3. In Sec. 4.4, can the authors detail on the filter strategy? > Once we retarget the motion from the Internet videos to humanoid trajectories, we filter the trajectories with the low optimization cost. 4. How many Internet data are used? This seems not stated in the paper.
Rating
8
Confidence
4
Soundness
4
Presentation
4
Contribution
3
Limitations
As the authors noted, data from human video come with the cost of being noisy as a result of our progress on the computer vision techniques. It would be helpful to elaborate on current results of using Internet videos.
Thank you for the rebuttal.
Thank authors for the detailed response. My questions are mostly addressed. My concern left is that will the largest model work at 50Hz, i.e. whether the increased model size with generative pretraining, hinders the inference. It would be great to discuss the minimum required frequency for functioning, and its relation to model size. I have raised the score according to the detailed open-source commitment of authors.
Thank you
Thank you for your response and for increasing the score. We think that the challenge of balancing model size and inference speed in real-time settings like ours is very interesting. Please find some comments below. The minimal frequency for functioning is determined by the highly dynamic nature of the problem (unstable bipedal robot with a big upper body with a lot of mass and inertia). We might be able to reduce it a bit (e.g., 40Hz) but probably not much. Our current models are relatively small (up to 8M parameters) and can run within 50Hz on the on-board CPU computer. We think that it is likely that the improvements in inference software (e.g., low precision inference), on-board compute (e.g., access to a GPU), training recipes (e.g., distillation), and modeling (e.g., sparse architectures) will offset these challenges and enable us to keep using larger models within the inference speed constraints. We find the modeling angle to be a particularly interesting direction for future work. For example, one could imagine architectures with more parameters than flops, where different parameters are activated at different frequencies and different parts of the model are executed asynchronously. We will include the additional discussion and an analysis of the model size and inference speed in the revised manuscript. Thank you for the suggestions.
Thank you very much for your response. I appreciate the effort in addressing my questions, and most of them have been satisfactorily resolved. However, I have some concerns regarding the reproducibility of the paper. It seems challenging to obtain the four datasets mentioned in the paper based solely on the descriptions provided. While I believe the paper offers valuable insights, the limited details, absence of detailed information in the appendix, and lack of accompanying code might affect its overall impact on the community. Considering these factors, I believe it may be best to maintain my previous score. Thank you once again for your thoughtful response.
Response by Authors
Thank you for your response. Please find our responses below: > I appreciate the effort in addressing my questions, and most of them have been satisfactorily resolved We are glad to hear you found our responses satisfactory. To ensure we have fully addressed all of your concerns, please let us know if there are any specific questions or points that you feel remain unresolved or require further clarification. We would be happy to address them. > However, I have some concerns regarding the reproducibility of the paper. It seems challenging to obtain the four datasets mentioned in the paper based solely on the descriptions provided. While I believe the paper offers valuable insights, the limited details, absence of detailed information in the appendix, and lack of accompanying code might affect its overall impact on the community. We commit to release all of the materials to fully reproduce the paper, including the four datasets, accompanying code, and detailed documentation. We will further include detailed descriptions of the datasets in the appendix, covering data collection methods, pre-processing steps, and any other relevant details. Please let us know if there is any additional information that you found lacking, and we would be happy to include it in the revised manuscript.
Thank you for your response. My remaining concerns mainly focus on the actual capabilities of this robot: The project page doesn't include videos of the robot turning; can it actually turn in the real world or walk for an extended period of time? However, the project page seems to be inaccessible now. It would be interesting to see its obstacle avoidance capabilities under control. Additionally, if the robot needs to go uphill or downhill slightly, would it fail? And it will be fine to analyze its robustness. If this method were extended to enable humanoid robots to go up and down stairs, what challenges would arise? Some of the suggestions may go beyond the scope of this paper, but considering the authors’ commitment to open-source, I have raised my score.
Thank you for the reply
Thank authors for the reply. My questions are mostly addressed. I have raised the score according to the open-source commitment of authors.
Thank you
Thank you for your response and for increasing the score. Please let us know if you have any further questions. We would be happy to address them.
Response by Authors
Thank you for providing the review. We are committed to maintaining the highest standards of ethics. Please find our responses below: - The videos that we use are a subset of two standard datasets in machine learning, Kinetics [1] and PoseTrack [2]. These datasets are in major use in the literature and have collectively been cited over 16,000 times. We have followed the standard practice in the research community and have not scraped any videos from YouTube. - We will not distribute or make any of these videos available (we did not source, collect, or scrape these videos). - We have extracted coarse human poses from these videos for deriving data for training our models. Specifically, we extract the geometric locations of 4 points (locations of the feet and the hands) and use them to derive robot poses for training. We would only provide the robot poses from which one cannot recover the original videos and which preserves the privacy of the individuals in the original videos. - Our models are trained on robot poses that are derived from abstracted geometric points derived from videos. Therefore our models never see real videos and one would not be able to reverse-engineer the original videos from our models. - We will include clarifications on the nature of our data, pre-processing, and model inputs, and include an extended discussion on ethics in the revised manuscript. Please let us know if this addresses your concerns and/or if you have any additional questions. We would be happy to answer them. --- [1] Carreira and Zisserman, Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset, CVPR 2017 [2] Andriluka et al, PoseTrack: A Benchmark for Human Pose Estimation and Tracking, CVPR 2018
Kind reminder for feedback - 1 day left for discussion
Thank you again for your time and effort spent on providing the ethics review. We clarified that we use videos from standard academic datasets, we are not distributing videos, our training data is derived from abstract geometric points that preserve privacy, and one cannot infer original videos from our models. Please let us know if there are any specific questions or points that require further clarification. We would be happy to address them.
Thank you very much for your response. After reading the author's response, my concerns have been addressed, and I believe that there are no ethical issues with this paper.
Kind reminder for feedback - 2 days left for discussion
Thank you again for your time and effort spent on providing a careful review of our paper. We performed new experiments on how different data sources affect performance. We find that (1) each of the data sources improves the performance individually and (2) that different data sources are complementary and lead to greater gains in aggregate. These detailed ablations demonstrate that our approach can benefit from different data sources. We also provided additional details on: - model inference - inference frequency - prediction errors - inverse kinematics - filtering strategy - number of videos To ensure we have fully addressed all of your concerns, please let us know if there are any specific questions or points that you feel remain unresolved or require further clarification. We would be happy to address them. Many thanks
Author response summary
We appreciate your positive review and helpful suggestions. We wanted to provide a summary of the additional information provided in our response for your reference. Additional experiments on: - MPC comparisons - ours > RL > MPC - Quantization ablation - continuous > discrete - Mask token ablation - learnable > constant - Pre-training ablation - joint > staged Additional details on: - Modality prediction - predict next token of the same modality - Modalities used - observations and actions - Prediction error - provided for reference - Simultaneous prediction - used in our models Thank you again for your valuable feedback. We would be happy to provide any further information if needed.
Thank you for addressing my comments, I am happy with the author’s rebuttal. However, the lack of comprehensiveness in discussing related works is slightly concerning. I was not knowledgeable about the work described by reviewer FS4P, which makes the contribution of this work less meritous. I believe the addition of an “Autoregressive pre-training for robotics” header in the related works section would be appropriate. Thus, I am lowering my score slightly. It’s worth noting that I still believe the work has a lot of merit, such as through masking actions and through applying autoregressive pre-training to humanoid locomotion. The results were also very impressive. I also agree with other reviewers that the eventual open source release is essential for this work. Thanks for addressing my concerns!
Decision
Accept (spotlight)