I saw your foundation model is trained on data from several different robots. Is the plan to eventually train a foundation model that can control any robot zero shot? That is, the effect of actuations on video/sensor input is collected and understood in-context and actuations are corrected to yield intended behavior. All in-context. Is this feasible?
More specifically, has your model already exhibited this type of capability, in principle?
Nearly 2 years ago I bet a roboticist $10 that we’d have “sci-fi” robots in 2 years.
Now, we didn’t set good criteria for the bet (it was late at night). However, my personal criteria for “scifi” are twofold:
1. Robots that are able to make peanut butter sandwiches without explicit training
2. Robots able to walk on sand (eg Tatooine)
Based on your current understanding, who won the bet? Also, what kind of physical benchmarks do you associate with “sci-fi robots”?
Hi! Very cool results. Are you able to share some numbers about the slope of the scaling curve you found, i.e. how performance responds to a growing nr of demonstrations?
Academically I'd also be very interested how much of a data efficiency improvement you achieved with the pretrained model + task specific post-training versus from-scratch task specific training - like, if post training requires say 50 additional demos, and from-scratch on smaller model requires say 250 demos (or whatever) to match performance, that would be an interesting quntification of the efficiency benefit of using the big foundation model
How does the post-training step work? In the case of t-shirt folding, does a supervisor perform the folding first, many times? Or is the learning interactive, where a supervisor corrects the robot if it does something wrong?
Happy to answer any questions on the model, hardware, etc