Ted Xiao
I'm a research scientist at Google DeepMind, where I work on making robots smarter. My research focuses on robot learning, internet-scale foundation models, and reinforcement learning. I am particularly interested in methods that can scale and generalize in the real world.
Ted Xiao is a Senior Research Scientist at Google DeepMind working on robot learning. His research agenda focuses on scaling robot learning in the real world, with a particular focus on approaches that can leverage internet-scale foundation models and methods that improve with more experience. Prior to joining Google DeepMind, Ted founded Machine Learning at Berkeley and worked at Adobe Research. Ted received his B.S. and M.S. in Electrical Engineering and Computer Science from UC Berkeley, where he was advised by Professor Claire Tomlin.
Formal Bio G. Scholar Twitter Blog LinkedIn
xiaoted at gmail dot com
Stop Regressing: Training Value Functions via Classification for Scalable Deep RL
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taïga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, Rishabh Agarwal
Preprint
PDF
PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
Soroush Nasiriany*, Fei Xia*, Wenhao Yu*, Ted Xiao*, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, Quan Vuong, Tingnan Zhang, Tsang-Wei, Edward Lee, Kuang-Huei Lee, Peng Xu, Sean Kirmani, Yuke Zhu, Andy Zeng, Karol Hausman, Nicolas Heess, Chelsea Finn, Sergey Levine, Brian Ichter*
Preprint
Website •
PDF •
Demo
Learning to Learn Faster from Human Feedback with Language Model Predictive Control
Jacky Liang, Fei Xia, Wenhao Yu, Andy Zeng, Montserrat Gonzalez Arenas, Maria Attarian, Maria Bauza, Matthew Bennice, Alex Bewley, Adil Dostmohamed, Chuyuan Kelly Fu, Nimrod Gileadi, Marissa Giustina, Keerthana Gopalakrishnan, Leonard Hasenclever, Jan Humplik, Jasmine Hsu, Nikhil Joshi, Ben Jyenis, Chase Kew, Sean Kirmani, Tsang-Wei Edward Lee, Kuang-Huei Lee, Assaf Hurwitz Michaely, Joss Moore, Ken Oslund, Dushyant Rao, Allen Ren, Baruch Tabanpour, Quan Vuong, Ayzaan Wahid, Ted Xiao, Ying Xu, Vincent Zhuang, Peng Xu, Erik Frey, Ken Caluwaerts, Tingnan Zhang, Brian Ichter, Jonathan Tompson, Leila Takayama, Vincent Vanhoucke, Izhak Shafran, Maja Mataric, Dorsa Sadigh, Nicolas Heess, Kanishka Rao, Nik Stewart, Jie Tan, Carolina Parada
Preprint
Website •
PDF •
Code •
Demo
AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents
Michael Ahn, Debidatta Dwibedi, Chelsea Finn, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Karol Hausman, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Sean Kirmani, Isabel Leal, Edward Lee, Sergey Levine, Yao Lu, Sharath Maddineni, Kanishka Rao, Dorsa Sadigh, Pannag Sanketi, Pierre Sermanet, Quan Vuong, Stefan Welker, Fei Xia, Ted Xiao, Peng Xu, Steve Xu, Zhuo Xu
Preprint
Website •
PDF •
Blogpost
RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches
Priya Sundaresan, Quan Vuong, Jiayuan Gu, Peng Xu, Ted Xiao, Sean Kirmani, Tianhe Yu, Michael Stark, Ajinkya Jain, Karol Hausman, Dorsa Sadigh*, Jeannette Bohg*, Stefan Schaal*
Preprint
Website •
PDF
How to Prompt Your Robot: A PromptBook for Manipulation Skills with Code as Policies
Montserrat Gonzalez Arenas, Ted Xiao, Sumeet Singh, Vidhi Jain, Allen Z Ren, Quan Vuong, Jake Varley, Alexander Herzog, Isabel Leal, Sean Kirmani, Dorsa Sadigh, Vikas Sindhwani, Kanishka Rao, Jacky Liang, Andy Zeng
IEEE International Conference on Robotics and Automation (ICRA) 2024,
★ Oral Presentation ★ Robot Learning Workshop (WRL) at the Conference on Neural Information Processing Systems (NeurIPS) 2023
PDF
Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation
Annie Xie, Lisa Lee, Ted Xiao, Chelsea Finn
IEEE International Conference on Robotics and Automation (ICRA) 2024
Robot Learning Workshop (WRL) at the Conference on Neural Information Processing Systems (NeurIPS) 2023
Website •
PDF
RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches
Jiayuan Gu, Sean Kirmani, Paul Wohlhart, Yao Lu, Montserrat Gonzalez Arenas, Kanishka Rao, Wenhao Yu, Chuyuan Fu, Keerthana Gopalakrishnan, Zhuo Xu, Priya Sundaresan, Peng Xu, Hao Su, Karol Hausman, Chelsea Finn, Quan Vuong, Ted Xiao
★ Spotlight ★ International Conference on Learning Representations (ICLR) 2024
Website •
PDF •
Blogpost
Open-World Object Manipulation using Pre-Trained Vision-Language Models
Austin Stone*, Ted Xiao*, Yao Lu*, Keerthana Gopalakrishnan, Kuang-Huei Lee, Quan Vuong, Paul Wohlhart, Sean Kirmani, Brianna Zitkovich, Fei Xia, Chelsea Finn, Karol Hausman
Conference on Robot Learning (CoRL) 2023
Webpage •
PDF •
Video
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alex Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, Brianna Zitkovich
Conference on Robot Learning (CoRL) 2023
Webpage •
PDF •
Video •
Blogpost
Language to Rewards for Robotic Skill Synthesis
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao-Tien, Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian Ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, Fei Xia
★ Oral Presentation ★, Conference on Robot Learning (CoRL) 2023
Webpage •
PDF •
Code •
Blogpost
Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models
Ted Xiao*, Harris Chan*, Pierre Sermanet, Ayzaan Wahid, Anthony Brohan, Karol Hausman, Sergey Levine, Jonathan Tompson
Robotics: Science and Systems (RSS) 2023
Webpage •
PDF
Scaling Robot Learning with Semantically Imagined Experience
Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Dee M, Jodilyn Peralta, Brian Ichter, Karol Hausman, Fei Xia
Robotics: Science and Systems (RSS) 2023
Webpage •
PDF •
Video
Deep RL at Scale: Sorting Waste in Office Buildings with a Fleet of Mobile Manipulators
Alexander Herzog*, Kanishka Rao* Karol Hausman*, Yao Lu*, Paul Wohlhart*, Mengyuan Yan, Jessica Lin, Montserrat Gonzalez Arenas, Ted Xiao, Daniel Kappler, Daniel Ho, Jarek Rettinghouse, Yevgen Chebotar, Kuang-Huei Lee, Keerthana Gopalakrishnan, Ryan Julian, Adrian Li, Chuyuan Kelly Fu, Bob Wei, Sangeetha Ramesh, Khem Holden, Kim Kleiven, David Rendleman, Sean Kirmani, Jeff Bingham, Jon Weisz, Ying Xu, Wenlong Lu, Matthew Bennice, Cody Fong, David Do, Jessica Lam, Yunfei Bai, Benjie Holson, Michael Quinlan, Noah Brown, Mrinal Kalakrishnan, Julian Ibarz, Peter Pastor, Sergey Levine
Robotics: Science and Systems (RSS) 2023
Webpage •
PDF •
Video •
Blogpost
RT-1: Robotics Transformer for Real-World Control at Scale
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, Brianna Zitkovich
Robotics: Science and Systems (RSS) 2023
Webpage •
PDF •
Video •
Code •
Blogpost
Token Turing Machines
Michael S. Ryoo, Keerthana Gopalakrishnan, Kumara Kahatapitiya, Ted Xiao, Kanishka Rao, Austin Stone, Yao Lu, Julian Ibarz, Anurag Arnab
Conference on Computer Vision and Pattern Recognition (CVPR) 2023
PDF •
Code
Jump-Start Reinforcement Learning
Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, Sergey Levine, Karol Hausman
International Conference on Machine Learning (ICML) 2023
Webpage •
PDF •
Blogpost
Inner Monologue: Embodied Reasoning through Planning with Language Models
Wenlong Huang*, Fei Xia*, Ted Xiao*, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, Brian Ichter
Conference on Robot Learning (CoRL) 2022
Webpage •
PDF •
Video •
2 Minute Papers
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, Andy Zeng
★ Oral Presentation, Special Innovation Award ★, Conference on Robot Learning (CoRL) 2022
Webpage •
PDF •
Video •
Code •
Google AI Blog •
Wired •
Washington Post •
CNET
PI-QT-Opt: Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scale
Kuang-Huei Lee, Ted Xiao, Adrian Li, Paul Wohlhart, Ian Fischer, Yao Lu
Conference on Robot Learning (CoRL) 2022
Website •
PDF •
Video
Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning
Dhruv Shah, Peng Xu, Yao Lu, Ted Xiao, Alexander Toshev, Sergey Levine, Brian Ichter
International Conference on Learning Representations (ICLR) 2022
PDF •
Blogpost
AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale
Yao Lu, Karol Hausman, Yevgen Chebotar, Mengyuan Yan, Eric Jang, Alexander Herzog, Ted Xiao, Alex Irpan, Mohi Khansari, Dmitry Kalashnikov, Sergey Levine
Conference on Robot Learning (CoRL) 2021
Webpage •
PDF
Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, Sergey Levine
International Conference on Machine Learning (ICML) 2021
Webpage •
PDF •
Video •
Blogpost
Thinking While Moving: Deep Reinforcement Learning with Concurrent Control
Ted Xiao, Eric Jang, Dmitry Kalashnikov, Sergey Levine, Julian Ibarz, Karol Hausman*, Alexander Herzog*
International Conference on Learning Representations (ICLR) 2020
Webpage •
PDF •
Video •
Yannic Kilcher
Learning Latent Plans from Play
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, Pierre Sermanet
★ Oral Presentation★, Conference on Robot Learning (CoRL) 2019
Webpage •
PDF
Adversarial Machine Learning
Phillip Kuznetsov, Riley Edmunds, Ted Xiao, Humza Iqbal, Raul Puri, Noah Golmant, Shannon Shih
Contributed Chapter, Artificial Intelligence Safety and Security (CRC Press)
Book
Goal-Driven Dynamics Learning via Bayesian Optimization
Somil Bansal, Roberto Calandra, Ted Xiao, Sergey Levine, Claire J. Tomlin
56th IEEE Conference on Decision and Control (CDC) 2017
PDF
A Literature Review of Learning and Optimization Methods Applied to Quadrotor Control
Ted Xiao
Technical Report
PDF
Generative Adversarial Networks for Model-based Reinforcement Learning with Tree Search
Ted Xiao*, Gautham Kesineni*
Technical Report
PDF
Frame Rate Upscaling with Deep Neural Networks
Ted Xiao, Raul Puri, Gautham Kesineni
Technical Report
PDF
2024
Cognitive Revolution, Podcast Guest2023
USC, CS699 Guest Lecture2021
UC Berkeley, Machine Learning at Berkeley Tutorial2018
Xoogler, Machine Learning Keynote2017
RobotX, NLP Tutorial2023
Co-Organizer, CoRL Workshop on Language and Robot Learning2022
Co-Organizer, NeurIPS Deep Reinforcement Learning Workshop2018+
Reviewer, CoRL, RSS, RA-L, ICRA, NeurIPS, ICML, ICLR