Weiyu Liu
I am a Postdoctoral Scholar working with Jiajun Wu in the CogAI group and Stanford Vision and Learning Lab (SVL) . I completed my Ph.D. in robotics at Georgia Institute of Technology, where I am advised by Sonia Chernova .
In 2021 and 2022, I interned at Nvidia Research, working with Dieter Fox , Tucker Hermans , Animesh Garg , and Chris Paxton .
Prior to joining the Ph.D. program, I received my Bachelor’s degree in Electrical Engineering from Georgia Tech.
The goal of my research is to develop robots that can effectively perceive, model, and interact with the real world.
I am particularly interested in the application where human users can easily command robots to complete long-horizon tasks via simple language commands, such as "make the tea".
Achieving this requires robust generalization across various dimensions, including objects, environments, actions, and tasks.
My core insight is that strong generalization can be achieved through a structured representation of world knowledge.
Rather than constructing this representation from scratch, robots should leverage the rich knowledge embedded in human language, grounding it through interactions with their environments to connect abstract knowledge to their sensorimotor capabilities.
My prior research has investigated core components of this structured knowledge representation, methods to extract and refine this knowledge extracted from humans, and algorithms that enable reasoning with this knowledge to support broad generalization.
Email  / 
CV (Nov 2024)  / 
Research Statement (2022)
Google Scholar  / 
Github  / 
Twitter
Your browser does not support the video tag.
Learning Compositional Behaviors from Demonstration and Language
Weiyu Liu* ,
Neil Nie* ,
Ruohan Zhang ,
Jiayuan Mao† ,
Jiajun Wu†
Conference on Robot Learning (CoRL) , 2024
paper /
website /
bibtex
Learning Planning Abstractions from Language
Weiyu Liu* ,
Geng Chen* ,
Joy Hsu ,
Jiayuan Mao† ,
Jiajun Wu†
International Conference on Learning Representations (ICLR) , 2024
arxiv /
website /
bibtex
Your browser does not support the video tag.
IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos
Yunong Liu ,
Cristobal Eyzaguirre ,
Juan Carlos Niebles ,
Vineeth Ravi ,
Saumitra Mishra ,
Weiyu Liu† ,
Jiajun Wu†
Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track , 2024
paper /
website /
code and data /
bibtex
Naturally Supervised 3D Visual Grounding with Language-Regularized Concept Learners
Chun Feng* ,
Joy Hsu* ,
Weiyu Liu ,
Jiajun Wu
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024
paper /
website /
code
bibtex
Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making
Manling Li* ,
Shiyu Zhao* ,
Qineng Wang* ,
Kangrui Wang* ,
Yu Zhou* ,
Sanjana Srivastava ,
Cem Gokmen ,
Tony Lee ,
Li Erran Li ,
Ruohan Zhang ,
Weiyu Liu ,
Percy Liang ,
Li Fei-Fei ,
Jiayuan Mao ,
Jiajun Wu
Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track , 2024
paper /
website /
code /
bibtex
MARPLE: A Benchmark for Long-Horizon Inference
Emily Jin ,
Zhuoyi Huang ,
Jan-Philipp Fränken ,
Weiyu Liu ,
Hannah Cha ,
Sarah A. Wu ,
Erik Brockbank ,
Ruohan Zhang ,
Jiajun Wu ,
Tobias Gerstenberg
Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track , 2024
arxiv /
website /
code and data /
bibtex
A Survey of Semantic Reasoning Frameworks for Robotic Systems
Weiyu Liu1 ,
Angel Daruna1 ,
Maithili Patel2 ,
Kartik Ramachandruni2 ,
Sonia Chernova
Robotics and Autonomous Systems , 2023
paper /
bibtex
Your browser does not support the video tag.
Composable Part-Based Manipulation
Weiyu Liu ,
Jiayuan Mao ,
Joy Hsu ,
Tucker Hermans ,
Animesh Garg ,
Jiajun Wu
Conference on Robot Learning (CoRL) , 2023
arxiv /
website /
bibtex
Your browser does not support the video tag.
StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects
Weiyu Liu ,
Yilun Du ,
Tucker Hermans ,
Sonia Chernova ,
Chris Paxton
Robotics: Science and Systems (RSS) , 2023
CoRL Workshop on Language and Robot Learning , 2022
arxiv /
code & data /
website /
bibtex
A Survey of Semantic Reasoning Frameworks for Robotic Systems
Weiyu Liu1 ,
Angel Daruna1 ,
Maithili Patel2 ,
Kartik Ramachandruni2 ,
Sonia Chernova
Robotics and Autonomous Systems , 2023
paper /
bibtex
Your browser does not support the video tag.
Latent Space Planning for Multi-Object Manipulation with Environment-Aware Relational Classifiers
Yixuan Huang ,
Nichols Crawford Taylor ,
Adam Conkey ,
Weiyu Liu ,
Tucker Hermans ,
Under Review , 2023
arxiv /
website /
bibtex
GraspGPT: Leveraging Semantic Knowledge from a Large Language Model for Task-Oriented Grasping
Chao Tang ,
Dehao Huang
Wenqi Ge,
Weiyu Liu ,
Hong Zhang
Under Review , 2023
arxiv /
code & data /
website /
bibtex
Your browser does not support the video tag.
StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects
Weiyu Liu ,
Chris Paxton ,
Tucker Hermans ,
Dieter Fox
International Conference on Robotics and Automation (ICRA) , 2022
arxiv /
code & data /
website /
talk /
bibtex
Your browser does not support the video tag.
Learning Instance-Level N-Ary Semantic Knowledge At Scale For Robots Operating in Everyday Environments
Weiyu Liu ,
Dhruva Bansal ,
Angel Daruna ,
Sonia Chernova
Robotics: Science and Systems (RSS) , 2021
paper /
code & data /
talk /
bibtex
Your browser does not support the video tag.
Towards Robust One-shot Task Execution using Knowledge Graph Embeddings
Angel Daruna ,
Lakshmi Nair ,
Weiyu Liu ,
Sonia Chernova
International Conference on Robotics and Automation (ICRA) , 2021
arxiv /
bibtex
Your browser does not support the video tag.
An Affordance Keypoint Detection Network for Robot Manipulation
Ruinian Xu ,
Fu-Jen Chu ,
Chao Tang ,
Weiyu Liu ,
Patricio Vela
IEEE Robotics and Automation Letters (RA-L) , 2021
paper /
code & data /
bibtex
Your browser does not support the video tag.
Same Object, Different Grasps: Data and Semantic Knowledge for Task-Oriented Grasping
Adithya Murali ,
Weiyu Liu ,
Kenneth Marino ,
Sonia Chernova ,
Abhinav Gupta
Conference on Robot Learning (CoRL) , 2020
arxiv /
code & data /
video /
talk /
project page /
bibtex
Your browser does not support the video tag.
CAGE: Context-Aware Grasping Engine
Weiyu Liu ,
Angel Daruna ,
Sonia Chernova
International Conference on Robotics and Automation (ICRA) , 2020
arxiv /
code & data /
video /
press /
bibtex
Your browser does not support the video tag.
Path Ranking with Attention to Type Hierarchies
Weiyu Liu ,
Angel Daruna ,
Zsolt Kira ,
Sonia Chernova
Conference on Artificial Intelligence (AAAI) , 2020   (Oral Presentation)
arxiv /
code /
bibtex
Taking Recoveries to Task: Recovery-Driven Development for Recipe-based Robot Tasks
Siddhartha Banerjee* ,
Angel Daruna* ,
David Kent* ,
Weiyu Liu* ,
Jonathan Balloch ,
Abhinav Jain ,
Akshay Krishnan ,
Muhammad Asif Rana ,
Harish Ravichandar ,
Binit Shah ,
Nithin Shrivatsav ,
Sonia Chernova
International Symposium on Robotics Research (ISRR) , 2019
paper /
code /
bibtex
Your browser does not support the video tag.
RoboCSE: Robot Common Sense Embedding
Angel Daruna ,
Weiyu Liu ,
Zsolt Kira ,
Sonia Chernova
International Conference on Robotics and Automation (ICRA) , 2019
arxiv /
code & data /
video /
bibtex
Your browser does not support the video tag.
Autonomous flying blimp interaction with human in an indoor space
Ningshi Yao ,
Qiuyang Tao ,
Weiyu Liu ,
Zhen Liu ,
Ye Tian,
Peiyu Wang,
Timothy Li,
Fumin Zhang ,
Frontiers of Information Technology & Electronic Engineering , 2019
paper /
bibtex
SiRoK: Situated Robot Knowledge - Understanding the Balance Between Situated Knowledge and Variability
Sonia Chernova ,
Vivian Chu ,
Angel Daruna ,
Haley Garrison,
Meera Hahn ,
Priyanka Khante,
Weiyu Liu ,
Andrea Thomaz ,
AAAI Spring Symposium Series (AAAI-SSS) , 2018
paper /
bibtex
Your browser does not support the video tag.
Situated Bayesian Reasoning Framework for Robots Operating in Diverse Everyday Environments
Sonia Chernova ,
Vivian Chu ,
Angel Daruna ,
Haley Garrison,
Meera Hahn ,
Priyanka Khante,
Weiyu Liu ,
Andrea Thomaz ,
International Symposium on Robotics Research (ISRR) , 2017
paper /
bibtex