[October 02 2024] Cutting edge papers in 2024 are avaliable!!!

• Survey Paper
• Language-conditioned Reinforcement Learning
• Language-conditioned Imitation Learning
  • Behaviour Cloning
  • Inverse Reinforcement Learning
• Diffusion Policy
• Neuralsymbolic
• Enpowered by LLMs
  • Planning
  • Reasoning
• Enpowered by VLMs
  • Vision-Language-Action Models (VLAMs)
• Comparative Analysis
  • Simulators
  • Benchmarks
  • Models

This paper is basically based on the survey paper

Language-conditioned Learning for Robot Manipulation: A Survey
Hongkuan Zhou, Xiangtong Yao, Oier Mees, Yuan Meng, Dhruv Shah, Ted Xiao, Yonatan Bisk, Edward Johns, Mohit Shridhar, Kai Huang, Zhenshan Bing, Alois Knoll

@article{zhou2023language,
  title={Language-conditioned Learning for Robotic Manipulation: A Survey},
  author={Zhou, Hongkuan and Yao, Xiangtong and Meng, Yuan and Sun, Siming and BIng, Zhenshan and Huang, Kai and Knoll, Alois},
  journal={arXiv preprint arXiv:2312.10807},
  year={2023}
}

• From language to goals: Inverse reinforcement learning for vision-based instruction following [paper]
• Grounding english commands to reward function [paper]
• Learning to understand goal specifications by modelling reward [paper]
• Beating atari with natural language guided reinforcement learning [paper] [code]
• Using natural language for reward shaping in reinforcement learning [paper]

• Gated-attention architectures for task-oriented language grounding [paper] [code]
• Mapping instructions and visual observations to actions with reinforcement learning [paper]
• Modular multitask reinforcement learning with policy sketches [paper]
• Representation learning for grounded spatial reasoning [paper]

• Lancon-learn: Learning with language to enable generalization in multi-task manipulation [paper] [code]
• Pixl2r: Guiding reinforcement learning using natural language by mapping pixels to rewards [paper][code]
• Learning from symmetry: Meta-reinforcement learning with symmetrical behaviors and language instructions [paper][website]
• Meta-reinforcement learning via language instructions [paper][code][website]
• Learning language-conditioned robot behavior from offline data and crowd-sourced annotation [paper]
• Concept2robot: Learning manipulation concepts from instructions and human demonstrations [paper]

• Language conditioned imitation learning over unstructured data [paper] [code] [website]
• Bc-z: Zero-shot task generalization with robotic imitation learning [paper]
• What matters in language-conditioned robotic imitation learning over unstructured data [paper] [code][website]
• Grounding language with visual affordances over unstructured data [paper] [code][website]
• Language-conditioned imitation learning with base skill priors under unstructured data [paper] [code] [website]
• Pay attention!- robustifying a deep visuomotor policy through task-focused visual attention [paper]
• Language-conditioned imitation learning for robot manipulation tasks [paper]
• Multimodal Diffusion Transformer for Learning from Play [paper]
• Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models [paper] [code] [website]
• PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Play [paper] [website]
• ChainedDiffuser: Unifying Trajectory Diffusion and Keypose Prediction for Robotic Manipulation [paper] [code] [website]
• GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fields [paper] [code] [website]
• DNAct: Diffusion Guided Multi-Task 3D Policy Learning [paper] [website]
• 3D Diffuser Actor: Policy Diffusion with 3D Scene Representations [paper] [code] [website]
• Vision-Language Foundation Models as Effective Robot Imitators [paper]
• OpenVLA:An Open-Source Vision-Language-Action Model [paper] [code] [website]
• Open X-Embodiment: Robotic Learning Datasets and RT-X Models [paper]
• 3D-VLA: A 3D Vision-Language-Action Generative World Model [paper] [code] [website]
• Octo: An Open-Source Generalist Robot Policy [paper] [code] [website]

• Grounding english commands to reward function [paper]
• From language to goals: Inverse reinforcement learning for vision-based instruction following [paper]

• Multimodal Diffusion Transformer for Learning from Play [paper]
• Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models [paper] [code] [website]
• PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Play [paper] [website]
• ChainedDiffuser: Unifying Trajectory Diffusion and Keypose Prediction for Robotic Manipulation [paper] [code] [website]
• DNAct: Diffusion Guided Multi-Task 3D Policy Learning [paper] [website]
• 3D Diffuser Actor: Policy Diffusion with 3D Scene Representations [paper]

• Hierarchical understanding in robotic manipulation: A knowledge-based framework [paper]
• Semantic Grasping Via a Knowledge Graph of Robotic Manipulation: A Graph Representation Learning Approach [paper]
• Knowledge Acquisition and Completion for Long-Term Human-Robot Interactions using Knowledge Graph Embedding [paper]

• Tell me dave: Context-sensitive grounding of natural language to manipulation instructions [paper]
• Neuro-symbolic procedural planning with commonsense prompting [paper]
• Reinforcement Learning Based Navigation with Semantic Knowledge of Indoor Environments [paper]
• Learning Neuro-Symbolic Skills for Bilevel Planning [[paper]](Learning Neuro-Symbolic Skills for Bilevel Planning)

• Learning Neuro-symbolic Programs for Language Guided Robot Manipulation [paper] [code] [website]
• Long-term robot manipulation task planning with scene graph and semantic knowledge [paper]

• Sayplan: Grounding large language models using 3d scene graphs for scalable task planning [paper]
• Language models as zero-shot planners: Extracting actionable knowledge for embodied agents [paper]
• Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents [paper]
• Progprompt: Generating situated robot task plans using large language models [paper]
• Robots that ask for help: Uncertainty alignment for large language model planners [paper]
• Task and motion planning with large language models for object rearrangement [paper]
• Do as i can, not as i say: Grounding language in robotic affordances [paper]
• The 2014 international planning competition: Progress and trends [paper]
• Robot task planning via deep reinforcement learning: a tabletop object sorting application [paper]
• Robot task planning and situation handling in open worlds [paper] [code] [website]
• Embodied Task Planning with Large Language Models [paper] [code] [website]
• Text2motion: From natural language instructions to feasible plans [paper] [website]
• Large language models as commonsense knowledge for large-scale task planning [paper] [code] [website]
• Alphablock: Embodied finetuning for vision-language reasoning in robot manipulation [paper]
• Learning to reason over scene graphs: a case study of finetuning gpt-2 into a robot language model for grounded task planning [paper] [code]
• Scaling up and distilling down: Language-guided robot skill acquisition [paper][code] [website]
• Stap: Sequencing task-agnostic policies [paper] [code][website]
• Inner monologue: Embodied reasoning through planning with language models [paper] [website]

• Rearrangement:A challenge for embodied ai [paper]
• The threedworld transport challenge: A visually guided task and motion planning benchmark for physically realistic embodied ai [paper]
• Tidy up my room: Multi-agent cooperation for service tasks in smart environments [paper]
• A quantifiable stratification strategy for tidy-up in service robotics [paper]
• Tidybot: Personalized robot assistance with large language models [paper]
• Housekeep: Tidying virtual households using commonsense reasoning [paper]
• Building cooperative embodied agents modularly with large language models [paper]
• Socratic models: Composing zero-shot multimodal reasoning with language [paper]
• Voyager: An open-ended embodied agent with large language models [paper]
• Translating natural language to planning goals with large-language models [paper]

• Cliport: What and where pathways for robotic manipulation [paper] [code] [website]
• Transporter networks: Rearranging the visual world for robotic manipulation [paper] [code] [website]
• Simple but effective: Clip embeddings for embodied ai [paper]
• Instruct2act: Mapping multi-modality instructions to robotic actions with large language model [paper] [code]
• Latte: Language trajectory transformer [paper] [code]
• Embodied Task Planning with Large Language Models [paper] [code] [website]
• Palm-e: An embodied multimodal language model [paper] [website]
• Socratic models: Composing zero-shot multimodal reasoning with language [paper]
• Pretrained language models as visual planners for human assistance [paper] [code]
• Open-world object manipulation using pre-trained vision-language models [paper] [website]
• Robotic skill acquisition via instruction augmentation with vision-language models [paper] [website]
• Language reward modulation for pretraining reinforcement learning [paper] [code]
• Vision-language models as success detectors [paper]

• A Generalist Agent [paper]
• RT-1: Robotics Transformer for Real-World Control at Scale [paper] [code] [website]
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control [paper]
• Vision-Language Foundation Models as Effective Robot Imitators [paper]
• OpenVLA:An Open-Source Vision-Language-Action Model [paper] [code] [website]
• Open X-Embodiment: Robotic Learning Datasets and RT-X Models [paper]
• 3D-VLA: A 3D Vision-Language-Action Generative World Model [paper] [code] [website]