Pragmatic Embodied Spoken Instruction Following in Human-Robot Collaboration with Theory of Mind

Sat, 31 Jan 2026 00:00:00 +0000

How can a robot infer the intended action when a spoken instruction is corrupted by noise? SIFToM combines a vision-language model’s symbolic descriptions of scenes, human actions, and speech with probabilistic theory-of-mind inference over collaborative goals and plans. This grounds ambiguous language in the task context.

Experiments use UnclearInstruct in VirtualHome and a Stretch robot in meal-preparation tasks, comparing instruction interpretation and collaboration against VLM and ablation baselines. The reported results concern these household settings; visual-to-symbolic grounding remains a source of failure. The paper connects embodied instruction following, pragmatic language understanding, and human-robot collaboration. Its earlier title, , refers to the September 2024 preprint; this page uses the revised v2 abstract and author list.

The records ICRA 2026; this page provides the arXiv v2 abstract and PDF.

Spoken Instruction Following | Yizirui Fang

Pragmatic Embodied Spoken Instruction Following in Human-Robot Collaboration with Theory of Mind