Pragmatic Embodied Spoken Instruction Following in Human-Robot Collaboration with Theory of Mind

Publication: ICRA 2026. The arXiv record provides the paper, full author list, and version history; the first author’s publication list also records the conference venue.

Research question

How can a robot infer a speaker’s intended goal when spoken instructions are noisy or ambiguous?

Method

SIFToM combines a vision-language model’s symbolic descriptions of scenes, human actions, and speech with probabilistic theory-of-mind inference over collaborative goals and plans. It uses the task context to interpret ambiguous instructions.

Evaluation and scope

Experiments use UnclearInstruct in VirtualHome and a Stretch robot in meal-preparation tasks, comparing instruction interpretation and collaboration against VLM and ablation baselines. The reported results concern these household settings; visual-to-symbolic grounding remains a source of failure.

The earlier preprint was titled SIFToM: Robust Spoken Instruction Following through Theory of Mind. The publication page includes the revised arXiv v2 abstract and author list alongside the ICRA 2026 venue.