Letting a robot look while it talks, through tool calls
Dong Won Lee, Cynthia Breazeal, Hae Won Park et al. · arXiv 2026
The authors propose a minimal system in which a real-time multimodal LLM calls tools to look at people and objects and scan the scene mid-conversation. On a Jibo robot across six scenarios, tool-decision accuracy was 0.72 for OpenAI and 0.77 for Gemini.
Looking less isn't a safe default for a talking robot. A model can be right most of the time it looks and still miss many moments that called for a look.
A Modern System Recipe for Situated Embodied Human–Robot Conversation with Real-Time Multimodal LLMs and Tool-Calling
Read the paper
