[2G4-OS-47a] OS-47

2:30 PM - 2:45 PM JST(5:30 AM - 5:45 AM UTC)

[2G4-OS-47a-05] Real-World Robot Control using World Model

〇Sota Nakamura¹, Atsuya Ishizu¹, Hiroaki Honda¹, Ritsuki Matsusunaga¹, Banjou Tahara¹, Toshiki Tsuzuku², Yasuhiro Noda², Makoto Kawano¹ (1. MATSUO INSTITUTE, INC, 2. Softbank Robotics Corp.)

Keywords:

Physical AI,World Model,Foundation Model,Real-world robot control

We propose a robotic control method that integrates a world model as a predictive model and a Vision-Language Model (VLM) as an evaluator, employing Model Predictive Control (MPC). The world model learns environmental dynamics through training, predicting future states based on provided observation inputs. By evaluating the predicted future states using the VLM, this approach enables action optimization even in complex and dynamic environments. In this study, we designed a Pick and Place task under visually constrained conditions using an actual robot and experimentally verified the effectiveness of our proposed method.