BAAI Wuji·RoboBrain Orca: From Predicting the Next Token to Predicting World States — Exploring Multimodal World Models

The BAAI Wuji·RoboBrain Orca team released the technical report "Orca: The World is in Your Mind," proposing a multimodal world model that aims to enable AI to learn unified world state representations rather than merely predicting single-modality outputs. Orca employs two training modes: unconscious learning (continuous video) and conscious learning (event annotations, language QA). It is trained on 125,000 hours of video, 160 million event annotations, and 11.5 million VQA data points to construct a scalable world latent space. Experiments show that as the pretraining data scale increases, the model loss consistently decreases. Moreover, freezing the backbone and using lightweight readout modules improves performance on text understanding, image prediction, and robot control tasks. Orca has been featured on the Hugging Face Daily Papers monthly list, sparking discussions in the overseas research community about the direction of world models.