Learning Object-Based World Models in MsPacman with DreamerV2

Cardenas J.D.; Munoz M.; Fabian J. · Proceedings - International Conference of the Chilean Computer Science Society, SCCC (2025) · DOI: 10.1109/sccc67219.2025.11420437

Resumen

Model-based reinforcement learning has achieved remarkable success by building internal world models for planning. However, these models often learn from unstructured, high-dimensional pixel data, leaving them "blind"to the semantic identity and significance of the objects they observe. This places a significant burden on the world model, which must infer crucial object-level dynamics from raw pixels - a process that can be sample-inefficient and may obscure the development of more direct, targeted strategies. To address this challenge, we introduce DreamerV2-OB, a targeted enhancement of the DreamerV2 agent. Our framework pioneers the fusion of structured, object-based representations directly into DreamerV2's latent world model. Specifically, we augment the agent's input with a dedicated channel encoding the spatial location of a key object class and employ a dual-reconstruction loss that forces the world model to explicitly learn this semantic information. Our evaluation in the challenging Atari game MsPacman reveals that this object-based guidance profoundly impacts agent performance and learning dynamics. More importantly, we demonstrate that the type of semantic information provided fundamentally shapes the agent's emergent strategy. Supplying information on collectibles cultivates an aggressive, high-scoring policy, while a focus on the agent's own location promotes a cautious, survival-oriented behavior. Conversely, providing an ambiguous signal - such as the position of ghosts without distinguishing whether they are threatening or vulnerable - disrupts the learning process. This work establishes that injecting targeted semantic knowledge is a powerful and computationally efficient approach to not only boost performance but also to actively sculpt the strategic biases of model-based RL agents. While our findings are currently demonstrated in MsPacman, we envision future extensions to multiple environments and object-detection pipelines, marking this as a first step toward more general, interpretable, and intelligent world models.

Instituciones: Universidad del Pacífico