World models and neurosymbolic approaches

Generative AI is now being applied to world models for interactive games, see, e.g. Deep Mind’s Genie 2:

 https://deepmind.google/discover/blog/genie-2-a-large-scale-foundation-world-model/

Generative AI struggles with semantic consistency especially over long contexts. Deep Mind claim consistency over 2 minutes for what was shown, but is now out of frame.  They do a reasonable job at animating behaviour and simulating physics.

However, to realise the promise of virtual reality, we need consistency over indefinite time periods for complex multi-player worlds.  I believe that this is where the potential for neurosymbolic approaches would be really strong.  The symbolic models would ensure longevity and could be generated from neural models. Training would involve the means to generate successive frames from 3D models and compare them to the actual frames from recorded games. The aim is to combine the strengths of generative AI with the strengths of traditional 3D models.

Symbolic models are also key to making augmented and extended reality experiences accessible to people with visual impairments, as well as to enable intent-based interaction that gives people the freedom to choose how they interact with the AR/VR application.

Cognitive models of behaviour would further support this along with autonomous virtual agents in the 3D world. I believe that neurosymbolic approaches would be useful to supplement explicit rule-based behavioural descriptions using Chunks & Rules.  This includes the means to apply reinforcement learning.

Dave Raggett <dsr@w3.org>