Researchers have made significant progress in developing AI agents that can interact with virtual environments, thanks to a new technology called Genie 2. This innovation allows for the rapid creation of diverse and rich 3D environments, enabling researchers to test AI agents in a wide range of scenarios. Max Cant, a concept artist, has created stunning environment concepts using Genie 2.
The technology is used to develop SIMA, an AI agent designed to complete tasks in 3D game worlds by following natural-language instructions. In collaboration with game developers, researchers have demonstrated SIMA’s ability to open doors, explore environments, and follow complex instructions. The work has significant implications for developing more general AI systems that can understand and safely carry out a wide range of tasks. Companies involved in this research include game developers, and key individuals include Max Cant and the researchers behind Genie 2 and SIMA.
The concept art and drawings turned into fully interactive environments are stunning. The ability to rapidly prototype diverse experiences is a game-changer for researchers, artists, and designers. It’s incredible how Genie 2 can take a single image prompt and generate a rich, immersive environment.
The AI agents acting inside the world model are equally impressive. The SIMA agent’s ability to complete tasks in unseen environments synthesized by Genie 2 remarkably demonstrates its capabilities. Genie 2 generates game frames, while SIMA controls the avatar via keyboard and mouse inputs.
The diffusion world model underlying Genie 2 is also intriguing. An autoregressive latent diffusion model trained on a large video dataset, combined with a transformer dynamics model, enables the generation of diverse environments. I’m curious to learn more about the classifier-free guidance used to improve action controllability.
External Link: Click Here For More
