Google DeepMind’s Latest AI Agent Can Play 3D Games: What We Know

Artificial Intelligence (AI) models specialising in one game and always playing to win are old news now. Google Deepmind has come forward with a fresh take, creating a new model that is designed to play multiple 3D games just like a human. Not only that, it will also understand and act based on the gamer’s instructions. Google DeepMind recently introduced its AI-based virtual gaming partner, SIMA, which will play just like professional gamers by responding to instructions and carrying out tasks accordingly.

The AI agent’s capability is not just limited to linear games where a specific path or story is to be followed; it will also be available in open-world settings.

Google DeepMind: The Virtual Gaming Partner

Standing for ‘Scalable, Instructable, Multiworld Agent’, SIMA has been trained on a number of video games and essentially understands natural language commands in reference to the 3D gaming world and image recognition to perform tasks effectively.

According to a Google blog post, the company has partnered with several game developers to train SIMA on a number of video games, like No Man’s Sky by Hello Games, Teardown by Tuxedo Labs, Valheim, and Goat Simulator 2 among others. Also, it has translated the capabilities of advanced AI models into useful, real-world actions through a language interface.

“SIMA is an AI agent that can perceive and understand a variety of environments, then take actions to achieve an instructed goal. It comprises a model designed for precise image-language mapping and a video model that predicts what will happen next on-screen,” it said, further adding that the AI agent doesn’t need access to a game’s source code or bespoke APIs but just requires the images on screen and simple, natural-language instructions from the user.

“SIMA uses keyboard and mouse outputs to control the games’ central character to carry out these instructions. This simple interface is what humans use, meaning SIMA can potentially interact with any virtual environment,” the post adds.

Presently, SIMA is evaluated across 600 basic skills, including navigation, object interaction, and menu use. Speaking on the same during a media briefing, DeepMind researcher Tim Harley shared that the AI agent has been trained to do what it’s told in the game. “We want our future agents to tackle tasks that will require high-level strategic planning and multiple sub-tasks,” he added.