Alibaba researchers have proposed a new VARP (Visual Action Role Playing) intelligent body framework that enables large models such as GPT-4o and Claude 3.5 to play Black Myth: Goku. The framework takes screenshots of the game directly as input and generates action-operated gameplay in the form of Python code through visual language modeling reasoning. Using Black Myth: Goku as the research platform, 12 tasks were defined and a human action dataset containing 1000 valid data was constructed, with each action consisting of a combination of atomic commands.The VARP framework mainly consists of an action planning system and a human-guided trajectory system, as well as scenario libraries, action libraries, and human-guided libraries. Comparing the human and AI performance, the AI of the small monsters is up to the level of human players, and the elite monsters, the GPT-4o, have the highest win rate, but they can't do anything about the ghosts. Due to VLMs reasoning speed limitations and the lack of a clear path to guide the game, the AI was inadequate. There are plans to release the relevant code and dataset in the future. At the same time, it is mentioned that AI playing the game is not a new thing, and it is unexpected that the pure large model can play the game, and the effective data of the dataset in this study is 1,000 pieces.
Paper address:
https://arxiv.org/abs/2409.12889
Project address:
https://varp-agent.github.io/