New York, NY (Top40 Charts) Youtuber Harrison Kinsley has used an AI tool named GameGAN to recreate a highway stretch from GTA 5. A video showing off the results - named GTA5: GAN Theft Auto. Using GameGAN, Kinsley collaborated with developer
Daniel Kukieła on this two-month project, which followed a similar idea last year that recreated Pac-Man by watching another AI play through the game.
Neither of the two collaborators came in with much knowledge of GANs, Kinsley told Eurogamer. "It was a lot of trial and error and just small tweaks and seeing if and how they improved. We didn't have much knowledge, so we had a lot of learning to do, and still do."
GameGAN is a generative adversarial network created by Nvidia that learns to visibly imitate a desired game by ingesting screenplay and keyboard actions during training.
Every GAN consists of two competing networks - a generator and a discriminator. The generator is trained on a sample dataset (the highway) and then told to produce content based on what it saw. Meanwhile, the discriminator will compare the output of the generator with the original dataset, and in the process train its counterpart to produce content close to the source material. In this case the source material being GTA 5.
In total, Kinsley and Kukieła had 30 upsampler models trained along with 15 GameGAN models before settling on a final one.
"The very first attempt worked way better than we expected though," Kinsley continued, "so it was overall pretty exciting. We were just trying to do a lot of experimentation."
The playable demo consists of driving down a short highway stretch in GTA 5. There are some impressive details visible in the demo, such as the shadow underneath the car, accurate sunlight reflections in the rear window that change as the car moves around, and the mountain in the distance, which gets closer as the car nears.
"We wanted something challenging, and cool if it worked. GTA fit that bill well," Kinsley said. "GTA 5 is also a bit of a throwback to a project we did a few years ago for doing self-driving in GTA 5, where I streamed the AI 24/7 on Twitch."
The pair trained the GAN with 12 feeds of the highway from the game, and based on the data it learnt how the car moves and responds to controls. To begin with the car didn't understand boundaries and would drive through the barriers on the side of the road, but eventually the GAN figured out what to do if it hit the side of the road or wall.
Along the way, however, not all tests were successful. The GAN struggled with crashing into other vehicles. In the video, Kinsley describes a test where the GAN split a police car in two when the main vehicle crashed into it head on.
If you want to try it out for yourself, the code and model are hosted in Kinsley's GitHub.
"We may also attempt to make it run in the browser so no-one would be required to have any programming knowledge at all," Kinsley said. "But it's unclear to us if this will be possible and what tweaks we'd need to make to run it via something like Tensorflow.js."
In future projects, Kinsley and Kukieła are more interested in moving toward modelling the real world. "But we may wind up doing more GTA 5 as a means of more rapid R&D than real life."
He added: "I think for both of us, we're just excited to see so many people, not just programmers and AI enthusiasts, as excited about this technology and what the future holds as we are."
