Inside a lab on the first floor of Edmunds Hall, Joseph Osborn, associate professor of computer science, and three rising Pomona seniors are using cutting-edge techniques to map the vibrant two-dimensional worlds of retro video games.
This summer, Emma Gandonou ’26, Steven Kim ’26 and Leo Torres ’26 have helped Osborn expand the scope of a project he’s been working on since graduate school—a computational system called Mappy that watches Nintendo Entertainment System (NES) games get played and constructs detailed maps of their terrain.
For their Summer Undergraduate Research Program (SURP) project, Gandonou, Kim and Torres are determining whether it’s possible to not just map NES titles, but any two-dimensional action game.
“That’s important if we want an AI agent to play a video game, or if we want to add visual or motor accessibility features to a game,” Osborn says. “It’s important to know the terrain and the layout and where the objects are and whether they’re solid or not.”
“All these features end up being useful for a lot of different tasks.”
Despite their varying interest in video games, Gandonou, Kim and Torres each sought a place in Osborn’s lab this summer to hone their coding chops in a practical, applicable manner.
“It’s been helpful over the summer to keep my coding skills fresh,” says Gandonou, a computer science and cognitive science double major from Dallas, Texas. “I’ve also gotten better with problem solving and become a better coder.”
Speaking from experience, Osborn says working on an existing project opposed to a school assignment “exercises a different set of skills.” Such work can be more efficient and effective than self-study, Osborn adds, and give students the confidence to pursue post-grad research.
Kim, a computer science major by way of Princeton, New Jersey, grew up playing video games and has been searching for ways to get involved in their development as he ponders a career in tech.
In Osborn’s lab, he and his lab mates have worked collaboratively to replace the mapping tool’s current approach, which is based on deep understanding of simulated game hardware.
“Since NES games are coded similarly, Mappy can only be used on these kinds of games,” Kim says. “Our work aims to improve Mappy by incorporating modern computer vision models and techniques.”
