Game designers are incorporating assessment tools into new learning games in ways that are useful for instructional decision-making, without overpowering teachers with irrelevant data.
Last fall, at Robinswood Middle School in Orlando, Florida, 65 students were learning about air pollution. They weren’t learning by textbook or even a video. Rather, they were part of a beta test using SimCityEDU: Pollution Challenge!, an adaptation of the popular commercial game SimCity from nonprofit game collaborative GlassLab.
Denise Cruz, the computer education teacher at Robinswood, watched students wrestle with the complex problem of how to manage air pollution while providing jobs, schools, homes, and transportation to SimCity citizens. She saw them collaborate with one another, try different strategies, and hop online to read more about issues raised in the game. What impressed her the most, however, was watching her fellow teachers assess their students’ learning in real time.
“It was not the normal assessment,” Cruz said. “Normally we give kids multiple choice tests. But they’ve been taught the process of test taking, so they may get something correct not because they know something but because they’ve mastered the skill of taking tests.”
Increasingly, teachers have accepted games-based learning as a way to excite and motivate students. What Cruz witnessed in her middle-school classroom, however, is more than just engagement. Learning games, when designed properly, can also provide teachers with a new approach to formative assessment. The people behind SimCityEDU are now trying to move this idea into the mainstream.
“There were two parts to the original vision for GlassLab,” said Jessica Lindl, general manager of GlassLab. “The initial focus was making learning games that were just as engaging as commercial games. The second part of the vision was that the data generated from game play could inform the assessment process in the classroom.”
GlassLab is a collaboration among the Institute of Play, Electronic Arts, Educational Testing Services, the Entertainment Software Association, Pearson’s Center for Digital Data, Analytics & Adaptive Learning, and others. It is funded by the Bill & Melinda Gates Foundation and the MacArthur Foundation.
In April, GlassLab announced it would make the formative assessment tools it has developed for games such as SimCityEDU available to other developers. Its first partners are iCivics, the game development nonprofit founded by former Supreme Court Justice Sandra Day O’Connor, and Filament Games, a for-profit learning-game company that has worked with iCivics for many years. Alpha testing of the game Argument Wars, beefed up with GlassLab’s formative assessment tools, is scheduled for this summer with a beta version to follow in the fall.
For Jeff Curley, executive director of iCivics, the new partnership is a relief. “For a long time the promise of assessment in games has been out there,” he said. “Everyone sees it theoretically. But it’s a challenge we haven’t been able to crack.”
iCivics had a Small Business Innovation Research grant from the US Department of Education to build assessment tools for teachers in their game, Do I Have a Right? Although the company had access to talented game designers and world-class legal experts, it didn’t have the expertise necessary to develop reliable, robust assessment tools. Eventually iCivics scrapped the project.
“You have to ask yourself, can you deliver the information in a way that is accessible and relevant to teachers?” Curley asked. “A teacher might have five classes of 30 kids each. And she might have a 30-minute break or an hour at night to look at all the information from the game and then come up with an actionable plan for adjusting her teaching. How do you make that work? That’s a radical design challenge.”
Lee Wilson, chief executive officer of Filament Games, shared Curley’s concerns.
“It is incredibly easy to overpower teachers with irrelevant data,” Wilson said. “And irrelevant data doesn’t help teachers make good instructional decisions.”
SimCityEDU, for example, pulls 3,000 data points for every player.
“How do you process that information and create visual symbols that let teachers navigate?” Lindl asked. “We’re trying to demonstrate visually for teachers the learning progression their students are on. We want teachers to be able to hover over the learning so they can adopt and personalize as they go.”
GlassLab is a 20-person team, made up of not only game designers, but learning scientists, assessment scientists, and psychometricians. Plus, through its partners Educational Testing Service and Pearson’s, it has access to evidence-based assessment research and expertise. In addition, it recently published a white paper, “Psychometric Considerations in Game-based Assessment,” outlining a new framework for making games-based assessment.
“Because of the resources at GlassLab, we can test these core ideas about what a game can do,” Lee, of Filament Games, said. “We know that in real time we can track how long it takes a student to reach a decision in a game as well as how they reached that decision. We can watch them try once, then a second time and a third time, and we can chart their different strategies. But most of that data is not going to actually be relevant for teachers. Sorting the wheat from the chaff is the promise of the tools from GlassLab.”
At Robinswood Middle School, Denise Cruz and her fellow teachers observed student progress through a dashboard that let her toggle between metrics such as “complex problem solving,” “integrating info-text and diagrams,” and “looking for evidence within the text.” She could track her students on each of their missions in the game, and color blocks helped her visualize students’ progress in different categories.
Cruz said some pleasant surprises, such as vibrant class discussion, emerged from their use of the assessment tools. Based on the data teachers got through the dashboard, questions about flawed reasoning arose. These questions turned into a conversation with students on their decisions and learning from their mistakes.
“The discussion across the board was exceptional,” Cruz said. “From a teacher’s point of view, it became not so much about a quick right or wrong answer but about understanding what the student was saying with their decisions in the game.”
And yes, when asked, students said if they could learn through a game in all their subjects, they would.
By Heather Chaplin