You’re in a dance club bathed in a neon purple glow, surrounded by ghostly silhouettes that sway to the music. A stranger at the bar greets you with a flirty wave: “Hey, you look so cute!” Sparks fly.
It’s all part of a virtual reality game called Rekindle, but the narrative seems real enough that you feel giddy. You break into what’s known as a genuine Duchenne smile — the kind where your mouth corners lift and your eyes squint a little. The biometric software in the headset you’re wearing scans your face and checks and your muscle movements against the 70 that are involved in typical expressions. You’re happy and your gaming PC knows it. You sink deeper into the story
Tapping into the emotional landscape and biometrics in gamingare both well-worn paths in the virtual reality, or VR, space, but Rekindle, a new first-person Interactive Digital Narrative (IDN) that is the brainchild of Interdisciplinary Design & Media PhD student Hector Fan and Northeastern professors Mark Sivak and Casper Harteveld, gives them a fresh spin
Set in a dystopian future where oppressive powers use memory manipulation to erase any trace of sexual identity that they view as being demory at a time. All the while, the player’s empathy for the LGBTQ+ community is being deepened by exploring the world through the eyes of the story’s hero. The narrative may be fictional, but the struggle is all too real, says Fan who hopes the game can foster “positive social change.”
At one level, the “memories” are collectibles that the player gathers to advance through the game. However, unlike your typical Mario coins or Pac-Man dots, they’re meant to be relatable situations that either jog your personal memories or remind you of significant events that are “culturally shared” in the LGBTQ+ space, Sivak explains. This shared archive includes some dark chapters, such as the 2016 Pulse Nightclub shooting, when a gunman killed 49 people and wounded 58 at the Orlando venue’s “Latin Night” event frequented by members of the gay community.
The game’s narrative is sprinkled with prompts that call on the user to actively display emotions corresponding to each collectible “memory fragment” in order to advance through the narrative while building an emotional tie to the protagonist.
Once “you’re in the helmet, you become that person,” Harteveld explains
In the past, emotions have been treated as “input” rather than “output,” Fan, Sivak, and Harteveld write in a paper that was presented recently at the Augmented World Expo, an XR and spatial computing community meet-up that took place this June in Long Beach, Calif. Responding to emotions of other characters on the screen, the player is passively “clicking through the game.”
Rekindle turns the tables in favor of active engagement in a fictional world that echoes our own. At one point, a fight scene brings to mind the Stonewall Riots of 1969 or the more recent West Hollywood Mass Brawls this March. “The memories … are culturally generic,” Fan explains. The emotional input you’re asked to provide further boosts the sense of “dramatic agency” and makes you feel like a participant.
Graham Page, EVP of Commercial Research and Innovation at iMotions, which pools multiple types of biometric data on a single platform, agrees that tracking user emotions adds an important dimension to the VR experience: “One of the frustrations of using, or interacting with tech, is that it knows what we click or say, but it doesn’t know necessarily how we feel or how we say it.” The ability to create tools sensitive to “non-verbal insight” paves the way for more meaningful interaction, he suggests. Technology that reads the room.
To do this, the biometric features of Rekindle’s Samsung Galaxy XR headset track the user’s facial expression and map them onto a set of 6 basic emotions, such as “happy,” “sad,” or “surprised,” and 15 compound emotions, which include combos such as “happily surprised,” “sadly angry,” or “fearfully disgusted.”
The headset picks up muscle movements and translates them into “action units” (AUs): for example, raised eyebrows or narrowing eyes are possible AUs. That information is then fed into a system that ranks each of the 21 possible emotions with a 0 to 1 score that indicates their relative strength, with 0 signaling a total absence of a particular emotion in that moment
While the technology adds another dimension to the gaming experience, it also opens the door for new levels of complexity and possible tension. Be it the flirtation at a gay bar or encounters with violence related to sexual identity, the situations within the Rekindle narrative will sit differently with a member of the LGBTQ+ community than someone who doesn’t know what the Pride Flag stands for. Those differences will naturally affect the degree to which the emotions displayed by the players will match the author’s intent and fit with the game’s progression.
But that’s also part of the point. Fan recognizes that there’s value to being put in someone else’s shoes, even if they pinch a little. “I think that the goal is to … foster empathy or attitude change towards the LGBTQ+ community.” By trying to enact the protagonist’s emotion — one that Fan had in mind when building the game — the player becomes “almost like an actor within a simulation,” he says. Taking on the protagonist’s quest “could help them to conduct perspective taking of other people’s experience that they might not have access to.”
In the future, Fan plans to lean into the differences in how people view and react to the situations in the game as a way to expand narrative progression. For example, if that friendly greeting at the bar gets a response that’s more “Ew!” than “Hey there!” from the user, the storyline and the interactions that follow could change accordingly. The result? An experience that feels even more immersive and realistic — one that also allows the player to grapple with their own biases and their possible effects in a way that feels safe, Fan hopes.
Ultimately, the technology has potential beyond the gaming world. “Books could also be interactive,” Harteveld suggests, describing texts with a nonlinear format similar to the “choose your own adventure” books that allow the reader to change the plot as they progress through the narrative
Sivak also mentions that virtual exposure therapy has been helping people grapple with phobias and post traumatic stress disorder, PTSD. Harteveld adds that an emotion tracking component could take it up a notch, for example by reading a patient’s stress levels instead of relying on their words alone. An emotion tracking camera could also boost safety on the road by sounding the alarm if somebody is falling asleep at the wheel, he suggests
Northeastern Global News, in your inbox
Harteveld recently wore eye-tracking glasses on a paragliding adventure that stirred up first-time jitters. While they could only track his eye movement to measure his stress levels, he envisions a version that could one day also respond to emotions akin to a virtual tandem jumper at his side. Tech that could talk you through a difficult task after sensing your anxiety could be especially useful, Harteveld thinks
“When you get into emergency situations, maybe it’s good that a system or a warning is trying to calm you down” and walk you through the steps, he says
