Semi Ryu’s project, SentimentVoice, uses technology to add emotion, depth and visual interpretation to narratives about immigration to the United States.
Two people sat on a stage. One read from a script: “The sense of joy and relief upon receiving our green cards was indescribable.” The other, wearing a virtual reality headset, responded: “I felt so happy to be able to start a new life in America.”
The performance, during an arts and technology symposium in March at Connecticut College, was a display of SentimentVoice. The project, created by Virginia Commonwealth University School of the Arts professor Semi Ryu, Ph.D., translates personal stories about immigration onto a large screen through machine-learning-powered emotion tracking and virtual reality.
“This project is about amplifying immigrant stories, and using AI as an empathetic tool to create a better chance to understand their emotional space and struggle,” said Ryu, a professor in the Department of Kinetic Imaging.
SentimentVoice draws from a subset of machine learning called Emotion AI, which uses artificial intelligence to understand human feelings and mental states. Ryu sees the project, which was developed alongside former VCU College of Engineering associate professor Alberto Cano, Ph.D., as a way to put AI and surveillance tracking to a positive purpose.
“Long before digital systems, institutions monitored, interpreted and shaped people’s emotional states in the service of control,” she said. “The project asks whether these tools can be directed toward listening, acknowledgement and empathetic encounters.”
To prepare for SentimentVoice, Ryu interviewed immigrants in the Richmond area about their experiences moving to the United States, and she compiled those testimonies into a script. The stories take place in one of five settings: airports, schools, workplaces, hospitals and immigration offices.
SentimentVoice takes two forms:
When exhibiting the technology as a live performance, Ryu selects actors or volunteers from the local community to read and experience an immigration-related story constructed from real-life interviews.
During the performance, one actor wears a VR headset, through which they can explore their surroundings in one of the pre-determined settings. The actors take turns reading from the script, and the VR headset detects the emotions exhibited by the wearer’s voice and facial movements. AI technology within the program then interprets those emotions into animations on the screen, overlaid on top of the background images.
“The AI is detecting their story, and then it responds,” Ryu said. “When they tell a sad story, there is rain, and for lonely stories, snow is coming. For angry stories, fire is flying.”
When Ryu displays SentimentVoice in a museum or exhibit space, anyone can experience the technology. First, participants read a short immigrant story. Then, they put on the headset and either retell the story or tell their own related story while AI interprets their emotions on the screen.
“I wanted to create a VR experience in which immigrant narratives could be heard, felt and amplified through responsive environments,” Ryu said.
Ryu has exhibited SentimentVoice around the country, as well as at the 2025 SIGGRAPH conference in Vancouver, Canada, and in the main exhibition hall of the Microwave International New Media Arts Festival at Hong Kong City Hall. Soon, her exhibit will travel to Dubai as a part of the International Symposium on Electronic/Emerging Art.
Ryu wants to take SentimentVoice even further. Mainstream AI tools, often developed using Western-centric AI datasets and frameworks, may not capture the nuance of how emotions are expressed or interpreted in other cultures.
“A lot of cultural emotion disappeared during the analysis process,” she said. “So we are really investigating cultural specificity.”
Now, Ryu is developing a version of SentimentVoice that is attuned to Korean cultural emotional norms. It is called SentimentVoice Han, for which Ryu is collaborating with VCU computer scientists Kostadin Damevski, Ph.D., and Bridget McInnes, Ph.D.
“Han is an extreme state of grief but also hope for overcoming the situation,” Ryu said. “It’s joy and sadness at the same time.”
Ultimately, her goal is to connect people to the experiences of marginalized communities worldwide.
“The world is marked by suffering, and I hope to foster broader conversations about the complexity and diversity of human emotional experiences,” Ryu said. “Particularly those that are often overlooked or underrepresented.”
SentimentVoice was developed by Semi Ryu (creative director), Miles Popiela (technical lead, AI and VR), Henry Bryant (AI and VR developer), Ariana Thomas (AI and VR developer), Matthew Labella (3D visual producer, Department of Kinetic Imaging class of 2025), Hanna Chou (video producer, Department of Kinetic Imaging class of 2025), Chrystine Rayburn (sound producer, Department of Kinetic Imaging MFA class of 2024), Ryan Flores (actor, Department of Sculpture MFA class of 2016), Katherine Nguyen (actor, Department of Musical Theatre class of 2026) and videographers Hanna Chou, Kiara Brown (Department of Kinetic Imaging class of 2024), Uday Illa (Department of Kinetic Imaging class of 2024) and Ryan Alvarado (Department of Kinetic Imaging class of 2024). The project was funded by theCommonwealth Cyber Initiative.
- Artificial intelligence
- Faculty
