This virtual reality system lets healthcare students practice with AI

The world is your classroom is one Northeastern University mantra students hear the minute they arrive on campus. Sometimes that means shrinking the world down to a simulation for students to experience it. 

That is precisely what a team of interdisciplinary faculty are doing in an effort to bridge classroom learning and the world of healthcare. They are building and refining a tool that lets healthcare students practice interacting with patients by simulating clinical environments similar to those they will encounter as professionals. 

The idea is to bring students into direct contact with an unscripted clinical environment that is difficult to reproduce outside of hospitals and doctors’ offices — one of the key challenges in health-profession pedagogy, said Eileen McGivney, assistant professor of art and design and communications studies at Northeastern.

Studies have found that clinical education can be difficult to standardize because real-world patient encounters are unpredictable and vary by setting and circumstances. Additionally, limited opportunities for hands-on clinical experience can make it harder to ensure students are adequately prepared before they enter real-world clinical settings.

With the help of a $900,000 grant from the National Science Foundation, faculty across the College of Arts, Media and Design, Khoury College of Computer Sciences and Bouvé College of Health Sciences are joining forces to further develop an AI-powered virtual reality system they call the Virtual AI Patient Simulator, or VAPS. 

The three-year award is part of the agency’s Research on Innovative Technologies for Enhanced Learning, or RITEL, program that supports research projects that use “AI, robotics and immersive or augmenting technologies for teaching and learning,” according to the foundation’s website. 

The Northeastern research team has been working to create customizable virtual patients within simulated environments with realistic speech, eye contact, facial expressions and gestures so that students can practice their clinical communication skills in real-time. 

The collaboration began a couple of years ago with a so-called Transforming Interdisciplinary Experiential Research, or TIER, seed grant, a competitive university-issued award meant to secure proof of concept for a project involving “multidisciplinary teams.” The TIER seed grant helps teams hone their concepts to eventually be able to compete for external funding. 

The team has since built and piloted the technology, including the headset and the virtual patient whose conversational programming is shaped by actual clinical case records, with plans to further test the platform and develop assessments to gauge student performance, McGivney said. 

“We really built it from the ground up,” said McGivney, who is the principal investigator on the project. The grant, she added, “will allow us to build an authoring tool, where clinical faculty who have no experience with virtual reality or AI can create their own patients that are fit for their own curricula and classrooms.”

Researchers are also using eye tracking and heart rate data collected through the VR headsets to measure students’ responses during the interaction. A separate animation system is being developed to make those digital patients’ appearance and nonverbal behavior more realistic, said Leanne Chukoskie, an associate professor at Northeastern with a joint appointment in the departments of art and design and physical therapy.

“I see a future of this kind of technical learning simulation opportunity for engaging students, potentially as co-ops,” Chukoskie said. 

Heidi Cheerman, an associate clinical professor in the department of physical therapy, said that one of the goals of the project is not simply to make health care training more technologically sophisticated, but to give students more chances to practice “the human side” of the profession.

That includes cultivating skills associated with “communication, empathy, listening,” as well as helping students become more adaptable to meet the moment, Cheerman said. 

“What excites us about VAPS is that there is an opportunity to give learners safe environments where they can practice navigating challenging conversations, receive feedback, reflect and try again,” Cheerman said. 

Several faculty members were brought on the project after McGivney and her colleagues achieved the initial seed funding, including Sheri Kiami, an associate clinical professor in the department of physical therapy, whose background is in simulation education. 

Having worked at the national level on simulation education in the field of physical therapy Kiami is helping to incorporate industry best practices into the project, which is another stipulation of the grant. 

“I greatly appreciate that the approach to using this technology is about filling an educational need,” Kiami said. “It really will help us overcome a lot of the barriers and obstacles we have in using simulation education to prepare our students for clinical practice.”