Mixed Reality for Managing Phantom Pain in the Multimedia Lab
Via UT Dallas Jonsson School Magazine — A person wanders into a mixed reality environment. A 3D mirror image avatar of the player appears, surrounded by a series of 3D games to choose from. Smashing brightly colored bubbles as they emerge from the floor is entertaining enough, but this immersive game has another purpose – physical therapy.
“Mirror box therapy is not engaging,” said Dr. Balakrishnan Prabhakaran, professor of computer science at the Erik Jonsson School of Engineering and Computer Science at the University of Texas at Dallas and director of the UT Dallas Multimedia Systems Laboratory. “It’s boring. But we have a series of games. We hope the patient will focus on playing rather than solving the pain problem.”
Prabhakaran, a Distinguished Scientist by the Association for Computing Machinery (ACM), is working with the Dallas VA Medical Center to develop mixed reality therapies to treat phantom limb pain (PLP), severe pain occurring after an amputation. Even after adjusting to a prosthetic limb, the patient may feel chronic pain where the limb was located. Nearly 85 percent of amputees experience PLP, leading to ongoing distress.
Prabhakaran’s research assistant Kanchan Bahirat (PhD ’18) consulted Dr. Thiru Annaswamy, section chief of electrodiagnostic and spine sections, and Dr. Gargi Raval, staff physician, at the Dallas VA Medical Center. Bahirat is the first author of an article on the subject published by the ACM in Proceedings of MM’17.
“When we move, that action is visualized by our eyes and connects to our brain,” Bahirat said. “But for the patient with limb loss, there is no visual feedback. That is the main reason for the pain.”
BEYOND THE BOX
Previous research indicates that PLP is related to neuroplasticity in the somatosensory cortex—brain changes caused by experiences in the part of the brain that processes tactile information. Because the brain causes a pain sensation from past experience, the body needs new experiences for new neural pathways to develop and the pain to subside.
Mirror box therapy is currently used to treat PLP, during which a mirror reflection of the intact limb is projected to the place of the amputated limb, allowing the brain to complete its circuit. However, this therapy requires the participant to stare at the reflection without moving. Other therapies use pre-constructed 3D models, which require using sensors to capture movement and do not always create a complete visual illusion.
Prabhakaran and his team proposed a novel solution – using commercial virtual reality devices and cameras to create a mixed reality therapy session. A camera captures the patient’s image, then a live 3D model of the phantom limb is generated using a mirror image of the current limb. The model displays in real-time, allowing the patient to move freely.
GAMING AS THERAPY
Prabhakaran’s team intentionally made the therapy interactive, with games embedded in a mixed reality environment. Bahirat consulted the physicians to select exercises for both upper and lower limb amputations.
“The first game for knee flexion and extension is called ‘Bubble Burst,’” Bahirat said. “The second one has a stomping action simulating a piano game, and the third is a pedal game where you make a ball go up as you’re making a pedal motion.”
Patient trials are currently underway. The therapy is completed daily at home for four weeks, so the team used readily available equipment. Each patient receives a Microsoft Kinect camera, an Oculus Rift headset and a gaming laptop.
VIDEO AND THE PATIENT EXPERIENCE
The benefits of mixed reality therapy extend beyond entertainment and distraction. Because each session is recorded, insights from the video can improve interventions.
“No doctor can sit and watch 10 hours of video,” Prabhakaran said. “An automated analysis can point out potential problems. ‘Did you do your exercises?’ Typically, a patient would say yes. I would say yes. But now, there is a recording. If there is no improvement, the physician can see what is going wrong.”
Video data may also improve each patient’s enjoyment of the process.
“The real feedback is from the patients,” Prabhakaran said. “We have to consider the non-computer science challenges to make the games as effective as possible.”
The research was funded by the U.S. Army Research Office and the National Science Foundation.
ABOUT THE UT DALLAS COMPUTER SCIENCE DEPARTMENT
The UT Dallas Computer Science program is one of the largest Computer Science departments in the United States with over 3,315 bachelors-degree students, more than 1,110 master’s students, 165 Ph.D. students, 52 tenure-track faculty members, and 44 full-time senior lecturers, as of Fall 2019. With The University of Texas at Dallas’ unique history of starting as a graduate institution first, the CS Department is built on a legacy of valuing innovative research and providing advanced training for software engineers and computer scientists.