"I see that her heart is in an irregular, irregular rhythm, so her atria are fluttering as her ventricles are trying to pump,” explained Ashlee Leshinski, a student in the University of Maryland School of Graduate Studies (UMSGS) Physician Assistant (PA) program. “And I can see nerves transmitting and see where the body is reactive versus nonreactive. And then I can also look at everything I'm seeing, and then I have a good visualization of where exactly in the brain is being affected.”
Leshinski’s amazing “view” of a patient experiencing stroke symptoms represents the newest level of medical education simulation. Called Holocamera, it’s the fruit of the collaboration between medical educators at the University of Maryland, Baltimore (UMB) and computer scientists at the University of Maryland, College Park (UMCP) under the University of Maryland Strategic Partnership: MPowering the State (MPower). MPower is a collaboration between the state of Maryland’s two most powerful public research engines, UMB and UMCP, designed to foster interdisciplinary research, increase educational opportunities, and address the state’s most critical issues.
(Above) Dr. Hendrix examines patient in Holocamera studio, examination with animated overlay, student Ashlee Leshinski watches virtually. (Below) Computer rendering of the Holocamera studio, a virtual subject in the studio, cameras line the walls of the Holocamera studio.
“Our students are so technologically savvy these days. How do I meet them somewhere where they can get those 'Ah-ha!' moments sooner, when I only have 24 months to prepare them as PAs?” said Cheri Hendrix, DHEd, MSBME, PA-C, associate professor and assistant dean, who leads the UMSGS Physician Assistant training program.
Cheri Hendrix and Amitabh Varshney
In simple terms, PAs are licensed clinicians who see patients and treat them under the supervision of a physician. They are also an increasingly important part of the nation’s health care system, with some 180,000 working in almost every aspect of patient care. The Bureau of Labor Statistics predicts the United States will add about 13,000 new PAs in each of the next 10 years. They will be expected to perform challenging, hands-on roles, diagnosing patients and prescribing treatment.
“We have to prepare them to get to that clinical phase. They've got to do family medicine, internal medicine, pediatric medicine, women's health, psychiatric medicine, behavioral health, emergency medicine, surgery,” Hendrix explained. “There's such a shortage of physicians right now. There's a massive nursing shortage as well. That's where physician assistants are coming in, trying to help bolster those numbers, because we're all over the place. We're so well trained, and we can change specialties at any time to help where the need is greatest.”
During their 24 months of training, PAs will perform rotations in each of the areas Hendrix outlined, but in just five weeks at a time. She admits it’s “very fast, not for the faint of heart,” but is optimistic that Holocamera can quickly and effectively bridge classroom learning to clinical experiences.
Explaining the technology is also not for the faint of heart. But if anyone can explain how Holocamera works, Amitabh Varshney, PhD, dean of UMCP’s College of Computer, Mathematical, and Natural Sciences, can. His groundbreaking 2018 study showed 8 percent greater recall of information received in an immersive environment rather than information on a computer or in a textbook.
Holocamera is just the latest in a long line of breakthroughs Varshney has brought to life in the realm of virtual and augmented reality. He and partner Sarah Murthi, MD, launched the Maryland Blended Reality Center in 2017 to advance visual computing tools — many of them based in immersive technologies like virtual, augmented, and mixed reality — that can be used for emergency medicine, health care, and innovative educational and training modules.
“Right now, if you look at all the content that is generated out there, they are of two kinds. In the video games, they are hand-drawn characters, which are not lifelike, but they have complete degree of freedom. You can move around and navigate around them as if you were looking at them in the real life," Varshney explained. "On the other hand, you have the cinema, the movies, the television, and others, where you have extremely high quality. However, your degrees of freedom of how much you can move are limited by what the director decided for you. So, what we wanted to do was to democratize the experience, give the user, the end user, the power to navigate through these environments at their own will and not leave it up to somebody remote deciding what you can and cannot see."
Varshney and his colleagues sought and received $1 million in funding from the National Science Foundation and built what is now the largest volumetric capture studio in the world. The studio space is lined with layers of cameras — 300 in all — that capture 100 billion visual samples each second. The trick, of course, is how to use all that data.
“Because it's 100 gigabytes, it’s way too large, no matter how you cut, and that's per second. So, if you're looking at 10 minutes, it runs two terabytes easily. So, for that we started looking at AI-based techniques to reduce the data, to fuse the data, and to make it be more accessible.”
Accessibility in this case means allowing students to don virtual reality goggles and virtually attend medical screenings and treatments. In that day’s scenario, student Leshinski and classmate Ryan Jetton watched Hendrix examine an actress, or standardized patient, portraying a 76-year-old woman complaining of classic symptoms of stroke.
“They put on the goggles and they see me with a patient and she's in a little bit of trouble. She's in a quandary. She's been having symptoms on and off for a few days, and I'm asking her some questions about those symptoms,” Hendrix said. Those questions involve basic medical history, as well as her current conditions, such as how the patient’s heart and breathing are doing and when she last experienced difficulty. But another very important part of the exam is what signs the patient is exhibiting, what the patient looks like, things the students can observe as well.
“And then she starts exhibiting the signs, not just telling me her symptoms, but the signs. And then we have this overlay of an animation of all her anatomy. They can see physiologically, what's going on with her as she's unable to talk, as she's unable to move limbs, or can move limbs as her eyes are deviated to this left side of her brain. She's right side dominant. None of her right side works because the left brain was affected.”
Leshinski says the combination of being able to observe the exam while seeing what’s going on inside the patient is a game-changer.
“I can see neurons transmitting motor and sensory information, represented by the blue and yellow dots, as well as I can see her heart beating currently," Leshinski said. "And the wonderful thing here, too, is that you can go ahead and stimulate a heart rhythm called atrial fibrillation, which leads to an increased ability or increased risk of developing a stroke. So, I'm seeing what that looks like, seeing how hard the ventricles are working in the atria, but it’s also being able to highlight where exactly the brain is being affected. And so, by seeing that, I can also see as we turn on the animation on the left side of her arm, I can see that she's not getting the neural impulses to her arm, which is why when we're doing the physical exam. We can't see any type of motor response when she's asked.”
Hendrix hopes students’ enhanced engagement with the use of Holocamera will lead to much greater retention. Typically, students learn through hours of classroom instruction, reinforcing their memories before exams by “cramming” facts and lessons.
“If I ask you about whatever was on that exam 30 days later, the recall is only 10 percent. And data shows that. With an immersive technology where it is, I'm seeing this, I'm doing this, I'm walking around and I'm maneuvering, I'm seeing everything, I'm going to tell you about it, It solidifies those concepts so that recall in 30 days I'd like to see around 90 percent. So there's a big difference, and it helps improve their clinical reasoning and their critical thought process.”
Leshinski agrees. “Of course, we're learning. But to have a new tool that helps take complex ideas and processes and see what's happening in the body, to see Dr. Hendrix work through how she would, with proper technique, assess this patient, and what you're seeing with the patient, I think is very sticky in your mind and you won't forget it in the future.”
Now that Leshinski and Jetton have put the technology through its paces, the plan is to put Holocamera to the test. Beginning in November, half of the 58-student PA class will work with Holocamera, while the other half will experience conventional training. Leshinski is betting on Holocamera.
“I think it bridges the gap between the classroom to the clinical experience. I think nothing, especially when we're looking at what technology is doing in health care, is ever going to be able to replace the true patient experience in front of you, interacting with patients. But when you think about accessibility and providing opportunities to areas that don't have access to large medical centers, who don't have the ability to spend hundreds of dollars on supplemental education resources to make them the best provider they can, to have something like this integrated in your education that's able to be utilized, we'll really see how we can adjust and safely practice techniques.”
Jetton, a former EMT, looks forward to a step even beyond Holocamera, where students will be able not only to see and hear, but feel virtual clinical examinations.
“I think that's the next horizon with this,” he said. “Where you can actually participate in skills practice and get that haptic feedback from what you're doing and being able to learn.” Much of what PA students learn, he added, concerns observation and diagnosis, but “there's a good percentage of what we're going to be doing in the field, it's actually hands-on, skills-based. So, understanding what's going on with a patient is only half the battle. A lot of times we have to do pretty complex tasks like chest tubes, intubations, things like that, and that's the future, right? Like being able to practice those things in an environment where you can hold a laryngoscope blade and see it in real time, and actually practice this kind of stuff.”
However the technology develops, Varshney is energized by Holocamera’s potential to virtually educate virtually anywhere.
“Imagine going to developing, underdeveloped countries, they will not be able to have folks like Dr. Cheri Hendrix among them. How can we communicate with them as to what are the highest, best possible standards of health care education? Virtual reality and immersive training and education provide that as a medium to educate the next generation of people, not just here in the United States, but across the world and around the planet. And that is what is so exciting about this collaboration.”