Abstract
Immersive health technologies are revolutionising the delivery of frontline healthcare, therapeutic techniques, and research. They also offer great potential to improve the training of healthcare professionals through reality-simulation training. This review paper summarises the current developments and uses of four types of immersive health technology: augmented reality, virtual reality, machine learning, and artificial intelligence. Current examples of their use in healthcare, opportunities and pitfalls, and how the use of these technologies could be improved further in the future are highlighted. How technology that once appeared to be only visionary is now part of day-to-day life for many patients and consumers is also addressed.
IMMERSIVE HEALTH
Immersive health is a term that encompasses technologies that interact with, or leverage, the neuroscience of the human brain. Within healthcare, immersive technologies have the potential to disrupt every medical specialty and collectively can also be thought of as ‘digital therapeutics’. These technologies allow health workers to treat or manage a medical condition more optimally than would be possible in traditional healthcare. A broad definition is proposed by the Digital Therapeutics Alliance (DTA): “Digital therapeutics deliver evidence-based therapeutic interventions to patients that are driven by high quality software programs to prevent, manage, or treat a medical disorder or disease. They are used independently or in concert with medications, devices, or other therapies to optimise patient care and health outcomes.”1 There are four main immersive modalities:
Virtual reality (VR): the computer-generated simulation of a three-dimensional image or environment that can be interacted with in a seemingly real or physical way by a person using special electronic equipment, such as a helmet with a screen inside or gloves fitted with sensors.2,3
Augmented reality (AR): an interactive experience of a real-world environment where the objects that reside in the real-world are ‘augmented’ by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory, and olfactory.4
Machine learning (ML): a process leveraging the computing power of modern architecture to learn relationships from data, with its emphasis on efficient computing algorithms and pattern recognition. ML algorithms build a mathematical model based on sample data, known as ‘training data’, in order to make predictions or decisions without being explicitly programmed to perform the task.5
Artificial intelligence (AI): the theory and development of computer systems able to perform tasks normally requiring human intelligence such as visual perception, learning, reasoning, and action-taking.4,6
One of the first uses of immersive health technology was in the mid-1990s, when the Oregon Research Institute Eugene, Oregon, USA, used VR to train mobility-impaired children on how to control their motorised wheelchairs.7 VR devices have evolved exponentially since initially being extremely cumbersome, expensive, and having issues including inducing motion sickness. In recent years, the use of immersive technologies has become mainstream, whether via voice recognition in systems such as Amazon’s virtual assistant Alexa™ (Amazon, Seattle, Washington, USA), or in video gaming in Pokémon Go™ (Niantic, Inc., San Francisco, California, USA), where players compete to collect VR animals overlaid on a real-world environment. Immersive technologies have become widely accepted by consumers for recreational use and accepted in part of their daily lives, consciously or not; however, their use in healthcare is currently not exploited to its full potential.
VIRTUAL MEDICINE TODAY
Today, the furthest that telehealth extends to in mainstream use is that of programmes like Simple Telehealth run by NHS Highland. This is a programmed protocol of text messages for patients with chronic, long-term conditions in the community, offering advice, tips, reminders, and questions for the patients to send in medical observations such as heart rate, blood pressure, and peak flow from home.8 Clinicians can log into the website to see readings and edit messages. A review in 2018 of multiple disease protocols found that patients felt less isolated, clinicians were better able to remotely monitor patients, and improved adherence by patients resulted from reinforcement of health advice.9
New mobile technologies are bringing the diagnosis to the patient enabled by AI analytics. Kardia’s AliveCor™ device (AliveCor Inc., Mountain View, California, USA) is one of the best known. AliveCor is a small device, half the size of a one-dollar bill, purchasable on the internet, and able to connect with a mobile phone in order to analyse whether the patient’s heart rhythm is irregular and indicative of atrial fibrillation (AF). Analysis of the rhythm is by AI within the application downloaded onto a mobile phone. Given that ≤50% of patients with AF are asymptomatic, and yet ≤25% of cardioembolic strokes can be attributed to AF, the positive benefits to healthcare systems are easily seen.10 In a review of 1 screening event of 954 people who had a basic AliveCor electrocardiography, 54 (5.6%) had an abnormality.11 Even without specialists on-site, detection of paroxysmal AF is enabled by the ability to email aberrant detections which patients can take as and when they have symptoms.
In the UK, the health-delivery company Babylon™ (London, UK) is expanding and now operates a remote service. This enables virtual consultations with doctors, nurses, and therapists by text and video messaging on a mobile application. They suggest that the majority of core primary care consultations are best managed at least initially by a video consultation, reducing human transport and movements.12 Telehealth also means that tests can be ordered and prescriptions dispensed without patients having to visit a general practice surgery.
The above technologies are all in current use and represent the apogee of healthcare delivery today. There has been little change to the way healthcare professionals are trained for decades, but immersive health technologies offer a chance to change that. By utilising immersive technologies, it is possible to deliver healthcare diagnosis and investigation remotely, with all the subsequent benefits to cost, outcomes, and compliance.
VIRTUAL REALITY AND AUGMENTED REALITY: “I KNOW IT’S NOT REAL, BUT IT FEELS REAL.”
Training Healthcare Professions
Training any healthcare professional is time and capital-intensive. To train the doctors of the future using traditional methods, which are partly implicated in the exponential growth in the societal cost of delivering healthcare, is an unrealistic expectation.13 Could immersive health technologies be a path both to cheaper and better training? VR training is not new; it has been used frequently in commercial applications for decades, from flight simulators in training air force pilots to Formula 1™ car racing, reducing both training duration and costs.14,15 This power has positioned VR and AR as vital teaching and research tool in multiple fields, including aviation, oil, shipping, and the military.16 The aviation industry credits immersive simulation education as a major contributor to the approximate 50% reduction in human error-related airline crashes since the 1970s. It has provided a safe space to place pilots in rare but critical situations, with enough immersion to replicate the physiological manifestations of stress that results in the error in real life scenarios, but which are not replicated in ordinary simulation.
In health education, learning involves multifaceted physiological systems, developed adaptive expertise, and the acquisition of collaborative skills. Learning in the medical domain is often situated in a real-life context yet training in this real-life context is not always possible. To truly achieve simulation-training that replicates medical environments, there needs to be not only analogous mental challenges, but also analogous physiological responses. To replicate and train for human factors in clinical decision making to date has been difficult to achieve, but the immersive nature of VR now provides the necessary tools.
A systematic review and meta-analysis of 31 studies by Khaw et al.17 found positive benefits for VR-trained groups with greater knowledge of a subject compared to traditional online or offline learning methods. They also identified improvements in healthcare professionals’ cognitive skills.17 However, these studies focussed heavily on the healthcare professional and none of the studies reported any patient-related outcomes or adverse effects, and cost-effectiveness was not addressed. VR has also been reviewed in improving nontechnical skills such as teamwork, situational awareness, and communication. A systematic review by Bracq et al.18 however, highlighted that there have been minimal measures of the outcome of the use of VR on nontechnical skills. Other studies have highlighted the potential for VR to supplement clinical training, especially set against increasing restrictions on the number of hours trainees are permitted to work, and the ability for observed review of VR practice. VR training compared to video training in laparoscopic surgery was reported by Gurusamy et al.19 as being at least equally effective as video training. In participants with limited laparoscopic experience, VR training resulted in a greater reduction in operating time, error, and unnecessary movements.
VR can also enhance learning performance in medical simulation.20 The sense of presence offered by immersive VR leads to much better learning from experience.21 “Virtual reality simulations can bridge the gap between theory and practice by immersing the learner in a realistic, dynamic, complex setting.”22 Both VR and AR solutions can teach clinicians complex procedures,23 are effective in CPR training,24 can improve communication skills,25 may enhance critical thinking,26 and have been shown to improve clinical decision making.27 They have also been shown to reduce cost and improve safety in advanced life support training.28
Advances in medical imaging over the last few decades have been exponential, from ultrasound to complex cross-sectional imaging using MRI. Yet the way the images are displayed has not changed. Imaging is still usually on a flat computer screen and requires skill, imagination, and lengthy training to convert this to an understanding that allows surgeons to operate on the patient in front of them. Research in the USA is working to prototype AR applications in medicine such that a surgeon using an AR headset would be able to see digital images and other data directly overlaid on their field of view.29 This has the potential to reduce intraoperative time, complications, and cost, ultimately leading to increased patient survival.
A systematic review by Munzer et al.30 on the use of AR on the training and delivery of care in the field of emergency medicine evaluated the outcomes of 24 papers across 3 themes; the user-environment interface, telemedicine and prehospital care, and education and training. Most of these studies were observational and had small cohorts but had a wide range of participants including nursing students, critical care doctors, and multiple specialties of surgeons. Overall positive effects were noted on improved practitioner engagement with training, reduced practitioner distraction, and cognitive load during procedures. However, issues including equipment cost and sterility were noted. All of the studies reviewed were in training only and none were carried out in the live patient environment.
Treating Patients
Off-the-shelf VR technology using simple headsets has promise for the prevention and treatment of health conditions, particularly in psychological care. The immersive and potentially entertaining nature of VR means that rehabilitation medicine has also been posited as an area which could benefit from VR technologies. In prevention, VR has been used by the military as part of predeployment training with the aim of reducing the incidence of post-traumatic stress disorder (PTSD). VR has been tested in real-life deployments of American soldiers to Afghanistan and Iraq as part of the Stress Resilience in Virtual Environments (STRIVE) project.31 Using VR headsets, soldiers are embedded in challenging combat contexts. The prevalence of PTSD in returning military personnel in the USA is estimated at 16.6%, substantially higher than the background lifetime population prevalence of 6.8%.32 The lifetime costs of PTSD are higher than other standalone mental health disorders and if predeployment resilience training using VR can reduce this, it is likely to be cost-effective.33
A systematic review of 50 studies by Valmaggia et al.34 demonstrated that VR offers a valuable method of assessing symptoms and the potential to facilitate learning new emotional and behavioural responses. One example of this can be found in a study in which 23 patients with arachnophobia were encouraged to gradually approach a virtual spider with increasing degrees of proximity. By the end, 83% of participants showed significant improvement in how they dealt with spiders, with some so desensitised they could approach real spiders without experiencing anxiety.35 Of equal relevance and despite the challenge of having to face their fears, there were no dropouts. Similar progress has been reported with the use of VR in tackling the fear of heights. A study of 100 subjects allocated participants between VR therapy and no treatment. All VR participants experienced a reduction in their fear of heights, with an average of 68% reduction in fears according to the Heights Interpretation Questionnaire (HIQ).36 More than 50% of participants experienced a 75% or greater reduction in their fears. This is comparable to, or better than, the best psychological intervention delivered face-to-face.36
Just as pertinently, while exposing those who have anxiety or post-traumatic disorders to powerful stimuli causing physiological response, therapists can be present alongside to guide, coach and, re-assure patients in situ. The effect or realism with VR allows a therapeutic window because scenarios can be repeated. The thought of facing a situation that triggers real physiological anxiety is never a pleasant one. The thought of facing it with support alongside, in what is recognised as a virtual scenario, is far more surmountable than it would be in real life.
The benefits of VR can be seen outside psychological therapy. In rehabilitation medicine, a study by Ho et al.37 in patients post-acute stroke showed that the use of VR in combination with conventional therapy improved outcomes compared with conventional therapy alone. The group of 100 case-matched patients receiving VR and conventional therapy had better functional independence at discharge and reduced medical costs. A 2018 study by Hanna et al.38 demonstrated the use of VR in remote supervision and annotation, and real-time pathology-radiology correlation during autopsy and examination of pathological specimens. VR has also been shown to enhance the enjoyment and intensity of physical activity, and by using omni-directional treadmills, potentially increase the activity of participants.39
ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
AI is the “theory and development of computer systems able to perform tasks normally requiring human intelligence.”40 It is necessary to understand exactly how AI and ML are individually defined. Deo defined ML in a 2015 paper as “the scientific discipline that focuses on how computers learn from data.” ML can be either supervised learning, which focusses on classification (e.g., sorting pictures into those with cats in and those with dogs in from a large set of images), and unsupervised learning. Unsupervised ML has no set outputs. The ‘machine’ seeks to find patterns or groupings within unsorted data.5 AI is the broader umbrella under which ML lies, and which encompasses the use of technology to perform tasks that would usually require human intervention including sensing, learning, reasoning, and taking action.6
The cumulative effects of Moore’s Law mean that we now have computational power in the palm of our hand which dwarfs that used to put mankind on the moon. This immense ability to perform billions of calculations a second allows the use of decades old mathematical tools to perform unique and highly accurate pattern recognition and analysis. In health data, this is allowing the ability to reveal associations and anomalies from previously thought random health data sets. Furthermore, the ability to create artificial neural networks and allow them to learn by trial and error, billions of times, creates software that evolves to solve complex problems in a nature akin to a child learning.
ML is already widely used in the consumer market, from recommendations by online film companies as to what film to watch next, to fraud detection and action. Self-teaching game-playing algorithms using re-enforcement learning can defeat the currently best-known chess playing program without any human intervention.41 AI and ML have an excellent performance in tasks involving image interpretation, suggesting that medical specialties such as dermatology and radiology are one of the most promising applications of the technologies.42 In dermatology, skin cancer is the most common human malignancy increasing in both incidence and prevalence. Current clinical reviews of simple skin lesions such as basal cell carcinomas and follow-up after a melanoma diagnosis require frequent follow-up in person. A convolutional neural network application demonstrated performance on par with all tested board-certified experts in classifying skin cancer.43
In radiology, it is expensive and time consuming to train radiologists, so radiographic image recognition is held to be one of the areas of AI ‘witness to the greatest gains’. In one study at Cornell University, Ithaca, New York, USA, a deep convolutional neural network was capable of automatically filtering CT head images and reporting with an error rate well below that for board-certified radiologists.44 A 2019 publication by Annarumma et al.45 found that AI systems can reduce the time needed for the most common radiological investigation, the chest radiograph, to be reviewed. Researchers collaborating with Guy’s and St Thomas’ NHS Foundation Trust, London, UK, used a data set of almost 500,000 anonymised adult chest radiographs to train an AI system to recognise abnormalities and then triage images for formal reporting, cutting the average delay in reporting from 11 days to 3 days. Normal chest radiographs were detected with a positive predicted value of 73% and a negative predicted value of 99%, and with a speed that meant abnormal radiographs could be prioritised to receive expert human opinion sooner. This is important in two ways: it recognises that AI systems presently have their limits and still require expert human input, and also that the use of AI can dramatically improve the service provided to patients.
A snapshot survey by the Royal College of Radiologists, London, UK, in 2015 estimated that at any one time in the UK, it is estimated >330,000 patients waited >30 days for their imaging to be reported.46 The situation has worsened with only 2% of radiology departments in UK hospitals in 2018 meeting their reporting requirements compared to 8% in 2014. Approximately half of departments reported leaving images auto-reported or unreported altogether, raising the spectre of potential delayed or missed diagnoses.47 An AI system capable of prioritising could make a significant contribution to improving the situation and reducing risk.
Moorfield’s Eye Hospital is an ophthalmological hospital based in London, UK. Clinicians at this hospital collaborated with DeepMind (London, UK) in using a deep learning architecture to interpret three-dimensional optical CT scans, reaching a performance level that met or exceeded that of experts after training on only 14,884 images.48 The next proposed step is to assess whether the same type of deep learning architecture would be able to diagnose conditions in advance of when a human expert would be able to, offering the option to assess the risk of complications before they formally present and stratify resources accordingly.49 The same team note that “volume and complexity of diagnostic imaging is increasing at a pace faster than the availability of human expertise to interpret it.”48 Saria et al.6 note that this is particularly true of low resource settings. The use of AI in this way offers a force multiplier for a limited workforce of clinicians.
In cardiology, there is increasing demand for imaging services such as echocardiography for the surveillance of heart function and valvular abnormalities. Commercial echocardiography software vendors have started incorporating automation to improve accuracy of images and to recognise required views.50 Deep learning has been tested at the initial step of evaluating imaging views, as well as for diagnosis in deliberately challenging images. The deep learning program was superior in terms of accuracy.51 Ultromics is a UK company spun out from the University of Oxford, Oxford, UK, which has developed AI capable of achieving better-than-human accuracy in interpretation of stress echocardiograms, at approximately 90% versus the 80% achieved with human interpretation.52 This has significant potential to reduce unnecessary investigations (initial interpretations are more accurate) and to reduce clinician time taken for interpretation of complex scans.
Beyond image analysis, there are ML algorithms in development to analyse physiological data and make predictions on when the patient may clinically deteriorate. These have proved to be extremely valuable and indeed picked up hitherto unknown patterns in patients’ physiology.53 One of the most recent applications of AI was in predicting early deterioration in renal function in patients developing acute kidney injury. Tomašev et al.54 used a data set from >700,000 patients in Veterans’ Affairs hospitals to develop an AI program capable of predicting 55.8% of all inpatient episodes of kidney injury, and 90.2% episodes which subsequently required dialysis. This was a retrospective study however, and the same program has not yet been validated in a live patient population.
Nonetheless, combining this approach with the exploding field of smart devices and the ‘always on’ connected world could reap enormous benefits in both health outcomes and resource efficiency. The ever-increasing volume of data in health, from key symptoms picked up in consultation, to blood test results and radiology investigations, are rich pickings for ML to assist with, including monitoring prognosis to suggesting possible diagnoses and treatments.
Patient monitoring is another application for AI. Oxehealth™, Oxford, UK, have created a medical device in Europe intended to perform spot observations of pulse and respiratory rate ‘contact free’ via optical and infrared sensors with no staff involvement. Its novelty means no published academic papers are available, but an audited case study of its use on an acute psychiatric ward showed improved in speed of observation collection and in the patient experience as they did not have to be woken to have their observations taken.55 Being woken to have observations taken can contribute to patients’ sleep deprivation and a poor patient experience.56
CONCLUSIONS
The technological advances of recent years have given us new tools and technologies to transform how we deliver healthcare. We can now educate and examine healthcare professionals with high fidelity immersive simulations at low cost and to a higher level of training. We can perform training and testing in human factors, tackling clinical errors not currently targeted effectively but that are associated with an almost ubiquitous element of bad outcomes. In healthcare delivery, no longer will the approach be central-down and population-based, but instead the patient becomes the centre of everything and the source of real-time data. The patient has the opportunity to become their own data controller, to contribute to their health analytics, and to leverage the power of smart devices. ML and AI algorithms can aid in prediction and diagnosis of disease, and for us as clinicians, enable the delivery of personalised healthcare in the home remotely using AR, with innovative VR treatments. In a world with a patient-centric record, data contributed comes from a wide variety of sources, forming part of our cultural heritage to improve health outcomes. The obstacles are no longer technical, but cultural and organisational. The era of immersive health is upon us.
