Virtual reality and augmented reality: Advances in surgery

Nearly 3% of the U.S.’s gross domestic product is spent on surgical care. Many are hopeful that technological advances will reduce morbidity and lower costs. One such technology that is beginning to play a role in many different surgical procedures is augmented reality (AR) where a virtual image can supplement the real-world scene to enhance surgery. In this article, we will discuss the role of AR in pre-operative planning and will discuss the types of AR used in surgery. We will examine the challenges of bringing AR into the operating room such as meeting the surgeons’ requirement of optimum ergonomic environments to exercise precise movements. Despite the challenges, many studies are showing significant benefits of the virtual image to the surgeon. As a result, more and more surgeons are attempting AR enhanced operations, even complex neurosurgical procedures. Correspondence to: David B Douglas, M.D., Department of Radiology, 300 Pasteur Drive, Room S047, Stanford, CA 94305-5105, USA, Tel: 650-723-4000; ext 14257; E-mail: ddouglas@stanford.edu


Introduction
According to 2015 data, the United States spends 17.2% of its gross domestic product on healthcare and 32.3% of that is spent on hospital stays [1]. Nearly one third of hospital stays include a surgery, and those procedures and post-operative care account for almost half of hospital costs [2]. Thus, nearly 3% of the nation's GDP is spent on care involving surgical procedures. Naturally then, there is a drive to improve efficiency in operating room, not only to improve patient outcomes, but also to bring down costs. Many factors play a role in determining hospital pricing including the imaging and lab work-up, medications, operating room instruments, post-surgical care, labor and specialist's fees [3]. Any advance in technology that decreases morbidity of the procedure, improves outcomes and decreases length of hospital stay will ultimately also have a beneficial outcome on a given hospital's bottom line. Technologies that hold promise in accomplishing these aims include virtual reality (VR) and augmented reality (AR) enhanced surgery.
VR technologies can be characterized as either non-immersive, such as desktop computers, semi-immersive or fully immersive VR [4]. In fully immersive VR, the head mounted display (HMD) presents a virtual image that completely occludes the real-world from the user's field of view, as is seen in the Oculus Rift and HTC Vive [5]. In semiimmersive VR, the HMD presents a virtual image that partially occludes the real-world from the user's field of view, as is seen in Samsung Gear VR [5]. In VR, the user can maneuver through the virtual world by head movements (via HMD tracking) or walking (via external camera tracking). The user can interact with the virtual environment through handheld devices with haptic feedback or voice gestures. However, one of the challenges of VR is lack of accurate head-tracking and motion sickness [6].
AR technologies can be characterized into AR or mixed reality (MR). Both AR and MR have a simultaneous display of a virtual image and a real-world image allowing the user to simultaneously interact with the real-world and the virtual image [7]. In both of these technologies, the user wears a HMD to display the virtual image and the real-world image. In AR, the virtual image is transparent like a hologram as is seen in Meta and DAQRI systems. In MR, the virtual image appears solid as is seen in the Microsoft Hololens.
Surgeons are increasingly recognizing the benefits of VR/AR. The objectives of this article are to provide a review of VR/AR technologies and techniques used in surgery, to illustrate by example VR/AR in preoperative planning, and to illustrate state-of-the-art display techniques available for surgeons. This paper will be divided into three sections. First, we will discuss the role of VR/AR in pre-operative planning. Second, we will discuss the role of VR and AR during surgery. Third and finally, we will conclude with a discussion on future uses of VR/AR in the field of surgery.

Pre-Operative planning
Pre-operative planning can be extremely complex. A surgeon must take into account the patient's medical condition (e.g., age, comorbidities, vital signs, labs, etc.), the anatomy based on physical examination, the patient's prior surgeries (if any), and pre-operative imaging to determine the surgical approach.

Vignette
A brain aneurysm is a balloon-like outpouching extending from a small blood vessel in the brain. The anatomy of the blood vessels of the brain is extremely complex and varies from one patient to the next, posing an additional challenge. If the aneurysm ruptures, the hemorrhage into the brain can manifest as a severe headache, coma or death. During the pre-operative planning period, the neurosurgeon must review all of the patient's pertinent clinical information and determine how to best care for the patient with surgical options including open surgery or closed endovascular surgery. In an open surgery, the surgeon makes an incision at the skin, performs a craniotomy through the skull, dissects down to the aneurysm and places a clip as a closure device so that the ruptured aneurysm sac stops bleeding. The neurosurgeon can also perform a closed endovascular procedure by accessing the femoral artery in the leg and traversing a catheter up to the brain where the aneurysm can be treated with a coil patch, which is analogous to patching a tire from the inside. How should the surgeon proceed?
Many times, such a decision relies on analysis of the precise anatomy of the aneurysm. Is it physically accessible from an open approach or is it in a location that is too hard to reach? How far and how deep will the dissection need to be? What is the width of the neck of the aneurysm sac like? Is it narrow where the coil patch has good purchase and can reliably stay in place? Or is it too wide such that the coil patch risks slipping out and risk either aneurysm bleeding or the coil patch causing a stroke by blocking a further downstream vessel? If there are multiple aneurysms, how does this affect the approach? Many questions run through the neurosurgeon's mind when making the decision on surgical approach.
Given the high stakes, complex anatomy and multiple possible treatment approaches, good pre-operative planning is essential. Many times, the most critical component of the planning is review of the preoperative diagnostic imaging. It is of paramount importance that the surgeon has the very best possible image quality and viewing methods. During the pre-operative imaging assessment, the surgeon can view the patient's images with conventional methods (e.g., 3 viewing planes including axial, sagittal or coronal imaging planes) or with advanced imaging methods to include VR/AR.

Conventional imaging-aided pre-operative planning
Conventional imaging includes axial, sagittal and coronal imaging planes. The surgeon reviews these 2D images on a high-resolution display monitor and mentally constructs a 3D image, which can be quite challenging [8,9]. As the complexity of the anatomy increases, it becomes ever increasingly difficult in mentally constructing the 3D image; yet the surgeon's mental map of the anatomy is critical for successful operations. The surgeon must also memorize the anatomy for recall of the precise details of the complex anatomy at a later time point while in the actual operating room. Thus, there has been an increasing trend for the use of advanced imaging in pre-operative planning.

AR/VR aided pre-operative planning
There are some FDA approved VR/AR systems for viewing of medical images. A quick view of the Brainlab Inc. website will give the reader a feel for the state of the technology. Navigation is a critical element of operations involving complex anatomy and there have been numerous approaches documented on ways to enhance that aspect of surgery. More can be done. As an example, VR/AR is currently undergoing research by DXC Technology (formerly Hewlett-Packard) and D3D Enterprises. As opposed to volume rendering images, which provide a 3D image on a 2D monitor, depth-3-dimensional (D3D) provides a unique image to each eye with accurate depth perception and capabilities for head tracking and joystick interface. This improves human machine interface [10][11][12][13]. See Figure 1.
Advantages of VR in pre-operative planning include an enhanced viewing of the complex anatomy. The surgeon experiences an immersive experience in viewing of the complex anatomy with 3D and depth perception. With external cameras, the surgeon could take a virtual walk through the patient's anatomy to better understand the anatomy. This would enable the surgeon to notice finer details that would not otherwise be noticed with conventional imaging. Furthermore, the surgeon may have a better ability to memorize the complex anatomy given the immersive experience and be able to recall it when in the operating room.
Segmentation is an important step in the AR/VR process. Through segmentation, organs can be isolated and individually examined. Segmentation can also help the surgeon visualize possible trajectories to pursue on the area being operated. Possible dissection pathways could be color coded, compared and evaluated. This will assist in simulation for the selected trajectory. False color can be added to indicate the need for caution for high-risk regions such as major vascular structures, which would be delineated through the segmentation process.
Multiple studies have demonstrated that virtual reality can improve the training and performance of intra-operative procedures. Surgical residents showed not only improved error rate when using virtual reality training to remove a gallbladder, but also shorter surgical time [14]. This translates into less time under general anesthesia and improved patient safety. One criticism of VR training has been that skills learned in the virtual setting do not necessarily transfer over to an actual surgical case. However, two different studies of 20 novice surgical trainees showed that basic surgical knot tying abilities and other basic laparoscopic skills could be significantly improved using only virtual reality simulators [15,16]. Again, this saves patients time under general anesthesia, improves safety in the training setting and reduces cost.
In the neurosurgical setting, very small or even microscopic distances can have profound impacts in critical outcomes, such as language processing and speech. The use of a virtual reality simulator has shown improvement in the learning curve for more complicated neurosurgical tasks, such as ventricular cannulation, again highlighting potentially improved patient outcomes [17].
Advantages of AR in pre-operative planning include similar advantages as in VR including depth perception, but also provide the ability to simultaneously view a real-world image. The real image could include dedicated pre-operative planning laboratories or even the actual patient at a time period prior to the surgery. VR/AR with D3D technology would be well suited in pre-operative planning as adequate image resolution, depth of field, depth of focus, field of view (FOV) and position tracking are achieved [5].

VR/AR technologies and techniques used to enhance surgical operations
Surgical procedures enhanced by VR/AR can be viewed as 'mixed procedures'. They are mixed in that on the one hand there is the reality of the patient present in front of the surgeon and on the other hand, there is VR/AR aspects, which involve use of data that generates the VR/AR environment. This creates both challenges and opportunities. The challenge is how does the surgeon best integrate the real and virtual environments. The opportunity created is potentially a safer and more efficient operation.
In recent years, VR systems have helped the surgeon navigate during the course of the operation. Some systems show in near real time the  showing the cerebral vasculature, which would be viewed with either VR or AR. Note the image on the left is left eye viewing perspective and the image on the right is the right eye viewing perspective, which provides depth perception. This depth perception cannot be appreciated on the paper format of this journal and requires a VR or AR headset. The red boxes represent the 3D cursors used. The white arrows point to middle cerebral artery branches, which course at varying depths.
location of surgical instruments with respect to the patient's anatomy. Example systems have a 2D display with 3D volume rendering, which the surgeon can refer to during the course of the operation. However, during surgery the surgeon's attention is primarily focused on the operating environment real image. AR is an excellent option given the simultaneous presentation of the virtual and real image.
AR has also been used in a wide variety of surgical procedures and is gaining significant popularity as it has helped make surgical interventions easier, faster and arguably safer [18]. Augmented reality has already been used for a wide variety of procedures, such as robotic liver resection, cholecystectomy (removal of the gallbladder), other laproscopic procedures and even neurosurgical procedures [18,19] • Salb [28] • Katic [29] • Sudra [30] • Bichlmeier [ Harders [35] Nicolau [36] • Nicolau [37] Albitar [38] • Commercial • Soler [23] • Soler [24] • Trevisan [26] • Birkfellner [48] • Katic [29]  The anatomy of the human body is complex and varies from one person to the next. Understanding the patient's unique anatomy is a major challenge in surgery and an opportunity for enhancement with image guidance. Specifically, AR can enhance the surgeon's understanding of the patient's anatomy during the surgery.
In this section, we will review the conventional operating room setup and four types of AR enhanced surgeries, including projection type AR, heads up display (HUD) type AR, operating microscope type AR and head mounted display (HMD) type AR.

Conventional operating room setup
The conventional operating room is equipped with an operating bed with an overlying bright light, table for equipment, anesthesia unit, multiple areas for storage, as well as a diagnostic imaging station. The diagnostic imaging station is typically positioned across the room from the operating table and the surgeon. The surgeon will alternate looking at the operating field of view and the diagnostic imaging monitor from across the room throughout the operation. Thus, surgeons perform innumerable switching -look up, look down, look up, look down, etc.

Projection type AR enhanced surgery
In projection-type AR, an image (e.g., CT scan) is projected by means of a mechanical arm and beamer directly onto the patient (e.g., skin surface) to co-display the virtual and the real-world images. Readyto-project images can be prepared within 10 minutes [18]. Landmarks such as the patient's umbilicus (belly button) have been used for registration [18]. The virtual image can be altered to display different anatomical elements, such as bone, blood vessels or internal organs. By a precise understanding of where the patient's organs lie beneath the skin, the skin punctures for the placement of the laparoscopic instruments or robotic trocars can be more precisely placed so that extreme movements of the instruments are avoided. Ultimately, patient safety is improved. If the abnormality that the surgeon is aiming to operate on is small, the overlaid images can be extremely beneficial for navigation. Furthermore, precision dissection through solid organs is advantageous in saving as much normal tissue as possible.
In one study by Besharati et al, a series of 10 brain tumors of different sizes and location were visualized by projection type AR with an image     projected directly onto the patient's skin [19]. The video projector was used in concert with 5 fiducial markers for spatial registration, with a mean time of image registration of 3.8 minutes and a mean projection error of 0.8 mm +/-0.25 mm [19]. In this study, there was no significant difference between the accuracy of the projection type AR system and standard neurosurgical navigation systems [19]. There is, however, a significant problem with this technique, which is parallax. In a letter to the editor of Journal of Neurosurgery, Ferrari points out the surgeon's viewing angle cannot be co-incident with that of the projected image and thus parallax occurs [22]. The parallax error increases with the depth of the region being operated on and with surgeon/projector misalignment. However, there have been other approaches used to correct for this parallax problem [50,51]

Heads up display (HUD) type AR enhanced surgery
In the HUD type AR enhanced surgery, the surgeon looks into a HUD containing both the real and virtual images. Some of the HUDs contain semi-transparent screens such that a portion of the natural light from the real image is seen along with the virtual image displayed on a hologram [52]. Other methods use a digital display that includes both a real-world image provided by a video camera and the virtual image presented on the same digital display.
In one study by Marker et al, a total of 23 bilateral paravertebral sympathetic nerve plexus was injected using HUD type AR into the thoracic, lumbar and hypogastric regions [53]. In this study, 46 of 46 (100%) of injections were on target. The mean error at the needle tip was 3.9 mm +/-1.7 mm and no critical non-target structures were hit.

Operating microscope type AR enhanced surgery
In operating microscope type AR enhanced surgery, the surgeon views both the real image and the virtual image through the operating microscope. The operating microscope is commonly used in a wide variety of surgeries today in the fields of neurosurgery, ophthalmology, otolaryngology and plastic surgery. The AR-display system can be integrated into an operating microscope, which surgeons are already accustomed to using during operations [32]. The operating microscope is mounted to a stable gantry; thus, changes in the relative positions between the surgeon, microscope and patient are minimized and the requirement for real-time head position tracking is minimized [48].
Since the operating microscope is already accepted in the operating room and the position of the operating microscope is relatively fixed, the operating microscope is an excellent method to further introduce AR into the field of surgery [48]. In a study by Raya et al, the traditional optical operating microscope was replaced with a digital operating microscope [54]. The neuronavigator virtual image was overlaid onto the real image of the surgical field of view to assist the neurosurgeon in the operation and the initial trials performed in a laboratory were successful.

Head mounted display (HMD) type AR enhanced surgery
Prior to discussing HMD type AR, we will provide a brief background on the surgeon's visual environment while in the operating room. It is a requirement for surgeons to wear goggles during a surgery to maintain a sterile operating field and to protect against possible blood or bodily fluid splashes into the eye. Some surgeons wear eyeglasses to meet both personal vision requirements and the protection/sterility requirement. For finer detail, many surgeons use magnifying surgical loops, which provide greater detail of small anatomical structures. Thus, all surgeons are accustomed to using eyewear while operating.
In HMD type AR enhanced surgery, the surgeon uses a HMD with computer-generated virtual image superimposed over the realworld scene of the patient's anatomy during the surgery [48]. One of the requirements is that the virtual image and real-world scene of the patient's anatomy must be appropriately registered with their positions aligned [48]. This requires accurate tracking of the AR user's HMD position and orientation with respect to the real-world scene [48]. This enhances a surgeons visualization of surgical anatomy [55] and enables the surgeon to focus on the operating field without the hindrance of switching back and forth to the monitor displaying the radiology images.
In HMD type AR, the user wears a HMD containing an AR-display system, which has historically been bulkier than the surgical loupes. The early HMDs displayed the virtual image and the real-world image in different focal planes and therefore could not both be in focus at the same time [48,56]. Complex systems utilizing a video camera solved the problem of differing focal planes, but suffered from heavier weight, parallax effects, and lower quality of the real-world image [57]. Fortunately, these problems have been solved through the use of miniature head-mounted binocular AR systems, such that the virtual image and the real-world images can be merged [48].
Two key challenges are facing HMD type AR. First is the challenge of surgeon acceptance of HMD into the operating room. Surgeons are accustomed to wearing glasses, but experience has suggested that bulky HMDs will not be accepted by surgeons [18]. Thus, improvements in making HMDs lighter and smaller are critical for advancing the field of HMD type AR surgery. Second is the requirement for synchronization of head position tracking so the virtual image displayed to the user aligns with the real-world image. Alignment preservation is necessary for precision of surgery and prevention of motion sickness [58]. Associated with this synchronization is the challenge of displaying a new virtual image at a frame rate adequate to match the changing realworld image. References or fiducial markers have traditionally been used to optimize registration of the two co-displayed images.

AR technologies in Laparoscopic/Endoscopic Surgery
Bernhardt et al. provides a comprehensive review of the latest techniques in augmented reality enhanced surgery [59]. Because laparoscopic/endoscopic surgeries are performed on non-rigid, hollow structures, the majority of laparoscopic/endoscopic surgeries do not involve the use of image-guidance for surgery since registration of the virtual image and the mobile organs in the operating field of view becomes exceedingly difficult. As a consequence, the majority of laparoscopic surgeries performed do not use AR for guidance. Surgeons typically rely solely on the laparoscopic image, but can refer to pre-operative imaging displayed separately.
As an example, laparoscopic liver surgery is inherently dangerous because large blood vessels can easily be hit and bleeding can be severe. Phutane et al. has performed preliminary trials of using an augmented reality guidance system (ARGS) in the assessment of hepatocellular carcinoma after it was resected laparoscopically [60]. In this trial, the authors found that the laparoscopic system aided in the finding of the transection plane, the nearby hepatic vein and the tumor. The authors have done 8 similar cases of viewing the tumor and now feel that the ARGS should be attempted intra-operatively.

Important considerations during AR enhanced surgery
The virtual image used in AR is most commonly a pre-operative image. Given the simultaneous presentation of both the virtual image and the real-world image, AR can be effective; however, the surgeon must proceed with the understanding that the virtual image represents the anatomy at the pre-operative state. New changes in the patient's anatomy (e.g., partial resection of tumor, new areas of bleeding, etc.) would not be reflected in the virtual image. While AR enhanced surgery may be appropriate for the beginning of an operation, it may not be as ideal for the mid-portions or later portions of an operation, given the fact that the patient's anatomy has changed. Intra-operative CT or fluoroscopic imaging would be beneficial to track changes in the patient's anatomy and condition during the surgery and could provide an updated virtual image. For example, it is possible during the course of the operation to employ 3D rotational angiograms under fluoroscopy to determine how the lesion (e.g., cerebral arteriovenous malformation) may have changed so that a more accurate registration can be obtained.
An additional consideration is registration of any projected imagery with that the real-time image the surgeon sees during the operation. For non-rigid soft tissues, registration is significantly more difficult. There are techniques such as deformable registration, which can be applied to reduce the error.

Future
In the future, pre-operative planning performed with advanced imaging techniques of VR/AR will be intimately linked to the AR enhanced surgery. The radiologist and surgeon will flag certain aspects of an operative site to be noted during the surgery. Examples of flagged items include the neck of the aneurysm, the boundaries of the tumor, a fragile blood vessel in close proximity to the operative site, certain landmarks in complex anatomy, and critical brain structures. These structures will be flagged to the surgeon during the operation through AR and the surgeon can be reminded of critical tasks while in the operation.
As the field of surgery enhances through methods such as robotic surgery with refining movements through scaling down the magnitude of the gestures, the role of AR will be ever increasing. Undoubtedly, multiple types of intraoperative AR will be used, but the HMD type offers depth perception, head tracking and improved HMI to facilitate the surgeon navigate the complex challenges. Now is the time for advancing research with VR/AR to improve surgical success.

Conclusions
Surgeons face significant challenges of how to operate on complex anatomical structures. Pre-operative planning is progressing with advanced imaging techniques including VR and AR, which hold promise in superior appreciation of finer details of complex structures and better surgeon recall of the complex anatomy while operating. AR enhanced surgery with simultaneous display of real and virtual images will be important in improving surgical outcomes. It is hopeful that continued advances in virtual and augmented reality technologies, as applied to surgery, will decrease morbidity, decrease length of hospital stay, improve patient outcomes, and also decrease overall hospital expenditures.