VIRTUAL SURGERY

1. INTRODUCTION

Virtual Reality (VR) is a widespread topic not only in computer graphics but also in many other

fields. In this paper, virtual reality and Augmented Reality(AR) in medicine especially in virtual

surgery is taken into consideration. This survey consists of a systematic collection of existing

methods and techniques for virtual surgery using Augmented Reality. The motive for doing this

study was the current stage of AR technology in virtual surgery. To figure out such problem, this

study attempts to critically review the application and limitation of well-known methods to

provide researchers with backgrounds on outdated and latest AR technologies in virtual surgery. In

this study widely used styles, which have been employed in virtual surgery, is classified and

systemized. The direction that each subject is going is introduced. We hope that this study could

help researchers to figure out what stage are the current AR researches not only in virtual surgery

but also in other parts of medicine.

The use of new technologies is well established for all areas such as industry,medicine, military

and etc. One of the latest technologies, which have been entering to medicine, is Mixed Reality

(MR). Mixed Reality is an integration of real and virtual world. Augmented Reality (AR) is a real

environment with some virtual objects, which are augmented in the real world. Virtual Reality

(VR) is a technology that allows users to interact with computer-simulated environments. Most

current Virtual Reality environments are essential for users to interact with video or audio using

tactile sensors.

Virtual Reality and virtual environments among the computer simulators for virtual surgery have

not been widely sued in professional surgery but they are incorporated into training and education

of surgery. Most of professional surgeons believe that this type of technology is very beneficial for

surgical education. The main problem is lack of information regarding related virtual reality and

augmented reality in the field. The developers of this new technology should focus on training and

CHINMAYA INSTITUTE OF TECHNOLOGY 1

 VIRTUAL SURGERY

educator users to figure out this concern. Currently, virtual reality is widely used in medicine most

of all for training. In medicine teachers can train students in virtual environments before doing

the surgery on real patients.

Advancements in computing power have enable continued growth in virtual reality, visualization,

and simulation technologies. Virtual reality is defined as human computer interface that simulate

realistic environments while enabling participant interaction, as a 3D digital world that accurately

models actual environment. The term virtual reality was coined by Jaron Lanier.

CHINMAYA INSTITUTE OF TECHNOLOGY 2

 VIRTUAL SURGERY

2. HISTORY OF VIRTUAL REALITY

The use of the term “virtual reality,” however, was first used in the mid-1980s when Jaron Lanier,

founder of VPL Research, began to develop the gear, including goggles and gloves, needed to

experience what he called “virtual reality.”

Even before that, however, technologists were developing simulated environments. One milestone

was the Sensorama in 1956. Morton Heilig’s background was in the Hollywood motion picture

industry. He wanted to see how people could feel like they were “in” the movie. The Sensorama

experience simulated a real city environment, which you “rode” through on a motorcycle.

Multisensory stimulation let you see the road, hear the engine, feel the vibration, and smell the

motor’s exhaust in the designed “world.”

Virtual Reality and virtual environments among the computer simulators for virtual surgery have

not been widely sued in professional surgery but they are incorporated into training and education

of surgery. Most of professional surgeons believe that this type of technology is very beneficial

for surgical education. The main problem is lack of information regarding related virtual reality

and augmented reality in the field. The developers of this new technology should focus on training

and educator users to figure out this concern. Currently, virtual reality is widely used in medicine

most of all for training. In medicine teachers can train students in virtual environments before

doing the surgery on real patients. In addition to real surgery and air flight, there are some other

fields such as in which nuclear power plant operation, simulators for military and mechanical

system maintenance needs virtual reality for training and education. These kinds of fields

represent high-risk performance in real actions in which a small failure may face a big result. One

of the new concept that recently has been defined is Computational Surgery in which the

CHINMAYA INSTITUTE OF TECHNOLOGY 3

 VIRTUAL SURGERY

application of mathematics and algorithm design, enabling imaging, robotics, informatics, and

simulation technologies, incorporating biological and physical principles, to improve surgery.

Fig 2.1 : Virtual Reality in medicine.

CHINMAYA INSTITUTE OF TECHNOLOGY 4

 VIRTUAL SURGERY

3. MEDICAL IMAGES

Although medical imaging has compiled about the date yet, we can take a look at the role of

scientists and engineers in physics and electronic. The first modern imaging techniques and

modalities can be referring to the discovery of X-rays by William Roentgen in 1895. More

advanced efforts to the radiography imaging systems such as X-ray screens of the resonator,

degree course, etc. occurred around 10–20 years later but the main improvements is originated

from the laboratory people compare to engineers in physics or electronics. A big revolution on

imaging took placed from the 1950s until the end of 1970s. Many new systems for anatomic

imaging and the process of the disease were developed in this era. This revolution began with

nuclear imaging and ultrasound inspite of the serious limitations on imaging. Computed

Tomography (CT) was another invention in medical illustration, which was introduced in the early

1970. By this technique the cross-sectional images came very good, which correspond to the

information obtained from the exploratory surgery. Virtual Reality Applications in surgery helps to

understand three-dimensional complex structures. Virtual reality applications can be categorized

into four types which are training and education, surgical planning, image guidance, and tele-

surgery.

CHINMAYA INSTITUTE OF TECHNOLOGY 5

 VIRTUAL SURGERY

4. VIRTUAL REALITY IN MEDICINE

In the last two decades, Augmented REALITY has been grown up in the area of medicine

dramatically. Augmented Reality including 3D objects and images equipped by sound and lighting

is increasingly being used not only for medical training and education but also in treatment and

diagnosis. Professional study on MRI (Magnetic Resonance Imaging), CT Scan and X-Ray is one

of the highlighted and widely used of virtual reality in medicine. Moreover, awareness on different

types of diseases is one other mainly used of virtual reality in medicine.

4.1 2D Images

Technique and process used to create images of the human body or its parts and its functions for

clinical purposes including medical procedures seeking recognition,treatment, and evaluation are

the disease or medical science includes anatomic and physiologic studies is called Medical

imaging. It is a conflict of several branches of science, such as medical physics, medical, biology,

engineering and optics. 2D images are one of the oldest human inventions in technology

century.Various different types of 2D images in medical are widely used more than 4 decades.

Radiography including projection radiography and fluoroscopy, Multi-Planar Rendering (MPR),

Nuclear medicine, Elastography, Tactile imaging, Photoacoustic imaging are some samples of 2D

images which are using in medical.

CHINMAYA INSTITUTE OF TECHNOLOGY 6

 VIRTUAL SURGERY

Fig 4.1 : 2D Image of a Brain

4.2 3D Images

Tele-surgery allows surgeons to operate on people who are physically separated from themselves.

This is usually done through robots provided with video cameras. More advanced system has been

used to perform Coronary Anastomosis on ex vivo swine hearts and in human undergoing

endoscopic Coronary Artery Bypass grafting. 3D images are the advanced images, which show

whole or some parts of body in three dimension. Body imaging can be included of CR Urography

and CT Colonography. Cardiovascular Imaging consists of some different parts such as Coronary

artery CT angiography, Vascular CT and MR angiography of the aorta,upper and lower

extremities, Cardiac MRI and MRA and Pulmonary vein and left atrial imaging for EP planning.

Neuroradiology is another part, which includes of CT and MR angiography of the head and neck,

MR spectroscopy of the brain and CT and MR brain perfusion. Musculoskeletal Imaging which is

CHINMAYA INSTITUTE OF TECHNOLOGY 7

 VIRTUAL SURGERY

3D reconstructions for preoperative planning. Last but not least is Thoracic Imaging which

includes 3D airway reconstructions and Lung nodule volume.

Fig 4.2 : 3d image of a brain

CHINMAYA INSTITUTE OF TECHNOLOGY 8

 VIRTUAL SURGERY

5. 3D VIRTUAL BODY STRUCTURES

A 3D virtual body is usually needed to be observed each part o body in any clinic or policlinic.

The 3D model cannot reveal all part of body. Thus, a 3D virtual body is more useful to be used for

this matter. Through this virtual model the doctor can interact with each part of the 3D model and

explore where the problem is,

Fig 5.1 : A virtual image of skeleton is projected on to a person’s body(right),surgeon is

performing surgery by watching the computer screen

Virtual Reality Medical Center (VRMC) in San Diego produced some related application in this

area. Pain Management which is a pain distraction application. Combating Phobias using

cognitive-behavioral therapy (CBT) in virtual reality to treat patients with a number of phobias.

CHINMAYA INSTITUTE OF TECHNOLOGY 9

 VIRTUAL SURGERY

6. DIFFERENT GENERATIONS OF SURGERY

6.1 Early Times

“Deformable model”is a term that refers to the mid of eighties regarding to the computer

imaging . Monserrat et al. in 2005 divided the deformable models of existing simulation for

surgery into three categories. Heuristic, hybrid and continuum mechanical were the all existing

model which belonged to the first generation at that time. All related approach to spring-masses

were categorized into the heuristic group. Most of the approaches at that time were included in the

continuum mechanics which were based on linear elasticity . One of the approaches that deserves

a special comment is surgical cutting.Continuum-based simulations have been preferred to be used

for GPU implementations and X-FEM technique from the time of cutting spring-mass systems.

6.2 Refining the Appearance

As the Early Times past quickly, some refining needed to be considered such as involving the

existing simulators with realistic sensations . One of the high importance model is Saint Venant-

Kirchhoff (SVK) which were not unstable under compression but has been used for many recent

works .Point-Associated Finite Field (PAFF) was developed which could approximated the large

deformation fields such as surgical tools .Deo presented a system which is trained with a wide set

of possible load states in the case of real-time performance based on neural network for execution

loop.

6.3 Adding Physiological Details

The third, fourth and fifth generation aimed to add more and more details. In addition of

quantitatively, realising the simulating physiology of human is a challenging part of virtual

surgery since two last decades. A clear paradigm can be observed in the project of Physiome .

On the contrary of physiology in simulation which was not strictly required,surgical planning was

in order . Guiatni et al. developed a new haptic simulation consists force and thermal feedback

CHINMAYA INSTITUTE OF TECHNOLOGY 10

 VIRTUAL SURGERY

which are very useful for the location of tumors. However, physiology in simulation is not

avoidable. To predict the outcome of vein graft surgery a physiology in simulation is taken into

account.

CHINMAYA INSTITUTE OF TECHNOLOGY 11

 VIRTUAL SURGERY

7. SURGICAL SIMULATOR REQUIREMENTS

7.1 Data Acquisition

Data collection is the first step to create a realistic model of the organ. An accurate data is required

to be able to show the organ like what can be seen and interacted with real one. In Virtual Reality

systems usually a realistic model is needed to be replaced by the real organ. The reaction and

movement of the model must be the real one. This requires an accurate date.

7.2 Imaging Modalities

The integration of advanced imaging technology, image processing and 3D graphical capabilities

has led to great interest in image guided and computer aided surgery. Navigation in surgery relates

on stereotactic principles, based on the ability to locate a given point using geometric reference. It

also proved useful in Robotic Surgery, a new technique in which surgeon remotely manipulate

robotic tool inside the patient body.

Variety types of organs need different type of data. Most of the organ issues can be diagnose using

different imaging modalities. Currently, there are some image modalities which the main ones are

as follows.

CT: CT or Computer Tomography usually gives a fast and obvious diagnosis of many different

conditions.

PET/CT: PET/CT is an integration of PET (Positron Emission Tomography) and CT which gives

images illustrating active pathology and anatomical location of astonishing.

MRI: MRI or Magnetic Resonance Imaging which is a magnetic resonance imaging. It is based

on magnetic resonance core. Currently MRI is widely used in medical and veterinary diagnostic.

Cardiac MRI: Cardiac MRI creates both still and moving pictures of your heart and major

blood vessels. The preferred investigation choice for many clinical indications and applicable for

cardiovascular system. Different types of heart diseases can be diagnosed by Cardiac MRI like:

CHINMAYA INSTITUTE OF TECHNOLOGY 12

 VIRTUAL SURGERY

Coronary heart disease, Heart failure, Damage caused by a heart attack, Heart valve

problems,Cardiac tumors, Congenital heart defects and Pericarditis.

X-ray: X-ray is one of the oldest image in medical. It creates a picture of the inside of the body

focusing on the bones.

Ultrasound: This method is based on ultrasound for subcutaneous tissues such as muscles, joints,

tendons and the internal organs of the body. There is also a widely used of ultrasound in

pregnancy.

Obstetric and maternal foetal ultrasound: This technology allows observing real-time 3D and

4D views of bone densitometry or DEXA.

Fluoroscopy: Fluoroscopy is a technology that creates a real-time moving image using X-ray and

reduces the patient’s radiation dose. This technology allows to scan all areas of the body in all

directions. Currently, Fluoroscopy is used in conjunction with MRI for MR Arthrography.

7.3 Segmentation

Segmentation is extracting and characterizing features of available data which are obtained from

patient. There are some different types of segmentation and they have been classified as structural

and statistical methods . Fast and accurate segmentation are more important to create VR-based

intra-operative surgery

7.4 Fusion of Multi-Modality Data

Fusion of different types of data is another part of some surgical simulators. By fusing and

registering multi-modal images multidimensional properties of human organs cab be created.

7.5 Registration

A common reference frame process between pre-surgical data and the corresponding patient

anatomy is called Registration. This pre-surgical data can be used in intra-operative when the

reference frame is established . Thus, the pre-surgical data can visualize the anatomical structures

CHINMAYA INSTITUTE OF TECHNOLOGY 13

 VIRTUAL SURGERY

not only for intraoperative data,but also for may other purposes such as surgical equipment, guide

a surgeon’s tool movements, position radio and guide robotic tool movement.

7.6 Modeling

One of the main steps regarding to have a realistic VR-based application in surgery is realistic

modeling. There are two different methods to creating the model. The first one is modeling based

on current data from existing images like MRI, CT and PET. As this method reconstruct the model

based on the real data it is accurate enough but it is different from different patients. The second

one is modeling using handcraft modeling tools. Kaye et al. presented a mechanical

cardiopulmonary which is interactive in 3D virtual environments. Sorensen et al. developed a

realistic model of cardiovascular which is interactive to be used for training of virtual surgery. The

purpose of this simulator was not only illustration of various elements of heart surgery but also

allowing surgeons to rehearse difficulty of open heart surgery before any cardiovascular surgery.

The morphologically model is reconstructed based on from 3D MRI data. Reconstruction of the

morphology for any patient is employed based on an post-processing invent. . In preoperative

planning process, an accurate reconstruction of intra and extra-cardiac morphology can be

integrated while computer visualization hardware allows interactive elastic tissue deformations

simulation. Different surgical strategies can be evaluated pre-operatively by visualizing the

morphology in respect for each incision.

7.7 Interaction

There are many different data which can be extracted from the patients. Variety of data in VR

systems needs and interactive application to be navigated by users or surgeons. Collision detection

and position tracking are the concepts that are in order to be employed in many of the surgical

applications. Interacting with different parts of body or even different parts of a system like

CHINMAYA INSTITUTE OF TECHNOLOGY 14

 VIRTUAL SURGERY

cardiovascular system needs tracking technique to be navigated by surgeons through some many

devices like optical, mechanical, magnetic and acoustic.

Fig 7.7 : A person performing virtual surgery

CHINMAYA INSTITUTE OF TECHNOLOGY 15

 VIRTUAL SURGERY

8. AREAS OF VR SIMULATION

8.1 Training

Training and education simulators have their own strategies and protocols. Level of difficulty and

psycho-motor skills are the main parameters which are taken into consideration. A VR-based

application is useful due to avoiding andy dangers from the patients. Training applications usually

have an elevation part for any step of training . However, the main challenge is to produce a step

with adequate fidelity to be simulated like what is performing on the patients. The other ability of

training and education simulators is preparing a situation for research surgeons to simply practice

what has been suggested by other colleagues and even reading form websites. In other word,

development of innovative for surgical instruments can be employed using training and education

simulators easily. For a more realistic and natural surgical training scenario, the position of the

thorax should be addressed. Young surgeons are allowed to do numerous amount of surgical

training scenario with limitless trial so that they can learn more from their mistakes.The time for

the training could become shorter and thus the cost will become lesser. For this project, to evaluate

potential corrective strategies for patients three-dimensional magnetic resonance imaging (3D

MRI), incision simulation has been performed.

Traditionally, textbook images or cadavers were used for training purposes. VR simulators allow

users to view the anatomy from a wide range of angles and “fly through”organs to examine bodies

from inside. The experience can be highly interactive allowing students to strip away the various

layers of tissues and muscles to examine each organ separately . Piromchai has reviewed the

virtual reality surgical training in ear, nose and throat surgery. The details include temporal bone

dissection surgery, middle ear surgery for myringotomy, Benign Paroxysmal Positional Vertigo

(BPPV) surgery,cochlear implantation, Endoscopic sinus surgery, phonosurgery, laryngeal

surgery,cricothyroidotomy, and bronchoscopic surgery.

CHINMAYA INSTITUTE OF TECHNOLOGY 16

 VIRTUAL SURGERY

8.2 Diagnosis and Pre-operative Planning

An accurate and reliable method for diagnosing the problem based on patient date can be provided

using VR-based visualization. In this case, study on the patient data before doing the operation is

the best way to figure out the surgery. Accurate data from the specific patient should be

automatically analyzed to be considered for diagnosing the issue before operating. To do the

preoperative planning of the heart surgery, the patients have to be scanned through the 3D MRI,

which was performed on an Intra 1.5T scanner (Philips Medical Systems, Best, Netherlands) .

Then, the virtual model of the myocardium and vessel borders were constructed using the “Virtual

Reality Heart”software (Systemic Software Engineering, Denmark) and also a custom software.

The elastic properties then applied to the tissue and have been described . In traditional surgery

planning, the surgeon calculates various parameters and procedure for surgery from his earlier

experience and imagination. This leads to lots of errors and even to the risk of losing the life of the

patients. The incorporation of the virtual reality techniques helps in reducing the errors and plans

the surgery in the most reliable manner. 3D reconstruction has proven particularly useful in

planning stereotactic and minimally invasive neurosurgical procedures.

8.3 Intra Operating

Accurate localization of the place of many different types of surgery on the tissue is needed for

adjacent the structures. Doing a test on the exact location is most of the time impossible or

difficult to do especially when the structure under treatment is deep. In these cases Intra-Operative

Assistance is more helpful and can be transferred to the operating rooms. Augmented Reality is

more suitable for these types of pre-operative surgical to guid surgeon using the pre-operative data

for intra-operative data.

CHINMAYA INSTITUTE OF TECHNOLOGY 17

 VIRTUAL SURGERY

9. ASSISTANCE SURGERY TOOLS

Currently, Image-guided surgery and Robotically-assisted surgery are the main tools in surgery

rooms. Image-guided has been used for many years while robotically-assisted surgery has been

sued for some decades. Surgeons can use VR-based systems with visual access of the location into

the location of the tissue. Through the assistance surgery tool the surgeons can access many parts

of patient inside. Collaboration and telemedicine are some of the extension of this tools which

can be used to remove the distance between surgeons and make it easy to communicate and

sharing the information for whole around the world. To give a 3D overview of the morphology

process, virtual reconstructions were introduced as a new supplementary tool to assist the process .

By using two Phantom Omnis by Sensible Technologies, two-handed interaction was achieved

which providing the exact position and orientation of the 3D hands. To provide a natural

sensation, the devices have been programmed to give feedback force when the 3D hands touching

the reconstructed 3D virtual heart which allow the user to feel the magnitude of the forces applied

during the simulations.

9.1 Touch Simulation

A haptic-touch sensation is simulated. Physicians rely a great deal on their sense of touch for

everything from routine diagnosis to complex, life-saving surgical procedure. A low update rate is

needed for soft tissues whereas hard tissues requirehigh update rate. For simulating touch

sensation, we have to calculate the forces applied to cut, puncture the various tissues. Physical

model is made assuming that tissues are polygon meshes that interact like an array of masses

connected by springs and dampers. The parameter values are derived using complex nonlinear

equations and the reaction forces are also calculated. The equations to solve such a complex

problem are known, but so far the calculations cannot be made fast enough to update a display at

CHINMAYA INSTITUTE OF TECHNOLOGY 18

 VIRTUAL SURGERY

30 Hz. Most difficult to simulate is two floppy objects interacting with each other, since the

mechanics of such interaction are complicated, because each object may deform the other.

Fig 9.1 : Touch Simulation

CHINMAYA INSTITUTE OF TECHNOLOGY 19

 VIRTUAL SURGERY

10. VIRTUAL REALITY AND AUGMENTED REALITY SIMULATORS

10.1 Virtual Reality

It consists of a powerful PC which runs the software and an interfacer-haptic interface. Usually

the haptic interfacer works on force feedback loop. Haptic interaction is a new field of research

that adds the sense of touch to virtual environments. Some haptic devices include joysticks, key-

board and gloves that all provide the user with some kind of sensation. The addition of haptics to

virtual environments has provided computer users with the ability of expression in multiple

dimensions.

A human hand moves the end effecter, causing the device to relay its position via sensors to a

computer running a VR simulation. In the left hand loop, forces on the end effecter are detected

and relayed to user’s brain. Force feedback occur in real time to convey a sense of realism. The

haptic device’s driver card plugs into usually a 500 MHz PC equipped with a standard graphic

card and a regular color monitor. The size is of desk-lamp with workspace of 6 ×5×5 in. It has a

robotic arm and two arms. One hand is used to interact with virtual world and another hand is to

touch and feel the virtual object. In addition, it has the position sensing with 6 degrees of freedom

and force-feedback with 3 degrees of freedom.

A software package aptly named GHOST, translates characteristics such as elasticity and

roughness into commands for the arm, and the arm’s actuators in turn produce the force needed to

simulate the virtual environment. Among the medical procedures that can be simulated are

catheter insertion, needle injection, suturing and surgical operations. A number of research works

have been done by using this interface . Different parts of human body make different application

to simulate each part. A taxonomy is proposed by which is based on the organ. You et al. and

simulated the Prostate. For Blood Vessels simulation there are many efforts for example .

Neurosurgery is studied for brain simulation such as . Face simulation which is called

CHINMAYA INSTITUTE OF TECHNOLOGY 20

 VIRTUAL SURGERY

Maxillofacial Surgery is another topic of these typemof application . Cardiovascular simulation is

the one which is taken into consideration more due to high risk level of this surgery . Paranasal

sinus surgery is the simulation for Nose which make it easy to interact with nose before surgery.

Liver is also taken into account.

Fig 10.1 : Haptic device

CHINMAYA INSTITUTE OF TECHNOLOGY 21

 VIRTUAL SURGERY

11. AUGMENTED REALITY

Blum et al. proposed a new system that allows the user to see a CT dataset that is augmented onto

the user’s body, creating an illusion that the user can see through the body. By using a depth

camera, the system can track the user’s pose when standing in front of the display. Different slices

from the CT and photographic dataset can be selected by using body gesture. 3D models of

organs, information and images of the organs can be showed by the system and can be interacted

using a new interaction metaphor, frosted glass interaction metaphor, which is manipulated by the

distance of the user from the depth camera. There are several methods that have been

implemented in the system. The first one is AR in situ visualization of human anatomy. The

concept of a mirror is implemented by flipping the captured images from the camera horizontally

and displayed on the screen creating a perspective that the user is standing in front of a mirror.

Then, the augmented organs are visualized onto the flipped user’s body. The concept is the same

with augmented shirts , shoes or knight’s armor which has been implemented previously. The

second method is gesture-based interaction for slices. A green rectangle will be augmented onto

the user’s body.The slice of the CT dataset that is being displayed on the screen is relatively with

the position of the green rectangle. User can control a transverse slice mode by moving the right

hand up and down. To switch to sagittal slice mode by moving right hand from left to right. The

third method is the frosted glass metaphor. By utilizing the function of the depth camera, this

method can be implemented to create a touch-free interaction.The frosted glass metaphor is the

closer an object to the frosted glass, the clearer the object can be seen. This will allow additional

information being displayed on the screen . The system has produced a large amount of positive

feedbacks from the users especially the children. The AR in situ visualization attracts people

attention and most users spend longer using the volume slicing feature to understand the human

anatomy. Blum et al. have proposed a system that allows the user to control the augmented reality

CHINMAYA INSTITUTE OF TECHNOLOGY 22

 VIRTUAL SURGERY

visualization by using brain signal and gaze through a brain-computer interface (BCI) and a gaze-

tracker. Data and feedback from medical doctors (MD) are gathered to access the potential of this

technology. By using Neural Impulse Actuator(NIA) by OCZ Technology, a head-band that can

read alpha and beta waves, electromyographic (EMG) and electrooculographic signals.To avoid

the MDs learning phase, alpha and beta waves is not used although the waves can control

additional parameters . Blum decided to focus on the use of EMG signal since it is the easiest yet

robust to control. A medical AR system that uses and optical tracking system and a video see-

through HMD to augment a volume rendering of a CT on a patient is the first system that applying

the integration of BCI and gaze-tracker.The second system that implemented the integration is a

monitor-based medical AR system where an optical camera is mounted beside a mobile X-ray

device, called CamC. Both medical AR system use brain signals and gaze for the system

interaction since touch-screen during a surgery is difficult as the surgeon has to stay sterile .This

system is mainly on the implementation of the integration of a BCI and a gaze-tracking in medical

AR system. As a, the use of BCI for non-AR application has higher rating (4.0 0.7) compared

with HDM-based (3.6 1.0) and monitor-based (3.8 0.7) AR. The use of gaze-tracker and EMG

signal are also highly rated which are 4.7 0.7 and 4.2 1.0 respectively.

Ma Meng et al. and his group have purposed a magic mirror for anatomy learning and have

integrated it with a depth sensor, Kinect. The improvements affect the quality of education and

learning of anatomy. While standing in front of the sensor, the user can touch the defined bone

landmarks on their body easily. It is very challenging to integrate virtual objects and real world,

which needs accurate calibration, advance virtualization and user interfaces . For current AR

magic mirror system, Mirracle, virtual anatomy information is augmented on the user’s body and

create an illusion that the user can see through their body . In this research, Ma Meng and his

group have suggested a first technique in improving the accuracy of Kinect sensor by using their

CHINMAYA INSTITUTE OF TECHNOLOGY 23

 VIRTUAL SURGERY

own researched anatomy within an AR setting. They have defined five bone landmarks, together

with orthopedic surgeons, which can easily be touched on the human body. The bone landmarks

are left and right acromion, the manubrium, and left and right anterior superior iliac spine.He

stated that this is a general method to interactively improve the Kinect skeleton for the purpose of

anatomy education. He also hoped that the idea and approach could result in new methods for

magic mirror technology. Traub et al. have purposed a new system combining both port

placement planning and intraoperative navigation. On a threedimensional virtual reconstruction of

the patient’s computed tomography scan, the optimal port placement is planned to perform an

accurate in vivo port placement by superimposing virtual thorax and the teleoperator arms models

in real world counterparts,which is achieved by AR techniques.

Fig 11.1 : dissecting an organ using augmented reality technique

CHINMAYA INSTITUTE OF TECHNOLOGY 24

 VIRTUAL SURGERY

Traub et al. have purposed several approaches for the system. One of the approached is virtual

simulation. This approach use computed tomography slices which can be displayed in sagittal,

coronal, and frontal directions, and aligned with the reconstructed model. Collision detection

techniques are also being applied and computed in real-time to avoid collision between the

teleoperator arms and their corresponding ports. By implementing the triangle–triangle

intersection detection,which is applied on the patient’s polygonal model , collision between the

arms and the anatomy can be avoided. The second approach is by implementing AR techniques

which enable the projection of the planned optimal poses for the port placement onto the patient’s

real world view . The matching of the 3D model with the patient is done using a RANSAC

approach . By using infrared tracking system, which does not rely on the actual teleoperator

parameters as related approaches , the endoscopic camera and the instrument arms can be tracked.

The position of the tracked arms, then is augmented on the 3D reconstructed model of the

patient’s thorax in real-time .As a result, precise rigid spatial registration of the pre-operative

acquired imaging data of the patient’s anatomy is realized. The expected result may lead to

significant reduction of operation time and improved quality due to a precise and collision-free

planning and placement of teleoperator arms.With the purposes of supporting medical students

and young children to learn bone anatomy, Stefan et al. have presented an edutainment system

called AR bone puzzle. The learning process is by assembling the augmented bone pieces on the

user’s body, which act as a puzzle frame. He quoted that regarding the content delivered and time

allocated for anatomy teaching, it has been significantly reduced. The learning environment is a

metaphor for bone learning using AR visualization and intuitive interaction. Several methods have

been purposed for the system. The first one is main puzzle game. There are three simple steps to

complete the game. Firstly, the system will randomly display one part of the bone. Secondly, the

user will be ask to grab the displayed bone. When the user has grabbed the displayed bone, lastly,

CHINMAYA INSTITUTE OF TECHNOLOGY 25

 VIRTUAL SURGERY

the user has to place the bone to the correct position of the bone on the user’s body. The second

method is AR visualization, which has two features: mirror-like display and visual hints. AR bone

puzzle has implemented a mirror-like display, which is proposed by the magic mirror system . The

second feature gives two different visual hints to the user during step three of the game. One of the

hint is displaying full skeleton in the reference window. The other hint is the synchronization of

the virtual bone with the motion of its corresponding body part. Means that the user can move

his/her body part individually to find the corresponding body part. A user study has been

conducted with non-medical students, medical students and an anatomy professor. The anatomy

professor suggested that the game difficulties should be adjustable to fit the level of the user that

use the system.

AR simulation systems have been developed for different field of surgery which the description

can be found in following parts of papers . Some of the systems that have been developed which

applying the AR techniques are ProMIS and CELTS. ProMIS was created by the collaboration of

experts from UK, USA and Ireland , allowing the users to learn, practice and hone skill with real

tools and its position, velocity and direction, ProMIS can provide unlimited degree of freedom and

tactile feedback anytime . On the other hand, the computer-enhanced laparoscopic training system

(CELTS) is another MIS surgery simulator, but still in prototype phase . Same with ProMIS,

CELTS also use real tools, real video display and laparoscopic light sources. The advantage of

using CELTS is it has an embedded matrices algorithm to indicate five critical indicators of

surgical skills: translocation, grasping, stretching/traction, touching and

navigation/coordination .Other AR surgery simulators for laparoscopic surgery are LTS3-e , Blue

DRAGON and a simulation system developed by Rhienmora et al. . Briefly, LTS3-e is a relatively

low cost system, developed by Society of American Gastrointestinal and Endoscopic Surgeon

(SAGES), to train and assess laparoscopic surgery skill of the user. The Blue DRAGON, on the

CHINMAYA INSTITUTE OF TECHNOLOGY 26

 VIRTUAL SURGERY

other hand, is a low fidelity simulator system to obtain the kinematics and dynamics of two

endoscopic tools.The simulation system that developed by Rhienmora et al. is used for dental

surgery. They have used ARToolKit and a video see-through head-mounted display (HMD) for

the system .MIS can reduce the surgery cost because it is a specialized surgical technique where

the surgical procedures are performed using small incision instead of large opening. Therefore,

patient will recover faster and stay less in the hospital . With the help of AR and VR technology,

the process will be better in the future.

CHINMAYA INSTITUTE OF TECHNOLOGY 27

 VIRTUAL SURGERY

12. CONCLUSION

Real-time simulation is a highlighted topic which has been taken into consideration since two last

decades. In this paper we have tried to present some prominent techniques and methods in not

only virtual surgery but also related to medical simulation. Intelligent computer backup minimizes

the number of medical mistakes. It provides a better training in anatomy and surgical skill, with

reduced need for cadavers. In addition, difficult surgery procedures can be practiced, new

procedures can be tried, and erroneous can be corrected. It gives the clear view of the outcome of

the surgery. The main disadvantage of employing the virtual surgery is involving the high cost for

the machine. Besides that, special training is needed to operate the machine.In terms of user

interactions and system responsiveness, there is a need to improve human-computer interfaces.

Time delays in the simulator’s response to the users movements (lagging issue). Furthermore,

there are shortcomings in the realism of the simulations. We hope this investigation on virtual

surgery and in general applying virtual reality and augmented reality in surgery can help

researchers to find the best suited way that they will face forward.

CHINMAYA INSTITUTE OF TECHNOLOGY 28

 VIRTUAL SURGERY

13. REFERENCES

1.Kolivand H, Sunar MS (2014) A quadratic spline approximation using detail multi-layer for

soft shadow generation in augmented reality. Multimedia Tools Appl 73(3):1225–1245

2. Bremner J, Lawrence J, Miall WE (1968) Degenerative joint disease in a jamaican rural

population. Ann Rheum Dis 27(4):326

3. Felson DT, Naimark A, Anderson J, Kazis L, Castelli W, Meenan RF (1987) The prevalence

of knee osteoarthritis in the elderly. The framingham osteoarthritis study. Arthritis Rheum 30

(8):914–918

4. Oliveria SA, Felson DT, Reed JI, Cirillo PA, Walker AM (1995) Incidence of symptomatic

hand, hip, and knee osteoarthritis among patients in a health maintenance organization.

Arthritis Rheum 38(8):1134–1141

5. Ku hnapfel U, Cakmak HK, MaaßH (2000) Endoscopic surgery training using virtual reality

and deformable tissue simulation. Comput Graphics 24(5):671–682

6. Clarke TL (1995) Distributed interactive simulation systems for simulation and training in

the aerospace environment. Society of photo-optical instrumentation engineers (SPIE)

7. Haluck RS, Marshall RL, Krummel TM, Melkonian MG (2001) Are surgery training

programs ready for virtual reality? a survey of program directors in general surgery. J Am

Coll Surg 193(6):660–665

8. Garbey M, Bass B, Berceli S (2012) Multiscale mechanobiology modeling for surgery

assessment. Acta Mech Sin 28(4):1186–1202

9. Satava RM (1996) Medical virtual reality. the current status of the future. Stud Health

Technol Inform 29:100

10. Delingette H, Ayache N (2004) Soft tissue modeling for surgery simulation. Handb Numer

Anal 12:453–550

CHINMAYA INSTITUTE OF TECHNOLOGY 29

 VIRTUAL SURGERY

11.Sorid D, Moore SK (2000) The virtual surgeon [virtual reality trainer]. IEEE Spectr 37

(7):26–31

12. BarbičJ, James DL (2005) Real-time subspace integration for st. venant-kirchhoff

deformable models. In: ACM transactions on graphics (TOG), vol 24. ACM, pp 982–990

13. Sederberg TW, Parry SR (1986) Free-form deformation of solid geometric models. In: ACM

Siggraph computer graphics, vol 20. ACM, pp 151–160

14. Meier U, López O, Monserrat C, Juan MC, Alcaniz M (2005) Real-time deformable models

for surgery simulation: a survey. Comput Methods Programs Biomed 77(3):183–197

15. Cotin S, Delingette H, Ayache N (2000) A hybrid elastic model for real-time cutting,

deformations, and force feedback for surgery training and simulation. Vis Comput 16

(8):437–452

16. Famaey N, Sloten JV (2008) Soft tissue modelling for applications in virtual surgery and

surgical robotics. Comput methods biomech biomed Eng 11(4):351–366

CHINMAYA INSTITUTE OF TECHNOLOGY 30