Virtual Reality Interaction Using Mobile Devices

Thesis by
Sahar Ali Aseeri

In Partial Fulfillment of the Requirements

For the Degree of

Master of Science

King Abdullah University of Science and Technology

Computer Science

Visual Computing

Thuwal, Makkah Province, Kingdom of Saudi Arabia

July, 2013
 2
The thesis of Sahar Ali Aseeri is approved by the examination committee

Committee Chairperson: Prof. Peter Wonka

Committee Co-Chair: Dr. Daniel Acevedo-Feliz

Committee Member: Prof. Markus Hadwiger

Committee Member: Prof. Xin Gao

 3

Copyright 2013

Sahar Ali Aseeri

All Rights Reserved

 4

ABSTRACT

Virtual Reality Interaction Using Mobile Devices

Sahar Ali Aseeri

With the use of an immersive display system such as CAVE system, the user is
able to realize a 3D immersive virtual environment realistically. However, interacting
with virtual worlds in CAVE systems using traditional input devices to perform easy
operations such as manipulation, object selection, and navigation is often difficult.
This difficulty could diminish the immersion and sense of presence when it comes to 3D
virtual environment tasks. Our research aims to implement and evaluate alternative
approaches of interaction with immersive virtual environments on mobile devices for
manipulation and object selection tasks. As many researchers have noted, using a
mobile device as an interaction device has a number of advantages, including built-in
display, built-in control, and touch screen facility. These advantages facilitate simple
tasks within immersive virtual environments. This research proposes new methods
using mobile devices like Smart-phones to perform di↵erent kinds of interactions both
as an input device, (e.g. performing selection and manipulation of objects) and as
an output device (e.g. utilizing the screen as an extra view for a virtual camera
or information display). Moreover, we developed a prototype system to demonstrate
and informally evaluate these methods. The research conclusion suggests using mobile
devices as a 3D-controller. This will be a more intuitive approach to interact within
the virtual environment.
 5

ACKNOWLEDGEMENTS

My initial and foremost thanks go to Allah for endowing me with health, patience,
and knowledge to complete this thesis.

Grateful acknowledgment is made to King Abdullah University of Science and

Technology, incorporated in the great education I have gained in the past two years
provided with the opportunity of writing this thesis which was created by the help of
Allah then and their support.

Further, I am deeply indebted and grateful to Prof.Peter Wonka, for the sup-
port, encouragement and facilities he has provided me. Thank you for everything.

Thereafter, I would like to thank, with deep gratitude and appreciation, the inspira-
tion, encouragement, valuable time and great guidance given to me by my respectable
supervisor Daniel Acevedo-Feliz. Thank you for everything.

Finally, I reserve my most special appreciation to my wonderful family. Firstly, to my

parents who have given me love and unconditional support during my life whenever I
was in success or loss, and also for their patience, encouragement and prayers. I hope
everything I have done can make you both happy and healthy. Secondly, to my three
lovely sisters as well as to my two wonderful brothers, for their love, encouragement,
emotional and moral support.
 6

TABLE OF CONTENTS

Examination Committee Approval 2

Copyright 3

Abstract 4

Acknowledgements 5

List of Abbreviations 8

List of Figures 9

1 Introduction 11
1.1 Problem Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.2 Research Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.3 Motivation and Importance . . . . . . . . . . . . . . . . . . . . . . . 15
1.4 Definition of Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.5 Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2 Literature Review 22
2.1 Object Manipulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.2 Object Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.3 Other Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

3 System Description and Architecture Design 34

3.1 Typical User Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.2 System Architecture Design . . . . . . . . . . . . . . . . . . . . . . . 35
3.2.1 CalVR Framework on Immersive Display System . . . . . . . 39
3.2.2 Networking Layer . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.2.3 OSG Framework on Mobile Device . . . . . . . . . . . . . . . 40
3.3 How the System Works . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.4 System Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
 7
3.4.1 Interaction Mechanism . . . . . . . . . . . . . . . . . . . . . . 42
3.4.2 Camera Manipulation . . . . . . . . . . . . . . . . . . . . . . 43
3.4.3 Object Selection . . . . . . . . . . . . . . . . . . . . . . . . . 46

4 Results and Evaluation 49

4.1 Presentation of the Prototype System . . . . . . . . . . . . . . . . . . 50
4.1.1 Camera Manipulation . . . . . . . . . . . . . . . . . . . . . . 50
4.1.2 Object Selection . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.2 Evaluation Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.2.1 Test Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.2.2 Test Environment . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.2.3 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.2.4 Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.2.5 Test Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.3.1 Comparison with State-of-the-Art Devices Results . . . . . . . 58
4.3.2 User Satisfaction Results . . . . . . . . . . . . . . . . . . . . . 59
4.3.3 Results Summary . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.3.4 Participant General Feedback . . . . . . . . . . . . . . . . . . 62
4.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

5 Conclusion and Future Work 65

5.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5.3 Lessons Learned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

References 70

Appendices 74
 8

LIST OF ABBREVIATIONS

CalIT2 California Institute for Telecommunications

and Information Technology

FOV Field-Of-View
FWIP Finger Walking in Place

IVE Immersive Virtual Environment

JNI Java Native Interface

KAUST King Abdullah University of Science and

Technology
KVL KAUST Visualization Laboratory

MR Mixed Reality

OSG OpenSceneGraph

PDA Personal Digital Assistant

UCSD University of California, San Diego

UIC University of Illinois at Chicago

VC Videoconferencing
VE Virtual Environment
VR Virtual Reality
 9

LIST OF FIGURES

1.1 Our prototype system . . . . . . . . . . . . . . . . . . . . . . . . . . 14

1.2 Example of a semi-immersive virtual environment (Flight simulator) . 18
1.3 Example of a fully-immersive virtual environment(CAVE system) . . 18
1.4 Cornea a CAVE system at (KAUST) . . . . . . . . . . . . . . . . . . 19
1.5 NexCAVE a semi-immersive virtual environment at KAUST . . . . . 20
1.6 An optical tracker device used to detect the user’s position in the VE 21

2.1 A system architecture design of the mobile animator system . . . . . 23

2.2 A user controlling a virtual world in the screen by using a mobile device 25
2.3 A user interacting with a virtual human . . . . . . . . . . . . . . . . 27
2.4 An example of a VC interface for viewing the virtual environment on
a PDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.5 A 3D navigation scene using a mobile device . . . . . . . . . . . . . . 32

3.1 A 3D truck model in the VE . . . . . . . . . . . . . . . . . . . . . . . 35

3.2 System architecture design . . . . . . . . . . . . . . . . . . . . . . . . 36
3.3 NexCAVE system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.4 A tracking system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.5 Tracker components attached with di↵erent devices . . . . . . . . . . 37
3.6 Di↵erent kinds of hand-held devices that are used with our system . . 38
3.7 Java Native Interface (JNI) communication . . . . . . . . . . . . . . . 38
3.8 CalVR menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.9 How the system works . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.10 Interaction mechanism per-frame . . . . . . . . . . . . . . . . . . . . 43
3.11 Camera manipulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.12 The FOV of the camera manipulation using a mobile devices . . . . . 44
3.13 Camera manipulation mechanism using a mobile device in the VE . . 45
3.14 Object selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.15 Object selection mechanism using a mobile device in the VE . . . . . 47
3.16 Calculating the new 3D object position matrix per-frame . . . . . . . 48
 10
4.1 Using hand-held devices with the NexCAVE . . . . . . . . . . . . . . 50
4.2 Example of using a mobile device as an output device to view the VE 51
4.3 Camera manipulation screen-shots . . . . . . . . . . . . . . . . . . . . 52
4.4 Object selection screen-shots . . . . . . . . . . . . . . . . . . . . . . . 54
4.5 Using the mobile device to manipulate the virtual camera and to see
the VE with no limitations . . . . . . . . . . . . . . . . . . . . . . . . 55
4.6 Direct and indirect manipulation task . . . . . . . . . . . . . . . . . . 56
4.7 Multiple users interacting with the VE using a mobile device and a
Gyro-mouse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.8 Results - Using mobile devices to facilitate the interaction with the VE 60
4.9 Results - Using mobile devices to reduce time needed to learn how to
deal with VE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
4.10 Results - Users preferring to interact themselves with the VE using
mobile devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

5.1 Local virtual environment with multi-users . . . . . . . . . . . . . . . 67

5.2 Global virtual environment . . . . . . . . . . . . . . . . . . . . . . . . 67
 11

Chapter 1

Introduction

Today, virtual reality (VR) is considered one of the most important research topics
in computer graphics [1], and it has become widely used as a functional tool in many
fields. In the past, VR has typically been restricted to research universities, large
corporations, and the military for training. However, fast progress in both hardware
and software technology has helped overcome these limitations and will allow VR to
be available and useful to a much wider population [2].

Virtual reality technology utilizes computer graphics to create a realistic world

that responds to user input. In other words, virtual reality is an experience in which
the user is e↵ectively immersed in a responsive virtual world, allowing the users to feel
they are part of the action. Another important meaning is that the VR environment
refers to how users can immerse themselves and feel part of the virtual world and
capable of interacting with this environment in meaningful ways [1]. Therefore, the
key features of virtual reality according to the above definitions are: virtual world,
immersion, sensory feedback, and interaction. The first three features depend upon
the last one being able to be executed properly. Thus, interaction is thought to be
one of the most important features in VR. It helps users feel truly involved within
the Immersive Virtual Environments (IVE) [3].
 12
Interaction in VR is usually described as the ability of the user to move and in-
teract with the objects inside the virtual world [1]. Interaction in IVEs is composed
of three elements: speed, range and mapping. Steuer (1992) [4] defined speed as
the system’s response time to user actions such as manipulation, object selection,
and navigation in the IVE. Therefore, late responses to the user’s action a↵ect the
interactivity in the 3D virtual environment. Additionally, he defined range as rep-
resenting interactions that can be used to manipulate objects in the environment.
This element increases interactivity in the IVE when the user is able to manipulate
large numbers of objects. Finally, he defined mapping as the connection between the
user and the actions executed in the IVE, with the goal of producing natural results
according to user’s interactions.

Based on the previous definitions of interaction, one may conclude that the success
of tasks in IVEs usually depends on the success of the interaction device [5].To mea-
sure the success of the interaction device within the IVE; we must consider immersion
and sense of presence [1]. Developers have discovered that the strongest sense of the
user feeling immersed emerges when the interaction device is easy and interesting.
This means that interaction with inadequate interaction devices could diminish the
immersion and sense of presence [6]. Therefore, the level of the user-controls over
interactivity of virtual objects with current state-of-the-art devices such as mouse,
keyboard, and joystick/wand varies from one device to another [1].

In this research, the problem area is that interaction with objects in virtual worlds
using traditional input devices (e.g. joystick or wand) is often difficult. This difficulty
could diminish the immersion and sense of presence in 3D immersive virtual environ-
ment tasks. In an attempt to increase immersion in the virtual world, mobile devices
can be used as an interaction device. This research will present di↵erent approaches
 13
for interacting with IVEs by using mobile devices for manipulation and object selec-
tion.

We hypothesize that using mobile devices as a 3D controller for interaction will

facilitate simple tasks such as manipulation, object selection, and navigation within
IVEs. Additionally, we hypothesize that using mobile devices as an interaction device
will increase the immersion and sense of presence within the IVE. Moreover, we pro-
pose that using a touch screen on a mobile device to interact with IVEs would reduce
the time needed for users to learn how to deal with the virtual world. To accom-
plish this, we disabled the button functions on the mobile device to encourage using
the touch screen. Finally, we hypothesize that using a tracked mobile device with
the capability of displaying a synchronized view of the IVE will allow users to manip-
ulate the virtual objects more easily in cases where multiple users are sharing the IVE.

Our prototype system works as a server-client application (see Figure [1.1]). It

consists of a CAVE system and a mobile device. On the CAVE system we use CalVR
framework [7] for our virtual environment and OpenSceneGraph (OSG) running na-
tively on an Android device. Two similar scenes are created, one for the server and
the other one for the client. To move through these two scenes concurrently, we con-
nected the CalVR system and the Android device via TCP/IP protocol to deliver
a single synchronized scene to the user. This occurs by enabling the user to inter-
act with objects in a single environment by sending and receiving transformation data.

The main contribution of this research is to provide new approaches to interact

with IVEs on mobile devices for manipulation and object selection. Additionally, we
deliver a single synchronized scene to the users, both on the client and server, en-
abling them to interact with a single IVE. In order to at least evaluate our hypothesis
 14
at an informal level and due to time limitations, our prototype system is limited to
working on one object and only allowing a single user to interact within the IVE
with a mobile device. Nonetheless, this proof-of-concept system demonstrates the
capabilities of mobile devices as VE interaction platforms.

Figure 1.1: Our prototype system: it allows the user to interact with a single syn-
chronized scene using a mobile device.

1.1 Problem Definition

1. Interaction with objects in virtual worlds using traditional input such as a

joystick or wand is often difficult and could diminish the immersion and sense
of presence for the user in 3D immersive virtual environment tasks.

2. The CAVE system or NexCAVE allows one user to interact with the IVE by
using traditional input devices.
 15
3. Usually, traditional input uses indirect manipulation techniques to manipulate
objects using buttons.

1.2 Research Objectives

1. To enable the use of mobile devices as a 3D-controller in IVEs.

2. To focus on mobile devices’ capabilities (i.e. multi-touch screen, monitor screen,

sensors, accelerometer, magnetometer, gyroscope, and finger gestures), which
provide new possibilities for interaction techniques on mobile devices within
IVEs.

3. To increase the immersion and sense of presence when using mobile devices in
IVEs.

4. To allow multiple users to interact with the virtual environment at the same
time using mobile devices.

1.3 Motivation and Importance

The wide spread use of 3D in movie theaters, gaming, education, and the military
has motivated researchers to be concerned with making the 3D virtual world more
e↵ective. As previously mentioned, interaction is one of the most important features
in VR, helping to increase the e↵ectiveness of the 3D virtual experience. VR in-
teraction tasks depend on the interaction techniques and the interaction devices [6].
Researchers have tried to combine these two components to produce more natural
interactions inside the IVE that are more intuitive.

Interactions make use of a vast variety of input devices, including joystick, data
gloves, and instrumented objects. The level of interactivity provided by using these
 16
devices - especially with 3D object manipulation tasks in the IVE - varies by device
[8]. This is important as the e↵ectiveness of the interaction techniques of 3D objects
manipulation tasks in the IVE are a↵ected by the e↵ectiveness of the interaction de-
vices. This means that there is a relation between the interaction techniques and the
interaction device to produce high-level interaction [6].

Several problems exist that relate to interaction in virtual reality. One of these
problems is object manipulation [6]. Indeed, manipulation is one of the most funda-
mental tasks in both real and IVEs. There are two kinds of manipulation interaction
techniques, direct manipulation and indirect manipulation. Indirect manipulation,
such as Nintendo Wiimote, uses indirect ways like buttons to select and manipu-
late the virtual objects. Direct manipulation, such as direct-hand manipulation, is
a natural technique that is intuitive for humans because it uses hands to select and
manipulate the virtual objects. However, using direct manipulation is not always
possible with all interaction devices [8]. If the user cannot manipulate the virtual
objects e↵ectively, then the user might be unable to complete other tasks in the IVE.

We propose to overcome this limitation using hand-held devices including mobile

phones and tablets. Hand-held devices have been spreading all over the world, and
many people are using them daily. Rapid developments on Smart-phones with re-
spect to both hardware and software have huge potential for using mobile devices
as 3D-controllers. This is because they have many capabilities such as multi-touch
screen, monitor screen, sensors, accelerometer, magnetometer, gyroscope, and finger
gestures, which provide new possibilities for interaction techniques within IVEs. This
scenario opens an interesting area for VR applications.
 17
In this work, our prime goals are to utilize the advantages of recent developments
in mobile devices, including multi-touch screens and monitor screens, to use mobile
devices as both input and output devices. We consider the mobile device as an input
device by using the touch screen to provide a direct means of interaction through
direct touch to allow the user to manipulate the object in the 3D virtual environment.
Additionally, we consider the mobile device as an output device by using a mobile
device with built-in display to allow the user to view the virtual environment on the
display screen. This gives the user the flexibility to move and explore the environment
freely. Finally, using mobile devices within the IVE allows multiple users to use their
own device to interact with the virtual world rather than allowing only one user to
interact with the environment. By using a mobile as an interaction device with the
IVE, we believe that the door will be opened to developers when considering them
as 3D-controllers to interact with the 3D virtual world.

1.4 Definition of Terms

Immersion
Immersion in the IVE is a term used to describe how much the user is involved
inside the virtual world. That is, how much does the user feel part of the virtual
world physically [1].

Immersive Virtual Environment (IVE)

One of the most important features of VR is the immersion in an artificial en-
vironment where the user feels the environment is reality. There are two types
of the IVE, semi-immersive virtual environments and fully- immersive virtual
environments. An example of a semi-immersive virtual environment is a flight
simulator in which the user does not become fully immersed (see Figure [1.2]).In
 18
a fully-immersive virtual environment (e.g. CAVE system) the user becomes
fully immersed within the virtual world (see Figure [1.3]).

Figure 1.2: Example of a semi-immersive virtual environment(Flight simulator) [9].

Figure 1.3: Example of a fully-immersive virtual environment (CAVE system) [10].

 19
CAVE System
The CAVE system is a kind of VR output device. It is a fully-immersive 3D vir-
tual environment (only if it has six sides) that was developed in the early 1990’s
by Professor Tom DeFanti and others at the University of Illinois at Chicago
(UIC) (see Figure [1.4]).

Figure 1.4: Cornea a CAVE system at King Abdullah University of Science and
Technology (KAUST)[11].

NexCAVE
The NexCave is a kind of VR output device. It is a semi-immersive 3D virtual
environment implemented at KAUST and designed by Professor Tom DeFanti
at the California Institute for Telecommunications and Information Technology
(CalIT2) at the University of California, San Diego (UCSD). It is configured as
a 21-tile system using monitor displays [12] (see Figure [1.5]).
 20

Figure 1.5: NexCAVE a semi-immersive virtual environment at KAUST [12].

Navigation
Navigation is a kind of interaction task used in the IVE; it provides the user
with information about her/his location by moving the user from one location
to a new target location in the virtual world [6].

Manipulation and Object Selection

Manipulation is a kind of interaction task in the IVE. Manipulation includes
selecting object tasks to move the 3D objects inside the virtual world [6].

Tracking System
The tracking system is one of the most important components in VR systems.
It allows the user to move in the IVE, using the physical space by detecting the
user’s position using trackers [6] (see Figure [1.6]).
 21

Figure 1.6: An optical tracker device used to detect the user’s position in the VE.

1.5 Organization

The rest of this research thesis is organized into several chapters. Chapter 2 provides
the review of related work in four main sections: studies related to object manipula-
tion, studies related to object selection, and studies concerned with di↵erent aspects
of VR and use of the mobile as an interaction device. Chapter 3 provides the research
methodology used in this work. It presents the system description and architecture
design. It also demonstrates the basic idea of the prototype and the tools used in the
implementation phase. Chapter 4 presents the informal evaluation and final results
phase of the system. This chapter presents a detailed description of the test evalua-
tion, test participants, and tasks, as well as final results and discussion. Chapter 5
concludes the research that has been done and proposes future works.
 22

Chapter 2

Literature Review

Research work related to the use of mobile devices for VR interaction has been of
great interest to many researchers since the early 1990s. With use of virtual reality
spreading, computer scientists introduced various approaches to overcome interaction
limitations within virtual environment (VE) by using a mobile device as an interaction
device. The available literature is composed of papers that have been published in sci-
entific journals or posters that have been presented at conferences. The presentation
of these studies will be organized under the following topics:

Studies related to object manipulation

Studies related to object selection

Studies concerned with di↵erent aspects of VR and the use of Smart-phones as

an interaction device

2.1 Object Manipulation

To facilitate the object manipulation task within the VE, we used mobile devices to
move 3D objects via touch screen. Gutierrez et al.(2004) [13] shed light on using
hand-held devices as interfaces in the VE. He focused on the manipulation of 3D
 23
objects, excluding articulated characters: translation, rotation and camera manage-
ment . This was coupled with creating a 3D graphical representation of characters
and objects in the VE right into the hand-held device. The name mobile animator
was given to the interface used, and a 3D representation of the objects was used in-
stead of 2D controls. Briefly, the 3D object was the hierarchical structure or skeleton
that is used to control the 3D virtual characters. The hand-held device allowed the
user to modify the character’s posture and its orientation at every joint shown in
the simulation screen (VE System) using a stylus in the hand-held. In case there
was more than one character in the VE, a list of virtual characters appeared for the
user to choose from. The system used consists of four main components: VE system,
network layer, character data, and the hand-held interface (see Figure [2.1]).

Figure 2.1: A system architecture design of the mobile animator system: the main
components of the system are VE system, network layer, character data, and hand-
held interface. [13]

The system implemented has shown efficiency by allowing more than one user
to interact in a shared VE. The main issue observed was the division of the user’s
 24
attention between the hand-held and the simulation screen. The hand-held device
has been shown to be a very e↵ective tool to modify the posture and location of
the virtual characters in the VE. In our study we built a system that has the same
components used in this research. The deference that we used Smart-phones as a
hand-held device to manipulate 3D objects and camera in the VE.

Another two studies have been done about object manipulation using mobile de-
vices. The first study, done by Katzakis and Hori (2009) [14], showed some initial
evidence that a mobile device can be e↵ectively used as a wireless hand-held 3-DOF
controller in VE tasks. They explored the use of accelerometer-and magnetometer-
equipped mobile phones, instead of using a mouse or a keyboard, as 3-DOF controllers
in a 3D rotation task to manipulate a 3D object. They compared three devices – mo-
bile, mouse, and touch panel. They indicated that a 3D rotation task in a VE would
take only five minutes of practice using a mobile device, almost half the time it took
with the other devices. The results indicated that using mobile devices for 3D virtual
environment rotation tasks seems to facilitate faster interaction compared to other
devices, such as mouse or touch panel devices.

The second study was conducted by Lee et al. (2011) [15]. It introduced a new
method for 3D object manipulation: a mobile device instead of the expensive trackers
previously used. Components used in the experimental environment are: normal PC,
tracker or remote (mobile device) and Wi-Fi network. Figure [2.2] shows the user
controls the mobile device to change its pose and position and the mobile device in
turn sends data to the computer via Wi-Fi. The virtual 3D space on the computer
screen will change accordingly. According to the paper, using this method for 3D ob-
ject manipulation is expected to be accustomed to everybody, can be applied easily,
and can be used for a lot of areas.
 25

Figure 2.2: A user controlling a virtual world in the screen by using a mobile device:
any movement applied on the mobile device will apply directly to the virtual world.
[15]

From the above two studies we can conclude that mobiles are e↵ective when used
as an interaction device for manipulation tasks to change the object transformations,
such as translation, rotation, and scaling. This research will concentrate on using
mobile devices for manipulation tasks to change camera and object transformations
for translation, rotation, and scaling.

2.2 Object Selection

To improve 3D interaction and overcome some difficulties when we interact within

the 3D environments, Miguel et al. (2007) [16] proposed the see-through the lens
interface technique using personal digital assistant (PDA) devices. They used this
technique with the PDA for spatial manipulation and some functionality such as se-
lecting objects in 3D space. They implemented a client-server application connected
 26
with two networking layers, TCP and UDP, to receive images of the environment from
the server to the client and to send the commands from the client to the server. They
determined the functionality of their approach through user opinions and observation
of the user behaviors. Such opinions and behaviors showed that the use of PDA-based
see-through the lens interaction was a powerful method for improving 3D interaction
in a more natural fashion. The main issue observed was a small latency when send-
ing the images to the client and some images were lost. Our study is similar except
for using Smart-phones instead of PDA devices and trying to avoid this latency by
creating the VE on the server (CAVE system) as well as on the client (Smart-phone).

To increase the immersion and sense of presence through the interactivity inside
the VE, we used touch screen to directly select and manipulate objects in the virtual
world. Chuah and Lok (2012) [8] examined the possibility of using Smart-phones as
a VR interaction device to select and manipulate objects in VEs. They discussed two
types of applications. The first application was a mixed reality (MR) game based
on general object selection and pose manipulation. It was created to evaluate the
Smart-phone’s usability as an interaction device in a (MR) environment. Overall, the
results indicated that 100 percent of the participants responded, positively to whether
they would use the mobile as an interaction device in the VE, and responded better
for both selecting and positioning objects. The second application was a Virtual Re-
ality Eye Exam, which is an example of adapting an existing real-world application
to use a Smart-phone as an interaction device (see Figure [2.3). This application
has been shown to improve the original interface (television as an output device and
Wiimote as an input device) in two ways. First, it reduces the need for training and
memorizing button mappings for each tool. Second, it combines the real and the
virtual worlds more seamlessly. Due to Smart-phone features, the adaptation of the
Smartphone was easy, quick and completely successful.
 27

Figure 2.3: A user interacting with a virtual human: the virtual human follow the
smart-phone movement using a tracker. [8]

2.3 Other Related Work

Several other studies covered other aspects related to mobile phone use as a 3D inter-
action device within VE, which are not covered in the previous studies and illustrate
that using mobile device will be a means that can be use to perform interaction
tasks. For instance, the contention between 2D and 3D interaction technique and
uncertainty as to which is appropriate and when, was known as one of the major
problems that plague the advancement of virtual world development. To overcome
these problems Watsen et al. (1999) [17] developed a new approach that can be used
to perform tasks appropriate to 2D interfaces with PDAs such as the 3Com Palm-
Pilot hand-held computer. They report the techniques that were implemented and
the specifics of using Bamboo and a Palm-Pilot for virtual world application. Despite
several limitations, the researchers indicated that the use of a PDA as an interface
 28
in a VE enabled the UI techniques without forcing 2D metaphor in a 3D space or
3D metaphor in a 2D space. They concluded that this approach was promising and
further research is needed to solve the limitations.

Marsden and Nicholas (2005) [19] wrote a paper about a cultural preservation
project with the District Six Museum in Cape Town, South Africa. They were en-
gaged to create a virtual District Six as it would have appeared before mass evictions.
They tried to build on Yee’s Peephole Display work [20] to examine the possibility
of using PDAs to create a peephole time-machine that would allow users to virtually
view the old District Six while standing in the current District Six. To accomplish
this project they investigated whether the PDA can provide an e↵ective level of in-
teraction and provide an intuitive navigation scheme for viewing virtual worlds. This
was done by using client-server system where the view is on the server and the PDA
receives the view of the environment over the network as 2D images. Specifically the
study focused on the development and evaluation of using the PDA as an interaction
device. The study result confirmed that using the PDA was e↵ective for interaction
for viewing virtual worlds.

In another related study, Weberet al. (2005) [18] discussed the interaction between
remote human computers and VEs. The research relied on a thin client solution that
in turn relies on videoconferencing (VC) protocol standards connected to a VE. The
former uses H.323 (standard VC protocol), which is a protocol that supports multi-
media communications to transfer audio and video data over networks. According
to this research, the proposal was a VC interface to a VE via a PDA. The remote
user could access the VE via sight and hearing. On the PDAs, seeing and hearing
the VE was made possible by equipping PDAs with VC software. Enabling remote
human computer interactions with a VE brings new applications to the mobile plat-
 29
form. This research aims to simplify the interaction between the mobile and the PC
by using the videoconferencing protocol (see Figure [2.4]). Our research will focus on
the use of mobile devices in order to access the VE visually and to interact with the
environment using the built-in display screen and touch screen facility. The VE and
mobile device will be connected using TCP/IP protocol.

Figure 2.4: An example of a VC interface for viewing the virtual environment on a

PDA. [18]

Another related work was carried out by Mountain and Liarokapis (2005) [21] as
part of the Locus Project. They introduced a new way of interacting with VR scenes
using mobile devices. In their paper they discussed three prototypes that adopt di↵er-
ent ways of interaction with VR scenes based upon both screen interaction, external
sensors and a mix of both approaches. They proposed that external sensors (interac-
tion via movement and gesture) may be a more intuitive approach for mobile device
within VR scenes. They evaluate and test this hypothesis during the Locus project.
 30
Research suggests there are di↵erences between using mobile devices and using desk-
top systems to interact with the VE, proposing that traditional interaction paradigms
may not be appropriate in this new context.

As a result of the need for expressive visualization for 3D data and interaction
with mobile devices, Knodelet al. (2008) [22] explored the possibility of using a new
technique that allows mobile users to interact together with di↵erent data using var-
ious hardware. As part of this research, they tried to take advantage of the recent
development of mobile technology such as multi-touch interface and overcome the
limitations, such as small buttons. They used this technique to move the camera po-
sition in the 3D environment with di↵erent sizes of hardware. The research concluded
that there is a need for more advances in using mobile devices for visualization and
interaction and proved the direct applicability of the research result with respect to
the collaborative scenario.

Another study by Kim et al. (2009 ) [5] provides new possibilities for interaction
techniques by using multi-touch screen hand-held devices such as the iPhone and the
iPod Touch for navigation in IVEs. They described implementation of a navigation
interaction technique called Finger Walking in Place (FWIP) [23], which was devel-
oped for a larger multi-touch device. The FWIP project was implemented for the
Lemur device. The project was improved to utilize the multi-touch screen hand-held
devices such as the iPhone/iPod Touch, which were used for navigation tasks in a
CAVE virtual environment. Initial results showed that the use of hand-held devices
for navigation tasks in IVEs provided sufficient precision. All in all, the paper re-
ported that using multi-touch hand-held devices is one of the approaches to navigate
tasks in 3D environment, and using it will increase the success of IVEs.
 31

Further research by Benzina et al. (2011) [24] introduced what they called Phone-
Based Motion Control as a technique to interact with the VE. They used a mobile
phone with integrated sensors as a 3D input device for travel interaction to provide an
easy way to track the user’s hand. To examine the e↵ectiveness of this technique, the
researchers tried to determine how many degrees of freedom are needed to perform
the travel task as easily as possible. They also tried to investigate di↵erent mapping
functions between the user’s action and the viewpoint reaction in the VR. To accom-
plish their goals they used a phone with multi-touch capability for VE translation
or viewpoint direct control and built-in sensors. Overall the results indicated that
users perform best using mobile device with 4 DOF metaphors and that the usage of
mobile device is the desired mapping.

Moreover, fast progress in both hardware and software technology means big po-
tential for mobile devices. Gang Pan et al. (2011) [25] concentrates on taking benefits
from capacitive mobile devices, such as touch screens, built-in MEMS (micro electro-
mechanical systems) motion sensors, built-in cameras, accelerometers, magnetome-
ters, gyroscopes and finger gestures, to interact and navigate in a VR environment,
transform viewpoint, move avatars and view camera zooming. They presented and
illustrated a new interaction called Tilt and Touch, using a mobile device as a 3D-
controller in a VE. They prototyped the Tilt and Touch system, using a mobile
application on the phone and a server application on the computer connected with
Wi-Fi. In order to interact with the 3D environment, or manipulate and rotate ob-
jects on the computer, you can Tilt the mobile phone and Touch with your finger or
fingers. Figure [2.5] shows the result of the navigation application in a VE using a
mobile device. Here, the user controls a bird soaring freely in the jungle.
 32

Figure 2.5: A 3D navigation scene using a mobile device: the user controls a bird in
the jungle using a mobile device. [25]

In addition Lee et al. (2011) [26] introduced a new approach for dual interactions
between multi-display and Smart-phones to share the content display views for indi-
vidual and private users. Also, they used Smart-phones for remote control and for
a dual visualization interface of the large display. The research team used Sam-sung
Galaxy based on the Android system as a mobile phone interaction and OSG library
for 3D graphics rendering and scene graph management. Further, two Wiimotes were
used to receive multi-touch inputs and capture the user’s interactions. Finally, a Wi-
Fi connection was used to establish communication between the Smart-phone and the
multi-display. The study results indicate that dual interaction between multi-display
and Smart-phone for collaborative design review and sharing among multi-users is
possible and can be adapted to various displays. Also, it is an easy way to share ideas
and interact with multimedia among the users.
 33

Overall, the previous studies’ results found some evidence of the possibility of using
mobile devices as a 3D-controller for VR interaction in processes such as manipulation
and object selection. Despite this, and the growing interest in research concerned with
utilization of mobiles as a device for VR interaction, the outcome is still limited in
size and quality and more research is needed. This research will be a continuation of
previous e↵orts and will tackle some problems and add some improvements to make
the mobile device an input and output device to perform interaction tasks within the
IVEs.
 34

Chapter 3

System Description and

Architecture Design

In this chapter, we present a description and an architecture design of the whole

system, including how it works. The primary goals of this chapter are to describe
the interaction methods, understand how the system works, and show how to utilize
these methods in the IVE using mobile devices. The chapter is organized into several
sections. First, a typical scenario that a user may perform is described. Then, a
system architecture design is explained including the subsections: CalVR framework
for the immersive display system, networking layer, and OSG framework on mobile
device. After that, an explanation is mentioned about how the system works. The
final section describes the system interaction mechanism, camera manipulation, and
object selection tasks.

3.1 Typical User Scenario

The users enter the CAVE system or NexCAVE. They wear the 3D glasses and carry
a mobile device attached with a tracker to see the 3D virtual environment through a
display screen on the mobile device. The users should find a 3D object on the scene
to apply camera manipulation and object selection tasks by the display screen and
 35
the touch screen on the mobile device. Figure [3.1] shows a truck model (3D object)
appears on the NexCAVE and the mobile screen. Any task applied on the truck
whether on the mobile device or on the NexCAVE will be applied to the other side by
sending and receiving the data transformation of the truck position over the network.
In the next sections we will explain and define each component in the system and
how the tasks are applied in more detail.

Figure 3.1: A 3D truck model in the VE: the user controls the 3D truck model in the
VE by using a mobile device.

3.2 System Architecture Design

The proposed system was implemented as a client-server application. The server

side was developed using CalVR [7]. CalVR is a VR framework based on OSG, a
3D graphics library, which was written in C++ and developed for Linux operating
system. The client side was developed using OSG running natively on the android
operating system. Our main mode of operation is to send and receive data transfor-
mations from CalVR to the android devices and vice versa using the network layer
TCP/IP protocol (see Figure [3.2]).
 36

Figure 3.2: System architecture design.

On the server side, we used an immersive display system (the NexCAVE system)
that runs the CalVR framework (see Figure [3.3]). Across the top of the NexCAVE
system is an optical tracking system (ART TrackPack4-infrared camera) that tracks
the users’ positions and movements inside the VE using tracker devices [27] (see
Figure [3.4]).

Figure 3.3: NexCAVE system: a semi-immersive virtual environment at KAUST.

 37

Figure 3.4: A tracking system: ART TrackPack4-infrared camera is used to detect

the mobile position in the VE [28].

Figure [3.5] shows di↵erent kinds of tracker components attached to di↵erent de-
vices. These trackers are used to find the device’s position inside the VE. For example,
head-tracker is a pair of 3D glasses attached with a tracker, Gyro-mouse-tracker is a
wireless input device attached with a tracker, mobile-tracker is a Sam-sung Galaxy S
Smart-phone attached with a tracker.

Figure 3.5: Tracker components attached with di↵erent devices - (a)Head-tracker

(b)Gyro-mouse-tracker (c)Mobile-tracker.

On the client side, the Sam-sung Galaxy S Smart-phone or Tablet runs Android
2.2 as a minimum platform because there are incompatibilities in some devices with
 38
the OSG library (see Figure [3.6]). Also, we used the Java Native Interface (JNI),
which is a programming framework that allows Java code to interact with code writ-
ten in other languages, such as C, C++, and assembly language. In our case, we
used Java and C++ languages. The JNI allows Java code to communicate with C++
code. This helps us to develop a Java interface based on OSG libraries to provide 3D
graphics applications on mobile devices (see Figure [3.7]).

Figure 3.6: Di↵erent kinds of hand-held devices that are used with our system.

Figure 3.7: Java Native Interface (JNI) communication.

 39
3.2.1 CalVR Framework on Immersive Display System

Because VR technology is developing constantly, the virtual reality companies and

researchers have been developing frameworks that can work with di↵erent hardware
and software platforms in order to make the process of creating software easier, both
in terms of development and maintenance. CalVR is a new VR framework based on
the scene graph conception, which was developed by Schulze and others (2013) in the
Immersive Visualization Laboratory in the California Institute for Telecommunica-
tions and Information Technology at the University of California, San Diego (UCSD)
[7]. It has several features, including a sound system and high-level programming
interfaces for interaction applications. CalVR includes object-oriented classes which
are written in C++. It was developed for CentOS Linux, but we can run it on other
Linux systems and Mac OS. CalVR integrates the OSG library for the graphics out-
put. CalVR has several built-in navigation methods, an extensible 3D menu system
(see Figure [3.8]), and features support for a variety of 3D display and tracking sys-
tems.

Figure 3.8: CalVR menu: an extensible 3D menu system [7].

 40
3.2.2 Networking Layer

The system is implemented as a client-server application. The CAVE server has one
open port waiting for a mobile device connection. We used the TCP/IP protocol to
send and receive reliable transformation data between the server and the client. The
TCP/IP protocol will start to send and receive segments every frame from the server
to the client and from the client to the server after the server accepts the client. The
transformation data sent from the server to the client contains two 4x4 matrices, one
for the object transformation (such as scaling, position, and orientation) and the other
for the tracker transformation (position and orientation). The transformation data
sent form the client to the server contains one 4x4 matrix for object transformation
(position and orientation).

3.2.3 OSG Framework on Mobile Device

OSG is one of the most commonly used high-performance 3D graphics toolkits. It

is used in the fields of VR, scientific visualization, games, simulation, and mobile
applications. It provides an object-oriented framework based on the scene graph
concept. With OSG, we can create applications with high-performance graphics and
high-speed rendering optimizations [29]. We used OSG library on a mobile device
because we can install this 3D graphics library on Android OS. Also, to create VE
similar to the one we created on the server because the server uses the same library
for the graphics output. Finally, to make sure that we can import objects with the
same extension files on the server as well as on the client.

3.3 How the System Works

Our system consists of a VE and a mobile device, both of which are based on OSG.
We use CalVR for our VE and OSG on the android. This creates two similar scenes:
 41
one on the CalVR system and the other on the android device. In order to move these
two scenes simultaneously, we need to connect the CalVR system and android device
via the TCP/IP protocol. This delivers a single synchronized scene to the user and
enables her/him to interact with a single environment. This all happens as a result of
sending and receiving the tracker transformation data that moves the virtual camera
on the Android device. We can also send and receive the 3D object transformation
on the scene to change the object’s position in the VE on both server and client sides
(see Figure [3.9]).

Figure 3.9: How the system works.

 42
3.4 System Description

Our system allows the user to perform the following two interaction tasks: camera
manipulation and object selection. In the camera manipulation task, we will use the
built-in display on the mobile device in order to see the environment on the mobile
screen. In the object selection task, we will use the screen surface on the mobile device
to select the object directly in order to move it from one place to another in the VE.
In the next sections we will explain in more detail about the interaction mechanism,
camera manipulation, and object selection.

3.4.1 Interaction Mechanism

When users enter the CAVE system or NexCAVE, they carry a mobile device con-
nected to the CAVE system via a Wi-Fi network and is ready to receive the data
transformation of the objects’ position and the tracker position coming from the
CAVE system. They are also ready to send the objects’ transformation data per-
frame to the CAVE system when clicking on the object (see Figure [3.10]). The
mobile device is attached to a tracker device to find the mobile device’s position in-
side the VE. When a user wants to select and move a 3D object from one place to
another using the mobile touch screen, she/he must first find the object by moving
the virtual camera inside the VE and then select the object to move it to another
place in the VE. Any changes applied to the 3D object will be applied immediately
to the 3D object on the CAVE system server side.
The main steps of our system are as follows:

Run the CalVR plug-in on the NexCAVE system as the server.

Run the application on the Android device as the client.

Connect the client with the server over the TCP/IP protocol to send and receive
the transformation data.
 43
Start to interact with two scenes, one on the CalVR system and the other on
the android device. Any changes in either system will be seen immediately in
the other one.

Figure 3.10: Interaction mechanism per-frame.

3.4.2 Camera Manipulation

To manipulate the camera inside the virtual world we need to manipulate the mobile
device to change the view that appears on the mobile display within the VE. This
allows the user to move freely inside the VE and see everything through the mobile
screen. This method permits the user to move the virtual camera around the object
inside the VE and see the object from any side (see Figure [3.11]). Additionally, some
of the semi-immersive display systems are shaped like a semi-circle which restricts the
user’s virtual field-of-view (FOV) when interacting with the VE; however, using the
virtual camera manipulation method will allow the user to see and interact with the
VE with no limitations (see Figure [3.12]).
 44

Figure 3.11: Camera manipulation: viewing the object from di↵erent orientations
using the mobile device.

Figure 3.12: The FOV of the camera manipulation using a mobile devices.
 45
Camera manipulation interaction is implemented by moving the mobile device in
order to change the virtual camera’s position in the VE using a tracking system. The
tracker component is attached to the mobile device; it helps the system determine
the mobile device’s position and orientation inside the VE. When we move the mo-
bile device, the tracker transformations (position and orientation) will be sent over
the socket from the server to the client to change the camera manipulator’s (virtual
camera) position inside the VE on the mobile device (see Figure [3.13]).

Figure 3.13: Camera manipulation mechanism using a mobile device in the VE.
 46
3.4.3 Object Selection

During the object selection interaction we change the object’s position in the VE
using the mobile touch screen by dragging and dropping the object from one place to
another (see Figure [3.14]).

Figure 3.14: Object selection: by the mobile touch screen we can drag and drop the
object into a new place.

Object selection interaction is implemented by using the ray-casting technique. To

manipulate 3D objects, the user should find the target object by moving the virtual
camera (i.e., the mobile device screen) inside the VE. Then, the user will be able to
select the object using the mobile touch screen. Any changes made to the 3D object
will be applied directly to the 3D object on the CAVE system server side (see Figure
[3.15]).
 47

Figure 3.15: Object selection mechanism using a mobile device in the VE.

Ray-casting is one of the most fundamental modes of 3D manipulation techniques

(pointing-based interaction techniques). This technique projects a 3D virtual ray to
the 3D object when the user selects the object via the touch screen on the mobile
device. This virtual ray is generated from the camera through the image plane to
define the direction of pointing and checks the ray intersection with the nearest object
[6]. After the ray intersects with any object in the scene, we can move the object
 48
from one place to another by selecting and calculating the new object position in
every frame and then sending the object matrix to the server side to change the ob-
ject simultaneously in real time. To calculate the object position matrix we use the
equation in Figure [3.16] to find the new object position.

Figure 3.16: Calculating the new 3D object position matrix per-frame: this is done
when a user selects a 3D object on the mobile device.
 49

Chapter 4

Results and Evaluation

We developed an interactive mobile phone application to turn mobile devices into a

3D interaction device to be used in the VE. This chapter presents the initial results
of the proposed system and those of an informal evaluation test, as well as comparing
this technique with the current state-of-the-art. We will compare the mobile device
with the Gyro-mouse device which we described it in Chapter 3. The evaluation test
will focus on the usability of employing mobile devices for interaction tasks. The
purpose of the evaluation was to investigate the users’ opinions and suggestions for
using mobile devices as a 3D-controller in the VE, the validity of our assumptions,
and the e↵ect of using mobile devices when the user interacts with the VE. The
chapter is organized according to the following structure: The first section provides
a presentation of the prototype system results for camera manipulation and object
selection. Then, the second section presents the informal evaluation test including
test objectives, test environment, participants, and tasks. The third section presents
the questionnaire results including the comparison with the state-of-the-art devices,
user satisfaction results, and participant general feedback. Finally, the last section
discusses the results.
 50
4.1 Presentation of the Prototype System

This section presents the initial results of the proposed system using the CalVR
framework on the server and the OSG framework on the client side. The server ap-
plication has been tested on Scientific Linux OS using the NexCAVE system, and the
client application has been tested on Android OS using either a Sam-sung Galaxy S
Smart-phone or a Galaxy Tablet running the Android 2.2 platform (see Figure [4.1]).
A Wi-Fi connection was used to connect the server-client applications. At the top of
the NexCAVE system is an ART TrackPack4 optical tracking system. It is used to
track the users’ positions and movements inside the VE. In the next sections, we will
discuss the initial results of the camera manipulation and object selection tasks.

Figure 4.1: Using hand-held devices with the NexCAVE: on the left is a mobile device
and on the right is a tablet device.

4.1.1 Camera Manipulation

Figure [4.2] gives an example of using the mobile as an output device in the system.
We can see that the mobile device is able to display the area covered by the mobile on
the NexCAVE screen. In this case, the mobile device is considered a virtual camera
that is used to view the VE on the mobile display screen.
 51

Figure 4.2: Example of using a mobile device as an output device to view the VE.

Figure [4.3] shows two examples of using a mobile device to move the virtual cam-
era around the object inside the VE in order to see the object from all sides. This
method allows the users to move freely inside the VE and see everything through the
mobile screen.
 52

Figure 4.3: Camera manipulation screen-shots: using the mobile device to manipulate
the virtual camera around the 3D object in order to see the object from all sides
(a)The front side of the object, (b)The right side of the object, (c)The top and down
sides of the object, and (d)The back side of the object.
 53
4.1.2 Object Selection

To select an object, the user needs to find the target object with the mobile device
and select the object directly using the touch screen. The selected object is then
highlighted by a bounding box and further operations can be performed. Figure [4.4]
shows an example of using the mobile device to select the object in order to move it
from one place to another inside the VE.

4.2 Evaluation Test

In this section, we will describe the test objectives, test environment, participants,
tasks, and test procedure.

4.2.1 Test Objectives

The main objective of the informal test was to assess the e↵ectiveness and subjective
satisfaction of using the mobile device as a 3D-controller to interact with the VE. The
participants were asked to interact with the VE to apply the camera manipulation
and object selection tasks, and then record their opinions of using the mobile device
for interaction instead of other devices such as the Gyro-mouse device.

4.2.2 Test Environment

All participants were given the same Sam-sung Galaxy mobile device containing the
application, and they were tested in the NexCAVE system. The experiment was
performed at KAUST in the KAUST Visualization Laboratory (KVL).
 54

Figure 4.4: Object selection screen-shots: using the mobile device to select and move
a 3D object from right to left.
 55
4.2.3 Participants

To be able to test the system, KAUST students and sta↵ was used as the test commu-
nity. The informal test was conducted with 10 participants, where five females and
five males were chosen randomly from KAUST colleges. Participants came from the
biology, engineering, and computer sciences divisions. They were familiar with Smart-
phone use, and there was no need for particular specifications or skills to perform the
test.

4.2.4 Tasks

Multiple tasks were designed to evaluate interaction using mobile and Gyro-mouse
devices. Two tasks were carried out for the mobile device. The first involved using
the mobile to manipulate the virtual camera inside the VE in order to see the object
from all sides and also to see the virtual FOV from the other side of the NexCAVE.
This will allow the user to see and interact with the VE with no limitations (see
Figure [4.5]).

Figure 4.5: Using the mobile device to manipulate the virtual camera and to see the
VE with no limitations (the picture on the corner is the VE on the display screen of
the mobile device).
 56
The second comprised using the touch screen on the mobile device for direct object
selection and manipulation to move the object from place to another. On the other
hand, one task was conducted for the Gyro-mouse device, which was using buttons
to indirectly select and manipulate the object in a 3D virtual space (see Figure [4.6]).

Figure 4.6: Direct and indirect manipulation task: on the left is the indirect manip-
ulation and on the right is the direct manipulation.

The last task was to allow two users to enter the VE, one user using a Gyro-mouse
and the other one using a mobile device to interact at the same time with the VE
(see Figure [4.7]).

Figure 4.7: Multiple users interacting with the VE using a mobile device and a Gyro-
mouse.
 57
4.2.5 Test Procedure

The informal test was conducted with each participant individually. Appointments
were scheduled with each participant 15 days before the test. Before starting the
test, participants were given a brief explanation of the purpose and procedure of the
test (see Appendix A). The client-side application had been previously installed in
a Sam-sung Galaxy S Smart-phone running the Android 2.2 platform. In the test,
participants were asked to carry the Gyro-mouse device to perform a list of tasks and
after they finish they will be asked to carry the mobile device to perform the same
tasks in order to compare between them. The time for the test was unlimited and the
duration for each participant varied from 10 to 15 minutes. After performing the test,
participants were asked to answer the user questionnaire and record their opinions
and suggestions. The questionnaire includes seven questions divided into three parts
(see Appendix A). The first part includes three comparison questions. This involves
a comparison between the mobile and the Gyro-mouse devices in terms of ease of
use, ease of learning, and which device increased the immersion and sense of presence
within the VE. These questions are as follows:

1. Which device is better to manipulate the object in VE using a touch screen or

a pointer button?

2. Which device increases the immersion and scene of presence within the VE?

3. Which of the two devices is easier to learn to perform the object manipulation
task?

The second part of the questionnaire includes three statements related to user
satisfaction. Participants are asked whether they agree or disagree with the state-
ments. These statements rate the general concept of the system which was using
mobile devices for 3D interaction within the VE. These statements are as follows:
 58
4. Using mobile devices as a 3D controller facilitates the interaction task’s manip-
ulation and object selection within the VE.

5. Using a touch screen on mobile devices to interact with VEs helps to reduce
the time it takes to learn how to manipulate objects in the virtual world by
disabling the buttons’ functions and using the touch screen.

6. I prefer to have my own mobile device to move objects by myself rather than
not using the mobile and asking the driver user to move objects from one place
to another.

The third part includes one open-ended question to know the feedback of the
participants regarding their opinions and suggestions. This question is as follows:

7. Write your opinion and suggestion? (Did you like or do not like).

4.3 Results

After performing the informal test, data were collected from the users’ questionnaires.
The questionnaire was used to investigate the comparison between the mobile and
Gyro-mouse devices. In addition, it explored the level of the user satisfaction of using
the mobile as an interaction device in the VE.

4.3.1 Comparison with State-of-the-Art Devices Results

The test evaluated the mobile as a 3D interaction device through an experiment

comparing the object selection task using two input devices, specifically the mobile
and the Gyro-mouse devices. A 3D object was presented on the NexCAVE and the
mobile screen; the user needed to select the object using the touch screen and move
it from one place to another. The experiment was repeated with the Gyro-mouse
 59
device. The results of the first part of the questionnaire regarding the comparison
between the mobile and the Gyro-mouse devices were as follows:
For the first question, 100% of the participants reported that the mobile device is
better for manipulating objects in VE using the touch screen. For the second question,
100% of the participants stated that the mobile device increases the immersion and
sense of presence within the VE. Finally for the third question, 80% of the participants
reported that using mobile devices are easy to learn how to use it while manipulating
the object in the VE, while 20% of them found that using mobile and the Gyro-mouse
devices are equivalent in ease of learning for manipulating the object in the VE.

4.3.2 User Satisfaction Results

The results of the questionnaire in the user satisfaction part were measured through
three statements (questions four, five, and six) given to the users after the comple-
tion of the tasks. The result in the first item, which was statement four, showed that
90% of the participants strongly agreed with this statement, while the other 10%
of the participants agreed that using mobile devices facilitate the interaction within
the VE (see Figure [4.8]). In addition, the result in the second statement, which
was question five, shows that 40% of the participants strongly agreed, 50% of the
participants agreed, and 10% of the participants were uncertain about whether using
mobile devices reduce time needed to learn how to deal within the VE (see Figure
[4.9]). Lastly, the result in the third item which was question six showed that all of
the participants prefer and strongly agree to interact themselves within the VE (see
Figure [4.10]).

In general, all of the participants judged that the interaction using mobile devices
within the VE is easy and comfortable to use, and they had positive attitudes toward
the system.
 60

Figure 4.8: Results - Using mobile devices to facilitate the interaction with the VE.

Figure 4.9: Results - Using mobile devices to reduce time needed to learn how to deal
with VE.
 61

Figure 4.10: Results - Users preferring to interact themselves with the VE using
mobile devices.

4.3.3 Results Summary

The results are set out in more detail as follows:

All the participants found that using mobile devices facilitate the interactions
within the VE rather than using traditional input devices.

All of the participants found that using the touch screen in mobile devices,
rather than using buttons in the traditional input devices, facilitates interacting
within the VE in an e↵ective way.

All of the participants found that using mobile devices with direct manipula-
tion by employing the touch screen to select objects by hand can increase the
immersion and sense of presence inside the virtual world.

More than half of the participants reported that using mobile devices can reduce
 62
the time needed to learn how to deal with the VE.

All of the participants preferred to use their own mobile devices to interact
with the VE rather than being close to the driver user and requesting objects
be moved in the VE.

4.3.4 Participant General Feedback

The participants’ answers, responses, and suggestions were collected from the third
part of the questionnaire (question seven). Based on the opinions of the participants
who tried our system, we could argue that using mobile devices for interaction within
the VE to select and manipulate objects is easy to learn and use, facilitates interaction
within VEs, and increases the immersion and sense of presence within the VE. We
realize this is only anecdotal evidence supported by the users’ comments. Some of
the comments were:

“Using the mobile has great features, learning slightly better than Gyro-mouse,
and overall using the mobile device with touch screen is better for interaction.”

“Using mobile devices with touch screen is easier for learning to select and
manipulate the object in the VE, because there is no way to know about the
CalVR menu on the server without guidance. Additionally, there is no doubt
that the idea of replacing the Gyro-mouse with the mobile device is an essential
progress for the VE usage, but some improvements should be considered in future
work to improve the object selection task.”

“The idea of using mobile devices as an interaction device will encourage people
to use the VE with their own devices rather than using VR devices to interact
with the virtual world.”
 63
“Using mobile devices increases the flexibility of the interaction, especially for
the users who do not know how to interact with the 3D virtual world.”

“I like the idea of allowing multiple users to interact with the VE with their own
screen to see the VE rather than using the big screen to see the VE.”

“I think using this kind of interaction method with multiple objects and users
for di↵erent application such as games, for example, will make this technique
more efficient for usage in the VE.”

4.4 Discussion

We attempted to examine the possibility of using mobile devices to interact with the
VE to perform simple tasks such as manipulation and object selection. We conducted
an informal test to compare use of the mobile and Gyro-mouse devices, and to eval-
uate user satisfaction concerning the use of the mobile device to interact within the
VE. The results demonstrated that all the tested users were willing to use their own
mobile to interact with the VE with di↵erent kinds of applications and with multiple
users.

From the previous results summary, we can say that the system can facilitate
interactions within the VE. That is because it is easy to use in terms of its simple
design, which disables the buttons’ functions and employs the touch screen for simple
tasks. Using the touch screen can also allow users to employ the interaction device
without memorizing the buttons’ functions.

Using a mobile device can increase the immersion and sense of presence inside
the virtual world by using the touch screen to move the objects. Because we used
direct manipulation interaction that allows users to become part of the VE, as they
 64
use their hands to select and manipulate the objects with the touch screen.

With regard to the usability of mobile and Gyro-mouse devices to select and ma-
nipulate objects in the VE, we can say that a few participants found that the usability
of both devices is equivalent, and this is because they already had experience using
Gyro-mouse devices before they performed the test. However, the other participants
found that the usability of the mobile was better than that of the Gyro-mouse device
in terms of ease of learning.

Moreover, all users prefer to interact with the VE themselves by using mobile
devices to feel that they are part of the virtual world and to allow them to interact
with multiple users with the VE at the same time. This happens by using their own
mobile devices rather than having only one user interacting with the virtual world
and others looking at the VE without any interaction. This provides users a more
interesting experience in virtual world and a strong sense of being part of the VE.

In brief, the use of mobile devices can solve a problem related to traditional
input devices, which use indirect manipulation interaction through the existence of
buttons. In addition, it can increase the immersion and sense of presence inside the
virtual world through the touch screen. Moreover, all participants agreed of using
mobile devices as a great interaction device within the VE because it has magnificent
features that will catch the attention of both users and developers.
 65

Chapter 5

Conclusion and Future Work

5.1 Conclusion

We have proposed alternative approaches to interact with the VE by using mobile

devices for camera manipulation and object selection tasks. With this approach, the
user is able to use the mobile device with di↵erent kinds of applications to interact
with the VE. Using mobile devices can solve the problem of using traditional input
devices within the VE, which uses indirect manipulation (interaction through but-
tons). Further, mobile devices enable direct manipulation using a touch screen which
can increase the immersion and sense of presence inside the virtual world and pro-
vides many possibilities for 3D interaction techniques in graphics and virtual world
visualizations because it o↵ers independence from traditional inputs while moving
and interacting with 3D space objects. We found that using mobiles is easy to learn
and use for the interaction within the VE. All the informal test results indicate that
using mobile device facilitates the interaction tasks and all participants agreed on the
usability of using it as a powerful interaction device within the VE.
 66
5.2 Future Work

As a future work we would perform a formal evaluation test to assess using mobile
devices from di↵erent aspects for comparing it with di↵erent kinds of VR interaction
devices. This evaluation will concern about comparing the accuracy, tasks completion
times and the e↵ectiveness of using touch screen or buttons to perform interaction
tasks within the VE. Additionally, we will build a full application to be installed on
any kind of operating system (Android OS/ iOS) and compatible with di↵erent kinds
of mobile devices. Also, further development could be done to interact with multiple
objects by allowing the user to import a full scene containing a number of objects on
the server and as well as on the client to interact with both scenes at the same time.
This needs to consider the mobile device memory limitation and the complexity of
the scene to see if the scene fits in memory on the mobile device.

Moreover, some improvements could be done to allow multiple users to interact

with one environment at the same time using their own devices locally or remotely.
Local VEs can be accomplished by adding and accepting real users to share one VE
at the same time (see Figure [5.1]). On the other hand, we can accomplish remote
VEs whether using NexCAVE or only optical tracking system. This happens by ex-
panding the network to connect between more than one local VE to add and accept
real and virtual users to the virtual environment to interact together (see Figure [5.2]).

Finally, to make the mobile phone the only device used to interact with the VE
to perform all the interaction tasks (such as manipulation, object selection, and nav-
igation), we need to add the navigation task to the previous two tasks: camera
manipulation and object selection. This will be accomplished by using two-finger
input on a touch screen to navigate the camera in the VE.
 67

Figure 5.1: Local virtual environment with multi-users.

Figure 5.2: Global virtual environment: remotely connecting between multi-local

virtual environments.
 68
5.3 Lessons Learned

We began our research project with the idea of using mobile devices to interact with
the VE. After we searched and read most of the published papers, posters, and ar-
ticles, we found that research in this field is limited in size and most of the research
projects are done with PDAs and old devices. Also, we found that most of the re-
search was concerned with using mobile devices in augmented reality (AR) and few
of it is using mobile devices for interaction. This limitation caused difficulties in col-
lecting information resources for writing. On the other hand, these few sources found
provided us with ideas and encouraged us to continue to search deeply in this field
and to focus more on some details such as mobile capabilities, allowing multi-users
to interact together, and increasing the immersion to interact with the virtual world
e↵ectively.

Accordingly, we decided to do a full application to be installed on mobile devices

to interact with the VE. This application has some features collected from existing
research and some of them our own ideas. This application is able to do all the in-
teraction tasks on one device such as manipulation, object selection, and navigation.
Also, we were aiming to do a formal study to publish the work in an international
conference. But since there was a time limit as we were restricted only to six months
for implementation and writing, we decided to do only initial results to prove that
using mobile devices as a 3D controller is able to do all tasks in VE.

We expect that using mobile devices as an interaction device will provide some
advantages. It can replace the VR devices (For example tracking systems and CAVE
systems) which are so expensive by using built-in sensors and built-in display. More-
over, it can allow people and organizations to use 3D virtual environment at any place
without restrictions.
 69

Regarding the implementation part, we have faced some obstacles with program-
ming in CalVR and OSG on the mobile device we used with this system. Here are
some lessons learned from these problems:

1. This project needs experience in CalVR library, and there is no resources for
CalVR frame work on the internet.

2. This project needs experience in JNI programming framework to build Java

interface in order to interact with OSG libraries on android OS using c++ and
Java languages.

3. Don’t use OSG with Android OS because there are few online resources to learn
about it and there are few experts in this technique. There are also some lim-
itations in programming and some mobile and tablet devices are incompatible
with this library.

4. We encourage using other graphic library platforms or tools on the mobile device
rather than using OSG library. This will make the programming easier and
without any limitations. Therefore, we recommend using Unity3D tool which
is a graphic engine that uses OpenGL ES for Android OS and other OS.

5. Some examples of the limitations in programming that we faced when we used

OSG with Android:

Only specific types of models that work with OSG in Android (e.g, .osg,
or osgt).

Wireframe modes, such as osg::PolygonMode::LINE, is not supported in

OSG for android.
 70

REFERENCES

[1] G. Nalbant and B. Bostan, “Interaction in virtual reality,” in 4th International

Symposium of Interactive Medial Design (ISIMD), 2006.

[2] E. Hodgson, E. Bachmann, D. Waller, A. Bair, and A. Oberlin, “Virtual reality

in the wild: A self-contained and wearable simulation system,” in Virtual Reality
Workshops (VR), 2012 IEEE. IEEE, 2012, pp. 157–158.

[3] W. R. Sherman and A. B. Craig, “Understanding virtual realityinterface, appli-

cation, and design,” Presence: Teleoperators and Virtual Environments, vol. 12,
no. 4, pp. 441–442, 2003.

[4] J. Steuer, “Defining virtual reality: Dimensions determining telepresence,” Jour-

nal of communication, vol. 42, no. 4, pp. 73–93, 1992.

[5] J.-S. Kim, D. Gracanin, K. Matkovic, and F. Quek, “iphone/ipod touch as input
devices for navigation in immersive virtual environments,” in Virtual Reality
Conference, 2009. VR 2009. IEEE. IEEE, 2009, pp. 261–262.

[6] D. A. Bowman, E. Kruij↵, J. J. LaViola Jr, and I. Poupyrev, 3D user interfaces:

theory and practice. Addison-Wesley, 2004.

[7] J. P. Schulze, A. Prudhomme, P. Weber, and T. A. DeFanti, “Calvr: an advanced

open source virtual reality software framework,” in IS&T/SPIE Electronic Imag-
ing. International Society for Optics and Photonics, 2013, pp. 864 902–864 902.

[8] J. H. Chuah and B. Lok, “Experiences in using a smartphone as a virtual re-

ality interaction device,” in Workshop on O↵-The-Shelf Virtual Reality, Orange
County, CA, 2012.
 71
[9] Sx02 flight simulator with surround screen. [Online]. Available: http:
//www.bornrich.com/sx02-flight-simulator-with-surround-screen.html

[10] How virtual reality gear works. [Online]. Available: http://electronics.

howstu↵works.com/gadgets/other-gadgets/VR-gear2.htm

[11] Kaust visualization core lab cornea. [Online]. Available: http://kvl.kaust.edu.

sa/Pages/CORNEA.aspx

[12] Kaust visualization core lab shaow case. [Online]. Available: http:
//kvl.kaust.edu.sa/Pages/Showcase.aspx

[13] M. Gutiérrez, F. Vexo, and D. Thalmann, “The mobile animator: Interactive

character animation in collaborative virtual environments,” in Virtual Reality,
2004. Proceedings. IEEE. IEEE, 2004, pp. 125–284.

[14] N. Katzakis and M. Hori, “Mobile devices as multi-dof controllers,” in 3D User

Interfaces (3DUI), 2010 IEEE Symposium on. IEEE, 2010, pp. 139–140.

[15] D. Lee, J.-I. Hwang, G. J. Kim, and S. C. Ahn, “3d interaction using mobile
device on 3d environments with large screen,” in Proceedings of the 13th Inter-
national Conference on Human Computer Interaction with Mobile Devices and
Services. ACM, 2011, pp. 575–580.

[16] M. M. Miguel, T. Ogawa, K. Kiyokawa, and H. Takemura, “A pda-based see-

through interface within an immersive environment,” in Artificial Reality and
Telexistence, 17th International Conference on. IEEE, 2007, pp. 113–118.

[17] K. Watsen, R. Darken, and M. Capps, “A handheld computer as an interaction

device to a virtual environment,” in Third International Immersive Projection
Technology Workshop (IPT 1999), Stuttgart, Germany, 1999.

[18] M. Weber, T. Pfei↵er, and B. Jung, “Pr@ senz-p@ ce: mobile interaction with
virtual reality,” in Proceedings of the 7th international conference on Human
computer interaction with mobile devices & services. ACM, 2005, pp. 351–352.
 72
[19] G. Marsden and N. Tip, “Navigation control for mobile virtual environments,” in
Proceedings of the 7th international conference on Human computer interaction
with mobile devices & services. ACM, 2005, pp. 279–282.

[20] K.-P. Yee, “Peephole displays: pen interaction on spatially aware handheld com-
puters,” in Proceedings of the SIGCHI conference on Human factors in computing
systems. ACM, 2003, pp. 1–8.

[21] D. M. Fotis, D. Mountain, F. Liarokapis, J. Raper, and V. Brujic-okretic, “Inter-

acting with virtual reality models on mobile devices,” in University of Salzburg.
Citeseer, 2005.

[22] S. Knödel, M. Hachet, P. Guitton et al., “Visualization and interaction with

mobile technology.” in Mobile HCI, 2008, pp. 557–558.

[23] J.-S. Kim, D. Gračanin, K. Matković, and F. Quek, “Finger walking in place
(fwip): A traveling technique in virtual environments,” in Smart Graphics.
Springer, 2008, pp. 58–69.

[24] A. Benzina, M. Toennis, G. Klinker, and M. Ashry, “Phone-based motion control

in vr: analysis of degrees of freedom,” in CHI’11 Extended Abstracts on Human
Factors in Computing Systems. ACM, 2011, pp. 1519–1524.

[25] Y. Du, H. Ren, G. Pan, and S. Li, “Tilt & touch: mobile phone for 3d in-
teraction,” in Proceedings of the 13th international conference on Ubiquitous
computing. ACM, 2011, pp. 485–486.

[26] J. Y. Lee, M. S. Kim, D. W. Seo, C.-W. Lee, J. S. Kim, and S. M. Lee, “Dual
interactions between multi-display and smartphone for collaborative design and
sharing,” in Virtual Reality Conference (VR), 2011 IEEE. IEEE, 2011, pp.
221–222.

[27] Nexcave interactive visualization system aids university research. [Online].

Available: http://www.kaust.edu.sa/media/features/nexcave.html

[28] Trackpack4. [Online]. Available: http://www.abs-tech.com/produto.php?id

produto=913
 73
[29] R. Wang and X. Qian, OpenSceneGraph 3.0: Beginner’s Guide. PACKT pub-
lishing, 2010.
 74

APPENDICES

A Appendix A

Testing and Evaluation

”Virtual Realty Interaction Using Mobile Devices”

Dear Participant, The objective of this study is to make mobile devices as a

3D-controller to interact and perform simple tasks such as manipulation and object
selection inside the virtual environment. The purpose of the informal test is to de-
termine the usability, the e↵ectiveness, and the subjective satisfaction of using the
mobile device as an interaction device in the 3D virtual world. The e↵ectiveness rep-
resents the immersion and sense of being part of the virtual environment by touching
and moving the 3D object directly by hand. The test will take at least fifteen min-
utes. You will be given few tasks to perform with the mobile device and then you will
repeat the same tasks with Gyro-mouse to compare between them. After completing
all the tasks, you will be asked to answer the questionnaire and write your opinion and
suggestions. Any comments and suggestions of your thoughts about the interaction
using mobile devices are supposed to contribute to this study to get better insights
and results.
Thanks for your participation,
Sahar Aseeri
 75
Tasks List:

1. Object Selection using Gyro-mouse device

Select and manipulate object using left button using Gyro-mouse device.

2. Camera Manipulation using mobile device

Find the object using mobile display screen in the virtual environment as a
virtual camera.

3. Object Selection using mobile device

Select and manipulate object using one-finger input on a touch screen by using
mobile device.

4. Using the mobile and the Gyro-mouse at the same time.

Two users one use mobile device and the other one use Gyro-mouse device.

Comparison Questions:

1. Which device is ”better” to manipulate the object in VE using touch screen

or pointer button?

Mobile (Direct interaction using touch screen)

Gyro-mouse (Indirect interaction using the pointer button)

2. Which device ”increases” the immersion and sense of presence within the vir-
tual world?

Mobile

Gyro-mouse
 76
3. Which of the two devices is ”easier” to learn to perform the object manipulation
task?

Mobile

Gyro-mouse

User Satisfaction:

4. Using mobile devices as a 3D controller ”facilitate” the interaction tasks ma-

nipulation and object selection within the virtual environment.

Strongly Agree Agree Uncertain Disagree Strongly Disagree

5. Using a touch screen on mobile devices to interact with IVEs helps to reduce
the time it takes to learn how to manipulate objects in the virtual world by
disabling the buttons functions and using the touch screen.

Strongly Agree Agree Uncertain Disagree Strongly Disagree

6. I prefer to have my own mobile device to move objects by myself rather than
not using the mobile and asking the driver user to move objects from one place
to another.

Strongly Agree Agree Uncertain Disagree Strongly Disagree

Open-ended Question:

7. Write your opinion and suggestion? (Did you like or do not like)