Evaluating Usability of VANET Applications

Vlada Emanuel George, SSA

emanuel_george.vlada@cti.pub.to
Ciprian Dobre
ciprian.dobre@cs.pub.ro

Abstract
A vehicular ad hoc network consists of smart
vehicle on the road and provides communication
services among nearby or with roadside infrastructure.
It is envisioned to provide numerous interesting
services in the near future. The major prupose of
VANET is to provide ubiquitous connectivity while on
the road to mobile users, tho are otherwise connected
to outside world through other networks at home or ar
the work place, and efficient vehicle-to vehiche
communicationd that enable the Intellingent
Transportation Systems. Today, VANET is very
important for researchers beacause they trying to find
solutions to
reduce traffic accidents and
inconveniences. VANET applications can distract
drivers attention and they can not drive safely and
properly. In this paper, we propose a simulation for
evaluating safe usability of VANET applications
together with a testing methodology to assess the
impact of their use on driving capabilities. The results
will be displayed graphically and will see what devices
affects more driving.

1. Introduction
A Vehicular Ad-Hoc Network, or VANET is a
technology that uses moving cars as nodes in a network
to create a mobile network. VANET turns every
participating car into a wireless router or node,
allowing cars approximately 100 to 300 metres of each
other to connect and, in turn, create a network with a
wide range. As cars fall out of the signal range and
drop out of the network, other cars can join in,
connecting vehicles to one another so that a mobile
Internet is created. It is estimated that the first systems
that will integrate this technology are police and fire
vehicles to communicate with each other for safety
purposes.
VANET offers several benefits to organizations of
any size. While such a network does pose certain safety

concerns (for example, one cannot safely type an email

while driving), this does not limit VANETs potential as
a productivity tool. GPS and navigation systems can
benefit, as they can be integrated with traffic reports to
provide the fastest route to work. A computer can turn a
traffic jam into a productive work time by having his
email downloaded and read to him by the on-board
computer, or if traffic slows to a halt, read it himself. It
would also allow for free, VoIP services such as
GoogleTalk or Skype between employees, lowering
telecommunications costs. Future applications could
involve cruise control making automatic adjustments to
maintain safe distances between vehicles or alerting the
driver of emergency vehicles in the area. VANET
applications need mobile devices for interacting with
car's drivers, the car itself, and woth other card and the
road infrastructure. Dervicers become more and more
besieged with information coming from a wode range of
surrounding sources. Studies show that mobile phone
has a negative influence on drivers and in some
countries is forbidden talking on the phone while driving
[8].
Besides talking on the phone, this situation of
distraction of the driver is enhanced with the
introduction of VANET applications. Such applications
are designed to assist the driver in making correct
decisions and they usually have an informative role,
presenting a particular situation to the driver, who can
further act upon the information and decide on the
correct action.
User of VANET applications is invaded by an
explosion of information and this can detemine some
psychological problems that the driver is no longer
concentrate to the traffic. Vehicle manufacturers that
introducing new VANET applications are more
interested in testing their ability to provide accurate
information than their driver influence.
In this paper, we present a way to evaluate the
effects of the use of vehicualar applications. Usability

problem is approached by using a driving simulator with

additional capabilities: an interface between the user and
various VANET applications and components necessary
to monitor and evaluate the influence of the application
on the test subjects. We also present how this solution
can be used with a usability testing methodology to
assess the psychological impact on driving ability of
drivers. Finally, we present the results graphically.
Both the simulator, the proposed methodology, and
results presented in usability evaluation are important
tools for researchers interested in validating the safe
usage of VANET applications in real situations. These
results together with possible solutions have a great
potential to save lives. Drivers, today, are becoming
more surrounded with technology and information and
these drawing his attention from driving.
This paper is organized as follows. Section 2
presents the advantages and disadvantages of driving
simulators. Section 3 presents the related work. In
Section 4, we present the application that we use for
simulation experiments. Below we present a
methodology for usability evaluation of VANET
applications. In Section 6 are presented obtained results
with the proposed solutions. Finally, in Section 7 we
present the conclusions and present future work.

2. Related Work
Driving is a complex task, requiring the concurrent
execution of various cognitive, physical, sensory and
psychomotor skills. Despite these complexities, it is not
unusual to observe drivers engaging in various non
driving-related activities while driving. These activities
range from conversing with passengers and listening to
the radio, to applying make-up and even reading. With
the advent of wireless communication, more
sophisticated entertainment systems and the
introduction of technologies such as route navigation
and the Internet into vehicles, preoccupation with
electronic devices while driving is also becoming
increasingly common (ex.: VANET applications). Any
activity that competes for the drivers attention while
driving has the potential to degrade driving
performance and have serious consequences for road
safety.
Research by the National Highway Traffic Safety
Administration (NHTSA) estimates that driver
inattention, in its various forms, contributes to
approximately 25 percent of police-reported crashes.
Driver distraction is one form of driver inattention and

is claimed to be a contributing factor in over half of

inattention crashes. However, as more wireless
communication, entertainment and driver assistance
systems proliferate the vehicle market, it is likely that
the rate of distraction-related crashes will escalate.
When driving, drivers must continually allocate
their attentional resources to both driving and nondriving tasks. Because many aspects of the driving task
become automated with experience, drivers are often
capable of dividing their attention between concurrent
tasks without any serious consequences to driving
performance or safety. Drivers are also capable of
adapting their driving to meet the demands of the
driving environment or compensate for a decrease in
attention to the driving task. Drivers can, however, be
distracted by an activity or event to the extent that
they no longer allocate sufficient attention to the
driving task and their driving performance is
compromised. In this sense, driver distraction results
when drivers normal cognitive processes and adaptive
strategies fail and drivers are no longer able to
adequately divide their attention between the driving
and secondary tasks and maintain driving performance
at a satisfactory level. Distraction can occur either
because the secondary task is so complex or
compelling that drivers fail to allocate (or prioritise)
sufficient attention to driving, or because the demands
of the driving task are so high that they do not allow
the performance of a secondary task at any level.
Driving means changing constantly location. This
means a constant demand for information on the
current location and specifically for data on the
surrounding traffic, routes and much more. This
information can begrouped together in several
categories.
A very important category is driver assistance and
car safety. This includes many different things mostly
based on sensor data from other cars. One could think
of brake warning sent from preceding car, tailgate and
collision warning, information about road condition
and maintenance, detailed regional weather forecast,
premonition of traffic jams, caution to an accident
behind the next bend, detailed information about an
accident for the rescue team and many other things.
One could also think of local updates of the cars
navigation systems or an assistant that helps to follow a
friends car.

Another category is infotainment for passengers.

For example internet access, chatting and interactive
games between cars close to each other.
Next category is local information as next free
parking space (perhaps with a reservation system),
detailed information about fuel prices and services
offered by the next service station or just tourist
information about sights.
A possible other category is car maintenance. For
example online help from your car mechanic when
your car breaks down or just simply service
information.
So far no inter-vehicle communication system for
data exchange between vehicles and between roadside
and vehicles has been put into operation. But there are
several different research projects going on [7] [4].
Driving the car is a ability that requires full attention
to safely control the vehicle and respond to events
happening on the road. When the driver is distracted
attention moves from driving to other activities. There
are three types of distraction [25]: visual (driver is not
paying attention to traffic), cognitive (when driver is no
longer focused on the road), and manual (when the
driver no longer take the wheel). Previous studies have
shown ([30], [31]) using real conditions, the existence
of a relationship between safety and activities that
distract the driver.
The evaluation of the usability of an application is
an activity involving both time and human resources.
Automating this activity can reduce costs enormously,
and can produce great results. This process involves
some tests on user's applications. These experiments
deal with both qualitative and quantitative aspects of
usability. For greater accuracy, are include measures
such as the time to complete an activity, number of
errors, and success and failure to complete a task. The
results are based on Creating a gauge to Which Further
the results are compared. The number of successful
completion of the tasks Further is used to determine the
difficulty of the activity.
The evaluation of the usability of an application is
an activity involving both time and human resources.
Automating this activity can reduce costs enormously,
and can produce great results. This process involves
some tests on user's applications. These experiments
deal with both qualitative and quantitative aspects of
usability. For greater accuracy, are include measures

such as the time to complete an activity, number of

errors, and success and failure to complete a task. The
results are based on Creating a gauge to Which Further
the results are compared. The number of successful
completion of the tasks Further is used to determine the
difficulty of the activity.
Simplicity is today an important aspect when
referring to designing interfaces between humans and
computers. The lack of content is no longer a problem;
but the lack of time to correlate, categorize, analyze
and act based on presented information is an important
issue. This is why an important research direction in
usability is towards estimating psychological and
physiological aspects affecting the user. Up until
recently, the evaluations followed a traditional
approach that separates the study of utility by the study
of emotions. Still, research showed that emotions play
an active role and should be considered when designing
interfaces between applications and users.
DRIVE is a project targeting the experimental
validation of the service performances in case of intervehicular applications [20]. Unlike our approach, the
DRIVE project assume hardware components designed
to test under realworld conditions the effect of using
inter-vehicular services. It actually includes several
services implemented using an OGSi based
communication framework. Examples include an
intelligent driving assistant capable of informing the
driver when various dangerous situations occur on the
road, and a service capable to monitor the driving
pattern of a particular subject and create a profile that
can be further used by insurance companies. However,
the focus of the DRIVE project is towards creating the
infrastructure to test the utility and performance of such
services in a framework providing real-world driving
conditions. Unlike this, we propose a solution to
evaluate the usability of such services, what impact
they have on the safety of driving.

3. Advantages and disadvantages of

driving simulators
I.

Advantages of driving simulators

Driving simulators offer various

compared to real vehicles, including:

advantages

a) Controllability,
reproducibility,
and
standardization. Behavior of virtual traffic, weather
conditions, and the road layout can be manipulated
(offline or in real-time) as a function of the training

needs or research aims. Purpose-developed

scenarios enable trainees to practice a large number
of dedicated maneuvers per time unit. Wassink et
al. [29] describe software architecture for
generating dynamic scenarios in a driving simulator.
With the aim of maximizing the effectiveness of the
training, the authors apply a metaphor from the
1998 movie The Truman Show: everything
surrounding the learner driver responds to the
drivers behavior. Using simulators, participants in
different physical locations can drive under the
exact same conditions. This is beneficial for
creating standardized driving tests and reproducible
research results. In contrast, the real traffic
environment is largely random.
b) Ease of data collection. A driving simulator can
measure performance accurately and efficiently.
With a real vehicle, it is far more cumbersome to
obtain complete, synchronized, and accurate
measurement data. It is a fundamental challenge to
get an accurate recording of where a real vehicle
actually is in the world. For example, in one study
using an instrumented vehicle and a driving
simulator, it was impossible to determine the
distance between the vehicle and a stop line on the
road, while in the simulator this information was
readily available [9]. Measurement of lateral
position is challenging as well, as this requires
visible lane markers while weather conditions,
reflection, and shades may affect the quality of the
measurement [22]. Santos et al. [23] found that
lateral position measurements of the instrumented
vehicle were of marginal quality while this
information was accurate in the simulator, leading
the authors to conclude that problems with field
studies in an instrumented vehicle have been
confirmed (p. 145). Because of the measurement
capabilities of simulators, new types of behavior
analyses come within reach, such as trigonometric
analysis of time-to-line crossing [27] or object
detection and hazard perception research using eyetracking [26].
c) Possibility of encountering dangerous driving
conditions without being physically at risk.
Simulators can be used to prepare trainees to handle
unpredictable or safety critical tasks that may be
inappropriate to practice on the road, such as
collision avoidance or risky driving [22]. In
addition, simulators make it possible to study
hazard anticipation and perception by exposing
drivers to dangerous driving tasks, which is an

ethically challenging endeavor in real vehicles [26].

Flach et al. [6] stated that simulators Proceedings of
Measuring Behavior 2012 (Utrecht, The
Netherlands, August 28-31, 2012) Eds. A.J. Spink,
F. Grieco, O.E. Krips, L.W.S. Loijens, L.P.J.J.
Noldus, and P.H. Zimmerman 47 offer an
opportunity to learn from mistakes in a forgiving
environment (p. 134). Allen et al. [1] made a
similar case: Motor vehicle crashes are
significantly higher among young drivers during the
first year of licensure, and crash risks decline with
increased experience. [] This produces an
interesting dilemma about how to provide young
drivers
with
driving
experience
without
significantly increasing their crash risk. Driving
simulation may be the solution to this dilemma.
d) Novel opportunity for feedback and instruction.
Simulators offer the opportunity for feedback and
instruction that is not easily achieved in real
vehicles. For example, it is possible to freeze, reset,
or replay a scenario [5]. Feedback and instructions
can also be delivered in other modalities besides
speech, such as visual overlays to highlight critical
features in the environment.
II.

Disadvantages of driving simulators

However, simulators have several

disadvantages and challenges, including:

known

a) Limited physical, perceptual, and behavioral

fidelity. Low-fidelity simulators may evoke
unrealistic driving behavior and therefore produce
invalid research outcomes. Simulator fidelity is
known to affect user opinion. Participants may
become demotivated by a limited-fidelity simulator
and prefer a real vehicle instead (or a more costly
high-fidelity
simulator
for
that
matter).
Interestingly, while safety is often cited as an
advantage of driving simulation (see above),
sometimes this same feature is interpreted as a
disadvantage. For example, Kppler [14] pointed
out that real danger and the real consequences of
actions do not occur in a driving simulator, giving
rise to a false sense of safety, responsibility, or
competence. Simply investing resources to increase
fidelity is not necessarily a desirable solution, as it
adds to the complexity of the device and might
hamper experimental control. In some cases,
deliberate deviations from reality yield valid results
[16][21]. Evans [18] provided an interesting
thought experiment, arguing against a blind focus

on high-fidelity driving simulation: Consider a

make-believe simulator consisting of an actual car,
but with the remarkable property that after it crashes
a reset button instantly cancels all damage to people
and equipment. What experiments could be
performed on such make-believe equipment that
would increase our basic knowledge about driving?
The answers provide an upper limit on what might
be done using improved simulators (p. 190).
b) Shortage of research demonstrating validity of
simulation. A growing body of evidence indicates
that driving-simulator measures are predictive for
on-the-road driving performance. However, only a
few studies have investigated whether skills learned
in a driving simulator transfer to the road. Note that
in the field of aviation, studies on the transfer of
training are far more common [12], but even in
aviation critical questions remain unanswered, for
example whether a motion base provides added
value for the effectiveness of flight training [24].
c) Simulator discomfort, especially in older people or
under demanding driving conditions. Simulator
sickness symptoms may undermine training
effectiveness and negatively affect the usability of
simulators. This is a serious concern, but
fortunately, useful technological and procedural
guidelines are available to alleviate it [15].
Research shows that simulator sickness is less of a
problem for young drivers [3]. Experience shows
that limiting the horizontal field of view, avoiding
sharp curves or stops during driving, and using
short sessions (10 min) with sufficient rest breaks
improves or even eliminates simulator sickness.

specific to a typical VANET application, in the lower

right corner of the screen. In the backend, there are
specific components for measuring the physiological
impact of the VANET application on the driving
behavior of the user. They measure the driving
accuracy, speed to complete a driving circuit, etc. In
addition, the system also includes a component that
facilitates the evaluation of the application when
involving communication with other distributed
workstations.
The architecture provides complete separation
between components. This facilitates the integration of
different VANET applications, the change of the
driving scenes, monitoring new psychological
parameters.
The driving simulator is based on the Vehicle
Dynamics Engine, provided by Insight Machines [2]. It
is a Java-based 3D engine, with a realistic vehicle
dynamics engine. The engine is built with a virtual 3D
simulation of a landscape to navigate around. The car
realistically reacts according to the environmental
physical forces. The landscape can feature even and
uneven grounds with hills and cliffs, obstacles such as
trees, rocks, building, and vehicles. The driving
simulator also includes realistic interactions between
objects in the scene (collisions, bumps, etc).

4. System Design
The architecture of the system is presented in Figure
1. It consists of several inter-connected drivingsimulators running on several workstations. On each
station a user runs a custom driving simulator, which
includes a user interface specific to a particular
VANET application. This emulates driving in a
realworld scenario, while having a mobile device (a
smart phone for example) running a particular VANET
application.
The main window of the application consists of a
roaddriving simulator. It serves to test the reflexes of
the user while performing specific driving activities:
drive straight, make turns, etc. In addition, while
driving, the user is also presented with a user interface

Figure 1. The design of the system

A driving simulator was chosen as a better

alternative then using real-world driving conditions
because of the costs involved, but also the possibility to
adjust and control driving conditions, as well as the
possibility to measure driving parameters more
accurately. As the authors in [28] previously shown, the
learner drivers' simulator measures actually relate to
on-road driving. In fact, according to the study, a
higher chance of passing the driving test the first time
was associated with making fewer steering errors on
the simulator and could be predicted in regression
analysis with a correlation of 0.18.
Our system allows the user to model the scene, and
to specify the user interface of a VANET application.
The user interface can include sounds, graphical
aspects, etc. We added, for example, a special text-tospeech component capable of emulating an application
where the user has to actively listen to various
indications.
A
typical
VANET
application
includes
communications components that allows for car-to-car
or car-to infrastructure communications. In our system,
we allow a distributed testing where we want to
evaluate the psychological impact of the
VANET application when used in scenarios
involving communications with other drivers. When
analyzing the driving behavior pattern, we also
correlate the results between the distributed involved
nodes.
The behavior analyzer looks at the users capability
to correctly finish a predefined driving circuit. It
monitors parameters such as time to finish, driving
accuracy in terms of collisions, and number of breaks.
The usability testing methodology, further discussed in
the next Section, is based on a number of statistical
results. The study group is composed of a population of
users and statistical results of their driving capabilities,
collected when users are and when they are not affected
by the VANET application. A profiler component
extracts statistical results and creates a profile for each
driving scenario. The analyzer then looks at statistical
deviations, based on these profiles, and reports back
inconsistencies.

5. Usability Test Methodology

The interface for Human-Computer Interaction is a
critical component when designing services designed
for use in cars. Today, many car manufacturers and

software companies are working together to provide

modern services to the driver. The driver faces a
complex human-computer interaction, which demands
occasionally the focus of the driver and can distract his
attention from driving.
Driving is a skill that requires full attention to safely
control the vehicle and respond to events happening on
the road ahead. The driver is said to be distracted when
its focus moves from driving to other activities.
The usability testing methodology considers both
visual and auditory factors affecting the driving
behavior. Using the proposed simulator, the driver has
to correctly finish, as fast as possible, a predefined
driving circuit. We are interested in the drivers
reactions when confronted with various obstacles, how
well he adapts speed. At the same time, the VANET
application can use a wide variety of distractions, such
as text, colors, and sounds. The methodology is based
on the one proposed in [29].
At the beginning of 90s, Jacob Nielsen proposed
testing the usability using small-size target groups
(usually up to five test subjects) and small-size tests
during the stages of software development [19]. He
later published his findings and proposed a
mathematical model for the heuristic evaluation of
usability. Behind this model, there is a formula that
states that the number of usability problems found in
test involving n users is N (1 - (1 - L)n ). In this
formula, N is the number of actual problems, and L is
the probability of finding problems when using one
user. A typical value for L is 31%, which yields a
usability detection pattern as presented in Figure 2.
The graph shows that one test subject can discover
almost a third of the usability problems. When testing
with a second subject, he will repeat many of the
actions already taken by the first subject. Therefore, the
number of discovered errors is reduced. But subjects
are different and, because they still take different
actions, it leads to an amount of different usability
problems being discovered. This continues for the
other test subjects. After the fifth test subject,
collecting more data will not lead to many new
findings.

Figure 2. Typical usability results [20].

J. Nielsen also introduced an heuristic usability
evaluation method based on several usability heuristics
and aspects [11]: visibility of system status means,
relation between the system and real-world, user
control and his freedom, consistency and standards
being used, error prevention, recognition, flexibility,
esthetics and minimalist design, help in error
recognition and recovery, and help and documentation.
For example, the system should be able to
communicate with the user using his language, and
using words and phrases that are familiar to the user,
rather than technocratic idioms. The user should not
waste time wondering if a particular context was
already deal with. The dialogs should communicate just
enough information as required.
Figure 3 shows the steps involved in designing the
usability experiments. We begin by carefully defining
the objectives of the evaluation experiment. The
objectives can be generic (improving user satisfaction)
or specific (evaluating the efficiency in using a
particular component). The next step consists in
choosing the subjects that participate in the experiment.
This involves decision on representative categories of
subjects, based on: experience level, age, experience in
using similar applications, etc. Various research
experiments show that using between three and five
subjects is enough to discover around 80% of the
usability problems.

Figure 3. The methodology for evaluation of usability

[12].

Subsequently, we decide on the types of tasks and

scenarios to use. The tasks must resemble real
situations, they should represent activities that users are
normally expected to perform when operating the real
application. The scenarios can be expressed by
defining the set of constraints, and usually include both
normal and unexpected situations.
The next step involves measuring the usability level
of the system under test. This involves collecting
subjective data (customer satisfaction, difficulty of use,
etc.) and quantitative data (time to finish a task, the
number of times the user requested help, etc).
Additional testing methods involve instructing the user
to narrate all of his actions (the "Thinking Out Loud"
procedure), the collaborative discovery procedure
where two participants execute the same tasks in
parallel, and the active intervention procedure where
one person stimulates the participants to reflect on the
events occurring during the test session. The
experiment can also involve other evaluation methods,
such as questionnaires and interviews [17].
The final step involves preparing the environmental
stage for the experiment. This means introducing
sensors and other devices to record information,
establish the role of each members of the testing team,
and prepare support materials.

6. Experimental Results
We conducted five sets of experiments: 1) simulation
using a simulated desturbing application (yahoo
weather); 2) simulation using a navigation application
with sound; 3) simulation using a navigation
application without sound; 4) simulation using a
navigation application running on the smart phone; and
5) simulation without any application.
These experiments involved five subjects that were
chosen such that to cover various degrees of driving
experiences and experiences in using modern
technology. The drivers have similar technical skills
and the environment used was the same for all of them,
one in which users do not have any other distractions
beside the ones generated in the test applications,
controlling their behavior.
For the simulation experiment we constructed a
circuit resembling one used in driving schools (Figure
4). The circuit is equipped with driving poles. The
monitored parameters were the time needed to
complete the circuit, number of collisions with the
poles, and the number of sudden breaks.

Figure 4. The simulation experiment (in the bottom right

corner is thedisturbing application simulating a navigation
application).

Figure 5. Results for all five simulations (time).

Figure 6. Results for all five simulations (collision).

Figure 7. Results for all five simulations (brakes).

The first experiments involved the use of the

driving simulator. The navigation application runs on
the same screen as the driving simulator, in the bottom
right corner (see Figure 4). We measured the effects of
using within the simulated navigation application of
texts with various colors for fonts and background
(measuring the effect of the contrast perception for
example), and sounds.

weather) and the most dangerous is navigation

application running on the smart phone.

The results obtained for these experiments are

presented above. The subjects were asked to use the
driving simulator and complete the circuit. To evaluate
the effect of driving using one hand while performing
some actions, we asked our subjects to interact with the
application and change certain parameters.

Finally, we can say that VANET applications

should not influence the driver when his driving. They
must be constructed to not attract the driver's attention
when they are used. They should help drivers to reach
their destination quickly, give a safe environment and
easy driving.

In the first experiment we used only the driving

simulator, without showing the simulated navigation
application. We next compared the obtained results
(see figures 5,6 and 7) with the ones obtained when the
subjects were presented with the simulated navigation
application. In this case, the subjects were again asked
to complete the same circuit, but now we used a
simulated desturbing application (yahoo weather).
Then, users need to complete the circuit when the
simulated navigation application involves text, but with
sounds and after that an another application like this
but without sounds. The fifth series of results were
obtained in similar conditions, but this time with a
simulation using a navigation application running on
the smart phone.
The increase in the time needed to complete the
circuit corresponds to a psychological effect on the
driving behavior previously noticed by a team from
Carnegie Mellon [13]. They too showed that sounds
leads to a dramatic effect over the capability of people
to concentrate on solving various tasks. The visual and
acoustics factors have, therefore, different capabilities
of affecting the driving accuracy.
According to the results, we can say that the route
was completed quickly without showing the simulated
navigation application (with an average of 119.92s),
and the slowest is simulation using a navigation
application running on the smart phone (with an
average of 232.78s).
About safety (collisions) , we can say that the most
circuit was completed quickly without showing the
simulated navigation application (almost on par with
using a simulated desturbing application - yahoo

Braking is a factor that slows the driver to reach

their destination but often used, can wear car.
Simulation using a navigation application running on
the smart phone had the greatest influence on braking
and driving without simulated application was safer.

7. Conclusions and future work

Because of the increasing potential of computer
technology, we foresee increasing use of driving
simulation in areas such as driver assessment, driver
training, research, and entertainment. Low-cost virtualreality applications will come within the reach of many
organizations. However, several research questions
may need to be answered before ubiquitous driving
simulation becomes feasible, particularly questions
related to simulator fidelity, predictive validity of
driving simulators, simulator-to-reality transfer of
learning, and simulator discomfort.
The distraction caused by interacting with invehicle devices while driving has been shown to
significantly impair a drivers ability to maintain speed,
throttle control and lateral position on the road. It can
also impair drivers visual search patterns, reaction
times, decision-making processes and can increase the
risk of being involved in a collision.
In this paper, we presented an approach to evaluate
the usability effects of introducing VANET
applications in cars. With the increased interest in
VANETs coming from scientists trying to find
solutions to increase safety of passengers, or solve
congestions and decrease pollution in urban road
scenarios, we believe it is important to properly
evaluate usability of such applications. Still, little
research was conducted in this area. This is surprising
as usability of VANET applications is ultimately a
safety issue. For example, previous studies showed the
negative effect of talking on mobile phones while
driving, and as a result in many countries this is
forbidden by law. Nevertheless, currently, people drive
while using intelligent assistants designed to help with

navigation, with fuel economy, or just entertain the

user. And yet, researchers failed to test their
psychological and physiological effects of their use on
the driving capabilities of users.
The simulator, the proposed methodology and the
obtained results in the evaluation of usability are
important tools for all researchers interested in
validating the safety of using VANET applications in
real-world situations. To our knowledge, no previous
experiments managed to evaluate the safety of using
real-world applications already in use on a large-scale
by drivers around the world. Our solution consists in
the use of a driving simulator augmented with
additional capabilities: an interface between the user
and various VANET applications, and the components
necessary to monitor and assess the influence of the
interface on the driving capability of the test subjects.
We presented how this approach can be used together
with a usability test methodology to evaluate the
psychological impact of using such applications on the
driving capabilities of persons..

we also plan to include in the usability methodology

solutions designed to test the effect of VANET
applications on the emotions dimension of usability.
Last, but not least we need to be able to modify test
case scenarios so that we can recreate a real life circuit,
could make possible the comparisons between real life
test bed data and statistical ones. A needed feature will
be to give the ability to introduce unexpected factors,
but when we do so, we will need to highlight the used
methodologies (E.g. if we have a scenario in which a
hole appears before the car we need to test an
application that warns or assists the user, not a
navigation application).

References
[1] Allen, R.W., Park, G.D., Cook, M.L., Fiorentino,
D. (2007). "The effect of driving-simulator fidelity
on training effectiveness. Proceedings of the
Driving Simulation Conference North America",
Iowa City, IA.

The results can be used by application designers, as

we demonstrated the different effects of using various
types of interface controls between applications and the
drivers. In particular, we showed how different sounds
and graphical controls affect the drivers behavior.
Such results can be the basis of a methodology that can
be used by the developers of VANET applications to
better assist the end user, the driver.

[2] T.D. Boham, Vehicle Dynamics Engine (VDE),

Computer Systems Engineering Course, Last
accessed December 20, 2010.

We also evaluated the usability of some of the most

popular today applications for navigation assistance.
Surprisingly, we revealed several problems when using
such applications while driving. We plan to investigate
further our discoveries, as we feel that the safety issue
has been insufficiently addressed before, resulting in
potential hazardous controlled applications. Such
scientific results, coupled with potential solutions to
addressing usability, have in our opinion a great
potential to saving lives, as the drivers, today, are
becoming more surrounded with technology and
information.

[4] CarTalk: www.cartalk2000.net

As future work, we plan to further develop the

approach. We plan to collect more information about
the usability of VANET applications, which can further
assist the developers of such applications. Also, in the
current area of usability, researchers try to approach the
third dimension of usability, namely emotions. Various
results were obtained in this area, based on tracking
face expression or using various sensors. In the future,

[8] G. Kolata, First proof: driving while talking on

phone is a hazard., The New York Times, p. A30, Feb. 13 1997.

[3] Brooks, J.O., Goodenough, R.R., Crisler, M.C.,

Klein, N.D., Alley, R.L. et al. (2010). "Simulator
sickness during driving simulation studies.
Accident Analysis and Prevention" 42, 788-796.

[5] Evans, L. (2004). Traffic safety. Bloomfield Hills,

MI: Science Serving Society.
[6] Flach, J.M., Dekker, S., Stappers, P.J. (2008),
"Playing twenty questions with nature (the surprise
version): Reflections on the dynamics of
experience. Theoretical Issues in Ergonomics
Science" 9, 125- 154.
[7] FleetNet: www.et2.tu-harburg.de/fleetnet

[9] Godley, S. T., Triggs, T.J., Fildes, B.N. (2002),

"Driving simulator validation for speed research.
Accident Analysis and Prevention" 34, 589-600.

[10] Hoeschen, A., Verwey, W., Bekiaris, E., Knoll,

C., Widlroither, H., De Waard, D., et al. (2001),
"TRAINER: Inventory of driver training needs and
major gaps in the relevant training procedures".
[11] E. Hvannberg, E. Law, M. Lrusdttir, Heuristic
Evaluation: Comparing Ways of Finding and
Reporting Usability Problems, Interacting with
Computers, 19 (2), pp. 225-240, 2007.
[12] Jacobs, J.W., Prince, C., Hays, R.T., Salas, E.
(1990), "A meta-analysis of the flight simulator
training research" (TR-89-006). Orlando, FL:
Naval Training Systems Center.

simulator using a concurrent telephone dialling

task". 42, 1015-1037.
[22] Roskam, A.J., Brookhuis, K.A., De Waard, D.,
Carsten, O.M.J., Read, L., Jamson, S., Victor, T.
(2002), "Development of experimental protocol".
[23] Santos, J., Merat, N., Mouta, S., Brookhuis, K.,
De Waard, D. (2005), "The interaction between
driving and in-vehicle information systems:
Comparison of results from laboratory, simulator
and real-world studies". Transportation Research
Part F 8, 135-146.
[24] Strategic HighWay Research Program.

[13] M.A. Just, T.A. Keller, J. Cynkar, A decrease in

brain activation associated with driving when
listening to someone speak, Brain Res., 1205:70-,
80, April 2008

[25] J.C. Stutts, D.W. Reinfurt, L. Staplin, E.A.

Rodgman, The Role of Driver Distractions in
Traffic Crashes, AAA Foundation for Traffic
Safety, New York, May 2001.

[14] Kppler, W.D. (1993), "Views on the role of

simulation in driver training" 5.12-5.17.

[26] Underwood, G., Crundall, D., Chapman, P.

(2011), Driving simulator validation with hazard
perception. Transportation Research Part F 14,
435-446.
[27] Van Winsum, W., Brookhuis, K. A., De Waard,
D. (2000), "A comparison of different ways to
approximate time-to-line crossing (TLC) during
car driving. Accident Analysis and Prevention" 32,
47-56.

[15] Kolasinski, E. M. (1995), "Simulator sickness in

virtual environments". Alexandria, VA: U.S. Army
Research Institute for the Behavioral and Social
Sciences.
[16] Lee, J.D. (2004), "Simulator fidelity: How low can
you go?" Santa Monica, CA.
[17] J.R. Lewis, IBM Computer Usability Satisfaction
Questionnaires: Psychometric Evaluation and
Instruction for Use, International Journal of
Human-Computer Interaction, 7(1), pp. 57-78,
1995.
[18] National Highway Traffic Safety Administration
(NHTSA), Last accessed January 02, 2011, from
http://www.nhtsa.gov/.
[19] J. Nielsen, Usability Engineering, San Diego:
Academic Press. pp. 115-148, 1994.
[20] C. Pinart, I. Lequerica, I. Barona, P. Sanz, D.
Garcia, D. Sanchez- Aparisi, DRIVE: a
reconfigurable testbed for advanced vehicular
services and communications, (WEEDEV), in
Proc. TRIDENTCOM 2008, Innsbruck, Austria,
March 2008.
[21] Reed, M.P., Green, P.A. (1999), "Comparison of
driving performance on-road and in a low-cost

[28] Vlakveld, W.P. (2005), "The use of simulators in

basic driver training".
[29] Wassink, I., Van Dijk, B., Zwiers, J., Nijholt, A.,
Kuipers, J., Brugman, A. (2006), "In The Truman
Show: Generating dynamic scenarios in a driving
simulator. IEEE Intelligent Systems" 21, 28-32.
[30] De Winter, J.C.F., Dodou, D., Mulder, M. (2012),
"Training effectiveness of whole body flight
simulator motion: A comprehensive meta-analysis.
The International Journal of Aviation Psychology"
22, 164-183.
[31] J.C. de Winter, S. de Groot, M. Mulder, PA.
Wieringa,
J.
Dankelman,
J.A.
Mulder,
"Relationships
between
driving
simulator
performance and driving test results", Ergonomics,
52(2), pp. 137-153.