ARTIFICIAL PASSENGER

Narendra Kumar Tripathi (1MV07CS064)

Piyush Anand (1MV07CS069)
Rishabh (1MV07CS088)
Vishnukumar PG (1MV07CS121)

OVERVIEW
Telematics device developed by IBM based
on inventions covered by a US patent
Used in a motor vehicle to ensure that the
driver stays awake while driving
Holds a conversation with a driver using
speaker and microphone.
Conversation based on a personalized
profile of the driver

 It uses concept of facial recognition to

evaluate the driver facial state.
Natural language e-companion.
It is also integrated with wireless services
to provide weather and road information,
driving directions.

WHY ARTIFICIAL PASSENGER?

According to a national survey in UK and USA, it is
observed that the driver fatigue annually causes
10000 crashes
1500 deaths
7100 injuries
which causes annual cost of $12.5 billions.
Majority of off-road accidents observed were preceded
by eye closure of half and even 2-3 seconds where
the normal human eye blinks at 0.2-0.3 seconds

ROAD ACCIDENT STATISTICS

38%

Human Error
Road Factor
Vehicle Factor

57%

Combination of the 3

3%

2%

ROAD ACCIDENT CAUSE

HUMAN ERROR
Driver feeling sleepy
Alcoholic driver
MEDICAL FACTOR
Driver having heart attack
Strokes
VEHICLE FACTOR
Brakes failure
Engine failure

HOW IT SOLVES ABOVE PROBLEMS?

Sleepy driver

DRIVER HAVING HEART ATTACK

 DRUNK DRIVER

 Allow the driver to speak naturally when

interacting with a car system to reduce a drivers
cognitive workload
Voice control interfaces reduce a drivers
distraction
Grab hold of the Driver's attention if he's showing
too much fatigue

FUNCTIONS OF ARTIFICIAL PASSENGER

Voice Control Interface
Embedded Speech Recognition
Driver Drowsiness Prevention
Workload Manager
Privacy and Social Aspects
Distributive Interface

VOICE CONTROL INTERFACE

Allow the driver to speak NATURALLY when
interacting with a car system
Difficult for a driver to remember a complex
speech command menu
This led to the development of Conversational
Interactivity for Telematics (CIT) speech systems
at IBM Research

 Development of full fledged Natural Language

Understanding (NLU) for CIT is a difficult problem
A quasi-NLU component - a reduced variant of
NLU has been developed, that can be run in CPU
systems with relatively limited resources
When the system gets a voice response, it
searches through special grammar files (starting
with the most relevant topic)
If it finds an appropriate command in some file, it
executes the command Otherwise the system
executes other options that are defined by a Dialog
Manager (DM)

EMBEDDED SPEECH RECOGNITION

Logically a speech system is divided into three
primary modules: the front-end, the labeler and
the decoder
When processing speech, the computational
workload is divided approximately equally among
these modules

SPEECH RECOGNITION(contd)
Speech
recognitionTypes
Isolated words
Connected words
Continuous Speech
Spontaneous speech

DRIVER DROWSINESS PREVENTION

Arousal (high, low) and valence (positive,
negative) are the two fundamental dimensions of
emotion
Common for drivers to talk to other people while
they are driving to keep themselves awake

 The purpose of Artificial Passenger part of the CIT

project at IBM is to provide a higher level of
interaction with a driver
A series of interactive modules within Artificial
Passenger increase driver awareness and help to
determine if the driver is losing focus
The scenarios for Artificial Passenger currently
include: quiz games, reading jokes, asking
questions, and interactive books

WORKLOAD MANAGER
It carry out moment-to-moment analysis of users
cognitive workload.
It monitors Driver actions and
characteristics(ex:applying brakes,eyelid status).
It monitors local and remote conditions(ex:Heavy
rain).
It controls distracting informations to
driver(ex:mob call,email,radio).

WORKLOAD MANAGER(contd)
It interprets the status of a drivers alertness,
based on his/her answers
It will make decisions on whether the driver needs
additional stimuli and on what types of stimuli
should be provided
verbal stimuli via speech applications
physical stimuli such as a bright light
whether to suggest to a driver to stop for rest

PRIVACY AND SOCIAL ASPECTS

It performs encryption of message traffic from the
vehicle, through a carrier's network such that the
drivers responses cannot be intercepted.
The social impact will be managed by making sure
that users clearly understand what the system is,
what the system can and cannot do

DISTRIBUTIVE USER INTERFACE

BETWEEN CARS
The safety of a driver depends on the behavior of
other drivers near him.
It informs driver about the conditions of drivers in
other cars passing near by.
Information includes
Driver profile(age,no of accident).
Mood(angry,calm).
Physical condition(tired,drowsy).
Attention(talking on phone,reading mail).
Driver can make use of these information to
estimate risk and drive carefully.

ARCHITECTURE(Blackboard)

WORKING COMPONENTS

WORKING COMPONENTS(contd)
CAMERA
This will track the lip
movements of the
driver
Used to improve the
accuracy of the speech
recognition

WORKING COMPONENTS(contd)
EYE TRACKER
Collecting eye
movement data
requires both hardware
and software.
Hardware:
Head mounted sys.
Remote sys.
Both systems measure
the corneal reflection of
the infrared LED

WORKING COMPONENTS(contd)
SOFTWARE USED IN EYE TRACKER
Data collection and analysis is handled by eyetracking software
Software catalogs eye-tracking in one of the two
ways
ERICAs Eye Gaze software uses a small red x to
represent eye movement
In other, data is stored as a series of x/y coordinates
related to specific grid points on the computer
screen

WORKING COMPONENTS(contd)
Automatic Speech Recognition
There are two ASRs used in system:
First one is speaker independent and used for
decoding voice signals of the driver
Second one operates with voice car media and
decodes tapes, audio , telephones, mails etc.

WORKING COMPONENTS(contd)
Natural Language Processor
Processes the decoded signals of voice or text data
provided from the ASR.
Identifies related meanings from the contents of
the decoded messages.
Produces variant of responses.
This output goes to the driver analyzer as an input.

NATURAL LANGUAGE PROCESSOR

WORKING COMPONENTS(contd)
Driver Analyzer
Receives the textual data and voice data.
Measures the time response using a clock.
Time responses, conclusions about drivers
alertness will be passed on to the conversational
planner

DRIVER ANALYZER MODULE

WORKING COMPONENTS(contd)
Conversational Planner
It is the heart of the system.
Instructs the language generator to produce the
response.
If the driver is in perfect condition CP instructs the
language generator to continue the conversation
otherwise it will be instructed to change the
conversation.

CONVERSATIONAL PLANNER

WORKING COMPONENTS(contd)
Alarm
If the CP (conversational planner)
receives information that the
driver is about to fall asleep an
alarm system is activated.

Microphone:
For picking up words and
separate them by some internally
used software for conversation.

Showing the dashboard of the car where the whole artificial system is generally attached.

APPLICATIONS
Prevents the driver, falling asleep during long and
solo trip.
In any problem it alerts the vehicles near by this,
so the driver there become alert.
Opens and closes the doors and windows of the car
automatically.
It is also used for the entertainment.

FUTURE APPLICATIONS

Will provide us with a shortest time routing based on

road conditions changing because of weather and
traffic,information of about the cars on the route,
destination requirement (as flight has been delayed).

CONCLUSION
Successful implementation of artificial passenger
would allow use of various services in car like
reading emails, navigation,downloading music
files, voice games without compromising a driver
safety.

DRIVE SAFELY

THANK YOU