Kinect2Connect

PROBLEM STATEMENT
Sign language is a visual gestural language which is used to communicate with the
people with speech and hearing impairment. It enables us to communicate with a whole
new group of people and it has been proven to improve alertness and brings other
neurological benefits. It is very difficult for any person to understand the nuances of the
technique of sign language without proper learning.
The problem becomes even more complex in the case of school children who are deaf
or cannot speak and have to be communicated with or to be taught. So, this project
aims at developing a system that is capable to capturing the common and widely used
gestures in different sign languages (especially the American Sign Language) using a
Kinect and delivering them as text and vice versa.
SCOPE
The deaf population in Unites States is more than 38 million*, which accounts to 13% of
the total population in US, so our system aims at creating a sign language recognition
system for deaf people. The system can be used in educational institutions for better
communication with deaf children. As our system can convert a sign language to text
and text further to speech, it can also be utilized by the family members of deaf people
to communicate with them efficiently. The primary users of this system will be deaf
people; potential clients can be schools, hospitals, workplaces etc.
BENEFITS
Gesture recognition can be employed in multiple areas in which will significantly
improve quality of life of users. A major area where it can be used is in recognizing sign
language, where the system can recognize a known sign and convert the same to audio
or text or even replicate the sign. This will allow communication between speech and
hearing impaired people and those who doesn't understand sign language. The gesture
to audio or text conversion module can be easily replaceable with a command handling
system, where simple gestures can be mapped to computer based programs which
helps to integrate other functionalities like controlling electronic devices using gestures.
The project is highly modularized to integrate multiple features as extensions. This can
be very effective and easy medium of communicating with the computer therefore
enhances the overall user experience. For example: the users hand can act as a paint

brush and use a paint program. Common gestures such as selecting a button or
movement of hand to recognize brush stroke etc.

FUNCTIONAL REQUIREMENTS
1. Indentifying the Common gestures and facial expressions or body movements to
be captured.
2. Designing a user interface which allows gesture recording, text and voice
processing.
3. Communication needs in a school environment. Some of the common
requirements are simple gesture translator, interactive educational applications
Instant gesture message translation system
Gesture based paint application
4. Grammar of the language and its mapping to simple English.
HARDWARE AND SOFTWARE REQUIREMENTS
1. Kinect and OpenNI framework.
2. Kinect based sensor, devices like an RGB color VGA video camera, a depth sensor,
and a multi-array microphone, Motorized Tilt, Display Screen
SAFETY REQUIREMENTS
The sensor should be placed in a well lit room and enough space should be provided for
the detection area. Determine how each child is able to use the sensor (translator,
interactive educational games). Also the Kinect software uses certain laser source to
extract a 3D pattern which may cause irritation to the user. Hence proximity to the
screen needs to be taken care of and long hours and need to be avoided.
CONSTRAINTS ASSUMPTIONS AND DEPEDENCIES
A major technical hurdle in this project would be to ensure fast performance with
respect to the processing of various forms of sign language. As the Kinect may need to
look up large databases of different sign languages and gestures, the processing
overhead may increase by a significant amount. Also, many sign languages include fast
and specific movements. Such movements may go unrecognized by the Kinect resulting
in a loss of accuracy. Thus, steps should be taken to minimize these inaccuracies.
Another technical hurdle would be to make the system customizable. Features must be
implemented to facilitate easy updating of gestures and signs recognized by the system.