Project 2

Tangible Interactive
Application for
LETS Storybooks

Guide: Prof. Jayesh Pillai

Co-guide: Prof. Alka Hingorani

Vishnu K
M.Des Interaction Design
IDC IIT Bombay
Introduction
Learn English Through Stories

Ongoing project by Prof. Alka Hingorani and

team

Tata Centre for Technology and Design

IIT Bombay

Image source: TCTD, IIT Bombay

LETS Storybooks

Stories and illustrations by students in their native

language

Stories are translated, illustrations are digitally

retouched and books are designed by LETS team

LETS storybooks are used by students in the

school libraries

English language is introduced through stories

and characters familiar to the students, therefore
bringing in context

LETS storybook The Monster and the Little Boy

Need

Need for use of mobile technology to create

interactive applications was identified by
LETS team from the field

Challenges in English language learning

● Lack of context in which language is

used outside structured classroom
environment
● Lack of skilled teachers in the schools
Need

Opportunities

● Penetration of smartphones in rural

areas
● Cheap and reliable internet connectivity
Objective

The objective of the project is to add interactivity to LETS storybooks to deliver

additional modalities and information that are missing in language learning in
resource-constrained rural environments.

The design shall augment the reading experience using LETS storybooks as well
as fit into the overall context of LETS project.
Research
User Study at Grammangal, Aina

• More focus on reading and writing skills

and very less on speaking skills

• School teachers own smartphones which

they sometimes share with students
User Study at Grammangal, Aina

• Teachers themselves lack expertise in

English language

• Students struggle at various language skills

including reading, writing and speaking, but
they have the least confidence about speaking.

Teaching aids at Grammangal, Aina

Discussions with LETS team

Workshops conducted at schools by LETS

team

Creating stories, drawing illustrations and

making books
LETS Reading Application

LETS reading application was developed by

Tactopus Learning Solutions

Narrate stories while students read LETS

storybooks
Secondary Research

Advantages of using AR in language learning

contexts:

● Could create word associations

● Higher memory retention [1]

Disadvantages:

● May cause attention tunneling [1]

Existing Products

Augmented Reality Storybooks

Augmenting 3D characters

Interactivity through manipulation, voice

input etc.

Mardles Augmented Reality Storybook

Existing Products

Tangible interactions with storybook

Interacting with contents inside the page

using additional cards

Focus on children of smaller age group

Tactopus Learning Application

Existing Products

Tangible interactions with storybook

Interacting with contents inside the page

using additional cards

Focus on children of smaller age group

Osmo App
Existing Products

Novel way of interaction possibilities

Embed illustrations in any page using RFID

sensors and different gestures for different
functions

OnObject, MIT Media Lab

Analysis

● Focus on engagement of children of smaller age

● Interactions with the contents of the page is limited
● Embedded technology provides novel ways of interactivity but are limited by its cost

What is missing?

● Interactions with the contents inside the storybook

Research
Redefined Objectives

Interaction with storybooks vs interaction with smartphone

Tangibility of physical books

Rich content already created by LETS team

Additional resources required to create animations, 3D characters etc.

Audio output vs 3D characters and animation

Design
Concept

Tangible Interactive Application for LETS

storybooks

● Students interact with the storybook to

explore additional information about
words
● Smartphone is used to understand user
interactions and deliver output
Components

● Bookmark
● Stand
● Android Application
● LETS Storybook
How to select a word?
How to play a function? (meaning, pronunciation etc.)
Input

Rectangular gap

Colored circles
(Virtual Button)

Slider
Selecting page

User turns the page

Selecting word

User selects the word by adjusting slider in

the bookmark

Selecting feature of a word

Using virtual buttons on bookmark

Output

Modes

Listening Interactive

Narration Pronunciation Meaning Synonyms

App Flow

App flow diagram

● Content
● Feedback
● Conditions
 Video Prototype

Link for video prototype:

https://drive.google.com/file/d/1Ehg7oLeA
e_Znk1U38_pPxk9rn1EbMR_4/view?usp=
sharing
Development

Using google cloud vision to detect word

How it works

- Vuforia Image tracking

- Google cloud vision package for unity from github
(https://github.com/comoc/UnityCloudVision)
- Vuforia virtual buttons
Evaluation
Affordance of bookmark

Goal: Activity:

Identify how users use the bookmark and
slider for the given tasks without telling them
about the interactions.
Number of users: 8
The setup of the application was explained to
the users. Users were told that application
gives meaning and pronunciation for the
words by identifying interaction with the
bookmark. Users were asked to interact with
the bookmark to find meaning and
pronunciation of words in the book. Users
were not told how the actual interactions
were. They were also not told the technology
used.
Observations

● Out of 8 users, 6 users adjusted the

slider to track the word and clicked on
the button to get pronunciation of
meaning.
● 1 user assumed bookmark was to cover
whole sentence
● 1 user assumed embedded technology
inside bookmark
Observations

● Length of the rectangular gap has to be

shorter, users created their own mental
model of interaction when the
rectangular gap was enough to cover
full sentence.
● Circles didn’t convey that it is a button.
User testing
Goal:
To understand how users interact with
the novel tangible interaction while
using LETS storybook
Method:
Wizard of Oz prototype
Key Questions:
Are students able to do simple task
using the design without making errors?
Do students interact with the tangible
interface in the same way as expected?
User testing

Training prior to the tasks

● Students were shown how to

interact with the bookmark
● The setup was demonstrated; both
listening mode and interactive
mode
User testing

Tasks ● Task 6 - When the user is in page, find

the synonyms for mango
● Task 1 - Read and listen to first five ● Task 7 - Ask the user to close the book
spreads of the storybook Clever Kalyani and find the meaning of word delicious
● Task 2 - When the user is in page, find
the pronunciation of Kakote and say it Observations were made while users
● Task 3 - When the user is in page three, performed tasks
find meaning of word cruel
● Task 4 - When the user is in page five,
ask the user to find synonyms for the
word small
Observation
● All the users were able to perform
simple tasks as expected, but the testing
revealed interactions which were not
foreseen.
● When text is near the edges inside the
page, virtual buttons goes out of the
frame
● Users pressed on virtual buttons as if it
was a real button, but there was no
tactile feedback to it. The earcon was
ineffective in case of button earcon
because of the delay
● Even though other two earcons also had
delay, users were not confused by it
may be because they were not response
to physical actions, but at system level
Conclusion

● A novel way of tangible interaction was

designed to interact with text inside
LETS storybook
● A small qualitative study was conducted
to evaluate the affordance of the tangible
interface and the interaction proved to
be intuitive with the tested users
● A wizard of oz test was conducted to
understand how children interact with
the new interaction and observations are
noted down and will inform the
iteration of the design
 References

1. Radu, I. Pers Ubiquit Comput (2014) 18: 1533. https://doi. org/10.1007/s00779-013-0747-y

2. Designing for Gesture and Tangible Interaction - Synthesis Lectures on Human Centred Informatics, Mary
Lou Maher, Lina Lee, 2017
 Resources

Audio earconsfiles used in the video prototype

Ding earcon: https://freesound.org/people/InspectorJ/sounds/411088/

Button earcon: https://freesound.org/people/JarredGibb/sounds/219472/

Error earcon: https://freesound.org/people/lluiset7/sounds/141334/