case studies compiled by Ada-Nkem Juwah

(text are direct quotes or based on quotes from sources)

 text input
punch card
object command line
mouse
keyboard
digital pen
joystick, wii wand, etc
gestural
cellphone, smartphone etc
human-machine connection
other

interaction

ambient
touchscreen
screen display screen
real time aggregated real time aggregated aggregated

environment web data

visual crowdsourcing
thermal
movement
audio
scent
HARDWIRED

punch cards

c://ommand line

(graphical)
»» Even the old can be
adapted to be new
»» multitouch for the
conventional keyboard
»» keyboard accepts
pressure-sensitivity data
»» can infer user intentions
»» Microsoft

http://www.fastcompany.com/blog/
chris-dannen/techwatch/multitouch-
future

http://www.youtube.com/
watch?v=PDI8eYIASf0
Data Visualization

»» Ben Fry
»» acquire – the matter of obtaining the data, whether
from a file on a disk or from a source over a network.
»» parse – providing some structure around what the
data means, ordering it into categories.
»» filter – removing all but the data of interest.
»» mine – the application of methods from statistics or
data mining, as a way to discern patterns or place the
data in mathematical context.
»» represent – determination of a simple representation,
whether the data takes one of many shapes such as a
bar graph, list, or tree.
»» refine – improvements to the basic representation to
make it clearer and more visually engaging.
»» interact – the addition of methods for manipulating
the data or controlling what features are visible.

http://benfry.com/phd/dissertation-050312b-acrobat.pdf
http://benfry.com/isometricblocks/
»» Narrative 2.0
»» visualizes music
»» music was segmented in single
channels
»» the angle of the line changes
according to the frequency
of the channel, while the
frequency reaching a high
level, the
»» channel becomes highlighted
by orange
»» music as an artist

http://www.matthiasdittrich.com/projekte/
narratives/visualisation/index.html
»» We Feel Fine
»» Jonathan Harris & Sep Kamvar
»» harvesting human feelings from
weblogs
»» (real-time) every few minutes
»» searches new blog entries with
the phrases “I feel” and “I am
feeling”
»» identifies the “feeling” expressed
in that sentence
»» age, gender, and geographical
location of the author can often
be extracted
»» local weather conditions also
captured
http://www.wefeelfine.org/index.html
»» visualizing intimate biometric data
and emotional experiences using
technology.
»» Bio Mapping project
»» Galvanic Skin Response (GSR), a
simple indicator of the emotional
arousal, is recorded in conjunction
with one’s geographical location
»» “Emotion Maps” were generated
of the city in which the participants
roamed around

http://infosthetics.com/archives/2009/04/emotional_
cartography_implications_of_visualising_intimate_
biometric_data.html
»» “Abundance”
»» Camille Utterback (2007)
»» video installation
»» converts the movement of
pedestrians into colourful
silhouettes and patterns that
are projected onto a nearby
3-storey rotunda

http://www.cartridgesave.co.uk/news/7-
incredible-computer-generated-works-of-art/
»» digg labs
»» real-time data visualization
http://labs.digg.com/stack/
»» apple store display
»» fake “real-time” data visualization of app purchases
»» employs metaphor of ripple
http://www.techcrunch.com/2009/06/08/apples-cool-matrix-style-app-wall/
»» Personas
»» Aaron Zinman , MIT
»» sophisticated natural language processing
and the Internet
»» data portrait of one’s aggregated online
identity

http://personas.media.mit.edu/
»» Ben Fry
»» Darwin Origin of
Species

http://benfry.com/traces/
»» experiments with systems that monitor traffic
patterns in real time and manage the use of lanes
and access accordingly
»» ability to map patterns as they happen, along with
the alternatives and consequences.
»» In Stockholm a system that tracks the movement of
every car has reduced carbon emissions by 25%
»» Pictured: Litmus (in UK)
»» monitors traffic, tide level, wind turbine activity

http://www.fastcompany.com/blog/michael-cannell/cannell/visualization-
new-frontier-design
http://www.youtube.com/watch?v=IkmpIXd9Q90&feature=related http://www.youtube.com/watch?v=jEGwsfpHevU&feature=related

»» environmental information
»» movement
»» speech patterns / conversation

http://www.fastcompany.com/blog/michael-cannell/cannell/visualization-
new-frontier-design
Data Interaction

»» touch
»» gesture
»» virtual
 http://www.perceptivepixel.com/applications.html
»» visualization in the public sphere
»» CNN wordle of President Obama’s
press conference
»» CNN’s magic wall (developed by
perceptive pixel)

http://www.fastcompany.com/blog/michael-cannell/
cannell/visualization-new-frontier-design
»» G-speak: a “spatial operating
environment.”
»» Oblong Industries (Minority Report's
science advisor, John Underkoffler, is
the founder)
»» Spatial operating environment (SOE)
»» combination of gesture-based input,
networking, real-world displays
»» operate with gloves
»» free-hand gestural input that can
include pointing, grabbing, and
several other hand poses from
several pairs of hands simultaneously
»» Anything on-screen can be
manipulated directly

http://www.fastcompany.com/news/2008/11/16-
minority-report-g-speak.html

http://www.youtube.com/watch?v=dyMVZqJk8s4
»» 'SixthSense'
»» MIT
»» wearable gestural interface
»» augments the physical world around us
with digital information
»» natural hand gestures interact with
information.

http://www.pranavmistry.com/projects/sixthsense/
metaphor
 play
... The interface sported almost no expanatory text or
conventionally meaningful icons... She wasn’t intimi-
dated, confused, or annyoyed; she seemed to consider
the lack of instructions part of the game. She merely
started the game and began clicking on objects...

“How do you know

which blocks to hit?”
“I just ... hit them.”
“So how do you figure out
what the rules are?”
“Just play.”
“Just play? And then
what happens?”
“You just ... play.”
Games are played for no other reason than for the experience
of playing them—unlike a software application, in which the
experience or enjoyment of the user is a by-product. If the
experience of the interface is not pleasing, players will walk away. By
contrast, the interfaces of many cell phones, software applications,
digital cameras, microwave ovens, cars, and even wayfinding
systems are maddening to use. In some situations—famously
the VCR—the interface has been bad for so long that we expect
operation to be frustrating and difficult.

A good game interface will not bombard the user with information
at the outset or rely on a complex instruction manual; it will teach
the user everything he or she needs to know on a need-to-know
basis. This convention is so entrenched, in fact, that gamers trust
the system and never read the manuals. Figuring out how it works,
whether it’s boosting your cyborg hero’s bomb-disposal skills or
downloading a cheat code that makes her invisible to flying aliens,
is part of the game. “A game’s system itself generates meaning, and
the way it changes over time begins to modify your understanding
of that system,” Salen says. “It’s a basic principle that can apply to all
kinds of design.”

The Principles of Play

http://www.metropolismag.com/story/20060911/the-principals-of-play
Sugar OS: http://laptop.org/en/laptop/software/index.shtml
kiosk system “head over heels”
White Void

http://www.whitevoid.com/portfolio.html
multi touch console “forward current”
White Void

http://www.whitevoid.com/portfolio.html
»» Media artist Toshio Iwai and Yamaha
»» digital musical instrument
»» TENORI-ON
»» 16x16 matrix of LED switches allows everyone to
play music intuitively, creating a "visible music"
interface
http://www.global.yamaha.com/tenori-on/what/index.html
»» Lemur, a multi-touch interface for creating real-time
light shows and music mixes
»» manipulate broad array of media in real time
»» mimicking the analog experience of futzing with mixing
boards and synthesizers
»» clever musicians to become more like visual artists or
stage directors, since the device can be linked up to light
outputs—giving music a real time visual interpretation
»» live performances at the hands of a single person
http://www.jazzmutant.com/videos/lemurlight.mov
»» developed by Potion
»» The Jump Station for a special Nike
exhibit at the 2008 Beijing Olympics
»» reactive piece that rewards physical
performance with a larger-than-life
payoff
»» eight foot tall stick figure reacts to the
person’s presence, begins warming up,
and waits for the visitor’s next move
»» a small jump is rewarded with only a
small hop by the figure
»» real effort is rewarded with a series
of high flying animations like Michael
Jordan style jump shots or acrobatic back
flips
http://www.potiondesign.com/index.php?page=2&project=
7&section=0&gallery=0&listPage=0&capPage=0
Display
»» GE
»» LED wallpaper
»» roll-to-roll processing
»» method of manufacturing
organic LEDs (OLEDs)
»» like a printing press
»» not quite practical/cost-
effective presently
»» tunable colour
»» transparency (windows)

http://dvice.com/archives/2008/04/
dvice_exclusive.php
»» programmable surfaces can be shaped
and textured in more flexible ways than
traditional LED displays
»» modular tiles as building blocks
»» displays can become an integral part of
objects, structures and spaces.
http://fluid.media.mit.edu/projects.php?action=details&id=23
»» MemTable
»» MIT
»» table with a contextual memory
»» supports meeting annotation,
process documentation, and
visualization of group work patterns
»» remember how it is used an
provide an interface for contextual
retrieval of information

http://fluid.media.mit.edu/projects.
php?action=details&id=71
Tangible Interfaces
»» DIS.PLAY
»» group of German hackers
»» Arduino and Processing!

http://www.engadget.com/2009/02/24/foam-
infused-impress-dis-play-encourages-touching/
 »» Trackmate
»» TaPuMa; digital, tangible public map
»» open source
»» allows people to use their own belongings or the
everyday objects they carry with them to access »» Trackmate Tracker allows any computer to recognize
relevant, just-in-time information and to find tagged objects and their corresponding position,
locations of places or people rotation, and color information when placed on a
surface
»» table-top environment where map and dynamic
content is projected on the table »» Trackmate sends all object data via LusidOSC (a
protocol layer for unique spatial input devices),
»» camera mounted above the table identifies and
allowing any LusidOSC-based application to work with
tracks the locations of the objects on the surface
the system
»» on the basis of identifications of the objects, the http://trackmate.media.mit.edu/
software program provides relevant information
visualization to be shown on the table
http://ambient.media.mit.edu/projects.php?action=details&id=53
»» SandScape
»» MIT
»» designing and understanding landscapes
»» sand
»» users view simulations as they are projected on the
surface of sand model
»» alter the form of the landscape model by
manipulating sand while seeing the resultant effects
of computational analysis generated and projected
on the surface of sand in real-time

http://tmg-video.media.mit.edu/sandscape/sandscape_352x240.mpg
Brain Machine Interface
»» Brain-Computer Interaction
»» Emotiv Systems
»» EPOC headset
»» electroencephalography (EEG) technology
»» measure the voltage produced by the combined activity of thousands of neurons in one area
of the brain
»» 16 electrodes can detect brain activity
»» recognize emotions, commands, and facial expressions
»» link them to operations or keystrokes in a wide variety of existing PC games and applications.
»» emotion recognition can be used to continuously adjust the game’s environment
»» i.e. the player is bored, the game can get harder; if the player gets excited, the background
music can adjust accordingly
www.emotiv.com
http://seedmagazine.com/content/article/standing_on_the_shoulders_of_giants/
»» Toyota
»» advanced brain sensing system that controls the movement of a wheelchair by reading a user’s
thoughts alone
»» processing patterns in brain waves
»» the system can propel a wheelchair forward, as well as make turns
»» virtually no discernable delay between thought and movement.
»» return a response from a thought stimulus in just 125 milliseconds
»» creating real-time responsiveness
»» Five electroencephalography sensors stationed above the regions of the brain that deal with
motor movement interpret patterns in the signals generated by the user
»» Further, the software interpreting the signals adapts to a particular user’s patterns of thinking,
achieving 95% accuracy after just one week of three-hour training sessions.

»» Honda
»» similar technology
»» allows its Asimo robot to be manipulated via brain signals

http://www.fastcompany.com/blog/clay-dillow/culture-buffet/mobility-mind-control-toyota-unveils-wheelchair-propelled-
thoughts-a
Questions based on a FastCompany article by David Sherwin

Can we listen & talk to the Internet?

»» “give me turn-by-turn directions to Pacific Place,”

Can a touch interface communicate

through sense of touch?
»» Blind people should be able to ask their phone, “What’s the temperature going to be tomorrow?” and have the phone
adjust its heat output in relation to today’s temperature to indicate the relative difference.

Can we feel directions?

»» I ask my phone (using my voice interface) how the traffic is on Road X. The steering wheel gets harder by 30%. Should I take
Road Y then? The steering wheel softens dramatically.

http://www.fastcompany.com/blog/david-sherwin/changeorder-business-process-design/move-beyond-gestural-interface-give-your-phon
http://www.fastcompany.com/blog/cliff-kuang/design-innovation/infographic-day-hype-cycle-computer-interface-design