Enhanced Reality:

A
A New
New Frontier
Frontier for
for
Computer
Computer Entertainment
Entertainment
Richard Marks

Enhanced
Enhanced Reality?
Reality?
Augmented reality but with an
entertainment focus
Minimize
Minimize encumbrance
encumbrance
Utilize
Utilize common
common hardware
hardware
Simplify
Simplify setup/calibration
setup/calibration
Create
Create an
an enjoyable
enjoyable user
user
experience
experience

Research
Research Goals
Goals
Natural User Interfaces
Allow
Allow people
people to
to interact
interact in
in aa more
more natural,
natural,
enjoyable
enjoyable manner
manner

Real-time Special Effects

Enable
Enable people
people to
to experience
experience for
for themselves
themselves
the
the kind
kind of
of special
special effects
effects seen
seen in
in the
the movies
movies

Related
Related work
work
Many years of SIGGRAPH
Myron
Myron Kruegers
Kruegers art
art exhibits
exhibits
MIT
MIT media
media lab
lab ALIVE
ALIVE system
system
Intervals
Intervals Magic
Magic Mirror
Mirror

Reality Fusion, ePlanet, etc.

Primarily
Primarily use
use motion
motion detection
detection or
or
background
background subtraction
subtraction to
to create
create sprites
sprites

Current
Current Setup
Setup
Standard television set
PlayStation2 for video processing and
graphical rendering
1394 webcam (<$90 retail)
30
30 frames/sec
frames/sec uncompressed
uncompressed video
video
320x240
320x240 YUV422
YUV422
640x480
640x480 YUV411
YUV411

Technologies
Technologies
Scene Interpretation
Participant
Participant tracking
tracking
3D
3D object
object tracking
tracking
Lighting
Lighting estimation
estimation

Rendering
Lighting
Lighting
Compositing
Compositing

System

Participant
Participant Tracking
Tracking
Segmentation

Background
Background subtraction
subtraction

Motion estimation
Optical
Optical flow
flow
Feature
Feature tracking
tracking

Part labeling
Face
Face detection/tracking
detection/tracking
A
A Survey
Survey on
on Face
Face Detection
Detection Methods.Yang,
Methods.Yang, Ahuja,
Ahuja, Kriegman
Kriegman

Limb
Limb finding/tracking
finding/tracking
A
A Survey
Survey of
of Computer
Computer Vision-Based
Vision-Based Human
Human Motion
Motion Capture.
Capture.
Moeslund,
Moeslund, Granum.
Granum.

3D
3D Object
Object Tracking
Tracking
Color-based tracking
Spheres
Spheres
3D
3D position
position from
from centroid
centroid and
and radius
radius
Rotation
Rotation rate
rate is
is also
also measurable
measurable using
using 2D
2D
visual
visual flow
flow at
at centroid
centroid
Illumination
Illumination unaffected
unaffected by
by rotation
rotation
Very
Very fast
fast and
and simple
simple

Sphere
Sphere and
and cylinders
cylinders
66 DOF
DOF tracking,
tracking, SIGGRAPH
SIGGRAPH 2000.
2000.

Lighting
Lighting
From a known sphere
Static
Static
Inspired
Inspired by
by Debevecs
Debevecs work
work
High
High dynamic-range
dynamic-range estimation
estimation
possible
possible

Dynamic
Dynamic
Real-time
Real-time light
light source
source estimation
estimation
Real-time
Real-time light
light map
map

Compositing
Compositing
Z-buffer rendering
Render
Render the
the tracked
tracked sphere
sphere to
to
Z-buffer
Z-buffer only
only

Alpha feathering
Render
Render CG
CG to
to texture,
texture, create
create
an
an alpha
alpha stencil,
stencil, blur
blur the
the alpha
alpha
stencil,
stencil, render
render to
to screen
screen
Still
Still have
have z-buffer
z-buffer aliasing
aliasing

Magic
Magic duel
duel

3D color tracking
Motion detection
Figure segmentation
Image distortions
Compositing

System
System
Use video as
texture for a mesh
Delay video to give
time for processing

Virtual
Virtual character:
character:
Misho the
the witch
witch

Misho stands on the red ball

Misho likes to watch the green ball
Misho tries to entertain herself (and you)

Virtual
Virtual character:
character:
Seymour
Seymour

Seymours plane follows the green ball

Seymour jumps out onto the red ball
Seymour loses his balance if you move
the red ball to fast
Seymour jumps back in if his plane
comes close
Seymours plane always rescues him

Issues
Issues
Lighting conditions
Insufficient
Insufficient ambient
ambient lighting
lighting
Extreme
Extreme back-lighting
back-lighting (windows)
(windows)

Visual distractions
Mirrors
Mirrors
Movement,
Movement, color
color

Conclusions
Conclusions
Real-time movie special effects are
coming soon
Video input will be a part of future
computer entertainment

Special
Special Thanks
Thanks to:
to:

Ohba-san for his fire/water demo

Tanya Scovill and Care Michaud
Tyler Daniel and Gabor Nagy
Scott Butler for helping with Magic Duel

Q&A