AR with head-mounted Displays

Vorlesung Augmented Reality Prof. Dr. Andreas Butz WS 2006/07

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 1

Head-mounted Displays (HMDs)

Optics of the human eye HMDs: Working Principles, Problems
Closed (video only) Optical see-through Video see-through

Examples of commercially available HMDs Head-up displays Proposed solutions to existing problems Research prototypes

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 2

A bit of history

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 3

Optical system of the human eye (1)

Simplified principle: the pinhole camera Only one light beam from each object point to the corresponding image point

Image distance

object distance

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 4

Optical system of the human eye (2)

Reality: a lens which has to be focused
all light rays from one object point have to meet in the same image point!

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 5

Objects out of focus (depth of field)

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 6

Focusing the eye by adjusting the lens

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 7

Resulting properties of the human eye

Focal length of the lens can be adjusted to
Objects at infinite distance Objects at ~20cm from the eye Everything between these distances Only one distance (range) at a time

Eye needs time to adjust between objects at different distances

Exhausting

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 8

Spatial vision: Depth Cues

Several different types of cues used by human visual system
Static monocular cues Stereopsis Motion parallax Oculomotor cues
Accommodation-convergence mismatch

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 9

Static Monocular Cues

Occlusion Relative Size Relative Height Linear Perspective Aerial Perspective Texture Gradient Shading

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 10

Stereopsis
Static, binocular cue Each eye gets a slightly different image
Monocular cues from each image

Only effective within a few feet of viewer

Useless if only distant objects

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 11

Motion Parallax
Dynamic, monocular cue Near objects move faster than far objects Generally more important than stereo! head tracking is very important!

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 12

Oculomotor Cues
Based on information from eye muscles Accommodation: lens shape Convergence: gaze direction HMDs confuse the brain with oculomotor cues
Accommodation focuses eye at one distance Convergence says objects are at different distance
LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 13

Principle: closed (video only) HMD

Monitor is mounted very close to the eye Additional lens makes it appear distant all images appear at the same distance
Usually at infinity or slightly less

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 14

Creating VR with a HMD

Rendering

Head tracker

3D scene

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 15

Challenges with HMDs in VR

Lag and jitter between head motion and motion of the 3D scene
Due to tracking predictive tracking Due to rendering nowadays mostly irrelevant

Leads to different motion cues from

Eye (delayed) and Vestibular system (not delayed)

Result: cyber sickness

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 16

nVision Industries
The Datavisor 80 contains wide field of view optics modules integrated with high-resolution CRTs. Designed to be worn for extended periods of time, the Datavisor 80 is built with optical, mechanical, and electrical components distributed around the unit for better balance and ergonomics. ;-)

Datavisor HiRes:
Field of view: 72 deg Resolution: 1280x1024
LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 17

SEOS HMD 120/40

Resolution: 1280 x 1024 Field of View: 80 x 67 per eye Overlap:50% (resulting in 120x67 deg FoV with a 40x67 deg stereo overlap) Weight: 1 Kg

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 18

Icuiti M920

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 19

Kaiser Electro Optics ProView SO35 Monocular

Field of View: 32x24 Resolution: 800x600 Mounting: Clip on to helmet (Display module); Clip on to belt (Display Controller) Temp.: Operating: -32 to +55C; Storage: -32C to +71C Humidity: Six 48-hour cycles, 20C to 55C, 95% RH Salt Fog: Four 24-hour cycles Vibration: Random vibration, 6 axis, 5 Hz to 2500Hz, up to 40 gs Immersion: Immersion in 1 meter of water for 2 hours $10,500
LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 20

Creating AR with optical see-through HMDs

Rendering

Head tracker

3D scene

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 21

Advantages of optical see-through HMDs Preserve the richness of the world

Very high resolution of physical image No lag between motion and phys. image Physical objects can be focused at their correct distance

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 22

Challenges with optical see-through HMDs

Lag and jitter between the physical and the virtual image Misalignment of physical and virtual image (registration) HMD can only add light to physical image
Looks like ghost images Always in front of physical objects

High dynamic range of the phys. image

Use in bright sunlight almost impossible

Virtual objects always focused at same distance

Permanent adaptation back and forth

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 23

Construction: Boeing, 1994

Assembly of wire harness for airplanes Assembled on a large board Traditionally tedious task Equip board with markers Show in HMD where to mount next wire

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 24

i-O Display Systems

Resolution: 110,000 pixels per LCD Panel = 230 x 173 lines of resolution Full color Stereo sound Field of view: 30 deg Price: 300$

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 25

Sony Glasstron
Initially built for watching DVDs Video resolution No longer manufactured Amount of see-through can be regulated

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 26

SAAB AddVisor 150

Field of view: 46 deg Eye overlap: 100% or 50% Resolution: 1280x1024 Full color

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 27

nVision Industries
Datavisor SeeThrough:
Field of view: 72 deg Resolution: 1280x1024

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 28

KEO Sim Eye XL100A

Resolution 1024x768 Contrast: > 20:1 Field of View: 50 x 100 with 30 Overlap Transmission: See through > 20% Collimation: Greater than 30ft. but less than infinity Weight: almost 3Kg Price: $87,500
LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 29

Creating AR with video see-through HMDs

Video Mixing Head tracker Parallax error 3D scene

Rendering

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 30

Advantages of video-based see-through

Lag between physical and virtual image can be compensated Camera can be used for tracking as well
Physical image = raw tracking data Perfect registration possible

Video mixer can add or subtract light

Virtual objects can be drawn in black Physical objects can be substituted Virtual objects can be behind physical objects

Just one image with a given focus distance

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 31

Challenges of video-based see-through Lag between physical and virtual image can be compensated
by delaying the physical image Leads back to the cyber sickness problem

Parallax error can not be corrected electronically

Wrong stereo cues when used for stereo

Richness of the world is lost

Video image just 0.5 megapixels Resolution of human vision is much higher (>10x)
LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 32

Video see-through examples

Here: just 1 camera between the eyes

No stereo Minimized parallax error
LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 33

Video see-through HMD without parallax error (e.g., eyetap device)

Video Mixing Rendering Head tracker

3D scene

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 34

Eyetap Technology

http://www.eyetap.org/ Computer mediated reality modify visual perception

Augment Diminish Alter
LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 35

Creating AR with Head-up Displays (HUDs)

Rendering

Head tracker

3D scene

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 36

Head-Up Display
Currently mostly military use limited applications in cars Fixed Display Very exact head or eye tracking needed
Easy for jet pilots

High brightness and dynamics needed

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 37

Optical see-through with occlusion

[Kiyokawa et al., ISAR 2000]

Rendering

Head tracker

3D scene

LCD

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 38

Optical see-through with occlusion

[Kiyokawa et al., ISAR 2000]

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 39

Optical see-through with occlusion

[Kiyokawa et al., ISMAR 2003]

LMU Mnchen Medieninformatik Butz Augmented Reality WS2006/07 Folie 40