SID S e r i e s i n D i s p l a y Te c h n o l o g y

Ernst Lueder

3D Displays

Ernst Lueder University of Stuttgart, Germany & Independent Consultant, USA

This book addresses electrical engineers, physicists, designers of flat panel displays (FDPs), students and also scientists from other disciplines interested in understanding the various 3D technologies. A timely guide is provided to the present status of development in 3D display technologies, ready to be commercialized as well as to future technologies.

Having presented the physiology of 3D perception, the book progresses to a detailed discussion of the five 3D technologies: stereoscopic and autostereoscopic displays; integral imaging; holography and volumetric displays, and:

Introduces spatial and temporal multiplex for the two views needed for stereoscopic and autostereoscopic displays;

Outlines dominant components such as retarders for stereoscopic displays, and fixed as well as adjustable lenticular lenses and parallax barriers for autostereoscopic displays;

Examines the high speed required for 240 Hz frames provided by parallel addressing and the recently proposed interleaved image processing;

Explains integral imaging, a true 3D system, based on the known lenticulars which is explored up to the level of a 3D video projector using real and virtual images;

Renders holographic 3D easier to understand by using phasors known from electrical engineering and optics leading up to digital computer generated holograms;

Shows volumetric displays to be limited by the number of stacked FPDs; and,

Presents algorithms stemming from computer science to assess 3D image quality and to allow for bandwidth saving transmission of 3D TV signals.

Cover design: Cylinder

PPC FINAL AW16mm

3D Displays

SID S e r i e s i n D i s p l a y Te c h n o l o g y

3D

Displays

Lueder

SID

Series Editor: Anthony C. Lowe, The Lambent Consultancy, Braishfield, UK

The Society for Information Display (SID) is an international society which has the aim ofencouraging the development of all aspects of the field of information display. Complementaryto the aims of the society, the Wiley–SID series is intended to explain the latest developments

in information display technology at a professional level. The broad scope of the seriesaddresses all facets of information displays from technical aspects through systems and

prototypes to standards and ergonomics.

RED bOx RuLES ARE FOR PROOF STAgE ONLy. DELETE bEFORE FINAL PRINTINg.

3D Displays

Wiley–SID Series in Display Technology

Series Editor:

Anthony C. Lowe

Display Systems: Design and Applications

Lindsay W. MacDonald and Anthony C. Lowe (Eds.)

Electronic Display Measurement: Concepts, Techniques, and Instrumentation

Peter A. Keller

Reflective Liquid Crystal Displays

Shin-Tson Wu and Deng-Ke Yang

Colour Engineering: Achieving Device Independent Colour

Phil Green and Lindsay MacDonald (Eds.)

Display Interfaces: Fundamentals and Standards

Robert L. Myers

Digital Image Display: Algorithms and Implementation

Gheorghe Berbecel

Flexible Flat Panel Displays

Gregory Crawford (Ed.)

Polarization Engineering for LCD Projection

Michael G. Robinson, Jianmin Chen, and Gary D. Sharp

Fundamentals of Liquid Crystal Devices

Deng-Ke Yang and Shin-Tson Wu

Introduction to Microdisplays

David Armitage, Ian Underwood, and Shin-Tson Wu

Mobile Displays: Technology and Applications

Achintya K. Bhowmik, Zili Li, and Philip Bos (Eds.)

Photoalignment of Liquid Crystalline Materials: Physics and Applications

Vladimir G. Chigrinov, Vladimir M. Kozenkov, and Hoi-Sing Kwok

Projection Displays, Second Edition

Matthew S. Brennesholtz and Edward H. Stupp

Introduction to Flat Panel Displays

Jiun-Haw Lee, David N. Liu, and Shin-Tson Wu

LCD Backlights

Shunsuke Kobayashi, Shigeo Mikoshiba, and Sungkyoo Lim (Eds.)

Liquid Crystal Displays: Addressing Schemes and Electro-Optical Effects, Second Edition

Ernst Lueder

Transflective Liquid Crystal Displays

Zhibing Ge and Shin-Tson Wu

Liquid Crystal Displays: Fundamental Physics and Technology

Robert H. Chen

3D Displays

Ernst Lueder

3D Displays

Ernst LuederUniversity of Stuttgart, Germany & Independent Consultant, USA

This edition first published 2012

� 2012, John Wiley & Sons, Ltd

Registered office

John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, United Kingdom

For details of our global editorial offices, for customer services and for information about how to apply for

permission to reuse the copyright material in this book please see our website at www.wiley.com.

The right of the author to be identified as the author of this work has been asserted in accordance with the

Copyright, Designs and Patents Act 1988.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted,

in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted

by the UK Copyright, Designs and Patents Act 1988, without the prior permission of the publisher.

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be

available in electronic books.

Designations used by companies to distinguish their products are often claimed as trademarks. All brand

names and product names used in this book are trade names, service marks, trademarks or registered trademarks

of their respective owners. The publisher is not associated with any product or vendor mentioned in this book.

This publication is designed to provide accurate and authoritative information in regard to the subject matter

covered. It is sold on the understanding that the publisher is not engaged in rendering professional services.

If professional advice or other expert assistance is required, the services of a competent professional

should be sought.

Library of Congress Cataloguing-in-Publication Data

Lueder, Ernst, 1932-

3D displays / Ernst Lueder.

p. cm.

Includes bibliographical references and index.

ISBN 978-1-119-99151-9 (cloth)

1. Three-dimensional display systems. I. Title. II. Title: Three D

displays.

TK7882.I6L84 2012

621.39087–dc232011032490

A catalogue record for this book is available from the British Library.

Print ISBN: 978-1-119-99151-9

ePDF ISBN: 978-1-119-96275-5

oBook ISBN: 978-1-119-96276-2

ePub ISBN: 978-1-119-96304-2

Mobi ISBN: 978-1-119-96305-9

Set in 10/12pt Times by Thomson Digital, Noida, India

To Helen

whose skills in computers and language were very helpful

Contents

Preface xi

Series Preface xiii

Introduction xv

1 The Physiology of 3D Perception 11.1 Binocular Viewing or Human Stereopsis 1

1.2 The Mismatch of Accommodation and Disparity and the Depths of Focus and of Field 3

1.3 Distance Scaling of Disparity 6

1.4 Interocular Crosstalk 7

1.5 Psychological Effects for Depth Perception 10

1.6 High-Level Cognitive Factor 10

Acknowledgments 11

References 11

2 Stereoscopic Displays 132.1 Stereoscopic Displays with Area Multiplexing 13

2.1.1 Retarders for the generation of polarizations 13

2.1.2 Wire grid polarizers for processing of the second view 20

2.1.3 Stereoscopic display with two LCDs 22

2.2 Combined Area and Time Division Multiplex for 3D Displays 26

2.3 Stereoscopic Time Sequential Displays 31

2.3.1 Time sequential viewing with an active retarder 31

2.3.2 Fast time sequential 3D displays by the use of OCB LCDs 33

2.3.3 Time sequential 3D displays with black insertions 33

2.4 Special Solutions for Stereoscopic Displays 41

2.5 Stereoscopic Projectors 48

2.6 Interleaved, Simultaneous, and Progressive Addressing of AMOLEDs and AMLCDs 60

2.7 Photo-Induced Alignment for Retarders and Beam Splitters 68

Acknowledgments 68

References 69

3 Autostereoscopic Displays 733.1 Spatially Multiplexed Multiview Autostereoscopic Displays with Lenticular Lenses 73

3.2 Spatially Multiplexed Multiview Autostereoscopic Displays with Switchable

Lenticular Lenses 85

3.3 Autostereoscopic Displays with Fixed and Switchable Parallax Barriers 95

3.4 Time Sequential Autostereoscopic Displays and Directional Backlights 104

3.4.1 Time sequential displays with special mirrors or 3D films 105

3.4.2 Time sequential displays with directionally switched backlights 109

3.5 Depth-Fused 3D Displays 115

3.6 Single and Multiview 3D Displays with a Light Guide 125

3.7 Test of 3D Displays and Medical Applications 129

Acknowledgments 129

References 130

4 Assessment of Quality of 3D Displays 1334.1 Introduction and Overview 133

4.2 Retrieving Quality Data from Given Images 135

4.3 Algorithms Based on Objective Measures Providing Disparity or Depth Maps 136

4.3.1 The algorithm based on the sum of absolute differences 136

4.3.2 Smoothness and edge detection in images 140

4.4 An Algorithm Based on Subjective Measures 146

4.5 The Kanade–Lucas–Toman (KLT) Feature Tracking Algorithm 153

4.6 Special Approaches for 2D to 3D Conversion 158

4.6.1 Conversion of 2D to 3D images based on motion parallax 159

4.6.2 Conversion from 2D to 3D based on depth cues in still pictures 161

4.6.3 Conversion from 2D to 3D based on gray shade and luminance setting 162

4.7 Reconstruction of 3D Images from Disparity Maps Pertaining to Monoscopic

2D or 3D Originals 165

4.7.1 Preprocessing of the depth map 165

4.7.2 Warping of the image creating the left and the right eye views 167

4.7.3 Disocclusions and hole-filling 172

4.7.4 Special systems for depth image-based rendering (DIBR) 176

Acknowledgments 182

References 183

5 Integral Imaging 1855.1 The Basis of Integral Imaging 186

5.2 Enhancement of Depth, Viewing Angle, and Resolution of 3D Integral Images 188

5.2.1 Enhancement of depth 189

5.2.2 Enlargement of viewing angle 193

5.2.3 Enhancing resolution 195

5.3 Integral Videography 196

5.4 Convertible 2D/3D Integral Imaging 207

Acknowledgments 214

References 214

6 Holography for 3D Displays 2176.1 Introduction and Overview 217

6.2 Recording a Hologram and Reconstruction of the Original 3D Image 218

6.3 A Holographic Screen 227

viii CONTENTS

6.4 Digital Holography Based on the Fourier Transform 229

6.5 A Holographic Laser Projector 232

Acknowledgments 235

References 235

7 Volumetric 3D Displays 2377.1 The Nature of Volumetric Displays 237

7.2 Accessing and Activating Voxels in Static Volumetric Displays 238

7.3 Swept Volume or Mechanical 3D Displays 245

Acknowledgments 252

References 252

8 A Shot at the Assessment of 3D Technologies 253

Index 257

CONTENTS ix

Preface

Flat panel display technology andmanufacture have now reached the level ofmaturity required to introduce

3D displays to the marketplace. The book covers five approaches to realize 3D perception, namely

stereoscopic and autostereoscopic displays, integral imaging, holography and volumetric displays.

I owe thanks to Dr. Tony Lowe who with his thorough understanding of scientific trends very much

supported the book on 3D technologies. I very much profited from Dan Schott’s excellent knowledge

about flat panel display technologies and I am very grateful for that. Based on his profound evaluation of

new display technologies, Dr. Christof Zeile drew my attention to various new publications. I very much

appreciate his support.

I would also like to express my appreciation of the excellent work performed by the typesetters.

The competent contribution to the index by Neil Manley is gratefully acknowledged.

As in earlier books, I am greatly indebted to Heidi Schuehle for diligently and observantly typing the

manuscript and to Rene Troeger for the professional and accomplished drawing of the figures.

Ernst Lueder

Scottsdale, USA, October 2011

Series Preface

Professor Lueder wrote his first book “Liquid Crystal Displays” for the Wiley-SID Series in Display

Technology in the year 2000. That bookwent on to become the best seller in the entire series and is now in

its second edition. I am therefore delighted to be writing a foreword to Ernst Lueder’s newest work, this

time on the topical subject of 3D Displays.

Most sighted human beings have a perception of what 3D means. We are familiar with what we see

around us, that we perceive some objects to be nearer than others, that distant objects traversing our field of

view appear to move more slowly than and are obscured by those nearer to us, and so on. A smaller but

growing fraction of the population is familiarwith 3Dmovies and television.However, amajoritywill have

only a vague understanding of how our brains operate on visual stimuli to create our familiar three-

dimensional view of the world. When it comes to creating 3D images on displays, further levels of

complexity are required not only to avoid eye strain by displaying inconsistent or misleading visual cues,

but to process prodigiously large quantities of data at sufficient speeds to enable real-time 3D visualisation.

This book sets out to present its subject in amannerwhich places it on a soundmathematical basis.After an

overview of the physiology of 3D perception, there follow detailed descriptions of stereoscopic and

autostereoscopic displays which are, after all, the most developed of 3D display technologies. Much

attention is given to the synthesis of 3D from 2D content, a most important topic, given the quantity of 2D

content already available. Quality issues are addressed next, with particular emphasis onmethods to improve

thevisual quality of 3D imagery and to reduce the bandwidth required to transmit it, with special emphasis on

a method known as depth image-based rendering. The book then describes three types of displays (integral

imaging, holography and volumetric displays) which, although less developed than stereoscopic and

autostereoscopic displays, are able to present real three-dimensional images in which the view changes -

with nearer objects obscuring more distant ones - as the viewer changes position. This is in contrast to

providing a mere illusion of three-dimensionality, as is the case with many stereoscopic images.

The book concludes with a chapter aptly named “AShot at the Assessment of 3DTechnologies” This is

not somuch a guess at what is coming next, but rather a logical in futuro extension of the technologies and

methods already described and, to my reading, a credible one.

This is a complete book, full of the necessary equations, with many illustrations and repletewith references.

The subject matter, whilst complex, is very clearly presented and will provide readers with a sound technical

basis from which to develop their skills further into the exciting field of three-dimensional display science.

Anthony Lowe

Braishfield, UK, 2011

Introduction

The design and manufacture of displays are now mature enough to introduce three-dimensional (3D)

displays into themarketplace. This happened first with displays for mobile devices in the form of near-to-

the-eye displays, but home TV will follow suit.

This book covers five approaches to realize 3D perception, namely, stereoscopic and autostereoscopic

displays, integral imaging, holography, and volumetric displays.

The intention guiding the book is to promote a well-founded understanding of the electro-optic effects

of 3D systems and of the addressing circuits. Equations are as a rule not simply stated but are derived, or, if

not fully done so, at least hints for the derivation are given.An example of this concept is the explanation of

the basics of holography by phasors, which will be outlined, but which are also known from electrical

engineering or from the Jones vector. This renders complex facts associated with holograms easier

to understand.

Emphasis is placed on stereoscopic and autostereoscopic displays as they are closest to being

commercialized. The basic components of stereoscopic displays are patterned retarders and to a lesser

degree wire grid polarizers. Autostereoscopic displays rely on beam splitters, lenticular lenses, parallax

barriers, light guides and various types of 3D films. All of these elements are explained in detail.

The glasses required for stereoscopic displays distinguish between the left and the right eyeviews either

by shutters or by circular polarization. Linearly polarized glasses have the disadvantage of being sensitive

to tilting of the head.

Special attention is given to 3D systems working in a spatial or temporal multiplex, as well as in a

combination of the two, and to novel fast addressing schemes. In order to suppress crosstalk and blur, a

240 Hz frame rate is preferred. The increased speed of addressing is handled by parallel processing and by

the recently published interleaved addressing, which also parallels the images. Special care is taken to

outline how the autostereoscopic approach is able to provide side views, the perspectives, of the object.

This paves the way for an understanding of integral images (IIs) with a pickup stage for information

similar to the lenticular lenses of the autostereoscopic displays. Very naturally this leads to the ingenious

design of an II projector working with real and virtual images where the viewer can walk around the

displayed object, thus enjoying a first solution for a true 3D display.

The chapter on holography leads the reader on to digital computer-generated holography, which is not

yet a real-time process.

Volumetric displays consist of a stack of LCDs, each of which is devoted to a particular depth, where

also the limitations of the fusion of the images become noticeable.

Notably, Chapter 4 is devoted to familiarizing designers of flat panel displays with the work done by

computer scientists on the assessment and improvement of 3D image quality. Algorithms are introduced

for evaluating the properties of 3D displays based on objective and subjective criteria and on tracking the

motion of selected special features. Special attention is drawn to establishing disparity maps and

preparing a 3D image ready for transmission with a bandwidth-saving “depth image - based rendering”

(DIBR). Head tracking for 3D reception by a group of single viewers is not included.

xvi INTRODUCTION

1

The Physiology of 3DPerception

1.1 Binocular Viewing or Human Stereopsis

As one eye is capable only of perceiving a planar image, 3D viewing is commonly achieved by the

cooperation of both eyes in providing each eyewith a view of the object. The images that the eyes receive

from the same object are different according to the different locations of the eyes. This binocular viewing

provides the perception of depth, the third dimension, as further explained by the horopter circle in

Figure 1.1. This circle serves as a reference from which the depth is determined [1,2]. If the eyes are

focusing, for which the synonyms fixating, accommodating, or converging are also used, on point M on

the horopter circle, the ciliarymuscles of the eyes rotate the eyeballs into such a position that the light from

Mpasses the pupils parallel to the axes of the lenses in the eyes. The axes intersect atM. Then the light hits

the retina in Figure 1.1 at the foveasml for the left eye andmr for the right eye. The foveas are in the center

of the retina and exhibit the highest density of light receptors. The rotation of the eyes is called the

vergence. Obviously the axes of the eyes are no longer parallel, which will provide the depth information

required by the brain [1,3]. In this situation light from point P hits the retinas at the points pl for the left eye

and pr for the right eye. The anglesa at the periphery of the circle are, as is known fromgeometry, the same

for all points P on the circle above the distance b between the pupils. As a consequence, also all the angles gfor points on the horopter circle are equal [4]. The angle g at the retina, measured as a rule in arcmin, is

called the disparity or the parallax. As all the pointsM and P on the horopter circle have the same disparity

g in both eyes, the difference d in the disparities of all points on this circle is zero. The further P is away

from M, but still on the horopter circle, the larger is the disparity [2,3]. Obviously the larger disparity is

associated with a smaller depth. The disparity information is transferred to the brain, which translates it

into a perceived depth. How the brain fuses the two disparities into a 3D image is not yet fully understood.

As all points on the horopter circle exhibit a zero difference in disparities, the circle serves as a reference

for the depth. The fusion of the disparities and the depth perception as described works only in Panum’s

fusional area in Figure 1.1 [3]. In this area, reliable depth perception decreases monotonically with

3D Displays, First Edition. Ernst Lueder.� 2012 John Wiley & Sons, Ltd. Published 2012 by John Wiley & Sons, Ltd.

increasing magnitude of the disparity. This relationship is called the patent stereopsis. For a point Q in

Figure 1.1 [3] not on the horopter circle but closer to the eyes and still in Panum’s area, the disparities on

the retina are given by the points ql for the left eye and qr for the right eye with the disparities g1 and g2.These points lie across the fovea on the other side of the retina and exhibit a so-called crossed disparity,

while the points farther away than the horopter have an uncrossed disparity. Their image points

corresponding to qr and ql for crossed disparities lie on the opposite side of the fovea.

For point Q the disparities g1 and g2 are no longer equal. The value g1� g2 6¼ 0 together with the

disparities themselves provide information to the brain on how much the depth of Q is different from

the depth on the horopter. However, how the brain copes with this difference of disparities is again not

fully known.

When moving an object from the horopter closer to the eye, the patent stereopsis is finally lost at a

distance of around 2mor less from the eyes. Fusion of the imagesmay no longer work and double images,

called diplopia, appear [3]. Due to overlarge disparities, the eyes perceive the object they are trying to

accommodate and its background separately. The brain unsuccessfully tries to suppress the background

information. On the other hand, the further away from the horopter the object is, the smaller is the

disparity, because the axes of the lenses become closer to being parallel. Finally, at distances beyond about

10m the differences between the small disparities can no longer be resolved and the depth information is

lost. This coincides with our inability to estimate the difference in depth of objects that are too far away.

The average distance b of the pupils in Figure 1.1 of adults in the USA is 6.5 cm, and for 90% of these

adults it lies between 6 and 7 cm [5]. The total range of disparity is about 80 arcmin for the perception of

spatial frequencies from 2 to 20 cycles per degree and about 8 arcdegrees for low spatial frequencies

around0.1 cycles per degree [3]. Thismeans that for low spatial frequencies larger disparities are available

than for larger spatial frequencies. As a consequence, the sensitivity of disparities for low spatial

frequencies is larger than for larger spatial frequencies. The same facts apply also for lower and larger

temporal frequencies of the luminance in an image.

The smallest still recognizable disparity, the stereoacuity Dmin, is 20 arcsec in the spatial frequency

range of about 2–20 cycles per degree,while themaximumperceivable disparityDmax is 40 arcmin for low

spatial frequencies [3]. As the values forDmin andDmax apply to both the crossed and uncrossed disparities

standing for different ranges of depths, the values can be added to a total of 80 arcmin for high and

8 arcdegrees for low spatial frequencies, as already given above [6,7]. Again this is also true for temporal

Figure 1.1 Horopter circle.

2 THE PHYSIOLOGY OF 3D PERCEPTION

frequencies in dynamic imageswith a larger sensitivity of disparities for lower temporal frequencies and a

lower sensitivity for larger temporal frequencies of luminance.

There are two visual pathways from the retina to the brain. The parvocellular-dominated dorsal–cor-

tical path connects the central retina to the ventral–cortical areas in the visual cortexwhere spatial patterns

and color are analyzed. The magno-cellular-dominated dorsal–cortical path leads from the central and

peripheral retina to dorsal–cortical areas in the visual cortex, where optical flow information for heading

control and biological motion are investigated. Further information on these paths can be found in [8–10].

The stereoanomalies are associated with defects in these paths of information where there are neurons

sensitive to only crossed or uncrossed disparities. The perception of depth is thought to involve responses

from both types of neurons. In stereoanomalous individuals, one type of these neurons fails to be sensitive

to their information. Then the other type of neurons dominates the response to all disparity information. In

the case where neurons are only sensitive to uncrossed disparities belonging to objects located further

away than the horopter circle, the information from crossed disparities stemming from objects closer to

the eye than the horopter is suppressed in favor of objects far away. The individual perceives the close-up

information as far away informationwith a far away depth.When the neurons are only sensitive to crossed

disparities, the individual perceives the far away information with a depth close to the eye [11,12].

Individuals who are stereoblind, as a rule resulting from a disease called strabismus, are assumed to be

entirely lacking in disparity-sensitive neurons.

Under degraded stimulus conditions such as brief stimulus exposure, stereoanomalies are found in 30%

of the population [13]. In addition, 6–8%of the population are stereoblind. The relatively large percentage

of people incapable of perceiving a 3D image would merit more attention.

Another physiological disturbance is binocular rivalry. In this case an individual views a stereo

display with a very large disparity or with interocular misalignment or distortion such that no fusion of

the two eyes’ image takes place [7,14]. One eye inhibits the visual activities of the other eye. One view

may be visible, as the other eye’s view is suppressed, which reverses over time. This is a problem

which may be experienced with headworn displays, where two images from different sources may be

misaligned or distorted [15].

Two physiological stimuli of depth can be detected by one eye alone. These are disparity and motion

parallax. Under this parallax the shift of a moving object toward a still background is understood. The eye

together with the brain extracts from this parallax a 3D perception with an associated depth.

Similar tomotion parallax is Pulfrich’s phenomenon [16].One eye is coveredwith a filter which darkens

the image. The processingof the dark image is delayed in relation to theprocessingof the bright image. This

leads to disparity errorswhen theviewermoves relative to an object. However, it can also be used to provide

a depth cue, as the delay renders the two eyes’ images differently as usually caused by depth.

1.2 The Mismatch of Accommodation and Disparityand the Depths of Focus and of Field

Nowwe are ready to consider a phenomenon explicablewith known stereoptic facts. Aswe shall see later,

in stereoscopic and autostereoscopic displays the two required views of an object are presented next to

each other on the screen of a display. The distance to the eyes of the viewer is constant for all scenes

displayed. That is the cause of a problem, as the eyes accommodate to the two images with a vergence

associated with the disparity. The disparity stimulates a depth perception in the brain. On the other hand,

the accommodation of points on the screen also conveys depth information, which is the constant distance

to the screen. The two depth details are contradictory, and are called themismatch of accommodation and

vergence or disparity. Thismay cause discomfort for viewers, manifested by eyestrain, blurred vision, or a

slight headache [7]. Fortunately the problems stemming from this mismatch are experienced mainly for

short viewing distances of around 0.5m. A quick and obvious explanation is the already mentioned fact

that for larger distances the disparities become smaller and are crowded together on the retina, so the

THE MISMATCH OF ACCOMMODATION AND DISPARITY 3

resolution of depth associated with disparity is diminished. Therefore the depth information based on

disparity no longer changes much with increasing distances and is more easily matched with the depth

information based on accommodation. In practice itwas found that a viewing distance of 2mormore from

a TV screen no longer leads to annoying discomfort [7].

A more thorough explanation is derived from the depth of focus and the depth of field, which is also

important for the design of a 3D system formoving viewers [17].We assume that the eyes have focused on

an object at point C in Figure 1.2, providing a sharp image. The depth of focus describes the range of

distance from a point P nearer to the eye thanC to a point D further away thanC inwhich an object can still

be detected by applying a given criterion for detection. If the distance of point P is p and that of D is d then

the depth of focus T in diopters is

T ¼ 1

p� 1

dð1:1Þ

where p and d are expressed in m. The depth of field is

F ¼ d � p ð1:2Þalso in m.

Diopters are defined by 1/f, where f is the focal length of a lens in m; in our case the lens is the eyewith

that f where the eyes experience a sharp image.

Possible criteria for the detectability of features in a display are:

(a) the deterioration of visual acuity or of resolving power;

(b) the discrimination of least perceptible blurring of the image;

(c) the loss of visibility or detectability of target details through loss of contrast; and

(d) the perceptual tolerance to out-of-focus blur which results in a stimulus for a change in

accommodation.

The first three criteria depend on the perception of out-of-tolerance blur, while the last one depends on

physiological tolerance. Point P is called the proximal blurring point, while D is the distal blurring point.

Below P and beyond D the image is no longer accepted.

The results reported now are based on criterion (a) and the out-of-focus blur in criterion (d) [17].

A checkerboard test pattern is used and test persons provide the percentage of correct answers in detecting

the correct pattern. The test pattern had a size of 1.25 arcmin corresponding to a Snellen notation of 20/25.

The diameter of the pupils was 4.6mm. The test result is shown in Figure 1.3. The abscissa represents the

displacement of the test pattern from the fixation point C measured in diopters. Hence the abscissa

indicates in diopters the degree to which the test pattern is out of focus. The ordinate represents the

percentage of the correct visual resolution perceived for the test pattern. This percentage exhibits a

Gaussian probability density.

Figure 1.2 Depth of focus and depth of field.

4 THE PHYSIOLOGY OF 3D PERCEPTION

The midpoint of the depth of focus is always slightly nearer to the eye than the focus point C.

For a 50% correct visual resolution, the depth of focus has awidth of 0.66 diopters, whereas for 99% the

width shrinks to 0.38 diopters. This shrinking is about 0.06 diopters for an increase in the visual

resolution of 10% of the proximal blurring. The depth of focus at the 99% level is an important one for the

out-of-focus blur at which the visual resolution begins to deteriorate.

The diagram in Figure 1.3 depends upon the location of the fixation point C. This is evident from

Table 1.1withmeasured distances for the fixation point C inm, the distances p of the proximal and d of the

distal blur also in m, as well as the resulting depth of focus T in diopters. Only if T were constant for all

points C would the diagram be independent of the location of C. The fixation point C for the diagram in

Figure 1.3 is about 1m from the eye. The depth of field, d� p, in m increases with increasing distance to

the fixation point C; it can even become infinite.

Further results in [17] relate to the influence of luminance, pupil diameter, and size of object in arcmin

on the depth of focus. The larger the luminance, the smaller the diameter of the pupil. At 0.03 cd/m2 the

diameter is 6mm, at 30 cd/m2 it is 3mm, and at 300 cd/m2 only 2mm.A linear decrease in the diameter of

the pupil is associated with a logarithmic increase in luminance. For a 1mm decrease of this diameter the

depth of focus increases by 0.12 diopters.

For an increase in the object by 0.25 arcmin the depth of focus increases by 0.35 diopters. At a size of

2 arcmin the depth of focus reaches 2 diopters.

Figure1.3 Percentage of correct resolutionperceived versus displacement of the test pattern from the fixation pointC

in Figure 1.2.

Table 1.1 Dependence of proximal and distal blur as well as depth of focus T on location of C

Distance of fixation

point C in m

Distance of proximal

blur p in m

Distance of distal

blur d in m

Depth of focus

T in diopters

1 0.75 1.5 1333� 0.666 ¼ 0.667

2 1 5 1� 0.2 ¼ 0.8

3 1.5 1 0.666

THE MISMATCH OF ACCOMMODATION AND DISPARITY 5

The results in Figure 1.3 are very important for those 3D displays where the viewer only has a sharp

picture at a given distance from the screen. Figure 1.3 reveals howmuch the viewer has tomove backward

and forward while still perceiving an acceptable image.

Newer values for the depth of field depending on the distance of the fixation point C are given in

Table 1.2 [18]. Obviously the depth of field increases strongly with increasing distance of the fixation

point. So fixation or accommodation on a given point is no longer so important for larger distances. As a

consequence for themismatch of accommodation and disparity, accommodation plays aminor rolewhich

also alleviates discomfort. This is no longer true for a fixation point at 0.5m or closer, meaning that

discomfort certainly is a problem for near-to-the-eye displays. For regular 3D displays a viewing distance

of at least 2m should sufficiently minimize discomfort, as already stated above.

In view of this result, discomfort when viewing 3D movies from larger distances should not occur as a

rule. This, however, is not the case, because there is a different effect identified as the cause of discomfort,

as discussed in Section 1.6.

Stereoscopic and autostereoscopic displays provide only an illusion of 3D perception. This is among

other effects due to the difficulty stemming from themismatch of accommodation and disparity, resulting

in a conflict of depth perception. Contrary to this, integral imaging, holography, and volumetric displays,

which will be treated later, do not exhibit this mismatch. There, the viewer, when moving, has the

impression of walking around the 3D object, thus experiencing true 3D. On the other hand the viewer

would always see the same image in the case of stereoscopic solutions.

1.3 Distance Scaling of Disparity

In stereopsis there are two definitions of perceived distance or depth. The egocentric view refers to the

conventional distance D between an observer and an object and is usually measured in m. On the other

hand, relative depth is based on the depth interval between a viewer and the reference point on the horopter

circle and ismeasured in radians of the disparity g on the retina in Figure 1.1. The disparity information g isconnected to D by a strongly nonlinear relation stemming from the geometry shown in Figure 1.1. This

relation has to be differently approximated or recalibrated or, in other words, scaled for different regions

of distance D [19,20].

For obtaining a veridical or true value, egocentric distance information D together with the relative

depth g are needed by the brain. It is assumed that the brain combines binocular disparity gwith egocentricdistance cues for the process of disparity scaling.

For a large distance D in real-world scenery, the magnitude of the disparity g varies, as we have seenintuitively from the geometry in Figure 1.1, approximately with the inverse of D2. It was found that g isalso proportional to the interpupillary distance b. This leads to the equation

g ¼ b d0

D2ð1:3Þ

in which d0, with the dimensions of cm arcmin, is an experimentally determined proportionality factor,

called the depth interval and sometimes also the predicted depth [21]; d0 is different for each D and is

approximated by a constant in an interval around D.

Table 1.2 Newer values for those in Table 1.1

Distance of fixation point C Distance of low end

of depth of field

Distance of high end

of depth of field

0.5 0.4 0.67

1 0.67 z

2 1 1

6 THE PHYSIOLOGY OF 3D PERCEPTION

In stereoscopic displays the disparity g depends approximately on the separation S between the two

images required for 3D perception and inversely on the viewing distance D. Thus

g ¼ S=D: ð1:4Þ

For Equation 1.4, a symmetrical convergence and targets on the retina close to the midsaggital plane

symmetrically dividing the body are required.

For a fixed value of S a stereoscopic display will provide the depth interval d0 as

d0 ¼ DS

b� S: ð1:5Þ

In the denominator the þ sign applies for crossed disparities and the� sign for uncrossed disparities.

In [22] it was found that this determination of d0 is very robust toward unequal luminances in the two

views. If the luminance exceeds 0.63 cd/m2 a 60% difference in the interocular luminance does not harm

the perception of the depth interval d0. However, for greater interocular luminance differences the

perceived depth may be far away from the correct value. Another luminance-related effect is the

discomfort created by interocular luminance differences of more than 25% [22]. Finally, interocular

differences in contrast of up to 83% did not affect depth perception, while the threshold for discomfort in

this case was between 25% and 50%.

1.4 Interocular Crosstalk

Information leaking from the view in one eye into that of the other eye is known as crosstalk, which as a

rule severely damages the quality of the perceived image but can also affect the fusion of the two images.

At no crosstalk the fusion is limited by 27 arcmin for crossed disparity and by 24 arcmin for uncrossed

disparity. For a 200ms stimulus, crosstalk has only a small effect on fusion,which is no longer true for a 2 s

stimulus [23]. In this case, 2–7% crosstalk can already hamper fusion and can cause discomfort [24].

Autostereoscopic displays may apply spatial multiplexing of the two views, for which an array of

lenticular lenses or parallax barriers is used. Lenticular lenses exhibit chromatic aberrations, while

barriers produce diffraction by which image content can leak into the wrong eye. The remedy is to limit

aberration and diffraction at least for a given position of the viewer.

For stereoscopic and autostereoscopic displays with temporal multiplexing, crosstalk occurs due to the

persistence of a display, inwhich the image content of one eye’s view is still visible in the next framewhen

that eye is exposed to a new view. This is shown in Figure 1.4. Temporal multiplexing can also induce

flicker seen in the visual periphery. This disrupts vision in large field-of-view immersive displays. The

cause is that these displays stimulate the magno-cellular-dominated dorsal–cortical area, which draws

connections from the peripheral retina, and above all have a transient response and high temporal acuity,

perceived as flicker. A remedy is a high frame rate enabling the visual system to integrate the intermittent

information in the periphery [6].

A further, very strong source of crosstalk is blurring of the edges of a moving image. Blur occurs in all

displays where the luminance of the image is held constant during the entire frame time. This occurs in

liquid crystal displays (LCDs) and in organic light-emitting diode displays (OLED displays). A relatively

brief description of this important phenomenon is given here, while a more detailed one can be found on

pages 298–300 of [25].

Blur is explained in Figure 1.5a, where a black stripe at rest on an LCD screen can be seen, while

Figure 1.5b shows the stripe moving to the right. The edges of the stripe in Figure 1.5a are perfectly sharp

but are blurred by themovement in Figure 1.5b. Themain cause is that an image on anLCD is held constant

during the frame time Tf, which for a frame frequency of f ¼ 60Hz is given by T ¼ 1/f ¼ 16.66ms.

INTEROCULAR CROSSTALK 7

This holding property does not occur in plasma display panels (PDPs) and in cathode ray tubes (CRTs) and

hence they are free of blur.

Now we consider what happens to the pixels in the column representing the left edge in Figure 1.5a.

Assuming black stands for fully white and white for fully black, then the luminance in the column has to

decay from fully white to fully black during the movement. This decay is gradual, as shown in Figure 1.6.

The reason is the delayed rotation of the liquid crystal molecules in response to an electric field applied in

the first frame in Figure 1.6. The display still provides full luminance at the beginning of the

frame time. One frame time later at time Tf the luminance is held at the value it had decayed to at

timeTf, as again indicated one frame time later in Figure 1.6. This stepwise decaying luminance continues

in Figure 1.6 until fully black is reached. The stepwise decay leads to the blurred left edge in Figure 1.5b

andwith the same explanation also for the right edge in Figure 1.5b. The duration of the decay is called the

blurred edge width (BEW). This duration can also be measured in the number of pixels that the first

Figure 1.4 Crosstalk due to persistence of luminance in an LCD.

Figure 1.5 (a) A stationary image and (b) the blurred edge of a moving image on an LCD.

Figure 1.6 Decay of luminance of a display and stepwise approximation representing the holding property of an

LCD.

8 THE PHYSIOLOGY OF 3D PERCEPTION

column has to travel to the right until a decay to black is reached. This number is n pixels per frame time

which providesBEW � n: ð1:6Þ

The stripe in Figure 1.5a becomeswider by the blur on each side. A severe consequence for 3D displays

is that the gap between the two views required for 3D is filled with blur, which may even extend into the

two original images. Thus blur represents crosstalk in both eyes.

Diminishing thewidth n of blur is mandatory for shrinking and even avoiding crosstalk in 3D displays.

In pursuing this goal we have to understand how blur depends on the frame frequency f and the frame time

Tf. This understanding is provided by the physiological law for the perception of a moving picture by the

human eye. It states among other things that the eyeballs track themovement of an object perfectly; this is

called the smooth pursuit along the track of the movement.

We apply this rule to Figure 1.7a with the coordinates t, with the frame times Tf, 2Tf, etc., and the

locationm, where themovement n per time Tf to the right of a black bar (white areas) is indicated. The eye

tracking of the movement is carried out in the direction of the arrows along the slanted line in Figure 1.7a.

We assume that after time 3Tf, the maximum luminance Lm is reached.

This generates the luminance V(x) over the location x in the diagram in Figure 1.7a, starting with a

luminanceof zeroatx ¼ 0andLmatx ¼ n. Thediagramrepresents the traceof the luminanceon the retina.

In order to determine the influence of the frame time, we plot in Figure 1.7b the same diagram but with

half the frame time Tf/2 and the same speed n of movement, resulting in an advance to the right by n/2

within Tf/2. The pertinent construction of V(x) reveals that the maximum luminance is reached after just

Figure 1.7 Speed of luminous response (a) for 60Hz and (b) for 120Hz frame frequencies.

INTEROCULAR CROSSTALK 9

half themovement at x ¼ n/2. As n represents thewidth of the blur, this blur is halved and hence crosstalkshrinks to half the width by doubling the frame frequency.

This result has enormous significance for reducing crosstalk by blur in 3D displays. As a rule for 3D

displays, a frame frequency of 240Hz is used for reducing crosstalk by a factor of four in comparison to a

60Hz frame. In this case crosstalk is virtually invisible, but the addressing circuits have to work at four

times the speed.

1.5 Psychological Effects for Depth Perception

The physiological depth perception discussed so far is complemented by a few monocular psychological

effects and experiences which are also encountered in 2D displays. They are learned while growing up.

With these effects, every 2D picture can induce a 3D impression.

The size of an image on the retina, the real size of which is known, indicates the depth by its actual

smaller perceived size. The hiding of an object by closer objects can also create a 3D impression. This

perception of so far hidden objects is called dynamic disclosure, while the occlusion of objects is called

dynamic occlusion. Both effects convey the sensation of depth. A 3D perspective is also created by two

straight lines which intersect at a vanishing point. This was discovered by medieval painters and is now

used extensively. Objects becoming bluish and misty with increasing distance also induce the perception

of depth. Further, one instinctively assumes that the illumination comes from above, so the length and

direction of the shadow also help to perceive depth. Finally, motion parallax, described in the next section,

is another strong trigger for depth. Further depth cues to be detailed later are the luminance, contrast, and

sharpness of an object.

1.6 High-Level Cognitive Factor

Immersive stereo displays such as 3Dmovies and 3D TV create real-world scenes by presenting a variety

of cues to depth and distance. These cues include binocular disparity, focusing on depth by accommoda-

tion, motion parallax, linear perspective, and texture perspective. For ease of viewing, all these cognitive

factors are supposed to provide the samemagnitude of depth, otherwise the viewer experiences high-level

cue conflict – high level because reasoning is involved, as we shall see below. Cue conflict induces

discomfort, as viewers may encounter in watching 3D movies [7].

This can be illustrated by a stereo scene of an American football game [26]. Binocular disparity

information may provide a depth perception of a few inches, while linear and texture perceptive

information could convey depths of several tens of yards consistent with the football field. A viewer

exposed to this conflict over a longer time will complain of discomfort. Psychologists explain this by

assuming that human reasoning is based on two modes of operation: activity in an analytical and in an

intuitive system [26]. The analytical system permits conscious rule-based reasoning, while the intuitive

system is based on situational pattern recognition. The latter approach uses information derived from

immersive stereo displays consisting of the perception of simultaneous redundant traits such as the

psychological features mentioned in Section 1.5. That way, immersive stereo displays stimulate

the intuitive system. However, there are exceptions, such as the perception of motion parallax, which

provides a stereo impression in the intuitive system that, as a rule, does not exist in immersive

stereo displays.

For a deeper understanding of this statementwe have to look at themotion parallax shown in Figure 1.8.

If a viewer is moving to the right with eyes fixed on the stationary point F, then the stationary objects

behind F are perceived as moving in the same direction to the right as the viewer, while those in front of F

are perceived as moving to the left. This motion parallax is part of the intuitive experience of the viewer

and provides the locomotive viewer with depth information relative to F.

10 THE PHYSIOLOGY OF 3D PERCEPTION

Now we consider single view immersive stereoscopic and autostereoscopic displays in which the 3D

impression can only be perceived from a single given position of the viewer. This implies that there is no

full motion parallax, because the entire scene is placed on the plane of the display screen. The eyes focus

on point F on that plane and all objects are perceived to be further away than F, that is, in an areawhere the

objects seem tomove in the same direction as theviewer.Objects closer than Fand theirmovement against

the viewer’s movement do not exist [27].

The fact that there is no full motion parallax in single view stereoscopic and autostereoscopic displays

contributes to confusion in the viewer’s brain, causing discomfort. This discomfort disappears in

multiview stereoscopic and autostereoscopic displays where the regular motion parallax can be detected

from a few specific viewing positions.

Obviously the discomfort originating from immersive stereoscopic and autostereoscopic displays is not

caused by the mismatch of accommodation and disparity which, as we know, plays virtually no role in

viewing from larger distances associated with immersive displays.

Acknowledgments

The author gratefully acknowledges permission to reproduce a figure granted by the institution named

below. The source of the figure is also listed below together with its corresponding number in this book.

Optical Society of America (OSA)

Journal of the Optical Society of America, vol. 49, no. 2, March 1959, p. 276, figure 2 Reproduced

as Figure 1.3

References

1. Wheatstone, C. (1838) Contributions to the physiology of vision. Philos. Trans. R. Soc. A, 128, 371.

2. Okoshi, T. (1976) Three Dimensional Imaging Techniques, Academic Press, New York.

3. Patterson, R. (2009) Human factors of stereoscopic displays. SID 09, p. 805

4. Bader, G. (1999) Elektrooptische Signalverarbeitung zur Darstellung autostereoskopischer Bewegtbilder und zur

Strukturerkennung. PhD thesis. University of Stuttgart.

5. Ferwerda, J.G. (1990) The World of 3D: 3D Book Productions, Borger.

6. Patterson, R. (2007) Human factors of 3D displays. J. SID, 151(11),861.

7. Patterson, R. (2009) Human factors of stereodisplays: an update. J. SID, 17(112),987.

8. Livingstone, M.G. and Hubel, O.H. (1988) Segregation of form, color movement and depth: anatomy, physiology

and perception. Science, 240, 740.

9. Schiller, P.H. et al. (1990) Role of the color opponent and broad band channels in vision. Visual Neurosci., 5, 321.

Figure 1.8 Motion parallax.

REFERENCES 11