applied perception in graphics
DESCRIPTION
Applied Perception in Graphics. Erik Reinhard University of Utah [email protected]. Computer Graphics. Produce computer generated imagery that cannot be distinguished from real scenes Do this in real-time. Trends in Computer Graphics. Greater realism Scene complexity - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/1.jpg)
Applied Perception in Applied Perception in GraphicsGraphics
Erik ReinhardErik ReinhardUniversity of UtahUniversity of Utah
[email protected]@cs.utah.edu
![Page 2: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/2.jpg)
Computer GraphicsComputer Graphics
• Produce computer Produce computer generated imagery generated imagery that cannot be that cannot be distinguished from distinguished from real scenesreal scenes
• Do this in real-timeDo this in real-time
![Page 3: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/3.jpg)
Trends in Computer GraphicsTrends in Computer Graphics• Greater realismGreater realism
– Scene complexityScene complexity– Lighting simulations Lighting simulations
• Faster renderingFaster rendering– Faster hardwareFaster hardware– Better algorithmsBetter algorithms
• Together: still too slow and unrealisticTogether: still too slow and unrealistic
![Page 4: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/4.jpg)
Algorithm designAlgorithm design
• Largely opportunisticLargely opportunistic
• Computer graphics is a maturing fieldComputer graphics is a maturing field
• Hence, a more directed approach is Hence, a more directed approach is neededneeded
![Page 5: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/5.jpg)
Long Term StrategyLong Term Strategy• Understand the differences between Understand the differences between
natural and computer generated scenesnatural and computer generated scenes
• Understand the Human Visual System Understand the Human Visual System and how it perceives imagesand how it perceives images
• Apply this knowledge to motivate Apply this knowledge to motivate graphics algorithmsgraphics algorithms
![Page 6: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/6.jpg)
This Presentation (1)This Presentation (1)
Reinhard et. al., “Color Transfer between Images”, IEEE CG&A, sept. 2001.
![Page 7: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/7.jpg)
This Presentation(2)This Presentation(2)
Reinhard et. al., “Photographic Tone Reproduction for Digital Images, SIGGRAPH 2002.
![Page 8: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/8.jpg)
IntroductionIntroductionThe Human Visual System is evolved
to look at natural images
Natural Random
![Page 9: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/9.jpg)
Human Visual SystemHuman Visual System
![Page 10: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/10.jpg)
RetinaRetina
![Page 11: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/11.jpg)
Color ProcessingColor ProcessingRod and Cone pigments
![Page 12: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/12.jpg)
Color ProcessingColor Processing
Cone output is logarithmic
Color opponent space
![Page 13: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/13.jpg)
Image StatisticsImage Statistics
• Ruderman’s work on color statistics:Ruderman’s work on color statistics:
– Principal Components Analysis (PCA) on Principal Components Analysis (PCA) on colors of natural image ensemblescolors of natural image ensembles
– Axes have meaning: color opponents Axes have meaning: color opponents (luminance, red-green and yellow-blue)(luminance, red-green and yellow-blue)
![Page 14: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/14.jpg)
Color Processing SummaryColor Processing Summary• Human Visual System expects images with Human Visual System expects images with
natural characteristics (not just color)natural characteristics (not just color)
• Color opponent space has decorrelated axesColor opponent space has decorrelated axes
• Color space is logarithmic (compact and Color space is logarithmic (compact and symmetrical data representation)symmetrical data representation)
• Independent processing along each axis Independent processing along each axis should be possible should be possible Application Application
![Page 15: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/15.jpg)
Color TransferColor Transfer • Make one image look like anotherMake one image look like another
• For both images:For both images:– Transfer to new color spaceTransfer to new color space– Compute mean and standard deviation along Compute mean and standard deviation along
each color axiseach color axis
• Shift and scale target image to have Shift and scale target image to have same statistics as the source imagesame statistics as the source image
![Page 16: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/16.jpg)
LL Color Space Color Space
Convert RGB Convert RGB triplets to LMS triplets to LMS cone spacecone space
Take logarithmTake logarithm
Rotate axesRotate axes
![Page 17: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/17.jpg)
Why not use RGB space?Why not use RGB space?Input images Output images
RGB
L
![Page 18: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/18.jpg)
Color Transfer ExampleColor Transfer Example
![Page 19: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/19.jpg)
Color Transfer ExampleColor Transfer Example
![Page 20: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/20.jpg)
Color Transfer ExampleColor Transfer Example
![Page 21: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/21.jpg)
Color Processing SummaryColor Processing Summary
• Changing the statistics along each axis Changing the statistics along each axis independently allows one image to independently allows one image to resemble a second imageresemble a second image
• If the composition of the images is very If the composition of the images is very unequal, an approach using small unequal, an approach using small swatches may be used succesfullyswatches may be used succesfully
![Page 22: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/22.jpg)
Tone ReproductionTone Reproduction
![Page 23: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/23.jpg)
Tone ReproductionTone Reproduction
![Page 24: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/24.jpg)
Global vs. LocalGlobal vs. Local
• GlobalGlobal– Scale each pixel according to a fixed curveScale each pixel according to a fixed curve– Key issue: shape of curveKey issue: shape of curve
• LocalLocal– Scale each pixel by a local averageScale each pixel by a local average– Key issue: size of local neighborhoodKey issue: size of local neighborhood
![Page 25: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/25.jpg)
Global OperatorsGlobal Operators
TumblinWard
Ferwerda
![Page 26: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/26.jpg)
Global OperatorsGlobal Operators
TumblinWard
Ferwerda
![Page 27: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/27.jpg)
Local OperatorLocal Operator
Pattanaik
![Page 28: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/28.jpg)
Spatial ProcessingSpatial Processing• Light reaches the retina and is detected Light reaches the retina and is detected
by rods and conesby rods and cones
• The number of rods and cones is much The number of rods and cones is much larger than the number of nerves larger than the number of nerves leaving the eyeleaving the eye
• Hence, data reduction occurs in the Hence, data reduction occurs in the retinaretina
![Page 29: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/29.jpg)
Spatial ProcessingSpatial Processing
• Certain aspects of natural images Certain aspects of natural images are more important than othersare more important than others
• For example, contrast edges need For example, contrast edges need to be detected with accuracy, to be detected with accuracy, whereas slow gradients do not whereas slow gradients do not need to be perceived at high need to be perceived at high resolutionresolution
![Page 30: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/30.jpg)
Spatial ProcessingSpatial Processing• Circularly symmetric Circularly symmetric
receptive fieldsreceptive fields
• Centre-surround Centre-surround mechanismsmechanisms– Laplacian of GaussianLaplacian of Gaussian– Difference of GaussiansDifference of Gaussians– BlommaertBlommaert
• Scale space modelScale space model
![Page 31: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/31.jpg)
Scale Space (Histogram Scale Space (Histogram Equalized Images)Equalized Images)
![Page 32: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/32.jpg)
Tone Reproduction IdeaTone Reproduction Idea
• Modify existing global Modify existing global operator to be a local operator to be a local operator, e.g. Greg operator, e.g. Greg Ward’sWard’s
• Use spatial processing Use spatial processing to determine a local to determine a local adaptation level for adaptation level for each pixeleach pixel
5.2
4.0
4.0max
max
5.2
4.0
4.0max
max
),(219.12219.1),(
219.12219.1),(
yxL
L
LyxLL
L
L
LyxLL
a
d
doutput
w
d
doutput
![Page 33: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/33.jpg)
Blommaert Brightness ModelBlommaert Brightness Model
22
2
22
1 skr
ii
iesk
R
ii RvuLsyxV ),(),,(
),,(2),,(),,()(),,(
12
21
syxVs
syxVsyxVsWsyxV
ns
s
syxVyxB0
),,(),(
Gaussian filter
Center/surround
Neural response
Brightness
![Page 34: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/34.jpg)
BrightnessBrightness
ns
s
syxVyxB0
),,(),(
![Page 35: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/35.jpg)
Scale Selection AlternativesScale Selection Alternatives
0
)),(,,(),(ssyxsyxVyxB
n
m
Mean value
Thresholded )),(,,(: yxsyxVs mm
How large should a local neighborhood be?
![Page 36: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/36.jpg)
Mean ValueMean Value
0
)),(,,(),(ssyxsyxVyxB
n
m
![Page 37: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/37.jpg)
ThresholdedThresholded
),,(: mm syxVs
![Page 38: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/38.jpg)
Tone-mappingTone-mapping
5.2
4.0
4.0max
max
1
),(219.12219.1),(
)),(,,(),(
yxL
L
LyxLL
yxsyxVyxL
a
d
doutput
maLocal adaptation
Greg Ward’s tone-mapping with local adaptation
![Page 39: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/39.jpg)
ResultsResults• Good results, but something odd about Good results, but something odd about
scale selection:scale selection:
• For most pixels, a large scale was For most pixels, a large scale was selectedselected
• Implication: a simpler algorithm should Implication: a simpler algorithm should be possiblebe possible
![Page 40: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/40.jpg)
Simplify AlgorithmSimplify Algorithm
),(),(
),(1),(
),(219.12219.1),(
5.2
4.0
4.0max
max
yxLLayxL
yxLyxLL
yxL
L
LyxLL
ww
output
a
d
doutput
Greg Ward’s tone-mapping with local adaptation
Simplify
Fix overall lightness of image
![Page 41: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/41.jpg)
Global Operator ResultsGlobal Operator Results
WardOur method
![Page 42: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/42.jpg)
Global Operator ResultsGlobal Operator Results
WardOur method
![Page 43: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/43.jpg)
Global Global Local Local
)),(,,(1),(
),(1),(
1 yxsyxVyxLL
yxLyxLL
moutput
output
Global operator
Local operator
![Page 44: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/44.jpg)
Local Operator ResultsLocal Operator Results
Global
Local
![Page 45: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/45.jpg)
Local Operator ResultsLocal Operator Results
Global Local Pattanaik
![Page 46: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/46.jpg)
SummarySummary• Knowledge of the Human Visual System Knowledge of the Human Visual System
can help solve engineering problemscan help solve engineering problems
• Color and spatial processing Color and spatial processing investigatedinvestigated
• Direct applications shownDirect applications shown
![Page 47: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/47.jpg)
Ongoing ResearchOngoing Research• Natural Image StatisticsNatural Image Statistics
• Applications:Applications:– Reconstruction filtersReconstruction filters– Perlin noisePerlin noise– Fractal terrainsFractal terrains
![Page 48: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/48.jpg)
Ongoing ResearchOngoing ResearchImpoverished environments
![Page 49: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/49.jpg)
Future WorkFuture Work
This presentation
![Page 50: Applied Perception in Graphics](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816336550346895dd3c345/html5/thumbnails/50.jpg)
AcknowledgmentsAcknowledgments• Thanks to my colaborators: Peter Thanks to my colaborators: Peter
Shirley, Jim Ferwerda, Mike Stark, Shirley, Jim Ferwerda, Mike Stark, Mikhael Ashikhmin, Bruce Gooch, Tom Mikhael Ashikhmin, Bruce Gooch, Tom TrosciankoTroscianko
• This work sponsored by NSF grants 97-This work sponsored by NSF grants 97-96136, 97-31859, 98-18344, 99-78099 96136, 97-31859, 98-18344, 99-78099 and by the DOE AVTC/VIEWSand by the DOE AVTC/VIEWS