parag k. mital - corpus-based visual synthesis an approach to artistic...
TRANSCRIPT
A C M S y m p o s i u m O n A p p l i e d P e r c e p t i o n | 2 3 A u g u s t 2 0 1 3
Corpus-based Visual Synthesis: An Approach to Artistic StylizationP a r a g K . M i t a l 1 M i c k G r i e r s o n 1 T i m S m i t h 2
Source Image Corpus Images Stylization
1D e p a r t m e n t o f C o m p u t i n g , G o l d s m i t h s , U n i v e r s i t y o f L o n d o n2D e p a r t m e n t o f P s y c h o l o g i c a l S c i e n c e s , B i r k b e c k , U n i v e r s i t y o f L o n d o n
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 712
Representation✤ Perception must be supported by pre-
attentive representations
✤ Abstract description of objects/scenes
✤ Numerous theories:✤ Gestalts✤ Geons (Biederman)✤ Object files (Wolfe)✤ Proto-objects (Rensink)✤ Indexicals (Pylyshyn)✤ Shapes (Marr)✤ Streams (Bregman)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 713
Representation✤ Perception must be supported by pre-
attentive representations
✤ Abstract description of objects/scenes
✤ Numerous theories:✤ Gestalts✤ Geons (Biederman)✤ Object files (Wolfe)✤ Proto-objects (Rensink)✤ Indexicals (Pylyshyn)✤ Shapes (Marr)✤ Streams (Bregman)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
4
✤ Perception must be supported by pre-attentive representations
✤ Abstract description of objects/scenes
✤ Numerous theories:✤ Gestalts✤ Geons (Biederman)✤ Object files (Wolfe)✤ Proto-objects (Rensink)✤ Indexicals (Pylyshyn)✤ Shapes (Marr)✤ Streams (Bregman)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
5
✤ Perception must be supported by pre-attentive representations
✤ Abstract description of objects/scenes
✤ Numerous theories: ✤ Gestalts✤ Geons (Biederman)✤ Object files (Wolfe)✤ Proto-objects (Rensink)✤ Indexicals (Pylyshyn)✤ Shapes (Marr)✤ Streams (Bregman)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
6
✤ Perception must be supported by pre-attentive representations
✤ Abstract description of objects/scenes
✤ Numerous theories:✤ Gestalts✤ Geons (Biederman)✤ Object files (Wolfe)✤ Proto-objects (Rensink)✤ Indexicals (Pylyshyn)✤ Shapes (Marr)✤ Streams (Bregman)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
7
✤ Perception must be supported by pre-attentive representations
✤ Abstract description of objects/scenes
✤ Numerous theories:✤ Gestalts✤ Geons (Biederman)✤ Object files (Wolfe)✤ Proto-objects (Rensink)✤ Indexicals (Pylyshyn)✤ Shapes (Marr)✤ Streams (Bregman)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
8
✤ Artists are well aware of the role of abstract representations in perception
✤ Influence how we look at and where we look in a scene
✤ Art Movements✤ Impressionism✤ Pointilism✤ Cubism✤ Orphism✤ Expressionism✤ Abstract-Expressionism
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
9
✤ Artists are well aware of the role of abstract representations in perception
✤ Influence how we look at and where we look in a scene
✤ Art Movements✤ Impressionism✤ Pointilism✤ Cubism✤ Orphism✤ Expressionism✤ Abstract-Expressionism
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
10
✤ Artists are well aware of the role of abstract representations in perception
✤ Influence how we look at and where we look in a scene
✤ Art Movements✤ Impressionism✤ Pointilism✤ Cubism✤ Orphism✤ Expressionism✤ Abstract-Expressionism
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
11
✤ Artists are well aware of the role of abstract representations in perception
✤ Influence how we look at and where we look in a scene
✤ Art Movements✤ Impressionism✤ Pointilism✤ Cubism✤ Orphism✤ Expressionism✤ Abstract-Expressionism
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
12
✤ Artists are well aware of the role of abstract representations in perception
✤ Influence how we look at and where we look in a scene
✤ Art Movements✤ Impressionism✤ Pointilism✤ Cubism✤ Orphism✤ Expressionism✤ Abstract-Expressionism
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
13
✤ Artists are well aware of the role of abstract representations in perception
✤ Influence how we look at and where we look in a scene
✤ Art Movements✤ Impressionism✤ Pointilism✤ Cubism✤ Orphism✤ Expressionism✤ Abstract-Expressionism
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Representation
14
✤ Artists are well aware of the role of abstract representations in perception
✤ Influence how we look at and where we look in a scene
✤ Art Movements✤ Impressionism✤ Pointilism✤ Cubism✤ Orphism✤ Expressionism✤ Abstract-Expressionism
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Related Work
15
✤ Filtering/Clustering
✤ Example-based Images
✤ Texture-transfer/Patch-based
✤ Dictionary methods/Collage approaches
Kyprianidis, J. et al., 2012. State of the “Art”: A Taxonomy of Artistic Stylization Techniques for Images and Video. IEEE transactions on Visualization and Computer Graphics. Available at: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6243138
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Problem Statement
16
✤ Our approach✤ Build corpus of abstract representations
from user chosen images
✤ Match target image’s abstract representations to nearest ones in corpus
✤ Synthesize target image using closest matches
✤ Interact with a simple set of parameters effecting representation detection and synthesis
✤ Create automated artistic stylizations of images/videos using an understanding of the role of abstract representations in art and perception
✤ Allow for a range of styles through a simple set of parameters
✤ Needs to be fast in order to explore different styles quickly / run in real-time
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Outline✤ CBVS Framework
✤ Results for Images/Videos
✤ Extensions
✤ Memory Mosaicing
✤ Augmented Reality Hallucinations
✤ PhotoSynthesizer (iOS app)
✤ Conclusion
17
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Outline✤ CBVS Framework
✤ Results for Images/Videos
✤ Extensions
✤ Memory Mosaicing
✤ Augmented Reality Hallucinations
✤ PhotoSynthesizer (iOS app)
✤ Conclusion
18
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
CBVS Framework
19
✤ Build corpus of abstract representations from user chosen images
✤ Match target image’s abstract representations to nearest ones in corpus
✤ Synthesize target image using closest matches
✤ Interact with a simple set of parameters effecting representation detection and synthesis
RC = {R1, R2, ..., RNC}
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Building the Corpus
20
✤ Need to represent:✤ Sparse/Dense strokes✤ Small/Large strokes
✤ Watershed? ✤ Posterization? ✤ Mean-Shift?
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Building the Corpus
21
✤ Maximally Stable Color Regions✤ No need for multiple scale detections✤ Implicit ordering of regions✤ Simple set of parameters for
discovering sparse/dense small/large strokes
✤ Fast/Robust across multiple views (used in video tracking)
✤ Similar process to the unconscious representations as theorized before
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7122
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7123
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7124
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7125
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7126
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7127
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7128
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7129
Increasing timesteps =
Denser detection More expressive corpus
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
CBVS Framework
30
✤ Build corpus of abstract representations from user chosen images
✤ Match target image’s abstract representations to nearest ones in corpus
✤ Synthesize target image using closest matches
✤ Interact with a simple set of parameters effecting representation detection and synthesis
RC = {R1, R2, ..., RNC}
RT = {R1, R2, ..., RNT }
?
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Matching
31
✤ Need to match regions in target to similar ones in corpus
✤ Describe Shape and Color
✤ Use Euclidean distance for shape values
✤ Use Perceptual distance for color values (CIEDE2000 formula)
✤ Nearest neighbor matching
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7132
✤ Need to match regions in target to similar ones in corpus
✤ Describe Shape and Color
✤ Use Euclidean distance for shape values
✤ Use Perceptual distance for color values (CIEDE2000 formula)
✤ Nearest neighbor matching
dRi =⇣µ00, ⌘11, ⌘20, ⌘02, L, a
⇤, b⇤⌘
⌘ij =µij
µ(1+ i+j
2 )00
Matching
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7133
✤ Need to match regions in target to similar ones in corpus
✤ Describe Shape and Color
✤ Use Euclidean distance for shape values
✤ Use Perceptual distance for color values (CIEDE2000 formula)
✤ Nearest neighbor matchingd(Rt, Rc) = ds(Rt, Rc) + dc(Rt, Rc)
Matching
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
CBVS Framework
34
✤ Build corpus of abstract representations from user chosen images
✤ Match target image’s abstract representations to nearest ones in corpus
✤ Synthesize target image using closest matches
✤ Interact with a simple set of parameters effecting representation detection and synthesis
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Synthesis
35
✤ From largest to smallest target region
✤ Find nearest neighbor
✤ Translate
✤ Rotate
✤ Scale
✤ Blend
d(Rt, Rc) = ds(Rt, Rc) + dc(Rt, Rc)
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Synthesis
36
T = centroidRt � centroidRc
✤ From largest to smallest target region
✤ Find nearest neighbor
✤ Translate
✤ Rotate
✤ Scale
✤ Blend
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Synthesis
37
⇥ =1
2⇤ arctan
2 ⇤ µ11
µ00
µ20
µ00� µ02
µ00
✤ From largest to smallest target region
✤ Find nearest neighbor
✤ Translate
✤ Rotate
✤ Scale
✤ Blend
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Synthesis
38
Sx
=width
RT
widthRC
Sy =heightRT
heightRC
✤ From largest to smallest target region
✤ Find nearest neighbor
✤ Translate
✤ Rotate
✤ Scale
✤ Blend
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Synthesis
39
✤ From largest to smallest target region
✤ Find nearest neighbor
✤ Translate
✤ Rotate
✤ Scale
✤ Blend
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Interaction✤ Spatial blending
✤ Temporal blending
✤ Motion tracking
✤ Timesteps
✤ Minimum region size
✤ Maximum region size
✤ Blending radius
40
Discrete Parameters
Continuous Parameters
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Spatial Blending
41
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Temporal Blending
42
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Temporal Blending
43
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7144
Increasing timesteps =
Denser layers, More expressive
Timesteps
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7145
Decreasing minimum size =
Finer brush strokes
Minimum Size
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7146
Increasing radius =
More of source texture
Blending Radius
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Outline✤ CBVS Framework
✤ Results for Images/Videos
✤ Extensions
✤ Memory Mosaicing
✤ Augmented Reality Hallucinations
✤ Conclusion
47
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7148
Target Image Corpus Synthesis
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7149
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7150
Target Image Corpus Synthesis
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7151
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7152
Target Image Corpus Synthesis
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7153
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7154
Target Image Corpus
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7155
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7156
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7157
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7158
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7159
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Outline✤ CBVS Framework
✤ Results for Images/Videos
✤ Extensions
✤ Memory Mosaicing
✤ Augmented Reality Hallucinations
✤ PhotoSynthesizer (iOS app)
✤ Conclusion
60
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Memory Mosaicing✤ Dynamic target (movie or webcam)
✤ Aggregate corpus over time using target, retaining only most recent N objects
✤ Only allow learning of objects with distance greater than threshold
61
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7162
Live demo
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Outline✤ CBVS Framework
✤ Results for Images/Videos
✤ Extensions
✤ Memory Mosaicing
✤ Augmented Reality Hallucinations
✤ PhotoSynthesizer (iOS app)
✤ Conclusion
63
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Augmented Reality Hallucinations
✤ Memory Mosaicing
✤ Uses Augmented Reality headset
✤ Exhibited during the Victoria & Albert Museum’s Digital Design Weekend co-located during the London Design Festival, 15,000 participants
✤ Short questionnaire for participants rating their experience (21 participants only)
64
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7165
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Outline✤ CBVS Framework
✤ Results for Images/Videos
✤ Extensions
✤ Memory Mosaicing
✤ Augmented Reality Hallucinations
✤ PhotoSynthesizer (iOS app)
✤ Conclusion
66
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
PhotoSynthesizer ✤ Free iOS app allows user to synthesize
target image
✤ No interaction besides selecting target and corpus
✤ Reveals synthesis process as painting regions over time
✤ Reached Top 50 app in Photo & Video in many countries
67
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
PhotoSynthesizer
68
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 7169
Live demo
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Outline✤ CBVS Framework
✤ Results for Images/Videos
✤ Extensions
✤ Memory Mosaicing
✤ Augmented Reality Hallucinations
✤ PhotoSynthesizer (iOS app)
✤ Conclusion
70
Corpus-based Visual Synthesis: An Approach to Artistic Stylization | Parag K. Mital | http://pkmital.com / 71
Conclusion✤ Simple shape representation affords
range of stylizations and a range of non/real-time applications
✤ Expressive control in a few parameters
✤ Future?✤ Better method of evaluation✤ Better metrics for shape description✤ Better temporal coherence
71