ai-sketcher: a deep generative model for generating high ... · ai-sketcher - cnn-based autoencoder...
TRANSCRIPT
![Page 1: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/1.jpg)
AI-Sketcher: A Deep Generative Model for Generating High Quality Sketches
Nan Cao, Xin Yan, Yang Shi, Chaoran Chen
Intelligent Big Data Visualization LabTongji University
![Page 2: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/2.jpg)
BACKGROUND - Cave Painting
2
![Page 3: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/3.jpg)
BACKGROUND - Children’s Drawings
3
![Page 4: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/4.jpg)
BACKGROUND - Design Drafts
4
![Page 5: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/5.jpg)
Can AI help designers create high quality sketches and boost their productivity and creativity?
5
![Page 6: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/6.jpg)
AI-Sketcher
6
![Page 7: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/7.jpg)
7
![Page 8: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/8.jpg)
Reference Pipeline for AI-Supported Sketching
Stroke Encoding Learning Generating
1 2 3
8
![Page 9: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/9.jpg)
[△x,△y,p1,p2,p3]
Start coordinates relative to the end of last stroke
one-hotcontinuous
continuousdrawing from the last point,
end of drawing a stroke
end of drawing a sketch
Stroke Encoding
Stroke Encoding
Learning
Generating
9——“Ha D, Eck D. A neural representation of sketch drawings[J]. arXiv preprint arXiv:1704.03477, 2017.”
![Page 10: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/10.jpg)
Learning (Sketch-RNN)
Stroke Encoding
Learning
Generating
10——“Ha D, Eck D. A neural representation of sketch drawings[J]. arXiv preprint arXiv:1704.03477, 2017.”
Encoder: Bidirectional RNN (BRNN)
![Page 11: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/11.jpg)
Learning (Sketch-RNN)
Random Sampling for Stroke Generation
Encoder: Bidirectional RNN (BRNN)
Stroke Encoding
Learning
Generating
11——“Ha D, Eck D. A neural representation of sketch drawings[J]. arXiv preprint arXiv:1704.03477, 2017.”
![Page 12: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/12.jpg)
Decoder: Autoregressive RNN
Random Sampling for Stroke Generation
Learning (Sketch-RNN)
Encoder: Bidirectional RNN (BRNN)
Stroke Encoding
Learning
Generating
12——“Ha D, Eck D. A neural representation of sketch drawings[J]. arXiv preprint arXiv:1704.03477, 2017.”
![Page 13: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/13.jpg)
Generating (Sketch-RNN)
Stroke Encoding
Learning
Generating
13——“Ha D, Eck D. A neural representation of sketch drawings[J]. arXiv preprint arXiv:1704.03477, 2017.”
![Page 14: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/14.jpg)
LIMITATIONS OF SKETCH-RNN
Low quality
2. dealing with multi-class situations1. generating sketches in one category (bird).
14
![Page 15: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/15.jpg)
AI-SKETCHER
15
![Page 16: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/16.jpg)
AI-SKETCHER
16
![Page 17: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/17.jpg)
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
AI-SKETCHER
17
![Page 18: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/18.jpg)
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER
18
![Page 19: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/19.jpg)
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
AI-SKETCHER
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
19
![Page 20: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/20.jpg)
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
• Loss function is modified.
20
![Page 21: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/21.jpg)
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
• Loss function is modified.
21
![Page 22: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/22.jpg)
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
• Loss function is modified.
22
![Page 23: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/23.jpg)
AI-SKETCHER - Conditional Sequence-to-Sequence VAE
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
23
![Page 24: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/24.jpg)
AI-SKETCHER - Conditional Sequence-to-Sequence VAE
Encoder: Bidirectional RNN
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
24
![Page 25: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/25.jpg)
AI-SKETCHER - Conditional Sequence-to-Sequence VAE
Xs: the sequences of strokes.
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
25
![Page 26: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/26.jpg)
AI-SKETCHER - Conditional Sequence-to-Sequence VAE
c is a k-dimensional one-hot conditional vector with k indicates the number of conditions.
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
26
![Page 27: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/27.jpg)
AI-SKETCHER - Conditional Sequence-to-Sequence VAE
hc is further transformed into two vectors to capture the distributions of the training strokes:
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
27
![Page 28: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/28.jpg)
AI-SKETCHER - Conditional Sequence-to-Sequence VAE
A latent vector has been randomly sampled from the distributions for generating the next strokes:
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
28
![Page 29: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/29.jpg)
AI-SKETCHER - Conditional Sequence-to-Sequence VAE
For decoding zs is concatenated together with the image feature vector zr, the latent influence vector ad, the conditional vector c and the last stroke vector si:
z = [zs; zr; ad; c; si]
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
29
![Page 30: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/30.jpg)
AI-SKETCHER - Conditional Sequence-to-Sequence VAE
predict the probabilities of the relative position of the next drawing point:
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
30
![Page 31: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/31.jpg)
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
• Loss function is modified.
31
![Page 32: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/32.jpg)
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
• Loss function is modified.
32
![Page 33: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/33.jpg)
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER - Influence Layer
33
![Page 34: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/34.jpg)
AI-SKETCHER - Influence Layer
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
It considers all the previous hidden node values until the latest drawing step in the RNN encoder.
34
![Page 35: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/35.jpg)
AI-SKETCHER - Influence Layer
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
The influence vector ad is a latent vector whose fields are sampled from these normal distributions:
35
![Page 36: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/36.jpg)
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
• Loss function is modified.
36
![Page 37: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/37.jpg)
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
• Loss function is modified.
37
![Page 38: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/38.jpg)
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
AI-SKETCHER - CNN-based Autoencoder
38
![Page 39: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/39.jpg)
AI-SKETCHER - CNN-based Autoencoder
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
Xr: the input raster image matrix.
39
![Page 40: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/40.jpg)
• three convolutional layers with the stride size as 2.
• the other three layers with the stride size as 1.
AI-SKETCHER - CNN-based Autoencoder
Encoder:
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
40
![Page 41: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/41.jpg)
AI-SKETCHER - CNN-based Autoencoder
Encoder:
• The last layer is a fully-connected neural network to produce the latent feature vector zr with 128 dimensions.
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
41
![Page 42: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/42.jpg)
AI-SKETCHER - CNN-based Autoencoder
Decoder:
• three deconvolutional layers with stride sizes equal to 2.
• the other three layers with stride sizes equal to 1.
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
42
![Page 43: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/43.jpg)
AI-SKETCHER - CNN-based Autoencoder
ReLU is used as the activation function in both convolutional and deconvolutional layers.
tanh is used as the activation function of the fully-connected neural network.
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
43
![Page 44: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/44.jpg)
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
• Loss function is modified.
44
![Page 45: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/45.jpg)
• A conditional vector is used to ensure a high quality generation of sketches from multiple categories.
• An influence layer is introduced to estimate how the previous strokes will influence on the next stroke.
AI-SKETCHER
• A CNN-based autoencoder is employed to capture the spatial information of a training set.
• Loss function is modified.
45
![Page 46: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/46.jpg)
AI-SKETCHER - Loss function
• Loss function is modified.
46
![Page 47: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/47.jpg)
AI-SKETCHER - Loss function
• Loss function is modified.
47
![Page 48: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/48.jpg)
AI-SKETCHER - Loss function
• Loss function is modified.
the reconstruction lossestimates the differences between the generated strokes and the training samples.
estimates the distribution differences between the generated strokes and the strokes in the training set modeled by the standard normal distribution.
48
![Page 49: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/49.jpg)
AI-SKETCHER - Loss function
• Loss function is modified.
49
![Page 50: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/50.jpg)
EVALUATION
We performed three experiments in purpose of validating the AI-Sketcher’s
• drawing quality
• capability of generating sketches from multiple classes
• generation diversity
50
![Page 51: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/51.jpg)
EVALUATION - Dataset
The QuickDraw dataset contains over 50 million sketches in 75 object categories and originally used for training Sketch-RNN.
https://quickdraw.withgoogle.com/ 51
![Page 52: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/52.jpg)
The FaceX dataset consists of 5 million sketches of both male's and female's facial expressions showing seven different types of emotions.
EVALUATION - Dataset
https://facex.idvxlab.com/ 52
![Page 53: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/53.jpg)
EVALUATION - Dataset
The FaceX dataset consists of 5 million sketches of both male's and female's facial expressions showing seven different types of emotions.
https://facex.idvxlab.com/
53
![Page 54: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/54.jpg)
Experiments based on FaceX Dataset
EVALUATION - 1st Experiment
Drawing quality
54
![Page 55: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/55.jpg)
Experiments based on FaceX Dataset
EVALUATION - 1st Experiment
Drawing quality
55
![Page 56: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/56.jpg)
Experiments based on FaceX Dataset
EVALUATION - 1st Experiment
Drawing quality
56
![Page 57: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/57.jpg)
Experiments based on FaceX Dataset
EVALUATION - 1st Experiment
Drawing quality
57
![Page 58: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/58.jpg)
Experiments based on FaceX Dataset
EVALUATION - 1st Experiment
Drawing quality
58
![Page 59: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/59.jpg)
EVALUATION - 1st Experiment
A within-subject user study
• 20 participants (10 females)
• The repeated measures one way ANOVA analysis showed that the generation quality of AI-Sketcher had an average rating of 3.9and was significantly better than that of the baseline models (with all p<.01).
Drawing quality
59
![Page 60: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/60.jpg)
EVALUATION - 2nd Experiment
Experiments based on QuickDraw Dataset
Generating sketches from multiple classes
60
![Page 61: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/61.jpg)
EVALUATION - 2nd Experiment
Experiments based on QuickDraw Dataset
Generating sketches from multiple classes
61
![Page 62: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/62.jpg)
EVALUATION - 2nd Experiment
Experiments based on QuickDraw Dataset
Generating sketches from multiple classes
62
![Page 63: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/63.jpg)
EVALUATION - 2nd Experiment
Experiments based on QuickDraw Dataset
Generating sketches from multiple classes
63
![Page 64: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/64.jpg)
EVALUATION - 2nd Experiment
Experiments based on QuickDraw Dataset
Generating sketches from multiple classes
64
![Page 65: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/65.jpg)
EVALUATION - 3rd Experiment
Generation Diversity
Experiments based on QuickDraw Dataset
In each set, the pairwised distances between sketches were calculated based on the perceptual hash.
The unpaired t-test showed that AI-Sketcher and Sketch-RNN had no significant difference.
65
![Page 66: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/66.jpg)
POTENTIAL APPLICATION
66
![Page 67: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/67.jpg)
CONCLUSION
67
We introduced AI-Sketcher, a hybrid deep
learning model to automatically generate high
quality sketch drawings .
Our model improves drawing quality by
• employing a CNN-based autoencoder to
capture the positional information.
• introducing an influence layer to more
precisely guide the generation of each stroke.
• provideding a conditional vector to support
multi-class sketch generation.
![Page 68: AI-Sketcher: A Deep Generative Model for Generating High ... · AI-SKETCHER - CNN-based Autoencoder Decoder: • three deconvolutional layers with stride sizes equal to 2. • the](https://reader030.vdocuments.mx/reader030/viewer/2022040310/5f38280e35525641d4719930/html5/thumbnails/68.jpg)
https://facex.idvxlab.com/
Thank You
Nan Cao, Xin Yan, Yang Shi, Chaoran Chen
Intelligent Big Data Visualization Lab
Tongji University
68