![Page 1: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/1.jpg)
Single Image Super Resolution of Textures via
CNNsAndrew Palmer
![Page 2: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/2.jpg)
What is Super Resolution (SR) ?Simple: Obtain one or more high-resolution images from one or more low-resolution ones
![Page 3: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/3.jpg)
![Page 4: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/4.jpg)
Many, many applicationsBiometric recognition, including resolution enhancement for faces fingerprints, and iris images
Medical diagnosis
Image compression
Text enhancement; preprocessing step for optical character recognition
![Page 5: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/5.jpg)
SR has diverse challengesUnderspecified problem; many solutions
No solid theory for determining what is 'good' enhancement. If it looks good, it looks good
![Page 6: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/6.jpg)
Computerphile: Bicubic Interpolation - https://goo.gl/wAvNtM
![Page 7: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/7.jpg)
Visualizing the problem for 2x upscaling
![Page 8: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/8.jpg)
Nearest Neighbour Upscaling
![Page 9: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/9.jpg)
Nearest Neighbour Upscaling
![Page 10: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/10.jpg)
Linear Upscaling
![Page 11: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/11.jpg)
Linear Upscaling
![Page 12: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/12.jpg)
Cubic Upscaling
![Page 13: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/13.jpg)
Cubic Upscaling
![Page 14: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/14.jpg)
Classical interpolation; most seen in practice
Image resizing options in Photoshop
![Page 15: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/15.jpg)
Other traditional methodsGaussian Smoothing, Wiener, Median filters (good at denoising)
Sharpening by amplifying existing image details (need to ensure that noise isn’t amplified)
![Page 16: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/16.jpg)
Texture Super ResolutionResults could use some improvement in texture quality
![Page 17: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/17.jpg)
The Describable Textures Dataset
The Describable Textures Dataset (DTD): textural images in the wild, annotated with a series of human-centric attributes, inspired by the perceptual properties of textures
5640 images, split into 47 classes, e.g. blotchy, polka dot, grainy
Describable Textures Dataset - https://goo.gl/YLtw2S
![Page 18: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/18.jpg)
Setup: Data synthesis by cropping
Low res (LR) image synthesis: crop ground truth images into several sub-images
![Page 19: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/19.jpg)
Setup: Generate a LR image from each patchFor each HR cropped image:
1. Apply a Gaussian convolution2. Sub-sample by the upscaling factor (produces a smaller image)3. Bicubic upscaling
![Page 20: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/20.jpg)
Input data going into the CNNTotal number of generated patches: 2,436,258
A subset (500) of the original images was used:Training set: 127,744 patchesValidation: 12,544 patches
![Page 21: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/21.jpg)
Common Evaluation MetricsPeak Signal to Noise (PSNR); >=30 dB for restoration = very good
Structural Similarity (SSIM)
Subjective perception
Time efficiency
![Page 22: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/22.jpg)
Setup: HyperparametersActivations: ReLU, ELU, tanh (popular choice for SR)
Loss function: MSE (old, but most common for SR)
Optimizer: ADAM
Iterations: 20,000
Learning rate: 10-3
https://xkcd.com/1838/
![Page 23: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/23.jpg)
Setup: Misc.Over the RGB colour space
Zero padded edges; each layer outputs same dimensions
No max pooling layers (might be bad for denoising and SR tasks https://arxiv.org/abs/1511.04491)
![Page 24: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/24.jpg)
SRCNN
The effectiveness of deeper structures for super resolution is found not as apparent as that shown in image classification
![Page 25: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/25.jpg)
CNN Architecture
ConvActivation: ReluFilter size: 9x9Filters: 64Padding: sameDim (out): 32 x 32
ConvActivation: ReluFilter size: 3x3Filters: 32Padding: sameDim (out): 32 x 32
ConvActivation: ReluFilter size: 5x5Filters: 32Padding: sameDim (out): 32 x 32
![Page 26: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/26.jpg)
Vanilla SRCNN (RELU)
Filter sizes: 9-1-5Max PSNR: 25.77
![Page 27: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/27.jpg)
Vanilla SRCNN (ELU)
Filter sizes: 9-1-5Max PSNR: 24.52
![Page 28: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/28.jpg)
Autoencoder
Source: https://goo.gl/R7itkL
![Page 29: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/29.jpg)
Autoencoder with skip connections
- Symmetric Skip connections Helps on recovering clean images Converges much faster and attains a higher-quality local optimum.
![Page 30: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/30.jpg)
AutoencoderMax PSNR: 28.4
![Page 31: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/31.jpg)
Other resultsTanh: Max PSNR: 24.68
Filter sizes: 9-3-5 max PSNR: 26.03
Todo: Denoising autoencoder (at the reconstruction phase)
Standard dataset using same configuration max PSNR: 30.49
![Page 32: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/32.jpg)
Todo: GANFrom Twitter
PSNR values worse than bicubic interpolation
Perceptual quality is the best by far
![Page 33: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/33.jpg)
Results for a sample cross-hatched texture
![Page 34: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/34.jpg)
Fibrous texture; 2x upscaling
![Page 35: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/35.jpg)
Interlaced texture; 2x upscaling
![Page 36: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/36.jpg)
Striped texture; 2x upscaling
![Page 37: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/37.jpg)
Porous texture; 2x upscaling
![Page 38: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/38.jpg)
TodoMore in-depth metrics on the performance against standard datasets
Evaluate performance on larger scaling factors, i.e. 4x, 8x etc.
Fix the GAN
(Maybe) Try a different down-sampling technique (some argue against bicubic preprocessing)
(Probably) Load a pre-trained model e.g. ResNet
![Page 39: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/39.jpg)
Conclusion and Future workStandard, shallow CNNs work alright for single image texture super resolution
Shallow autoencoders work better than CNNs
Quantify the performance for each texture class
Evaluate how effective these models are for classification
![Page 40: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/40.jpg)
ReferencesBoosting Optical Character Recognition: A Super-Resolution Approach - https://arxiv.org/abs/1506.02211
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network - https://arxiv.org/abs/1609.04802
Image Super-Resolution Using Deep Convolutional Networks - https://arxiv.org/abs/1501.00092
Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion - http://www.jmlr.org/papers/volume11/vincent10a/vincent10a.pdf
Super-Resolution via Deep Learning - https://arxiv.org/pdf/1706.09077.pdf
Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections - https://arxiv.org/pdf/1606.08921.pdf
![Page 41: Resolution of Textures via Single Image Super CNNsmli/palmer-super-resolution.pdfStandard, shallow CNNs work alright for single image texture super resolution Shallow autoencoders](https://reader034.vdocuments.mx/reader034/viewer/2022042213/5eb675a9e39ca4630f35be4e/html5/thumbnails/41.jpg)
Questions, suggestions, ideas?