Andrew Liu, Ryan Shen

Deep Learning Solution Architect

DEFECT INSPECTION FROM SCRATCH TO PRODUCTION

2

AGENDA

Defect Inspection and its challenges

NGC Docker images

Model set up - Unet

Data preparation - DAGM

Production - Speed up with TensorRT

3

INDUSTRIAL DEFECT INSPECTION

4

INDUSTRIAL DEFECT INSPECTION

5

NGC DOCKER IMAGES

Benefits for Deep Learning WorkflowHigh Level Benefits and Feature Set

Single software stack

Develop once, deploy anywhere

Scale across teams of practitioners

Developer, DevOp, QC

7

Defect inspection Workflowfrom scratch to production within container

Training Inference

NGC optimized docker image TENSORRT GRE

docker pull

nvcr.io/nvidia/tensorflow

:18.02-py3

docker pull

nvcr.io/nvidia/tensorrt:

18.02-py2

8

MODEL SET UP

9

WAYS FOR DEFECT INSPECTION

Depends on domain adaptation

Image SegmentationObject Detection

Image Classification

Label: 0 / 1 (Defect / Non Defect)

Label: Bounding-Box Label: Polygons Mask

10

CNN STRUCTURELeNet

Source: Design of Deep Convolutional Neural Network Architectures for Automated Feature Extraction in Industrial Inspection, D. Weimer et al, 2016

11

“defect

class1”

1000-dim vector

< 1 millisecond

convnets perform classification

end-to-end learning

https://arxiv.org/abs/1605.06211

12

~1/10 second

end-to-end learning

???

lots of pixels, little time?

https://arxiv.org/abs/1605.06211

51

22

51

22

51

22

1 16 16

16 32 32

32 64 64

64 128 128 256

128 256 256

25

62

25

62

25

62

12

82

12

82

12

82

64

2

64

2

64

2

32

2

32

2

32

2

12

82

12

82

12

82

128 64 64

128 128

64

2

64

2

64

2

64 32 32

25

62

25

62

25

62

51

22

51

22

51

22

32 16 16

3X3 Conv2d+ReLU

2X2 MaxPool

2X2 Conv2dTranspose

copy and concatenate

U-Net structure

14

KERAS IMPLEMENTATIONConvolution

15

DECODINGdeconvolution

16

Image segmentation on medical imagesSame process among various use cases

MRI image

Left ventricle

heart disease

Data Science BOWL2016

Data Science BOWL 2017

CT image

Nodule

Lung cancer

Image

Nuclei

Drug discovery

Data Science BOWL2018

17

DATA PREPARATION

18

INDUSTRIAL OPTICAL INSPECTIONGerman Association for Pattern Recognition

• http://resources.mpi-inf.mpg.de/conferences/dagm/2007/prizes.html

19

INDUSTRIAL OPTICAL INSPECTIONGerman Association for Pattern Recognition

Pass NG Pass

NGNGPass Pass

NG

20

DATA DETAILS

• Original images are 512 x 512 grayscale format

• Output is a tensor of size 512 x 512 x 1

• Each pixel belongs to one of two classes

• Training set consist of 100 images

• Validation set consist of 50 images

21

MORE EXAMPLES

22

IMBALANCE DATA

23

Dice Metric (IOU)• Metric to compare the similarity of two samples:

2𝐴𝑛𝑙________________________________

𝐴𝑛 + 𝐴𝑙

• Where: • An is the area of the contour predicted by the network • Al is the area of the contour from the label• Anl is the intersection of the two

• The area of the contour that is predicted correctly by the network

• 1.0 means perfect score.

• More accurately compute how well we’re predicting the contour against the label

• We can just count pixels to give us the respective areas

24

LOSS FUNCTION WITH KERAS

24

25

LEARNING CURVES

25

LOSS

26

APPLICATION: INDUSTRIAL INSPECTIONNVIDIA

• 1000 defect-free, 150 defect images

• Challenges: Not all deviations from the

texture are necessarily defects.

27

PRODUCTION

28

FINAL DECISIONPlus Human Logic

Size, Position, … etc

29

INFERENCE PIPELINE

Post-processing and decision makingInference

Camera

Machine

P40

DGX Server

TensorRT+GRE

Fetch image

TensorRT+GRE

Detectors/Classifiers/Segment Composite Result Metadata

DataCenter

Edge

Defect Pattern Ratio Defect Level Defect region size Defect counts

…

Domain Criteria Determine threshold

Precision/ Recall

30

Development flow

Called UFF, Universal Framework FormatONNX

31

Defect inspection Workflow

from scratch to production within container

Training Inference

NGC optimized docker image TENSORRT GRE

docker pull

nvcr.io/nvidia/tensorflow

:18.02-py3

docker pull

nvcr.io/nvidia/tensorrt:

18.02-py3

32

SUMMARY

Challenges Delivers

Training , inference environment is hard to build, maintain, share. Using NGC Docker images.

Model optimizations and speed up throughput. TensorRT + GRE SDK

So many deep learning model out there, how to choose the right

model?

If your dataset, demand requirement fit the scenario

like we do. U-Net model is great choice for

segmentation task.

33

Thank You