automatic detection of rail surface defects using … faculteit...microsoft powerpoint - 2019 05 10...
TRANSCRIPT
![Page 1: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/1.jpg)
1
Lunch lecture series of Monitoring Community10th May 2019
Automatic detection of rail surface defects using
video image: A case study in the Dutch railways
Dr. Alfredo Núñez VicencioAssistant Professor, TU Delft
Section of Railway Engineering
![Page 2: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/2.jpg)
2
1) Video Image based Monitoring of Rail
2) Deep Neural Networks for Rail Monitoring
3) Making use of the data: Risk analysis
4) Conclusions
Outline
![Page 3: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/3.jpg)
3
1) Video Image based Monitoring of Rail
2) Deep Neural Networks for Rail Monitoring
3) Making use of the data: Risk analysis
4) Conclusions
Outline
![Page 4: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/4.jpg)
4
Railway Infrastructure
![Page 5: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/5.jpg)
5
Railway Infrastructure
Risk
RiskRisk
Risk
Risk
Risk
Risk
Risk
Risk
Risk
Risk
Risk
![Page 6: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/6.jpg)
6
Defects in rails
Light Moderate Severe Two squats at a thermite weld
Initiating squat
Severe
Squats
CorrugationInsulated joint with
plastic surface degradation
Wheel burns
Damaged welds
…..
![Page 7: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/7.jpg)
7
In The Netherlands (about 7000 km of tracks)
Almost no time for monitoring and maintenance
![Page 8: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/8.jpg)
8
Video
![Page 9: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/9.jpg)
9
What do we need?
![Page 10: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/10.jpg)
10
What do we need?A method that can tell us whether the image is:
Healthy rail
Insulated rail joint
Rail surface defect
![Page 11: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/11.jpg)
11
What do we need?A method that can tell us whether the image is:
Healthy rail
Insulated rail joint
Rail surface defect
![Page 12: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/12.jpg)
12
1) Video Image based Monitoring of Rail
2) Deep Neural Networks for Rail Monitoring
3) Making use of the data: Risk analysis
4) Conclusions
Outline
![Page 13: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/13.jpg)
13
Images: dimensions
How many possible images do we have?
Grey scale (265 levels)
Dimensions266 x 224
Dimensions12 x 10
Answer: (12 x 10) 265
Grey scale
About 1.8 x 10532
possible images
![Page 14: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/14.jpg)
14
Images: dimensions
How many possible images do we have?
Grey scale (265 levels)
Dimensions266 x 224
Dimensions12 x 10
Answer: (12 x 10) 265
Grey scale
About 1.8 x 10532
possible images
The estimated number of atoms in the observable universe (1080)
M87's black hole is 53.5 million light-years away
(5.014 × 1026 millimeters)
But don’t worry, the Interesting features of the
image lies in a lower dimension
![Page 15: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/15.jpg)
15
Features hierarchy
Image Contour Edge Shapes
Smaller Dimensions
![Page 16: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/16.jpg)
16
Feat
ure
1
Feature 2
-
--
-
-
-
-
+
+
+
+
+
++
+
Classification methods
![Page 17: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/17.jpg)
17
Feat
ure
1
Feature 2
-
--
-
-
-
-
+
+
+
+
+
++
+
Classification methods
![Page 18: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/18.jpg)
18
Feat
ure
1
Feature 2
-
--
-
-
-
-
+
+
+
+
+
++
+
0 0x 0 0x
Decision boundary
Classification methods
![Page 19: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/19.jpg)
19
Feat
ure
1
Feature 2
-
--
-
-
-
-
+
+
+
+
+
++
+
0 0x 0 0x
Decision boundary
Classification methods
?
![Page 20: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/20.jpg)
20
Feat
ure
1
Feature 2
-
--
-
-
-
-
+
+
+
+
+
++
+
0 0x 0 0x
Decision boundary
Classification methods
-
![Page 21: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/21.jpg)
21
Feat
ure
1
Feature 2
-
--
-
-
-
-
+
+
+
+
+
++
+
0 0x 0 0x
Decision boundary
Classification methods
-
![Page 22: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/22.jpg)
22
Feat
ure
1
Feature 2
-
--
-
-
-
-
+
+
+
+
+
++
+
0 0x 0 0x
Decision boundary Fe
atur
e 1
Feature 2
-
--
-
-
-
+
+
+
+
+
++
+ +
+
--
-
-
Classification methods
-
![Page 23: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/23.jpg)
23
Feat
ure
1
Feature 2
-
--
-
-
-
-
+
+
+
+
+
++
+
0 0x 0 0x
Decision boundary Fe
atur
e 1
Feature 2
-
--
-
-
-
+
+
+
+
+
++
+ +
+
--
-
-
Decision boundary
, 0f x , 0f x
Classification methods
-
![Page 24: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/24.jpg)
24
Possible methods
Source: http://yann.lecun.com/exdb/mnist/2015
0
2
4
6
8
10
12
14
LinearClassifiers
K-NearestNeighbors
BoostedStumps
SVMs Neural nets Convolutionalnets
test
err
or (
%)
MNIST 28x28 images, 60,000 train, 10,000 test
![Page 25: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/25.jpg)
25
Deep learning
Classic
Input
Output
Mapping (Hand designed)
ClassicMachine Learning
Input
Output
Mapping (Hand designed)
Deep Learning
Input
Output
Learned mapping
Tim de Bruin and Robert Babuska, “Artificial Neural Networks 1”. Course: Knowledge-Based Control Systems (SC42050), TUDelft.
Learned mapping
Learned features
Learned features
……
![Page 26: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/26.jpg)
26
Deep learning: learned features
https://ai.googleblog.com/2017/11/feature-visualization.html
![Page 27: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/27.jpg)
27
Deep learning: learned mapping
R. Babuska, Knowledge based Control Systems, Lecture notes.
McCulloch and Pitts modelled the behavior of a single neuron in
1943. They called this mathematical model a
Perceptron.
![Page 28: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/28.jpg)
28
1x
2x
nx
1w
2w
nw
z z y...
.
.
.
Artificial neuron
Deep learning: learned mapping
![Page 29: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/29.jpg)
29
1x
2x
nx
1w
2w
nw
z z y...
.
.
.
Inputs of the neuron
Adaptive weights (synaptic strength)
Activation function
For Perceptron
z
Output of the neuron
Deep learning: learned mapping
![Page 30: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/30.jpg)
30
Deep learning: learned mapping
R. Babuska, Knowledge based Control Systems, Lecture notes.
Paul Werbos, the father of backpropagation, since 1974 we
can train neural networks
![Page 31: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/31.jpg)
31S. Faghih-Roohi, S. Hajizadeh, A. Núñez, R. Babuska, and B. De Schutter, “A deep learning approach for detection of rail defects”. Proceedings of the IEEE World Congress on Computational Intelligence, IEEE WCCI 2016, 2016 International Joint Conference on
Neural Networks (IJCNN), Vancouver, Canada, 25-29 July, 2016, pp. 2584-2589.
Detection of rail defects
![Page 32: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/32.jpg)
32
Image data• The dataset consists of 4220 samples, of which 3170 are
normal, and roughly 1000 are defects.
• We train a convolutional neural network model with 80% of the data, and test with the remaining 20% (in 5 folds). Here is the averaged result of the test:
Accuracy = 0.9870
Predicted normal Predicted defect
Normal samples 635 1
Defects 10 197
32
![Page 33: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/33.jpg)
33
False detections (image data)
Defects not detected
Images from INSPECTATION
![Page 34: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/34.jpg)
34
Hits 1 (image data)True Positive
TU DELFT
Images from INSPECTATION
![Page 35: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/35.jpg)
35
Hits 2 (image data)
TU DELFT
Images from INSPECTATION
True Positive
![Page 36: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/36.jpg)
36
Classification of types• We also tried to classify the defects into 2 categories of
spots/light vs. medium/severe.
TU DELFT
![Page 37: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/37.jpg)
37
1) Video Image based Monitoring of Rail
2) Deep Neural Networks for Rail Monitoring
3) Making use of the data: Risk analysis
4) Conclusions
Outline
![Page 38: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/38.jpg)
38A. Jamshidi, S. Faghih-Roohi, S. Hajizadeh, A. Núñez, R. Babuška, R. Dollevoet, Z. Li and B. De Schutter, “A big data analysis
approach for rail failure risk assessment”. Risk Analysis, Volume 37, Issue 8, August 2017, Pages: 1495-1507.
A big data analysis approach is used to automatically detect squats from rail
images.
A Bayesian model is employed to estimate the failure probability.
![Page 39: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/39.jpg)
39
Architecture of the proposed DCNN model
A. Jamshidi, S. Faghih-Roohi, S. Hajizadeh, A. Núñez, R. Babuška, R. Dollevoet, Z. Li and B. De Schutter, “A big data analysis approach for rail failure risk assessment”. Risk Analysis, Volume 37, Issue 8, August 2017, Pages: 1495-1507.
![Page 40: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/40.jpg)
40
1) Video Image based Monitoring of Rail
2) Deep Neural Networks for Rail Monitoring
3) Making use of the data: Risk analysis
4) Conclusions
Outline
![Page 41: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/41.jpg)
41
Conclusions
• “Fancy” algorithms will not perform 100% if the knowledge of the railway system is not included explicitly.
• Purely data-based methods do not guarantee physical meaning. A combined approach, data-based with physical modelling would be preferred.
• There is a great potential for using Deep Learning to facilitate maintenance decisions on Dutch railways. Further research: head-checks, corrugation, wheel-burns, indentations.
![Page 42: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/42.jpg)
42
Conclusions
• Self-learning, transfer learning and new architectures could be tested.
• Higher resolutions cameras, including 3D measurements, can allow a complete digitalization of the railways assets.
• Many open challenges: Fusion of data, velocity, etc.
![Page 43: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/43.jpg)
43
A. Jamshidi, S. Hajizadeh, Z. Su, M. Naeimi, A. Núñez, R. Dollevoet, B. De Schutter and Z. Li, “A decisionsupport approach for condition-based maintenance of rails based on big data analysis”. Transportation
Research Part C: Emerging Technologies, Volume 95, October 2018, Pages: 185-206.
Video and ABA to detect squats.
Other signals for influential factors used for modelling.
How to keep improving?
![Page 44: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/44.jpg)
44
Deep learning: learned features
Just for fun:
https://affinelayer.com/pixsrv/index.html
https://playground.tensorflow.org/
![Page 45: Automatic detection of rail surface defects using … faculteit...Microsoft PowerPoint - 2019 05 10 ANunez CiTG-ES Monitoring Community Author anunezvicencio Created Date 5/10/2019](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e290155280b706304e8d2/html5/thumbnails/45.jpg)
45
Lunch lecture series of Monitoring Community10th May 2019
Automatic detection of rail surface defects using
video image: A case study in the Dutch railways
Dr. Alfredo Núñez VicencioAssistant Professor, TU Delft
Section of Railway Engineering