slide 1 illinois - railroad engineering development of machine vision technology for inspection of...

Slide 1ILLINOIS - RAILROAD ENGINEERING

Development of Machine Vision Technology for Inspection of Railroad Track

John M. Hart*, J. Riley Edwards, Steven V. Sawadisavi, Esther Resendiz*

Christopher P. L. Barkan, Narendra Ahuja*Railroad Engineering Program and *Beckman Institute

University of Illinois at Urbana-Champaign

AAR Technology Scanning Meeting - 24 September 2009

Slide 2ILLINOIS - RAILROAD ENGINEERING

Presentation Outline

• Background

• Prioritization of Inspection Tasks

• Current Camera Views

• Initial Work on Turnout Inspection

• Data Collection

• Algorithm Development

• Lateral View Processing

• Over-the-Rail View Processing

• Future Work and Inspection Tasks

• Summary and Conclusions

Slide 3ILLINOIS - RAILROAD ENGINEERING

Background

• Inspection of track and track components is a critical, but labor intensive task resulting in large annual operating expenditures

• There are limitations in speed, quality, objectivity, and scope of the current inspection methods

• Machine vision provides a robust solution to facilitate more efficient and effective inspection of many track components

Slide 4ILLINOIS - RAILROAD ENGINEERING

Machine Vision System Outline

Image Acquisition

System

MachineVision

Algorithms

Railroad Track

Data AnalysisSystem

PertinentInformation

forRailroads

Slide 5ILLINOIS - RAILROAD ENGINEERING

Top Track Related Accident Causes from 2001-2005 (Class I Mains & Sidings)

Slide 6ILLINOIS - RAILROAD ENGINEERING

Current Component Inspection Priorities

• Rail anchors– Missing– Shifting

– Inappropriate patterns • Cut spikes

– Missing– Raised– Inappropriate patterns

• Turnouts– Missing bolts– Missing cotter pins– Damaged switch and

frog points

Slide 7ILLINOIS - RAILROAD ENGINEERING

Future Component Inspection Priority

• Future inspection tasks:

– Turnouts (continuation of current work)

– Crossing diamonds

– Crib ballast / curve breathing

• Other inspection possibilities (may require additional camera views):

– Measuring tie spacing

– Detecting insulated joint slippage

– Monitoring wayside rail lubricator performance

– Recording rail manufacturing markings

– Determining thermite weld integrity

– Monitoring track circuit bond wire condition

– Inspecting grade crossing surface condition

– Measuring flangeway width and depth

Slide 8ILLINOIS - RAILROAD ENGINEERING

Current Camera Views: Over-the-rail View

• Used for measuring the distance between the spike head and the base-of-rail and verifying spiking patterns

• Also used to measure crib ballast level with respect to the top of the tie

Slide 9ILLINOIS - RAILROAD ENGINEERING

Current Camera Views: Lateral View

• Optimal view for measuring the distance between the rail anchors and the edge of the ties and verifying anchor patterns

• Used to verify spike heights measured in the over-the-rail view

• Conducive to panorama generation

Slide 10ILLINOIS - RAILROAD ENGINEERING

Turnout Background - FRA Data Analysis

• According to the FRA (Federal Railroad Administration) Accident database, the top three causes of turnout-caused derailments are:

– Broken or Worn Switch points

– Broken Switch

– Switch point gap

• Switch points account for almost 13% of all rail accidents

• Frogs account for about 3%, in particular, worn or broken frog points

Slide 11ILLINOIS - RAILROAD ENGINEERING

Turnouts: Areas of Focus• Switch

– Distance of switch points from stock rail when open and closed

– Missing bolts and cotter pins

– Worn points

• Frog

– Worn or broken point

• Guardrail

– Gauge check clearance

Slide 12ILLINOIS - RAILROAD ENGINEERING

Field Data Collection

• Initially used handheld cameras to take images

– Obtained insights on later challenges, such as variability in component appearance

• Developed Video Track Cart for field data acquisition

– Used on low density track for verifying camera views and experimenting with lighting

• Optimizing lighting is a challenging task

– Reviewed lighting solutions from existing machine-vision systems

Slide 13ILLINOIS - RAILROAD ENGINEERING

Video Track Cart

• Equipped with two camera mounts for the over-the-rail view and a gauge-side lateral view

• Geared tripod heads for precise adjustment of camera views

• Deep-cycle marine battery supplies power for all electronics

• Rugged laptop resistant to environmental conditions

Slide 14ILLINOIS - RAILROAD ENGINEERING

Initial Field Video Data Collection• Visit to Advanced Transportation Research and Engineering Laboratory

(ATREL) in Rantoul, IL - Tested on track panel

• Monticello Railway Museum (MRM) visits

– Improved lateral view and over-the-rail view mounts

– Manually measured and catalogued track defects for algorithm verification

Slide 15ILLINOIS - RAILROAD ENGINEERING

NS Field Video Data Collection• Collected 20 videos on NS track west of Champaign, IL using both

camera views

• Components captured: Turnout, panelized concrete grade crossing, continuously welded rail (CWR), tangent and curved track, anchors, and cut spikes

• Captured videos under a variety of lighting conditions

Slide 16ILLINOIS - RAILROAD ENGINEERING

Lighting Considerations

• Reviewed lighting systems of similar machine-vision systems

– University of Central Florida’s track inspection system

• Lasers and timed strobe lights

• Sun shields

– ENSCO’s joint bar inspection system

• High-powered xenon lights

• No sun shields

– Georgetown Rail’s Aurora system

• Lasers with light filters on cameras

• Lighting solutions are unique to each system’s data collection requirements, with our system requiring even illumination over a large area of track

Slide 17ILLINOIS - RAILROAD ENGINEERING

Track Component Isolation

• Delineate ties, ballast, and rail using edge detection

• Incorporate texture classification to make edge detection robust

– Extract patches of texture from a hand-labeled training image

– Compare new texture patches against known textures

• Use Gabor filter outputs to classify patches from new video frames

Original image Edge image (base-of-rail highlighted)

Tie patches Ballast patches Rail patches

Slide 18ILLINOIS - RAILROAD ENGINEERING

Base-of-Rail Delineation

The original image, where we will isolate the base of the rail

Slide 19ILLINOIS - RAILROAD ENGINEERING

Base-of-Rail Delineation

Strong horizontal edges are detected (candidate edge is shown)

Slide 20ILLINOIS - RAILROAD ENGINEERING

Base-of-Rail Delineation

Surrounding pixels above and below candidate edge are examined

Slide 21ILLINOIS - RAILROAD ENGINEERING

Base-of-Rail Delineation

Ballast below edge

Ballast below edge

Base of the rail is confirmed

Slide 22ILLINOIS - RAILROAD ENGINEERING

Component Detection

• Detect tie plates using similar edge/texture method– Edge detection, verified with texture classification

• Hypothesize the spike locations (and missing spike locations) using prior knowledge of tie plate structure

– Spikes (and missing spikes)

are located with template

filters

– Spike height is measured

relative to the base of the rail

Original Image Delineated components

Spikes and missing spikes

Slide 23ILLINOIS - RAILROAD ENGINEERING

Component Detection: Anchors

• Detect anchors

– Use prior knowledge of anchor location (since tie plate and rail bottom have been delineated)

– Find parallel edges to achieve anchor detection

• Robust to changes in anchor orientation

Isolated anchors

Slide 24ILLINOIS - RAILROAD ENGINEERING

Defect Detection Results

• Individual components are located and measured

Tie with shifted anchor

Slide 25ILLINOIS - RAILROAD ENGINEERING

Defect Detection Results

• Individual components are located and measured

Tie with shifted anchor

Slide 26ILLINOIS - RAILROAD ENGINEERING

Defect Detection Results - Shifted Anchor

• Defect detected if anchor displacement with respect to the edge of the tie exceeds a threshold

Tie with shifted anchor

Slide 27ILLINOIS - RAILROAD ENGINEERING

Component Detection on Field Video

Field data video

Slide 28ILLINOIS - RAILROAD ENGINEERING

Panoramic Generation

Slide 29ILLINOIS - RAILROAD ENGINEERING

Track Panorama

• Visualization of cumulative track components using panoramic stitching

Tie / Tie plate delineation on test panorama

Component detection on test panorama

Slide 30ILLINOIS - RAILROAD ENGINEERING

Performance

• Lateral-view video from NS track (54 consecutive ties)

0

10

20

30

40

50

60

70

80

90

100

Percent (%)

Tie Detect Anchor Detect

Precision

Recall

Tie and anchor detection rates (precision and recall)

Slide 31ILLINOIS - RAILROAD ENGINEERING

Performance

0

10

20

30

40

50

60

70

80

Percent (%)

Tie Plate Holes Spikes

• Lateral-view video from NS track (54 consecutive ties)

Tie plate hole and spike localization success

Slide 32ILLINOIS - RAILROAD ENGINEERING

Over-the-rail Inspection - Original Frame

Original frame

• Locate track components from over-the-rail video

Slide 33ILLINOIS - RAILROAD ENGINEERING

Over-the-rail Inspection - Rail Location

• Locate rail by spatial self-similarity across consecutive frames

Locate rail

Slide 34ILLINOIS - RAILROAD ENGINEERING

Over-the-rail Inspection - Texture Classification

• Classify texture in frame as ballast/non-ballast

• Detect ties with tie filter

Locate ties with filter

Tie Filter

Tie Filter

Slide 35ILLINOIS - RAILROAD ENGINEERING

• Localize tie and tie plate using edge boundaries

• Ballast detection used as preliminary step

Over-the-rail Inspection - Tie and Plate Location

Delineate ties with gradients

Slide 36ILLINOIS - RAILROAD ENGINEERING

Over-the-rail Inspection - Spike Location

• Locate spike and spike hole with spatial template

Locate spike and tie plate hole

Slide 37ILLINOIS - RAILROAD ENGINEERING

Over-the-rail Inspection Results

Inspected frame

Slide 38ILLINOIS - RAILROAD ENGINEERING

Over-the-rail Inspection Video

Field data video

Slide 39ILLINOIS - RAILROAD ENGINEERING

System Implementation

• Future system likely to be mounted on either:

– Track geometry car

– Rail defect detector car

– High-rail vehicle

• Possible initial implementation on defect detector car

– Inspections occur with acceptable frequency

– Crews better trained with advanced inspection equipment

– Already equipped for massive data storage and upload

• The system will be initial adapted for experimental testing high-rail vehicle

Slide 40ILLINOIS - RAILROAD ENGINEERING

Future Work

• Plan for the remainder of 2009

– Continue testing spike and anchor algorithms on field-acquired video

– Begin lighting experimentation on Video Track Cart

– Complete study on methods to inspect for turnout defects

– Develop an approach for detecting turnout components

– Record additional videos at MRM and on other local tracks

• Approach for 2010

– Investigate cameras which can run at higher speeds and in a greater variety of environmental conditions

– Develop and test algorithms for turnout defects

– Begin adaptation of acquisition system for trials on a track vehicle

Slide 41ILLINOIS - RAILROAD ENGINEERING

Future Component Inspection Discussion

• Future inspection tasks:

– Turnouts (continuation of current work)

– Crossing diamonds

– Crib ballast / curve breathing

• Other inspection possibilities (may require additional camera views):

– Measuring tie spacing

– Detecting insulated joint slippage

– Monitoring wayside rail lubricator performance

– Recording rail manufacturing markings

– Determining thermite weld integrity

– Monitoring track circuit bond wire condition

– Inspecting grade crossing surface condition

– Measuring flangeway width and depth

Slide 42ILLINOIS - RAILROAD ENGINEERING

Acknowledgements

• Research Sponsors

– AAR Technology Scanning Program

– NEXTRANS Center

• Industry Assistance

– David Davis of TTCI

– Technology Scanning Committee

– CN

– UP

– NS

– BNSF

– Monticello Railway Museum

42

Slide 43ILLINOIS - RAILROAD ENGINEERING

UIUC Multidisciplinary Research Team

Computer Vision and Robotics Laboratory:

Narendra Ahuja, Professor, Beckman Institute

John M. Hart, Senior Research Engineer, Beckman Institute

Esther Resendiz, Graduate Student, Electrical and Computer Engineering

Railroad Engineering Program:

Chris Barkan, Associate Professor, Director of Railroad Engineering Program

Riley Edwards, Lecturer, Railroad Engineering Program

Luis Fernando Molina Camargo, Graduate Research Assistant

Matt Toussaint, Undergraduate Research Assistant

Randy Nolden, Undergraduate Research Assistant

Brennan Caughron, Undergraduate Research Assistant

Slide 44ILLINOIS - RAILROAD ENGINEERING

Appendix

Additional Turnout Data

Slide 45ILLINOIS - RAILROAD ENGINEERING

Turnouts: Switch Point Inspection

• Rail point closed tightly

• Chipped

• Gap on open rail point is wide enough for wheel flange

• Missing cotter pins

Slide 46ILLINOIS - RAILROAD ENGINEERING

Spring Frog Inspection

• Missing cotter pins

• Gauge

• Springs working

• Tightly fit

Slide 47ILLINOIS - RAILROAD ENGINEERING

Bound Manganese Frog (BMF) Inspection

• Gauge

• 6” of point missing

• worn down 5/8”

• Flangeway depth

• Worn tread point

6” max4” defect