slide 1 illinois - railroad engineering development of machine vision technology for inspection of...

ILLINOIS - RAILROAD ENGINEERING

Development of Machine Vision Technology for Inspection of Railroad Track

John M. Hart*, J. Riley Edwards, Steven V. Sawadisavi, Esther Resendiz*

Christopher P. L. Barkan, Narendra Ahuja*Railroad Engineering Program and *Beckman Institute

University of Illinois at Urbana-Champaign

AAR Technology Scanning Meeting - 24 September 2009


Presentation Outline

• Background

• Prioritization of Inspection Tasks

• Current Camera Views

• Initial Work on Turnout Inspection

• Data Collection

• Algorithm Development

• Lateral View Processing

• Over-the-Rail View Processing

• Future Work and Inspection Tasks

• Summary and Conclusions


Background

• Inspection of track and track components is a critical, but labor intensive task resulting in large annual operating expenditures

• There are limitations in speed, quality, objectivity, and scope of the current inspection methods

• Machine vision provides a robust solution to facilitate more efficient and effective inspection of many track components


Machine Vision System Outline

Image Acquisition

System

MachineVision

Algorithms

Railroad Track

Data AnalysisSystem

PertinentInformation

forRailroads


Top Track Related Accident Causes from 2001-2005 (Class I Mains & Sidings)


Current Component Inspection Priorities

• Rail anchors– Missing– Shifting

– Inappropriate patterns • Cut spikes

– Missing– Raised– Inappropriate patterns

• Turnouts– Missing bolts– Missing cotter pins– Damaged switch and

frog points


Future Component Inspection Priority

• Future inspection tasks:

– Turnouts (continuation of current work)

– Crossing diamonds

– Crib ballast / curve breathing

• Other inspection possibilities (may require additional camera views):

– Measuring tie spacing

– Detecting insulated joint slippage

– Monitoring wayside rail lubricator performance

– Recording rail manufacturing markings

– Determining thermite weld integrity

– Monitoring track circuit bond wire condition

– Inspecting grade crossing surface condition

– Measuring flangeway width and depth


Current Camera Views: Over-the-rail View

• Used for measuring the distance between the spike head and the base-of-rail and verifying spiking patterns

• Also used to measure crib ballast level with respect to the top of the tie


Current Camera Views: Lateral View

• Optimal view for measuring the distance between the rail anchors and the edge of the ties and verifying anchor patterns

• Used to verify spike heights measured in the over-the-rail view

• Conducive to panorama generation


Turnout Background - FRA Data Analysis

• According to the FRA (Federal Railroad Administration) Accident database, the top three causes of turnout-caused derailments are:

– Broken or Worn Switch points

– Broken Switch

– Switch point gap

• Switch points account for almost 13% of all rail accidents

• Frogs account for about 3%, in particular, worn or broken frog points


Turnouts: Areas of Focus• Switch

– Distance of switch points from stock rail when open and closed

– Missing bolts and cotter pins

– Worn points

• Frog

– Worn or broken point

• Guardrail

– Gauge check clearance


Field Data Collection

• Initially used handheld cameras to take images

– Obtained insights on later challenges, such as variability in component appearance

• Developed Video Track Cart for field data acquisition

– Used on low density track for verifying camera views and experimenting with lighting

• Optimizing lighting is a challenging task

– Reviewed lighting solutions from existing machine-vision systems


Video Track Cart

• Equipped with two camera mounts for the over-the-rail view and a gauge-side lateral view

• Geared tripod heads for precise adjustment of camera views

• Deep-cycle marine battery supplies power for all electronics

• Rugged laptop resistant to environmental conditions


Initial Field Video Data Collection• Visit to Advanced Transportation Research and Engineering Laboratory

(ATREL) in Rantoul, IL - Tested on track panel

• Monticello Railway Museum (MRM) visits

– Improved lateral view and over-the-rail view mounts

– Manually measured and catalogued track defects for algorithm verification


NS Field Video Data Collection• Collected 20 videos on NS track west of Champaign, IL using both

camera views

• Components captured: Turnout, panelized concrete grade crossing, continuously welded rail (CWR), tangent and curved track, anchors, and cut spikes

• Captured videos under a variety of lighting conditions


Lighting Considerations

• Reviewed lighting systems of similar machine-vision systems

– University of Central Florida’s track inspection system

• Lasers and timed strobe lights

• Sun shields

– ENSCO’s joint bar inspection system

• High-powered xenon lights

• No sun shields

– Georgetown Rail’s Aurora system

• Lasers with light filters on cameras

• Lighting solutions are unique to each system’s data collection requirements, with our system requiring even illumination over a large area of track


Track Component Isolation

• Delineate ties, ballast, and rail using edge detection

• Incorporate texture classification to make edge detection robust

– Extract patches of texture from a hand-labeled training image

– Compare new texture patches against known textures

• Use Gabor filter outputs to classify patches from new video frames

Original image Edge image (base-of-rail highlighted)

Tie patches Ballast patches Rail patches


Base-of-Rail Delineation

The original image, where we will isolate the base of the rail



Strong horizontal edges are detected (candidate edge is shown)



Surrounding pixels above and below candidate edge are examined



Ballast below edge

Ballast below edge

Base of the rail is confirmed


Component Detection

• Detect tie plates using similar edge/texture method– Edge detection, verified with texture classification

• Hypothesize the spike locations (and missing spike locations) using prior knowledge of tie plate structure

– Spikes (and missing spikes)

are located with template

filters

– Spike height is measured

relative to the base of the rail

Original Image Delineated components

Spikes and missing spikes


Component Detection: Anchors

• Detect anchors

– Use prior knowledge of anchor location (since tie plate and rail bottom have been delineated)

– Find parallel edges to achieve anchor detection

• Robust to changes in anchor orientation

Isolated anchors


Defect Detection Results

• Individual components are located and measured

Tie with shifted anchor


Defect Detection Results - Shifted Anchor

• Defect detected if anchor displacement with respect to the edge of the tie exceeds a threshold

Tie with shifted anchor


Component Detection on Field Video

Field data video


Panoramic Generation


Track Panorama

• Visualization of cumulative track components using panoramic stitching

Tie / Tie plate delineation on test panorama

Component detection on test panorama


Performance

• Lateral-view video from NS track (54 consecutive ties)

0

10

20

30

40

50

60

70

80

90

100

Percent (%)

Tie Detect Anchor Detect

Precision

Recall

Tie and anchor detection rates (precision and recall)


Performance

0

10

20

30

40

50

60

70

80

Percent (%)

Tie Plate Holes Spikes

• Lateral-view video from NS track (54 consecutive ties)

Tie plate hole and spike localization success


Over-the-rail Inspection - Original Frame

Original frame

• Locate track components from over-the-rail video


Over-the-rail Inspection - Rail Location

• Locate rail by spatial self-similarity across consecutive frames

Locate rail


Over-the-rail Inspection - Texture Classification

• Classify texture in frame as ballast/non-ballast

• Detect ties with tie filter

Locate ties with filter

Tie Filter

Tie Filter


• Localize tie and tie plate using edge boundaries

• Ballast detection used as preliminary step

Over-the-rail Inspection - Tie and Plate Location

Delineate ties with gradients


Over-the-rail Inspection - Spike Location

• Locate spike and spike hole with spatial template

Locate spike and tie plate hole


Over-the-rail Inspection Results

Inspected frame


Over-the-rail Inspection Video

Field data video


System Implementation

• Future system likely to be mounted on either:

– Track geometry car

– Rail defect detector car

– High-rail vehicle

• Possible initial implementation on defect detector car

– Inspections occur with acceptable frequency

– Crews better trained with advanced inspection equipment

– Already equipped for massive data storage and upload

• The system will be initial adapted for experimental testing high-rail vehicle


Future Work

• Plan for the remainder of 2009

– Continue testing spike and anchor algorithms on field-acquired video

– Begin lighting experimentation on Video Track Cart

– Complete study on methods to inspect for turnout defects

– Develop an approach for detecting turnout components

– Record additional videos at MRM and on other local tracks

• Approach for 2010

– Investigate cameras which can run at higher speeds and in a greater variety of environmental conditions

– Develop and test algorithms for turnout defects

– Begin adaptation of acquisition system for trials on a track vehicle


Future Component Inspection Discussion

• Future inspection tasks:

– Turnouts (continuation of current work)

– Crossing diamonds

– Crib ballast / curve breathing

• Other inspection possibilities (may require additional camera views):

– Measuring tie spacing

– Detecting insulated joint slippage

– Monitoring wayside rail lubricator performance

– Recording rail manufacturing markings

– Determining thermite weld integrity

– Monitoring track circuit bond wire condition

– Inspecting grade crossing surface condition

– Measuring flangeway width and depth


Acknowledgements

• Research Sponsors

– AAR Technology Scanning Program

– NEXTRANS Center

• Industry Assistance

– David Davis of TTCI

– Technology Scanning Committee

– CN

– UP

– NS

– BNSF

– Monticello Railway Museum

42


UIUC Multidisciplinary Research Team

Computer Vision and Robotics Laboratory:

Narendra Ahuja, Professor, Beckman Institute

John M. Hart, Senior Research Engineer, Beckman Institute

Esther Resendiz, Graduate Student, Electrical and Computer Engineering

Railroad Engineering Program:

Chris Barkan, Associate Professor, Director of Railroad Engineering Program

Riley Edwards, Lecturer, Railroad Engineering Program

Luis Fernando Molina Camargo, Graduate Research Assistant

Matt Toussaint, Undergraduate Research Assistant

Randy Nolden, Undergraduate Research Assistant

Brennan Caughron, Undergraduate Research Assistant


Appendix

Additional Turnout Data


Turnouts: Switch Point Inspection

• Rail point closed tightly

• Chipped

• Gap on open rail point is wide enough for wheel flange

• Missing cotter pins


Spring Frog Inspection

• Missing cotter pins

• Gauge

• Springs working

• Tightly fit


Bound Manganese Frog (BMF) Inspection

• Gauge

• 6” of point missing

• worn down 5/8”

• Flangeway depth

• Worn tread point

6” max4” defect

slide 1 illinois - railroad engineering development of machine vision technology for inspection of...

Documents