video based system for railroad collision warning
Post on 18-Mar-2023
0 Views
Preview:
TRANSCRIPT
Video Based System for Railroad Collision Warning
Jonny A. Uribe1, Luis Fonseca
2, J. F. Vargas
3
1GITA – Grupo de Investigación en Comunicaciones Aplicadas
2GEPAR – Grupo Electrónica de Potencia, Automatización y Robótica
3Grupo de Investigación Microelectrónica y Control
Departamento de Ingeniería Electrónica
Universidad de Antioquia,
Medellín - Colombia
{jauribe, luis.fonseca, jfvargas}@udea.edu.co
Abstract— Autonomous systems can assist humans in the
important task of safe driving. Such systems can warn people about
possible risks, take actions to avoid accidents or guide the vehicle
without human supervision. In railway scenarios a camera in front
of the train can aid drivers with the identification of obstacles or
strange objects that can pose danger to the route. Image processing
in these applications is not easy of performing. The changing
conditions create scenes where background is hard to detect,
lighting varies and process speed must be fast. This article describes
a first approximation to the solution of the problem where two
complementary approaches are followed for detecting and tracking
obstacles on videos captured from a train driver perspective. The
first strategy is a simple-frame-based approach where every video
frame is analyzed using the Hough transform for detecting the
rails. On every rail a systematic search is done detecting obstacles
that can be dangerous for the train course. The second approach
uses consecutive frames for detecting the trajectory of moving
objects. Analyzing the sparse optical flow the candidate objects are
tracked and their trajectories computed in order to determine their
possible route to collision. For testing the system we have used
videos where preselected fixed and moving obstacles have been
superimposed using the Chroma key effect. The system had shown
a real time performance in detecting and tracking the objects.
Future work includes the test of the system on real scenarios and
the validation over changing weather conditions.
Index Terms—Obstacle detection, Object tracking, Autonomous
train driving, Digital image processing, Hough transform, Optical
flow
I. INTRODUCTION
Safe autonomous driving is a huge technical challenge which still lacks general solution. Researches had explored different artificial vision methods for implementing automatic driving algorithms but many issues like changing lighting, changing background, process speed, etc., turns this problem into a complex task. Autonomous train driving is a particular case where the previous problems are present too but where the specific characteristics suggest the use of different
methodologies for the design and implementation of vision algorithms.
We create a first approximation to the solution of detecting both fixed and moving objects in a railway system. Our intention was to explore different techniques for creating a system capable of warning a driver about possible threats in the route. We employed two complementary approaches for detecting and tracking the objects. The first strategy detects the rails and scans the area near them in a bottom-up way searching for possible obstacles. This method was effective in detecting fixed objects in front the train and obstacles just near the rails. The second approach is based on the optical flow between frames. Discarding background moving elements our algorithm finds candidate dangerous objects, tracks their trajectories and foresees their paths for determining if exist a course to collision. This method has the advantages of warn with anticipation when an object could pose danger to the safety of the train.
Our work is a first local approximation to the creation of an automatic train driving system and a continuation of the work described in [1]. Nowadays metropolitan train systems has been forced to find new strategies for keeping high user experience quality levels and comply with requirements for increasing the number of people using the transportation. The high rate of passing trains puts the driver in a delicate situation where he must keep a continuous high level of attention on the railway scenes. In these environments is quite easy to lose concentration and the possibilities of making a human mistake are increased. To provide the train driver an automatic aid system could alleviate the effort required and contribute with the safety of the transportation system.
The article structure is as follows: section II describes some related work in the field of artificial vision and automatic driver assistance in the railway domain. Section III shows the proposed system and implementation details. Experiments and results are described in section IV. Conclusions are in Section V and the future work is detailed in section VI.
II. RELATED WORK
Primarily two strategies guide the design and implementation of autonomous train driving systems: the active and passive approaches. In active systems emitting devices and associated sensors are placed in train's head for scanning the near area. For example some systems use infrared lasers that emit a beam pattern and its reflection is then analyzed in order to detect obstacles or judge safety route. Works that follow this approach includes [2][3]. The main disadvantages of these kinds of systems are their difficult to identify obstacles boundaries in a reliable way, their short range of action and their low accuracy on curve zones. In contrast passive systems use video cameras in front of the train and rely on image processing algorithms for obstacle detection. A complementary approach consist in mixing active and passive system and integrating the multiple sources of information (see for example [4][5][6]). Our work will be focused in passive system where the only source of data is the video obtained from a single camera attached in the front of the train.
Related to this approach the work proposed by Ukai [7] used the Hough transform for tracking the rails and compute the vanishing point using a moving camera. The camera is dynamically adjusted in such way that this point is always near the center of the image. For obstacle detecting the optical flow is employed with several algorithms for mitigating the adverse effect of the moving background. In order to detect fixed objects any rail interruption on the images is labeled as a possible obstacle. A low luminosity lens in the camera and anti-blurring techniques complement the system.
In the work described in [8] the authors use clothoids segments for the rail detection. In each segment several algorithms are used for detecting obstacles, these included: discontinuities in the lines (rails), gray scale variability between adjacent segments, optical flow between video frames and statistic of textures in the segments.
In [9] dynamic programming is used in order to extract the rails and the area between them but the obstacle detection problem is not considered. Calculating the image gradient using the Sobel operator the near driver segments of the rails are extracted. After that the Hough transform is employed for estimating the rails vanishing point and delimiting the area enclosed by the rails. The remaining segments in the upper part of the image are treated in a recursive way.
In [10] a projective transform is used for creating a new image where the rails become parallel or with a constant separation between them like in a birds view perspective. The Hough transform is employed on this image for tracking and detecting the railway. If the system fails to track the rails at any point the possibility of it being cover with an obstacle arise. In this system an intensive previous configuration is required for the projective transform. In the configuration phase manual operator must indicate through a complete train route where the rails are, which is the area between them and estimates near and far distances.
In [11] the rail detection is considered but not the obstacle identification. Using the inverse mapping transform [12] a bird eye view is compute. With this transformation the distance between rails remains constant from the bottom to the upper part of images altering the train driver perspective. Using Gaussian filters blurring is mitigated and every frame is
segmented in ten consecutive parts. On every segment the rails detection task is initiated using a polynomial model. This model computes the distance between pair of lines and selects the most likely candidates for representing the rails. Information from every segment is then mixed and the central imaginary line between rails is obtained. Using this information the area of interest is identified but the obstacle identification is relegated as future work.
All the previous works had problems identifying the rails in changing environmental conditions, scenes with low luminosity and in pronounced curve zones. The obstacle detection in front of the train is such hard problem that many author prefer not confront it, limiting their approaches to the rail and near area identification.
Our work deals with the railway identification through the Hough transform and initiate the obstacle finding using artificial vision algorithms. Autonomous train driving is still an open research area with not given general solution. Our work is a first exploration in the topic where we initiate the design and implementation of signal processing methods in order to extract the rails and the area of interest near them, to detect obstacles in front of the train and to identify moving objects with route to collision. To our knowledge this is the first initiative in our country in this important and relevant area of research.
In the next section we will describe the proposed system its processing phases and the resources used for adjusting and testing the system.
III. METHODOLOGY
A. Motivation
Fig. 1 shows an image where the rails are capture from the train driver perspective.
We can say that the rails look like lines projecting from the bottom of the image to the horizon and they seem to converge in a far point known as the vanishing point. In the near to driver zone the rails appearance is quite monotonous in all the route while in the horizon do exist greater variability mainly in curve sections. The presence of the rails can be used as a first input for computer vision algorithms.
Fig. 1: Typical railway appearance from the driver perspective
One strategy for detecting the rails consists in extracting the lines in the image and selecting the most likely candidates to rails using criteria such as length, angle and position in the image. After the rail extraction the area near the railway and between the rails become the zone of interest for detecting obstacles to the route.
Nevertheless this strategy is insufficient to deal with moving objects that are traveling in a path to collision but are still far from the rail area. To detect these kinds of objects before they reach the rails or hit the train is highly important in order to take actions as soon as possible.
But accomplish this early detection impose huge challenges to artificial vision algorithms, mainly because from the train perspective, all the background is moving too and to discriminate dangerous objects from mere background elements who are in a perspective normal displacement, requires a fine tuning that can fail when faced to changing environments.
For detecting moving objects we used the optical flow algorithm proposed by Lucas-Kanade [13] and we created a set of rules for discriminating the features belonging to the moving background from characteristics owned by the passing objects. The next section describes our system and its complementary approaches for dealing with fixed obstacles over the rails and moving objects in route to collision.
B. System Diagram
As said we followed two approaches in order to detect obstacles posing danger to the train traveling. The first strategy deals with obstacles that are on the area near the rails. Fig. 2 shows a general diagram of this part of the system. The input to the systems is a video recorded from the train driver perspective but here we process the video in a frame to frame basis. The first task is a preprocessing stage where a filter for improving contours is applied. Then a gray scale conversion and dimensioning reduction phase take place.
The Hough transform was the mechanism used to find the lines representing the rails. We found that using a smaller image could improve the process speed without affecting precision considerably. So the rail detection task resizes the image and applies a rectangular mask for eliminating irrelevant background objects. After a Canny edge process and a morphological closing the Hough algorithm find candidate lines. Applying several criteria like length and position the lines that more likely represent the rails are selected.
In relation with the obstacle detection stage the first task is to use the Canny algorithm. A morphological closing follows and then the contours in the image are identified. Small and disconnected objects are then eliminated. After filling the contours we start a systematic search using the rails as guide.
An independent tracking in each rail is conducted by using a small dynamic area that moves from the bottom of the rail to the upper part. In each analysis step the area and centroid are computed. When the metrics are far from the expected values the selected area grows and modified its centroid with the intention of enclosing the strange object (See Fig. 3).
In order to detect moving objects that could represent a future danger to the train route but which are not near the railway zone yet, we use an strategy based on the optical flow. Fig. 4 shows an scheme of the steps followed for detecting and
tracking these kind of obstacles. After preprocessing every image the main characteristics are extracted using the algorithm described in [14].
Fig. 3: Rails tracking for finding obstacles in front the train
These characteristics represents zones in the image with more chances of be found in consecutive frames. Often zones where the corners of objects are clear allow for tracking their displacement and rotation in different frames. Appliyng the
Preprocessing
Border
Detection
Close
Contour
Detection
Contour
Filling
Residues
Elimination
Resize
Mask
Hough
Transform
Border
Detection
Rail Detection
Rail Tracking
Image
Obstacle
Identification
Fig. 2: System diagram for detecting obstacles over the railways
algorithm proposed by Lucas-Kanade [13] the sparse optical flow between frames is computed. After that, a new set of point of interest is obtained, where several features previously found still persist. If the background in the video were relatively static it could be easy to find the set of characteristics experimenting motion or rotation and therefore to find the foreground objects. Unfortunately the train movement creates the visual effect that almost all the objects are experimenting movement and in this cases the discrimination between foreground and background becomes more challenger.
So the set of characteristics found with the optical flow and under tracking, includes several elements from the background that appears to be moving, but their movements is just apparent and don’t represent danger to the train. This is why we analyzed the pattern of motion from every characteristic found in order to discriminate the elements from the background that are harmless. The first task for this procedure consists in getting the field velocity of every relevant feature in the image. Those characteristics with low velocity magnitude are then discarded. The direction of displacement is then considered. Analyzing the pattern of moving objects we found that many objects from the background often seems like if they were moving away from the rails. Quantifying this appreciation give us the following rules:
1. Features in the left moving with an angle between 180 and 270 degrees belong to the background.
2. Features in the right moving with an angle between 270 and 360 degrees belong to the background.
Althoug these can not be considered general rules give us a first criteria for the discrimination of typical foreground-background movement patterns. The remaining featurs are then clustered around three possible centers. Close centers are glue together as representing the same object. Using the media and variance of features velocity, points are associated as representing a foreground object. If the points velocity seems to be consistent in their displacement a candidate obstacle arises and is enclosed for signaling its detection. After this the trajectorie of the object is compute and its intersection with the rails predicted. Objets in route to collision generate a warning and are tracked in all their trajectories.
C. Data set
Due to the difficulty of using real videos where obstacles appear in front of the train, initially we choose to use freely available recordings on the Internet. We searched high resolution videos recorded from the driver's perspective where the route followed had no problems. After that the obstacles were digitally added through the superimposition of Chroma key videos. For this task we use the Kdenlive [15] software. The system works with image resolutions of 640x480 pixels All the system implementation was accomplished using the C++ libraries of OpenCv [16].
IV. RESULTS
Our system was tested using several modified videos from the Internet but the video used for measuring the system performance shows the route from city of Landskrona to Ramlösa (Sweden) [17]. We were granted permission for modification of the original video from the author. This video was recorded from the driver perspective with favorable weather conditions, at day, without turnouts and with splendid visibility.
For testing the system we used 7 minutes of recording where we added a total of 31 digital obstacles of different nature, shape and obstruction trajectory.
The first strategy, which scans the rails and detect objects
over them successfully identified 30 of the 31 obstacles, but
still has a high occurrence of false positives (15). Most false
detections were due to the presence of objects near the road
but actually do not represent real risk for the train travel, these
includes signs, pass levels, bridges over the rail, some
platforms, etc.
Figs. 5 and 6 show some examples of positive obstacle
identification. One of the false positives found is displayed on
Fig. 7, where a signal is erroneously classified as an obstacle
due its proximity to the rail. The second strategy, which finds objects approximating to
the rails, correctly identified 20 obstacles before they reach the rails or the train. We didn’t get false detections in this case. Fig. 8 shows an example of the moving obstacle detection.
Yes No
Feature Extraction
Sparse Optical Flow
Features Tracking
Displacement Analysis
Obstacles Selection
Trajectory Prediction
Identification Objects
in route to Collision
Features
Lost?
Fig. 4: System diagram for detecting obstacles in route to collision
Fig. 5 Image example showing moving obstacle over the rails and its
detection
Fig. 6: Image example showing fix obstacle over the railway and its
identification
Fig. 7 Incorrect classification of signal as obstacle due to its close position
to the rail
Table 1 summarizes the results for both strategies considered separately, where M1 represents the first approach, M2 the second approach, TP is the number of true positive cases, FP stands for false positives and FN for false negatives.
TABLE I. RESULTS
TP FP FN Precision Sensibility
M1 30 15 1 0.67 0.97
M2 20 0 11 1 0.65
The real time performance of the system reached rates around 30 fps using image resolutions of 640x480 pixels.
Fig. 8: Image example with positive approximating obstacle detection
V. CONCLUSIONS
Our work represents a first approach to the complex task of
creating an automatic tool for supporting train drivers labor.
Two related challenges were considered:
1. Detection of fixed or moving obstacles on the rails or
in the near area.
2. Detection of moving obstacles far from the rails and
in a route to collision.
Both tasks are highly complex and current research topics.
Particularly the detection and tracking of moving obstacles has
many difficulties due to the high speed motion, the
dissemination of forms (blurring) and the changing nature of
the background which greatly complicates its extraction.
The strategy used in our system for the obstacles identification
is still very simplified. Surprisingly the results obtained show
that it has high potential. It is clear that this strategy need to be
complemented with additional methodologies in order to allow
a more precise identification of objects and recognize those
who pose no threat to the train travel and in this way decrease
even more the false positive rate.
VI. FUTURE WORK
Addressing this work showed us that there exist the
potential to build support systems for automatic train driving.
In order to continue in the pursuit of this objective multiple
strategies can be followed to improve the system. Here we list
some of them.
Correct identification of the rails is an important step for the
system. In this work we used the Hough transform for line
detection. However, when the road is curved the method is
quite inaccurate in the remote areas of the rails. To
compensate for this it is necessary to evaluate alternative
strategies to capture more accurately the inclinations of the
rails and find a more appropriate vanishing point according to
the actual scene. A candidate method is the use of clothoids or
parabolic segments that can much better fit to the rail.
Many objects in the railway can affect the monotony of the
rails images and the nearby area without posing a real danger
to the train course. This includes turnouts, signs, pass levels,
tunnels, bridges, etc. Recognition of these objects is essential
for a reliable behavior of a train driver assistance system.
Learning and identification of benign object provides
interesting challenges for computer vision algorithms. Another important aspect that should be explored is the
robustness of the algorithms to changes in light and weather conditions. Driving at night or amidst storms creates a vast number of difficulties for automated driving systems based on vision. It is possible that the creation of strategies based on adaptive parameters can provide a solution but its use is currently complex and difficult to implement.
REFERENCES
[1] L. A. Fonseca, J. A. Uribe, y F. Vargas, «Obstacle Detection over Rails
Using Hough Transform», presented at the XVII SIMPOSIO DE
TRATAMIENTO DE SEÑALES, IMÁGENES Y VISIÓN ARTIFICIAL ,STSIVA-2012, Medellín, Antioquia, COL, 2012.
[2] J. R. Jamieson y M. D. Ray, «Railway obstacle detection system and
method», U.S. Patent App. 10/251,422sep-2002. [3] R. Passarella, B. Tutuko, y A. P. P. Prasetyo, «Design Concept of Train
Obstacle Detection System in Indonesia», IJRRAS, pp. 453-460, dic-
2011. [4] F. Kruse, S. Milch, y H. Rohling, «Multi Sensor System for Obstacle
Detection in Train Applications», Proc. of IEEE Tr., June, pp. 42–46,
2003. [5] S. Sugimoto, H. Tateda, H. Takahashi, y M. Okutomi, «Obstacle
detection using millimeter-wave radar and its visualization on image
sequence», in Pattern Recognition, 2004. ICPR 2004. Proceedings of the 17th International Conference on, 2004, vol. 3, pp. 342 - 345 Vol.3.
[6] J. J. D. Garcia, J. U. Urena, A. A. Hernandez, M. Q. Mazo, J. F.
Vazquez, y M.-J. Diaz, «Multi-sensory system for obstacle detection on
railways», in Instrumentation and Measurement Technology Conference
Proceedings, 2008. IMTC 2008. IEEE, 2008, pp. 2091 -2096.
[7] M. Ukai, «A New System for Detecting Obstacles in Front of a Train»,
vol. 2006, p. 73, otoño 2006.
[8] M. Ruder, N. Mohler, y F. Ahmed, «An obstacle detection system for automated trains», in Intelligent Vehicles Symposium, 2003.
Proceedings. IEEE, 2003, pp. 180–185.
[9] F. Kaleli y Y. S. Akgul, «Vision-based railroad track extraction using dynamic programming», in 12th International IEEE Conference on
Intelligent Transportation Systems, 2009. ITSC ’09, 2009, pp. 1-6.
[10] F. Maire y A. Bigdeli, «Obstacle-free range determination for rail track maintenance vehicles», in 2010 11th International Conference on
Control Automation Robotics & Vision (ICARCV), 2010, pp. 2172-2178.
[11] M. Gschwandtner, W. Pree, y A. Uhl, «Track detection for autonomous trains», Advances in Visual Computing, pp. 19–28, 2010.
[12] H. A. Mallot, H. H. Bülthoff, J. J. Little, y S. Bohrer, «Inverse
perspective mapping simplifies optical flow computation and obstacle detection», Biological cybernetics, vol. 64, no. 3, pp. 177–185, 1991.
[13] B. D. Lucas y T. Kanade, «An iterative image registration technique
with an application to stereo vision», in Proceedings of the 7th international joint conference on Artificial intelligence, 1981.
[14] J. Shi y C. Tomasi, «Good features to track», in Computer Vision and
Pattern Recognition, 1994. Proceedings CVPR’94., 1994 IEEE Computer Society Conference on, 1994, pp. 593–600.
[15] «Kdenlive | Free and open source video editor for GNU/Linux, Mac OS
X and FreeBSD». [Online]. Available: http://www.kdenlive.org/. [Accessed: 08-jul-2012].
[16] «Welcome - OpenCV Wiki». [Online]. Available: http://opencv.willowgarage.com/wiki/. [Accessed: 08-jul-2012].
[17] «Train Driver’s View: Landskrona - Ramlösa - YouTube». [Online].
Available: http://www.youtube.com/watch?v=ParZVLmuxbo. [Accessed: 08-jul-2012].
top related