1

Autonomous Real-time Vehicle Detection from aMedium-Level UAV

Toby P. Breckon, Stuart E. Barnes, Marcin L. Eichner and Ken Wahren

Abstract— A generic and robust approach for the detectionof road vehicles from an Unmanned Aerial Vehicle (UAV) is animportant goal within the framework of fully autonomous UAVdeployment for aerial reconnaissance and surveillance. Here wepresent a novel approach to the automatic detection of vehiclesbased on using multiple trained cascaded Haar classifiers (adisjunctive set of cascades). Our approach facilitates the real-time detection of both static and moving vehicles invariant toorientation, colour, type and configuration. The results presentedshow the successful detection of differing vehicle types undervarying conditions in both isolated rural and cluttered urbanenvironments with minimal false positive detection. The tech-nique is realised on aerial imagery obtained at 1Hz from anoptical camera on the medium UAV B-MAV platform with resultspresented to include those from the MoD Grand Challenge 2008.

Index Terms— vehicle detection, UAV image analysis, threatdetection, cascaded Haar classifier.

I. INTRODUCTION

Within the common deployment roles of Unmanned AerialVehicles (UAV) the detection and location reporting of keyground assets such as vehicles and people are found to becommon to both military and civil scenarios. Whilst thedetection of people has received considerable attention withinthe literature [8], [13], [10], work on the detection of vehicleshas until recently been dependent on movement of the vehiclesin the scene [6], [8], [13].

Here we consider the detection of vehicles from a mid-range aerial platform that provides a perspective aerial viewdownward from an acute camera angle on the platform (e.g.Figure 4). Compared to a top-down planar view, such acamera view is better suited towards the detection of peoplefor dual role detection strategies [13] but poses additionalchallenges for vehicle detection. From a perspective viewpointthe shape and size of a vehicle is less invariant to changes incamera position and orientation which vary considerably withUAV approach angle. A planar viewpoint reduces the vehicledetection problem somewhat to the detection of a simplifiedrectangular object at an angular offset to the horizontal. Bycontrast, from a perspective viewpoint the available vehicleview is one that includes both side-on and top-down vehicledetail. The result is a complex representation of a vehicle thatis highly variant to approach angle of the camera to the target

Toby P. Breckon and Stuart E. Barnes are with the School of Engineer-ing, Cranfield University, Bedfordshire, UK. toby.breckon@cranfield.ac.uk /s.e.barnes@cranfield.ac.uk

Marcin L. Eichner is with the Department of Information Technology andElectrical Engineering, ETH Zurich, Switzerland.

Ken Wahren is with Blue Bear Systems Research, Bedfordshire, UK.

in the x/y plane and to minor changes in the angle of horizontalview in z. Such a perspective view, however, facilitates botha significant improvement in situational awareness for humanoperators/users and aids greatly in the automated or manualidentification of human assets within the same imagery. Unlikethe vehicle case, the aerial detection of people benefits fromthe consistent projection of the upright human form to avertical trace within a perspective view image - an assumptionnotable even in the state of the art approaches within thisdomain [1]. As can be seen from our examples (Figures 4 - 8)no such assumption is applicable within the domain of aerialvehicle detection from a perspective viewpoint. The generalproblem of aerial vehicle detection is also made significantlymore challenging by the non-uniformity of vehicle colour,localised shape characteristics and overall dimension. Ther-mal/IR vehicle detection is similarly limited by the variancein thermal signature in relation to time of day (sunlight tovehicle heat transfer) and operation (engine to vehicle heattransfer) meaning that additional scene information is oftenrequired to detect vehicles in addition to thermal signaturebased approaches [4].

The key challenge is a robust approach for the orientationinvarient detection of both moving and static vehicles in bothcluttered and uncluttered aerial views. We propose the useof a novel adaptation to the established real-time detectionapproach of [11], already investigated for ground based vehicledetection by [9], to this problem.

II. PRIOR WORK

Recent work on UAV based vehicle detection appears tolargely focus on the detection of moving vehicles [5], [6], [13]or the specific tracking of identified ground objects [3]. Thework of [5] uses an approach based on identifying consistentlymoving subsets of edges within an overall flight sequenceas a moving vehicle using a graph cuts driven technique.Previously [6] followed a similar methodology through theuse of camera motion estimation and Kalman filter basedtracking of a moving object within the scene but extended overoptical/IR sensing. In [13] the authors present an approachbased on layered segmentation and background stabilizationcombined with real-time tracking which then leads to theclassification of identified moving objects as {people | vehicle}based on [1]. The more general work of [3] makes use of theclassical mean-shift tracking approach to track generic groundobject descriptors, including but not limited to vehicles, froma UAV image sequence but does not explicitly tackle the initialobject detection problem. In all of these cases [5], [6], [13]

2

the detection of vehicles is primarily driven by the isolationof a moving component from the overall scene. By contrastrecent work in people detection [10] investigates the problemof people detection in UAV aerial imagery independently ofmovement using modern classifier approaches [11] aided bymulti-spectral (optical/IR) imagery. However, [10] is aided bythe IR temperature characteristics for human bodies that cannot be readily relied upon in the vehicle detection case.

Overall work on the specific detection of vehicles, encom-passing both static and dynamic vehicles, within UAV imageryis limited. Current work specifically addressing this problem[4] relies upon auxiliary scene information and additionalthermal/IR sensing. Here we present an approach for theapplication of the object detection methodology of [12] forthe detection of both static and dynamic vehicles using onlyan optical camera based on a perspective viewpoint from amedium level UAV platform.

III. AUTONOMOUS VEHICLE DETECTION

We propose a two stage approach to autonomous vehicledetection: 1) primary detection using multiple trained cascadedHaar classifiers [11] and 2) secondary verification based onUAV altitude driven vehicle size constraints.

A. Primary Detection

The primary means of vehicle detection makes use ofmultiple cascaded Haar classifiers [11], [7]. Cascaded Haarclassifiers were firstly proposed by [11] and later improvedby [7] with a primary application domain for the detectionof faces [12]. The concept is to use a conjunctive set ofweak classifiers to form a strong classifier - in this instance,a cascade of boosted classifiers applying Haar-like features.These Haar features are essentially drawn from the spatialresponse of Haar basis functions and derivatives (hence Haar-like features) to a given type of feature at a given orientationwithin the image (Figure 1). In practice they are computedas the sum of differences between differing rectangular sub-regions at a localised scale which although limited in scopeas individual features can be computed extremely efficiently.Individually, they are weak discriminative classifiers but whencombined as a conjunctive cascade a powerful discriminativeclassifier can be constructed capable of recognising commonstructure over varying illumination, base colour and scale [12].

A cascaded Haar classifier is trained using a set of a fewhundred positive object and negative object training images.The use of boosting techniques then facilitates classifiertraining to select a maximally discriminant subset of theseHaar-like features, from the exhaustive and overcomplete set,to act as a multi-stage cascade [11]. In this way, the finalcascaded Haar classifier consists of several key simpler (weak)classifiers that all form a stage in the resultant complex(strong) classifier. These simpler classifiers are essentiallydegenerative decision-tree classifiers that take the Haar-likefeature responses as input to the weak classifiers and returna boolean pass/reject response. A given region within theimage must then achieve a pass response from all of theweak classifiers in the cascade to be successfully classified

Fig. 1. Haar-like features. A. Edge features. B. Line features. C. Centre-surround features.

as an instance of the object the overall strong classifier hasbeen trained upon. The classifier is then evaluated over aquery image at multiple scales and multiple positions using asearch window approach [11]. Despite this apparent exhaustivesearch element of the classifier the nature of the cascade(sorted in order of most discriminative feature) allows theearly rejection of the majority of such windows with a minimalevaluative (and hence computational) requirement. In this waythe Haar cascade classifier thus combines successively morecomplex classifiers in a cascade structure which eliminatesnegative regions as early as possible during detection butfocuses attention on promising regions of the image. Thisdetection strategy dramatically increases the speed of thedetector, provides an underlying robustness to changes in scaleand maintains achievable real-time performance [11], [12].

For our application to aerial vehicle detection four separatecascaded Haar classifiers where trained based on sample vehi-cle images categorized into one of four positional orientationsto the horizontal - the resulting set of clockwise angular offsets{0◦, 45◦, 90◦, 135◦}. A large training set was empiricallysub-divided based on the perceived nearest angular offset ofthe vehicle wheelbase with vehicles assumed to be symmetricabout the axis perpendicular to the angle of the wheelbase. Aseparate cascaded Haar classifier was trained for each subset ofvehicle orientations in a similar manner to the use of separate{side, front} facial and pedestrian {torso, full-body} detectionapproaches of [11], [7]. An example of the subset of used fortraining the 135◦ offset classifier is shown in Figure 2.

Fig. 2. Subset of the training data used for training the 135◦ offset classifier.

B. Secondary Verification

The results obtained from the evaluation of this set of trainedclassifiers is then post-processed through a series of rule-based

3

logic to confirm the presence of vehicles within the identifiedcandidate regions.

From the output of the primary detection a set of potentialvehicle locations within the image is identified in each of thefour orientations as a rectangular region of interest. Thesedetections are treated as a set of disjunctive vehicle detections- a vehicle may exist at any one of these image locations ifone or more of the orientation specific classifiers has beenactivated at that position subject to additional constraints. Inthis way we develop a fuzzy logic style approach to detectionsuch that a given vehicle within the image may belong to thedetection set returned from one or more of the orientationspecific classifiers. This is justifiable due to the subjectivenature of the orientational set boundaries and is found to workwell in practice.

The returned set of regions within the image is spatiallymerged to resolve multiple-classifier detections and detectionoverlaps to produce a singular set of vehicle detections withinthe image. A series of logic constraints then aims to eliminatenon-vehicle candidates. Each rectangular region is rejectiontested against maximal height maxH , width maxW and areamaxA constraints with similar minimums imposed using theminimum search window size used in the initial primarydetection phase. This restricts the size a vehicle can be withinthe overall image driven from a priori knowledge of camerafield of view and UAV platform altitude. Due to the earlierstate of merging, multiple spatially co-incident vehicles withinthe scene may be merged into a single (large) detection causingelimination by this latter size based constraint rejection. Res-olution of this problem is left as an area for future work andis an issue encountered in other applications of the cascadedHaar classifier approach [2]. Instead we concentrate on isolatedvehicle detection in the general case.

IV. UAV PLATFORM CHARACTERISTICS AND OPERATION

The UAV platform used for this work was a Blue Bear Sys-tems Research MAV (B-MAV) platform with a 1m wingspanand operating weight of 1.8kg. This platform operates a 40minute flight duration based on either hand or catapult launchand can carry a single camera payload. Control is via the on-board Surveillance & Navigation Auto Pilot (SNAP) controlsystem from which the platform can either fly under remotepilot or GPS way-point control. For this work the platformwas equipped with a 25mm Sony 1/3" Sony Interline CCDbased camera positioned at 45◦ angle to the horizontal in thedirection of flight.

Operationally the platform was controlled by GPS way-points uploaded over IP radio link as part of the Stellar TeamSATURN system developed for the MoD Grand Challengecompetition (2008). These way-points tasked the platform tofly set raster search patterns over a given search area withinthe environment. Images were captured at 1Hz on-board theplatform at PAL resolution, compressed to 32 kilobytes usingthe JPEG2000 compression scheme and transmitted back tothe SATURN ground control station via the same IP radio link.Automated vehicle detection was performed in real-time at theground control station on an Intel 8-core CPU workstation that

Fig. 3. Blue Bear Systems Research MAV (B-MAV) platform

was concurrently tasked with the processing of image datafrom two other remote platforms.

V. RESULTS

The results of the proposed vehicle detection approach isshown in Figures 4 - 8 over environments of differing com-plexity in terms of scene clutter with detected vehicles (andfalse positives) highlighted using red boxes. Figure 4 showsthe successful detection of both static and moving vehicles ina common un-cluttered environment (Figure 4: rows 1 and 2)with additional detections at alternative vehicle orientations inaddition to a couple of illustrative false positives (Figure 4:rows 3 and 4). One vehicle is additionally missed in Figure 4(row 3 left) but notably detected in the other images of Figure4 during the same flight. This brings us to the definition ofsuccess criteria for detection.

Whilst traditional object detection approaches are driven bythe goal of the correct detection of every occurrence of anobject in every image frame within a sequence the problem ofvehicle detection and subsequent location reporting is subtlydifferent. We do not actually require to detect every vehicle inevery frame, instead we require to locate every vehicle withina given search area. This means that we only require to detect avehicle once at most as we pass over it on a given flight. Thissubtlety of definition facilitates the tuning of our techniquetowards minimizing false positives whilst maximizing vehicledetections per flight not per image frame. Returning to theimage frames extracted from the flight sequence of Figure 4 wesee that indeed every vehicle within the (same) environmentis successfully detected in at least one frame despite a numberof inconsistent false-positive detections.

Figure 5 shows the successful detection of vehicles withina varying rural/industrial setting. Notably some vehicles aremissed in the car park scenes (Figure 5, row 1 left and row 2right) which is attributable to the multi-detection grouping andconstraint rejection criteria of the second phase of processing.Detections containing multiple vehicles are also attributableto this behaviour and whilst the accuracy of the number ofdetections may be inaccurate the presence of the vehicleshas correctly been identified for position reporting. Accurateprocessing of multiple vehicle clusters is left as an area forfuture work. The examples of Figure 5 also illustrate the

4

Fig. 4. Rural environment detection results (West Wales UAV Centre, Parc Aberporth, UK).

5

robustness of the detection approach to changes in orientation,scale and vehicle colour.

Figures 6 to 8 show the successful vehicle detections andillustrative false positives obtained during the finale and prov-ing events of the MoD Grand Challenge event 2008 where theapproach presented formed part of the Stellar Team SATURNsystem. Figure 6 shows the successful detection of variousvehicles from various angles of UAV approach and showsthe approach is both robust to variations in scale and partialocclusion (Figure 6, row 1 left). Notably a secondary vehicleis not detected in Figure 6 (row 2, row 3 right) but is laterdetected within the same flight as shown in Figure 7. Falsepositive detections are illustrated in Figure 6 (row 4 left) andin Figure 7 (row 3 right, row 4 left). Despite this, Figures 6and 7 show the consistent detection of multiple static vehiclesindependent of variation in orientation, colour and moderatechanges in scale within an urban environment.

Figure 8 shows additional vehicle detections within the sameurban environment where we can see both detection of vehiclesin multiple frames (Figure 8, row 3) and the principle of everyvehicle detected at least once per over flight (e.g. red vehiclenot detected in centre of Figure 8 row 1 right is detected inrow 2 left). False positive detections are also present in Figure8 (row 2, left, lower) and Figure 8 (row 1, left, upper). Theambiguity of detection is also illustrated in Figure 8 row 4with regard to image quality deterioration. Here whilst Figure8 row 4 left is arguably a vehicle, the image quality of Figure8 row 4 right makes it indeterminable. Figures 6 to 8 containlargely static vehicles with minimal moving vehicles with theenvironment.

All of the examples shown (Figures 4 to 8) were collectedand processed using the setup described in Section IV. TheB-MAV platform was flown at an auto pilot target altitudeof 40m for all of these examples with minor changes incamera angle and altitude attributable to GPS error and windconditions within the environment. The vehicle size constraintparameters minH , maxH , and maxA were set empiricallybased on calculations made from sample imagery captured atthis altitude.

VI. CONCLUSIONS

Overall from the results presented we can see the successfuldetection of vehicles from imagery obtained in both unclut-tered rural, industrial and urban environments independently ofvehicle orientation, type characteristics and movement withinthe scene. This is achieved using a two stage approach basedon multiple cascaded Haar classifiers tuned for different vehi-cle orientations over a fuzzy set paradigm. This disjunctive setof Haar cascades offers explicit vehicle detection, in place ofthe movement dependent approaches of [6], [5], [3], [13], thatis capable of detecting both static and dynamic vehicles withinthe scene. This is demonstrated over a range of environmentsfrom the medium level B-MAV platform. Within our defined“detection per over flight” criteria the detection of vehiclesperforms well using this approach although improving individ-ual vehicle occurrence detection rates (i.e.“detection per imageframe”) and the reduction of false positive detections remain

goals for future work. Additional work to address multiplevehicle clusters within scenes and the possible integration ofthermal sensing and additional scene information [4] wouldalso be of benefit.

Acknowledgments

The work presented in this paper was funded by the UKMoD as part of the MoD Grand Challenge 2008 and formedpart of the Stellar Team SATURN system that was awardedthe R.J. Mitchell Trophy in winning the 2008 challenge.Stellar Team comprises of Cranfield University, Blue BearSystems Research, Selex Galileo, Marshall System DesignGroup, TRW Conekt and Stellar Research Services.

REFERENCES

[1] N. Dalal and B. Triggs. Histograms of oriented gradients for humandetection. Proc. Conf. on Computer Vision and Pattern Recognition,1:886–893 vol. 1, June 2005.

[2] J.W. Han and T.P. Breckon. Radicular cysts and odontogenic keratocystsepithelia classification using cascaded haar classifiers. In Proc. 12thAnnual Conference on Medical Image Understanding and Analysis,pages 54–58, July 2008.

[3] H. Helble and S. Cameron. Oats: Oxford aerial tracking system. Robot.Auton. Syst., 55(9):661–666, 2007.

[4] S. Hinz and U. Stilla. Car detection in aerial thermal images by localand global evidence accumulation. Pattern Recogn. Lett., 27(4):308–315, 2006.

[5] K. Kaaniche, B. Champion, C. Pegard, and P. Vasseur. A visionalgorithm for dynamic detection of moving vehicles with a UAV. Proc.Int. Conf. on Robotics and Automation, pages 1878–1883, April 2005.

[6] J. Kang, K. Gajera, I. Cohen, and G. Medioni. Detection and tracking ofmoving objects from overlapping EO and IR sensors. Proc. Int. Conf. onComputer Vision and Pattern Recognition, pages 123–123, June 2004.

[7] R. Lienhart and J. Maydt. An extended set of haar-like features forrapid object detection. Proc. Int. Conf. Image Processing, 1:I–900–I–903 vol.1, 2002.

[8] A. Miller, P. Babenko, M. Hu, and M. Shah. Person tracking in UAVvideo. In Proc. Int. Workshop on Multimodal Technologies for Perceptionof Humans, pages 215–220, Berlin, Heidelberg, 2008. Springer-Verlag.

[9] P. Negri, X. Clady, S.M. Hanif, and L. Prevost. A cascade of boostedgenerative and discriminative classifiers for vehicle detection. J. Adv.Signal Process, 8(2):1–12, 2008.

[10] P. Rudol and P. Doherty. Human body detection and geolocalizationfor UAV search and rescue missions using color and thermal imagery.Aerospace Conference, IEEE, pages 1–8, March 2008.

[11] P. Viola and M.J. Jones. Rapid object detection using a boostedcascade of simple features. Proc. Conf. on Computer Vision and PatternRecognition, 1:I–511–I–518 vol.1, 2001.

[12] P. Viola and M.J. Jones. Robust real-time face detection. Int. J. Comput.Vision, 57(2):137–154, 2004.

[13] J. Xiao, C. Yang, F. Han, and H. Cheng. Vehicle and person trackingin aerial videos. In Proc. Int. Workshop on Multimodal Technologiesfor Perception of Humans, pages 203–214, Berlin, Heidelberg, 2008.Springer-Verlag.

6

Fig. 5. Rural/industrial environment detection (Blue Bear Systems Research facility, UK).

7

Fig. 6. Detections during MoD Grand Challenge 2008 (Copehill Down FIBUA Training Facility, UK).

8

Fig. 7. Detections during MoD Grand Challenge 2008 (Copehill Down FIBUA Training Facility, UK).

9

Fig. 8. Detections during MoD Grand Challenge 2008 (Copehill Down FIBUA Training Facility, UK).