Multi-Sensor Data Fusion for Checking Plausibilityof V2V Communications by Vision-based

Multiple-Object Tracking

Marcus ObstBASELABS GmbH

Ebertstr. 10Germany, Chemnitz 09126

Email: marcus.obst@baselabs.de

Laurens HobertHitachi Europe SAS

955, Route des LuciolesFrance, Sophia Antipolis 06560

Email: laurens.hobert@hitachi-eu.com

Pierre ReisdorfTechnische Universitat Chemnitz

Reichenhainer Str. 70Germany, Chemnitz 09126

Email: pierre.reisdorf@etit.tu-chemnitz.de

Abstract—Automated vehicles require an adequate and re-liable perception of the surrounding world in order to makegood decisions. Using vehicle-to-vehicle (V2V) communication toexchange location data (i.e. time, position, heading and speed)can improve the perception beyond the capabilities of traditionalon-board sensors (e.g. radar, lidar). However, it is vital to trustthe data before it is being used. Cryptographic mechanismscan protect the exchange and authenticity of data but do notguarantee the correctness of the content. In this paper we presenta vision-based multi-object tracking system for checking theplausibility of V2V communication. The system is addressing thechallenge of fusing relative sensor observations as provided by aMobilEye vision-system with time-delayed absolute GNSS-basedmeasurements from Cooperative Awareness Messages (CAMs) asprovided by V2V. The plausibility check is implemented in aprototype and based on a state-of-the-art multiple-object trackingalgorithm. The proposed system is evaluated and validated underreal-world conditions by conducting several test drives underurban conditions.

I. INTRODUCTION

A clear trend in the automotive industry is the automationof driver tasks. Early technologies such as ABS and ESPare supporting the driver respectively in braking and unstablesituations whereas more recent technologies such as the lane-keep assistant and adaptive cruise control go one step furtherby taking over driver tasks. Considering the intention of manyvehicle manufacturers to introduce fully automated vehicles bythe year 2025, we may expect a continuation of this trend inthe future.

Many of these technologies depend on an adequate andreliable perception of the surrounding world. For example,the lane-keep assistant needs to know the vehicle’s positionin a lane and a fully automated vehicle requires a viewof surrounding obstacles in order to navigate around them.Traditional automotive sensors (e.g. radars, lidars, vision-systems) can sense the environment with a high frequencyand high reliability. However, they are limited in range, field ofview (FOV) and target measurements (e.g. distance and angle).

With the introduction of vehicle-to-vehicle (V2V)communication—that is the exchange of data between vehicles—and standards like the Cooperative Awareness Message (CAM),the perception of vehicles may be improved beyond thecapabilities of traditional sensors by the exchange of locationdata (i.e. time, position, heading and speed), vehicle dimensions,

vehicle type, yaw rate, acceleration and many other parameters.However, V2V communication involves by definition a sendingand receiving vehicle and an important question is whetherthe received data can be trusted; especially if the data isused to control the vehicle. For example, an attacker mighthave compromised the transmitting system or the system isunintentionally sending wrong data due to a broken sensor.Cryptographic mechanisms can protect the exchange andauthenticity of data but do not guarantee the correctness of thecontent so we need another way of checking the plausibilityof V2V data.

In this paper, we develop and verify a system to checkthe plausibility of received CAM data by cross correlating itwith data obtained from a MobilEye camera. This system isimplemented as a probabilistic multi-sensor multi-object datafusion algorithm and is addressing the challenges of fusingglobal and vehicle-centric position coordinates as providedrespectively by CAMs and the MobilEye camera and out-of-sequence measurements due to V2V communication delays.The presented work deliberately builds on existing technol-ogy such as commercial sensors. Furthermore, the effortsof implementing the probabilistic multi-sensor data fusionalgorithms are limited by using and relying on a generic datafusion framework BASELABS Create. By that approach, amulti-sensor data fusion system which comprises the state-of-the-art performance of single sensors can be developedand implemented in a prototypical way1 rather efficiently.Consequently, this paper will not present theoretical detailsof Bayesian data fusion nor implementation details of typicalmulti-object tracking (MOT) approaches. In [2] the wholedevelopment cycle for a probabilistic vehicle localizationsystem, which was implemented with the aforementioned tools,is presented and elaborated in detail.

The paper is structured as follows: in section II a briefoverview of related work is given. Afterwards, the subsequentsection III introduces the principles of the used sensors andgives a glance of multi-sensor data fusion in the context ofmulti-object tracking. Section IV elaborates on the proposedapproach to derive a decision from the tracking measureswhether a CAM message is spoofed by an attacker or can be

1BASELABS provides a further tool named CODE which automaticallytransforms the prototyping code of CREATE to production-ready ECU sourcecode. This process is described in more detail in [1].

assumed trustworthy. The implementation details are given insection V. Special emphasize is put on the task of transformingthe absolute positions from the CAMs to the local vehiclecoordinate frame. Finally, section VI presents quantitativeresults from real-work test drives and evaluates the capabilityof a multi-object tracker for plausibility checking. The paperconcludes with a summary in section VII.

II. RELATED WORK

The VSC-A research project has identified in collaborationwith NHTSA, 6 potential safety application concepts that canaddress imminent crash scenarios using V2V communication.Two of these promising concepts are the Electronic EmergencyBrake Light (EEBL) and the Intersection-Movement Assist(IMA) which warn the driver respectively for hard brakingvehicles and potential intersection crashes. For EEBL, Szczureket al. have developed and compared in [3] two methods toestimate the relevancy of a hard braking vehicle. These methodsdepend on the exchange of position, time, vehicle speed, length,road and lane identifiers using V2V communication. In [4], arisk assessment is presented for an IMA which heavily relieson V2V communication to overcome the limitations of classicalon-board vehicle sensors with limited field of view. In bothaforementioned publications, the authors assume accurate andconsistent data. However, no dedicated plausibility check wasforeseen.

Typically, heterogeneous sensor information are combinedto either implement some kind of plausibility checking or toimprove the observation area and therefore provide a betterperception. Apparently, both tasks are often implemented bycompletely different approaches. Examples for multi-sensorplausibility checking for VANETs are given in [5] and [6].The former work introduces an integrated particle filter-basedapproach to implement plausibility checking in VANETs whichincorporates several information such as radar observations ordigital map data. The authors of the latter work propose touse multiple behavior analysis modules in order to enhance thesecurity in VANETs. Another approach, which was developedin the simTD project, utilizes a Kalman filter-based approachto track and assess individual CAMs. The aforementionedpublications have in common that they rely on fixed stationidentifiers, require periodic CAMs with fixed latencies and makestrong assumptions on the number of involved participants.

On the contrary, there are several studies which investigatethe potential of V2V communication as part of a multi-sensordata fusion framework to extend the perception of the on-boardsensors. Typically, an accurate and comprehensive knowledgeof the vehicle’s environment is a crucial requirement for safety-relevant ADASs and autonomous driving in general. In [7]a high-level fusion architecture which considers CAMs as akind of virtual sensor is presented. The authors propose touse a track-to-track fusion based data fusion scheme. In [8] apractical implementation of a tracking system which uses theposition messages from V2V in combination with an on-boardradar sensor to provide an adaptive cruise control application ispresented. The authors conclude that the rather limited accuracyof the position messages complicates the data associationstep between both sensor systems. A simulative study basedon artificial radar measurements and communicated vehiclepositions in order to demonstrate the successful extension ofthe sensor field of view was performed in [9].

The paper at hand wants to demonstrate that both tasks,that is plausibility checking and extended perception, can beimplemented based on a unified multi-object tracking frame-work. Therefore, a consumer-grade on-board perception sensorwill be combined with the information coming from CAMsas communicated using ITS-G5 devices. The heterogeneoussensor information are processes by a Bayesian data fusionalgorithm under real-time conditions. In addition to the dynamicparameters of the surrounding vehicles a confidence estimate isnaturally provided as well. Furthermore, each object is assignedan existence probability which will be used in order to realizethe plausibility checking.

III. FUNDAMENTALS

In this section, the used sensor hardware is briefly explainedand a proposal how to integrate it with Bayesian multi-sensordata fusion is given.

A. Vision Based Vehicle DetectionTypically, radar or lidar sensors are used in order to

perceive the local environment around the vehicle. In thispaper, a commercial off-the-shelf vision-based system whichcomprises a mono-camera of MobilEye is used. The MobilEyesystem is mounted behind the windshield and provides relativemeasurements (distance, angle and relative speed) as well asthe width of other vehicles with respect to the ego vehicleover a CAN interface. This type of detections are sometimesalso called stick objects. More details on the internal imageprocessing algorithms of this system can be found in [10]. It isworth mentioning that the detection quality is far from perfect.For example, under difficult illumination conditions or on wetroads the system is sensitive to clutter measurements. Thisneeds to be considered inside of the data fusion.

B. Perception by CommunicationWith the deployment of V2V communication, a new class

of sensors is available which can be used for innovative ADASapplications and intelligent vehicles. Several standards for V2Vand V2I (vehicle-to-infrastructure) communication have beendefined by organisations like IEEE, SAE and ISO. In Europe,ETSI and CEN have defined a set of standards for V2V andV2I (V2X) communication. This set includes the CooperativeAwareness Basic Service which mandates each road userparticipating in a V2X network to transmit periodically theirstation type, time, position, velocity and many other parametersin a CAM. This rather comprehensive description of an objectstate in combination with a communication range of up to1000 m [11] makes CAM data appealing candidates for multi-sensor data fusion.

However, V2V communication in general and CAM trans-missions in particular have some distinctive characteristicscompared to classical on-board perception sensors. The follow-ing section lists these characteristics briefly. Later in SectionV, they will be addressed inside the proposed implementationof the multi-sensor data fusion system.

1) Coordinate frame: While on-board sensors usually per-ceive the environment in a vehicle-centric coordinate frame,CAMs describes the object’s position in absolute coordinates. Itis therefore inevitable to translate to a single coordinate systemin order to associate CAMs with on-board sensor detections.

2) Update rates: On-board sensors usually have a fixedand high-frequent update rate of 20 Hz-50 Hz. CAMs arehowever generated with a rather low dynamic rate of 1 Hz-10 Hz which depends on the vehicle dynamics. Moreover, CAMtransmissions using ITS-G5 are unreliable and may get lostwhich leads to an even lower update rate at a receiver. It ishence considered a challenging task to observe high-dynamicmaneuvers solely by using CAMs.

3) Field of view: The FOV of on-board sensors is usuallyrather limited. Although, long-range radars might observe thearea in front or behind the ego vehicle up to 150 m, theyprovided only limited performance to perceive objects in thenear proximity. Moreover, on-board sensors are quite sensitiveto temporary occlusion which might be caused by objects whichare blocking the sensor’s FOV. V2V communication basedon ITS-G5 may allow omnidirectional communication with arange up to 1000 m and relative speeds up to 30 m/s [11]. Suchvalues have been observed even under real-world conditions asreported in [12]. Furthermore, V2V communication can providemeasurements even under non-line-of-sight conditions.

4) Observation latencies: The time span between thegeneration of a measurement and its availability to the multi-sensor data fusion is described as observation latency. Themeasurements of in-vehicle perception sensors are usuallysubject to a fixed sensor-specific observation latency of upto 100 ms for average scenarios. However, for CAMs, variablelatencies of up to 1000 ms are observed. There are severalreasons for this behaviour: time synchronization errors, alatency between the position measurement and the CAM packetgeneration, the processing latency of the sending and receivingcommunication stack and medium contention and transmissionlatencies. Without and further precautions, the CAMs willarrive out of sequence—thus, they are called out of sequencemeasurements (OOSM)—at the data fusion system which mightlead to inconsistent tracking results [13].

5) Missing detections: The event of missed detections ofon-board sensors is usually only caused by non-systematicerrors. Hence, on average each object in the sensor’s FOV isdetected as long as it is not occluded. For V2V this is differentas the reception of CAMs is known to be more unreliable.For example, the probability of a successful CAM receptiondepends on the physical distance [14], package size [11], themounting position of the antennas [15] and the link congestion.Moreover, the environmental conditions (urban, suburban) haveto be considered as shown in [12] where a reliability analysisof ITS-G5 in the context of the DRIVE-C2X project wasconducted.

6) Measurement refinement: The behaviour of on-boardsensors, whether they provide raw data or already applysome kind of refinement (e.g. based on Bayesian filtering),is documented in the sensor manual and typically alreadyavailable when designing a multi-sensor data fusion system.However, the CAM standard does not define how the positioninformation of CAMs is generated. For example, one vendormight simply connect a low-cost GPS receiver, while anotherone implements a sophisticated vehicle positioning algorithm.Obviously, in this particular case the quality and statistics of thepositions contained in both CAMs will be completely differentand are unfortunately not known to the data fusion at thereceiving vehicle. This applies to the quality indicators andconfidence values transmitted in the CAMs as well.

C. Multi-Object Tracking and Multi-Sensor Data FusionObject tracking refers to the task of using noisy sensor

measurements to derive several characteristics of objects ofinterest over time [16]. The characteristics typically includedynamic parameters such as position, velocity and headingas well as discrete states (e.g. the existence/relevance of anobject2). Moreover, object tracking aims to estimate the correctnumber of objects in the surveillance area. If the number ofobjects to track is larger than one and several perception sensorsare utilized, a multi-object tracking (MOT) problem whichnaturally includes multi-sensor data fusion is present. The mainchallenge in MOT is to correctly perform the measurementassociation. In literature several approaches for multi-sensordata fusion in object tracking have been proposed in [13]. Thepaper at hand rigorously relies on the concept of recursiveBayesian filtering to exploit temporal correlation between theobservations as provided by the modular perception frameworkBASELABS Create [1]. Independent of the selected MOTalgorithm, the domain-specific design (e.g. the considerationof particular sensors and the targeted application) of a MOTtypically involves several tasks which have to be fulfilled bythe practitioner. In the subsequent text these tasks are brieflyexplained.

1) Identification of target variables: Besides the estimationof the total number of objects, one objective of MOT is toestimate dynamic parameters. Which particular parameters areof interest typically depends on the final application or use-case.Moreover, a coordinate frame for the estimation is selected[17].

2) Identification of support variables: In addition to thetarget variables which are directly required by the application,several so-called support or auxiliary variables need to beestimated as well to support the integration of observationmodels. Both, the target and support variables comprise thefinal state vector to estimate.

Additionally, for each sensor which is part of the multi-sensor data fusion, the following decisions have to be made:

3) Selection of fusion schema: For sensor observationswhich are directly derived from raw measurements ameasurement-to-track fusion by performing a Bayes filterupdate step is used. If the sensor observations are alreadypreprocessed and refined inside of the sensor a track-to-trackfusion approach is more appropriate [17].

4) Implementation of observation model: The Bayesianfiltering framework incorporates new observations at time k by aprobabilistic transformation of the predicted (time synchronized)estimate xk, which is described by the target and supportvariables, to the measurement space zk. Hence, this sensor-specific transformation function h(·) needs to be defined.

5) Design and implementation of detection model: Thedetection model is the conditional distribution

P (nz|x,∃x = ∃), (1)

where nz denotes the number of measurements that are expectedfrom an object and x is the state of that object. Simply stated,the detection model gives the probability that an existing objectwith state x generates 0, 1, . . . , n measurements. Typically, the

2This work assumes that a MOT algorithm is used which estimates theexistence of an object along with the dynamic states.

detection model is used to roughly describe the sensor FOV orthe characteristics of the transmission channel.

6) Track management: The initialization of new object/trackhypotheses is tightly coupled with the addressed use-case. Forexample, a typical choice is to propose new tracks based onthe not associated measurements after each sensor’s duty cycle.The same applies to the deletion of already existing tracks.

IV. PLAUSIBILITY CHECKING OF CAMS

The main objective of this work is to demonstrate how multi-sensor data fusion can leverage consistency and plausibilitychecking for perception sensor data. In particular, we areinterested in independently validating the contents of CAMsas delivered by connected vehicles with on-board perceptionssensors. Hence, our approach does not rely on a cooperativereputation system to assess individual V2V entities. Insteadwe aim to derive a significant measure of trustworthinesssolely based on the sensor information available in the hostvehicle. Consequently, the proposed approach only deliversrepresentative results when the V2V entity which should bevalidated is inside of the surveillance area of the on-boardperception sensors. Thus, the system is able to identify threedifferent states of a V2V entity: neutral which indicates thatthe V2V entity is not inside of the sensor’s FOV and thusno plausibility checking can be performed. A valid stateimplies that the sensor information transmitted by the V2Ventity does comply with the observations of the in-vehicleperception sensors. Finally, the invalid state signals that a V2Ventity’s sensor information is not consistent with the on-boardobservations.

In MOT there is already a concept called track scoreof assigning a probabilistic confidence measure to a trackhypotheses which is usually computed over time and considerssensor characteristics [17]. According to different authors,practical implementations use this measure to derive theexistence or relevance of tracks. The principle idea is thatsuccessful consecutive assignments of sensor measurements toa track hypotheses should lead to a high confidence and finallyconfirm a track hypotheses to represent a real object. Dependenton the selected implementation of a MOT, the existence is eitherestimated separately from the dynamic parameters or in anintegrated manner. Basically, the existence measure is expectedto increase in the surveillance area of a sensor by consideringthe sensor characteristic such as the detection probability PD.On the other side, if an expected sensor observation of a trackhypotheses is not available over a period of time, the hypothesesis likely to be not relevant and should be removed.

This concept naturally applies to the generic case of multi-sensor data fusion as well. Here, the existence of a track isupdated by all available sensors which are observing the area thetarget is supposed to be. Again, if a particular sensor observationis not available (considering PD) the existence/relevance ofthe track will be lowered. Consequently, this can be used toperform a plausibility check of sensor observations.

As we are assessing the plausibility of the V2V messages,only the CAMs are used for proposing new track hypothesesinside of the MOT. Afterwards, the local perception system isin charge of validating theses newly created objects. Of course,subsequent CAMs will be used to update the existing trackhypotheses as well. Apparently, this only works as long as theseobject are within the FOV of the front camera. Consequently,

the plausibility checking concept implemented in the paperat hand is restricted to a rather limited FOV. However, thiscould be easily extended by plugging-in addition perceptionssensors such as rear or blind sport radars to achieve a 360degree coverage. In order to correctly implement the existenceestimation it is important to provide an appropriate detectionmodel which considers the true detection and false alarm rateof each particular sensor. Generally speaking, the detectionmodel gives a probability which specifies how likely it is thata particular object hypothesis causes an observation in theconsidered measurement domain. The detection model usedto describe the behavior of connected vehicles is different toconventional perception sensors: for example, a classical FOVdoes not exists. At best, an approximation can be providedby an assumed maximum communication range drange whichdescribes the radius of a circle centered at the position of thecommunication antenna at the ego vehicle. In section VI-A, anempirical detection model which considers the observed packeterror rates in the conducted test drives is presented.

Finally it is worth mentioning that with this approach twotypes of inconsistencies can be detected: First, an artificiallyinjected ghost vehicle which is not really present on the roadas shown in [5] can be detected. Second, temporary sensorfailures can be detected. For example, the low-cost GPS devicesusually used to provide the absolute position information forthe CAMs might be subject to multipath as present in urbanareas. This temporary disturbance may significantly offset theposition inside of the CAMs.

V. IMPLEMENTATION

The paper at hand follows a similar approach as recentlypresented in [18] where a system design for multi-sensor datafusion with a flexible configuration is proposed. However, theaforementioned system does not consider sensor input fromCAMs and thus has not to deal with the challenge of integratingCAMs with absolute position measurements.

A. State Space QuantitiesA fundamental fist step is the selection and definition

of the state space quantities, that is the target and supportvariables, which should be estimated for each object by theMOT algorithm. We propose to use a Constant Velocity (CV)state space [19] which is augmented with the dimensions (widthw and length l) of the object, thus describing a planar box:

x = (x y ϑ v | w l)ᵀ, (2)

where x and y denote the relative position of the tracked objectwith respect to the ego vehicle frame (which originates at themiddle of the rear axle) and ϑ represents the heading deviationof the track with respect to the ego heading. Moreover, the statespace contains the absolute magnitude of the track’s velocityindicated by v. For the sake of simplicity the distance betweenthe vehicle origin and the rear bumper is assumed a fixedvalue of -0.5 m. After defining the state space quantities, adynamic model which describes the evolution of a track overtime needs to be selected. For the dynamic parameters the CVvehicular motion model as described in [19] can be directly usedwhile the dimensions are assumed to be constant. Finally, eachtrack is hybridized with an additional binary random variable∃x = [∃, ∃] which represents the existence of the track givenits associated noisy sensor observations. In this work, ∃x is

estimated based on sequential probability ratio testing (SPRT)as explained in [17]. Basically, this represents an independentdiscrete Bayes filter which estimates the existence based onassociated and missed sensor observations.

B. MobilEye Sensor1) Observation Model: The MobilEye camera is an after-

market system which implements an acoustic forward collisionwarning. Therefore, the detected objects are described in aparticular coordinate frame which is appropriate to fulfill theanticipated function. The system is calibrated in such a way,that independent of the particular mounting position of thesensor, each vehicle detection mz describes the distance r andangle ϕ from the rear bumper of the detected object to thefront bumper of the ego vehicle. Moreover, a range rate ris included. It is worth mentioning that the provided widthobservation was neglected, thus, the implemented observationmodel for the MobilEye sensor is rather similar to a classicalradar observation model:

mz = mh(x) =

r

ϕ

r

=

√

(x− xr)2 + (y − yr)2

arctan y−yr

x−xr

v · cos(ϑ− ϕ)− ev · cos(ϕ)

,

(3)

where xr and yr are the known offsets between the frontbumper and the vehicle origin of the ego vehicle. It should benoted that the variable ev which denotes the velocity of theego vehicle is explicitly available inside of the measurementmodel and not taken from the state vector.

2) Detection Model: According to [10] the MobilEyesystem has a sensor FOV of 47 ◦ with a detection probabilityof PD = 0.995. However, for the used test vehicle this angularrange was not symmetric. By empirical tests the minimum andmaximum angle have been set to -30 ◦ and 17 ◦, respectively.Moreover, the minimum and maximum lateral detection rangewas set to 10 m and 50 m. As the MobilEye system typicallyonly emits one detection per real object, the detection modelwas implemented for nz = 1 only (observation inside FOV)and nz = 0 (outside of FOV or occluded).

C. Incorporation of CAM Detections1) Transformation of CAMs to Relative Measurements: Due

to the fact that the MOT is estimating the objects in relativecoordinates with respect to the coordinate frame of the egovehicle as proposed in [18], the absolute position measurementscontained in the CAMs cannot be directly incorporate by thetracker with a classical observation model. Therefore, an addi-tional transformation step is introduced which transforms theCAMs to relative measurements which are compatible with thepreviously defined state space of the tracker. In order to performthis transformation the position and heading of the ego vehicleneeds to be known in absolute coordinates (e.g. UTM). In ourwork we are taking these values xego = (xego yego ϑego)

ᵀ

at time k from a high-frequent GNSS/IMU-based groundtruth sensor which is installed at the ego vehicle. The spatialalignment of CAMs at time k, that is, the transformation ofthe absolute position from the remote vehicle’s CAM to arelative distance vector dk = (∆x ∆y ∆ϑ)

ᵀ, is based onthe idea proposed in [7]. Basically, this is an adoption ofthe absolute position differencing (APD) as explained in [20]that accounts for the measurement uncertainties. Therefore, a

temporary joint state space xk which comprises xego of theego vehicle and the position and heading from the receivedCAM xcam = (xcam ycam ϑcam)

ᵀ in Cartesian coordinates(e.g. UTM) is constructed by concatenation:

xk =

(xego

xcam

)Pk =

(Pego Cego,cam

Cego,cam Pcam

). (4)

Apparently, the mean and covariance parameters inside of(4) can be directly taken from the individual densities ofthe ego and remote vehicle. However, the correlation Cego,cambetween both estimates, which are typically based on the GNSSobservations at each vehicle, is generally not known and istherefore considered as a process parameter. In [7], a ratherpessimistic choice with Cego,cam = 0 is made. Based on thisconcatenated state xk, a transformation function f(·) whichprojects the absolute CAM position and heading to the distancevector dk ∼ N (µd,Pd) is given by f : xk 7→ dk:

dk =

∆x

∆y

∆ϑ

= f(xk) (5)

=

sinϑego(xcam − xego) + cosϑego(ycam − yego)

− cosϑego(xcam − xego)− sinϑego(ycam − yego)

ϑego − ϑcam

(6)

Although, both states are represented by Normal distributions,the transformation function (6) is non-linear. Hence, thecomputation of the PDF for dk can be performed by anunscented transformation in a consistent way which correctlyconsiders the uncertainties of the 2D positions and the heading[21]. In [22], typical accuracies or these relative distance vectors,which are calculated from two absolute GPS fixes, are given.

2) Observation Model: After the absolute position andheading measurements from the CAM have been transformedinto the local coordinate frame of the ego vehicle as describedin the previous section, the observation model of the relativeCAM can be formulated in a straightforward manner:

camz =

∆x

∆y

∆ϑ

v

w

l

= mh(x) =

x

y

ϑ

v

w

l

. (7)

Apparently, this one-to-one relation of the state and observationspace is only true as long as the position inside of the CAMsis related to the same vehicle origin (middle of rear axle). Forthis work, this was ensured by configuration of the utilizedITS-G5 devices.

3) Detection Model: Compared to a typical on-boardsensor, the communication range of ITS-G5 devices is ratherlarge and can be approximately described by a circular areaaround the ego vehicle with the radius dmax [14]. However, incontrast to the MobilEye sensor, there are no exactly defineddetection areas with a fixed detection probability PD. Instead,the detection probability is dependent on several parameterssuch as the communication distance d and environmentalconditions. Consequently, the actual dmax might be smallerin high congested urban areas than on rural motorways. Inthis work the test drives where conducted under constant

sub-urban conditions. Thus, we will use a fixed PDF whichdescribes the reception probability of a CAM dependent onthe communication range. The particular statistics of this PDFdescribing PD(d) for the conducted test drives are derived insection VI-A from empirical data.

D. Time Synchronization & Fusion SchemaDue to the asymmetric sensor latencies between the Mo-

bilEye system and CAMs, the multi-sensor data fusion willreceive the individual observation out of order regarding theirtime of validity—an OOSM scenario is present. In [7] theOOSM problem was mitigated by predicting the object statecontained in the CAMs at reception time tr to time of themost recent state estimate tk inside of the tracker. The CAMprediction was implemented by using a constant turn rate andacceleration (CTRA) motion model and afterwards incorporatedinto the track list by track-to-track fusion. Unfortunately, sucha temporal alignment strategy cannot be applied in this workas the CAMs of the used ITS-G5 devices do not containthe momentary acceleration and turn rate. Thus, a forwardprediction of the CAM solely based on the position, the headingand the velocity for a time span of up to 800 ms will generaterather inaccurate results [23]. Therefore, we decided to use abuffering strategy as explained in [24] to generate first baselineresults which can be used later on for further evaluations andcomparisons. Due to the fact that the used ITS-G5 devicesderive the vehicle dynamics from an attached low-cost GPSdevice and do not apply any filtering refinement, the CAMsare incorporated by measurement-to-track fusion.

VI. QUANTITATIVE RESULTS

In this section the results of the empirical evaluation arepresented. The three utilized test vehicles are equipped withdifferent sensors for vision, range, high-accurate localizationand communication. In [25], a comprehensive descriptionof these prototyping platform is available. In addition, allvehicles have been equipped with a communication devicebased on the Atheros AR5414A-B2B chipset with customfirmware to support communication on the ITS-G5 controlchannel. On top of this device we ran an ETSI compliantimplementation of GeoNetworking, Basic Transport Protocol(BTP) and Cooperative Awareness Basic Service (CA BS) fromHitachi Solutions. This implementation was used as referenceimplementation for the ETSI conformance tests and was one ofthe best scoring implementations during all ETSI PlugtestsTM todate.

A. Packet Delivery RatioThe packet delivery ratio (PDR) is the ratio of received

packets to the number of sent packets. Let senti be the numberof sent packets by vehicle i and rcvdji the number of receivedpackets from vehicle i by vehicle j. The packet delivery ratioat vehicle j for vehicle i is then calculated as rcvdji

senti. Figure 1

shows the PDR as a function of the inter-vehicle distance.The results show as expected a decreasing PDR for increasingdistances. Our results are however limited to 50 m because thevehicles were driving too close in our tests to collect statisticallysignificant results for any larger distance, even though we haveseen transmissions and receptions at larger distances. Our resultsshow a steeper decrease of the PDR compared to [12]. Thisresult is likely caused by the used chipset which was originally

0 20 40 60 8040

60

80

100

Distance [m]

PD

R [

%]

Packet delivery ratio per distance

PDR Carai 2

Fig. 1. Packet Delivery Ratio (PDR) of Carai2. Each distance bin is 10meters with the mean plotted as a data point in the center of each bin. Theerror bars indicate the one standard deviation around the mean. For statisticalrelevance only bins with 40 or more measurements are considered.

Fig. 2. The target vehicle is constantly tracked. After leaving the field ofview of the MobilEye sensor (highlighted by red area), the CAMs are solelyused to update the track. However, plausibility checking of the CAMs is notpossible anymore as the on-board sensor cannot assess the track.

not designed for high-speed scenarios and shadow fading causedby the vehicle roof bars. Finally, this data was use to derive thecharacteristics of the reliability of the ITS-G5 wireless channelby giving a detection probability PD that is dependent on thecommunication range.

B. Assessment of Valid SituationsIn a first step, the proposed MOT approach for plausibility

checking is evaluated for a scenario which contains consistentmeasurement data of both sensor systems. The target vehicle,which is observable by both sensors at the beginning, is drivingin front of the ego vehicle. After a period of a straight motionthe target vehicle turns left and thus leaves the FOV of theMobilEye system. However, the CAMs are still received as thecommunication range is rather large compared to the on-boardperception sensor. In Fig. 2, the scenario is depicted after theleft turn maneuver of the leading vehicle. In order to comparethe influence of the multi-sensor data fusion we first evaluatedthe tracking results which can be achieved with using CAMsonly. Figure 3 illustrates the existence estimate of the V2Ventity. Every time a new CAM is received (indicated by thered circles) and successfully assigned to an existing track the

0 2 4 6 80

5

10

15

20

Time [s]

Exis

tenc

e [L

ogO

dds]

ExistenceCAM

Fig. 3. Existence for tracked vehicle when using CAMs only. The largestobserved existence is app. 15 log-odds which corresponds to a neutralplausibility checking result.

0 2 4 6 80

5

10

15

20

Time [s]

Exis

tenc

e [L

ogO

dds]

ExistenceCAMMobilEye

Fig. 4. Existence for tracked vehicle when using both sensors. Apparently, anexistence estimate of 16.5 log-odds is reached. In our work this correspondsto a regular behavior of the sensor under plausibility checking.

existence estimate is updated and increased to a maximumvalue of app. 15 log-odds. Apparently, in between of the CAMupdates the existence is lowered according to the implementedpersistence model. Thus, a decreasing existence does not meanthat a track is invalid, instead it just indicates that we have notobserved the target. It is worth mentioning that in the first partof the sequence (between 0 s < t < 2.2 s) and starting fromt > 6s the frequency of received CAMs, and thus also theexistence, is lower compared to the middle part. This can beexplained by the decreasing reliability of the wireless channelas already stated in section VI-A which directly depends onthe distance between both vehicles. It can be summarized thatfor this use-case the V2V entity has a rather high existencevalue in the range of 15 log-odds but is still only confirmed asa neutral object as the leading vehicle is solely observed by theV2X sensor. In addition, Fig. 4, illustrates the same sequencebut now with a multi-sensor data fusion among CAMs andthe MobilEye system (vision detections are indicated by greencrosses). Apparently, the MobilEye systems only perceives theleading vehicle for t < 3.8 s which is before the left turnmaneuver. As both sensors deliver consistent observations themax. estimated existence now reaches a higher value of app.16.5 log-odds. Thus, the CAMs of the V2V entity can beconfirmed as valid. Starting from t > 3.8 s, the leading vehicleleaves the FOV of the MobilEye system and the existenceestimate reverts to 15 log-odds again. In terms of plausibilitychecking this corresponds to a neutral classification result.

R

A

Acam

Fig. 5. Evaluation scenario with attacker who injects a ghost-vehicle.

0 2 4 6 8−20

−10

0

10

20

Time [s]

Exis

tenc

e [L

ogO

dds]

ExistenceSpoofed CAMMobilEye

Fig. 6. Estimated existence for tracked vehicle with spoofed CAMs andMobilEye on-board plausibility checking. When the ghost-vehicle A enters theFOV of the MobilEye system at t = 3.8 s the plausibility checking instantlyclassifies the V2V entity as invalid which is indicated by a low log-odds value.

C. Assessment of Invalid SituationsIn the evaluated scenario, which is similar as in [5], we

assume an attacker which has tampered a system and is ableto send CAMs with valid certificates and signatures whileyet being able to control the message content. Without anyplausibility check, a receiver could wrongly consider thesemessages as valid and as such, an attacker is able to generatea number of attacks, including the generation of ghost vehiclesby changing the location data of broadcasted CAMs. Suchsituation is illustrated in figure 5. An attacker A is positionedalongside the road and is broadcasting CAMs with forgedposition ACAM. Vehicle R is driving on the road and receivesthe broadcasted CAMs of attacker A. In addition, vehicle Rdetects objects in front of the vehicle using its built-in camera.In this situation, the camera does not detect vehicle A at positionACAM and for that reasons the plausibility check of Section Vshould classify the CAM data as implausible. In Fig. 6 theplausibility checking result for the spoofing CAM use-case isillustrated. At the beginning (t < 3.8 s) the faked CAM pretendsthat the injected vehicle is outside of the FOV of the on-boardperception system. In fact, it is approaching from behind andstarting to overtake the host vehicle. For this first part of thissequence the proposed system classifies the V2V entity asneutral similar to VI-B. However, starting from t > 3.8 s theinjected vehicle performs a cut-in maneuver and thus entersthe FOV of the MobilEye system. Instantly, the MOT starts toestimate a low existence value for this track as the object cannotbe confirmed by the MobilEye system which does not detect acorresponding vehicle. Apparently, the track’s existence is stillupdated by the faked CAMs but never reaches similar valuesas observed in section VI-B—the V2V entity is classified toinvalid. Consequently, the ghost-vehicle is reliably detectedand can be marked as non trustworthy.

VII. CONCLUSION & OUTLOOK

In this paper, we evaluated a MOT-based approach forplausibility checking of CAMs as provided by ITS-G5. There-fore, we applied a multi-sensor data fusion among a MobilEyeon-board perception sensor and V2V communication whichis typically used to achieve an extended perception. Bothsensors are considered off-the-shelf hardware and have beenfused with a MOT algorithm as provided by BASELABSCreate. We demonstrated that the existence estimate which isnaturally included in MOT applications can be directly used toinfer the plausibility of CAMs. Therefore, we evaluated threedistinct use-cases which comprise plausible and implausiblescenarios. It has been proven that the estimated existence valuesclearly correspond to the three used sensors states neutral, validand invalid. Besides demonstrating that MOTs can leverageplausibility checking of sensors, we faced several challengesin multi-sensor data fusion: for example, the particular sensors’behavior were characterized by dedicated sensor models whichdescribe the FOV and the detection probability. For the V2Vsystem an empirical detection probability that is dependent onthe communication range has been deduced. One shortcomingof the presented system is the rather limited overlapping FOVof both sensors which leads to a rather narrow surveillance areawhere plausibility checking can be performed. Nevertheless,the proposed system correctly classifies V2V entities outsideof that area as neutral. By adding more perception sensorsthis limitation can be removed without any need to modifythe plausibility checking strategy. Moreover, the paper at handaimed to evaluate the capabilities of MOT for plausibilitychecking in general. Therefore, we deliberately mitigated theissue of OOSM by using a buffering strategy. For a plausibilitychecking which has to operate under real-time conditions,another OOSM strategy—perhaps similar to [7]—is moreappropriate. Finally, it is worth mentioning that the APD methodused to transform the absolute position information of theCAMs to a relative observation as required by the MOT, dependto a big extend on the number of common satellites betweenboth vehicles. In our work we took the observed statistics forthe relative measurements from [22]. For further work, thisstatistics should be derived online.

ACKNOWLEDGMENT

This work was supported by the European Commissionunder AutoNet2030, a collaborative project part of the SeventhFramework Programme for research, technological developmentand demonstration (Grand Agreement no 610542). The testvehicles used to collect measurement data were provided byTechnische Universitat Chemnitz.

REFERENCES

[1] N. Mattern and R. Schubert, “A Hybrid Approach for ADAS AlgorithmDevelopment - From High-Level Prototypes to ECUs,” in Proceedingsof the 10th ITS European Congress, Helsinki, Finland, 2014.

[2] R. Schubert, M. Obst, E. Richter, and N. Mattern, “Software-supporteddesign of probabilistic sensor processing techniques for vehicle localiza-tion,” in ITS European Congress 2013, 2013.

[3] P. Szczurek, B. Xu, O. Wolfson, and J. Lin, “Estimating relevance for theemergency electronic brake light application,” ITS, IEEE Transactionson, vol. 13, no. 4, pp. 1638–1656, Dec 2012.

[4] S. Lefevre, C. Laugier, and J. Ibanez-Guzman, “Risk assessment atroad intersections: Comparing intention and expectation,” in IntelligentVehicles Symposium (IV), 2012 IEEE. IEEE, 2012, pp. 165–171.

[5] N. Bismeyer, S. Mauthofer, K. M. Bayarou, and F. Kargl, “Assessmentof node trustworthiness in vanets using data plausibility checks withparticle filters,” in Vehicular Networking Conference (VNC), 2012 IEEE.IEEE, 2012, pp. 78–85.

[6] R. K. Schmidt, T. Leinmuller, E. Schoch, A. Held, and G. Schafer,“Vehicle behavior analysis to enhance security in vanets,” in Proceedingsof the 4th IEEE Vehicle-to-Vehicle Communications Workshop, 2008.

[7] A. Rauch, F. Klanner, R. Rasshofer, and K. Dietmayer, “Car2x-basedperception in a high-level fusion architecture for cooperative perceptionsystems,” in Intelligent Vehicles Symposium (IV), 2012 IEEE. IEEE,2012, pp. 270–275.

[8] G. Thomaidis, K. Vassilis, P. Lytrivis, M. Tsogas, G. Karaseitanidis,and A. Amditis, “Target tracking and fusion in vehicular networks,”in Intelligent Vehicles Symposium (IV), 2011 IEEE. IEEE, 2011, pp.1080–1085.

[9] M. Rockl, T. Strang, and M. Kranz, “V2V communications in automotivemulti-sensor multi-target tracking,” in Vehicular Technology Conference,2008. VTC 2008-Fall. IEEE 68th. IEEE, 2008, pp. 1–5.

[10] E. Dagan, O. Mano, G. P. Stein, and A. Shashua, “Forward collisionwarning with a single camera,” in Intelligent Vehicles Symposium, 2004IEEE. IEEE, 2004, pp. 37–42.

[11] F. A. Teixeira, V. F. e Silva, J. L. Leoni, D. F. Macedo, and J. M.Nogueira, “Vehicular networks using the {IEEE} 802.11p standard: Anexperimental analysis,” Vehicular Communications, vol. 1, no. 2, pp. 91– 96, 2014.

[12] P. d’Orey and M. Boban, “Empirical Evaluation of Cooperative Aware-ness in Vehicular Communications,” in Proc IEEE Vehicular TechnologyConf. - VTC-Fall. IEEE, 2014, pp. 1–p.

[13] B. Khaleghi, A. Khamis, F. O. Karray, and S. N. Razavi, “Multisensordata fusion: A review of the state-of-the-art,” Information Fusion, vol. 14,no. 1, pp. 28–44, 2013.

[14] F. Martelli, M. Elena Renda, G. Resta, and P. Santi, “A measurement-based study of beaconing performance in ieee 802.11 p vehicularnetworks,” in INFOCOM, 2012 Proceedings IEEE. IEEE, 2012, pp.1503–1511.

[15] G. Gavilanes, M. Reineri, D. Brevi, R. Scopigno, M. Gallo, M. Pannozzo,S. Bruni, and D. Zamberlan, “Comparative characterization of fourantennas for vanets by on-field measurements,” in COMCAS, 2013 IEEEInternational Conference on. IEEE, 2013, pp. 1–5.

[16] S. Challa, Fundamentals of object tracking. Cambridge UniversityPress, 2011.

[17] S. Blackman and R. Popoli, “Design and analysis of modern trackingsystems,” Boston, MA: Artech House, 1999.

[18] F. Sandblom and J. Sorstedt, “Sensor data fusion for multiple configura-tions,” in Godkand for publicering pa Intelligent Vehicles Symposium(IV), 2014, pp. 8–11.

[19] R. Schubert, C. Adam, M. Obst, N. Mattern, V. Leonhardt, andG. Wanielik, “Empirical evaluation of vehicular models for ego motionestimation,” in Intelligent Vehicles Symposium (IV), 2011 IEEE. IEEE,2011, pp. 534–539.

[20] F. de Ponte Muller, E. M. Diaz, B. Kloiber, and T. Strang, “Bayesiancooperative relative vehicle positioning using pseudorange differences,”in Position, Location and Navigation Symposium-PLANS 2014, 2014IEEE/ION. IEEE, 2014, pp. 434–444.

[21] S. J. Julier, “The scaled unscented transformation,” in American ControlConference, 2002. Proceedings of the 2002, vol. 6. IEEE, 2002, pp.4555–4559.

[22] V. Elisson and G. Gassler, “Low cost relative gnss positioning with imuintegration,” 2014.

[23] S. Lefevre, C. Sun, R. Bajcsy, and C. Laugier, “Comparison ofparametric and non-parametric approaches for vehicle speed prediction,”in American Control Conference 2014. IEEE, 2014, pp. 3494–3499.

[24] M. M. Muntzinger, M. Aeberhard, S. Zuther, M. Mahlisch, M. Schmid,J. Dickmann, and K. Dietmayer, “Reliable automotive pre-crash systemwith out-of-sequence measurement processing,” in Intelligent VehiclesSymposium (IV), 2010 IEEE. IEEE, 2010, pp. 1022–1027.

[25] R. Schubert, E. Richter, N. Mattern, P. Lindner, and G. Wanielik,Advanced microsystems for automotive applications 2010. Springer,2010, ch. A Concept Vehicle for Rapid Prototyping Of Advanced DriverAssistance Systems, pp. 211–219.