data fusion framework: concurrent architecture for real ... fusion... · data fusion framework:...

424 Int. J. Information and Communication Technology, Vol. 1, Nos. 3/4, 2008

Copyright © 2008 Inderscience Enterprises Ltd.

Data Fusion Framework: concurrent architecture for real-time processing

Stefano Laoreti*, Davide Renzi*, Raffaele Parisi and Aurelio Uncini Department INFOCOM, University of Rome ‘SAPIENZA’, Via Eudossiana, 18 – 00184 Rome, Italy E-mail: [email protected] E-mail: [email protected] E-mail: [email protected] E-mail: [email protected] *Corresponding authors

Abstract: In the last few decades, a new way of interpreting multi-sensor array processing techniques and artificial intelligence systems has been introduced. This theory focuses on interactions and fusion between heterogeneous/homogeneous information at different layers in order to guarantee data completeness and robust resolution of hard-to-solve problems. Our research investigates this theory, well known as Data Fusion (DF). In this paper, we present an operating framework that permits DF during the elaboration flow at different levels of computation. We focused our attention on several key aspects of real-time implementation.

Keywords: data fusion; artificial intelligent system; multi-thread; software framework; multi-layer processing.

Reference to this paper should be made as follows: Laoreti, S., Renzi, D., Parisi, R. and Uncini, A. (2008) ‘Data Fusion Framework: concurrent architecture for real-time processing’, Int. J. Information and Communication Technology, Vol. 1, Nos. 3/4, pp.424–436.

Biographical notes: Stefano Laoreti received the ‘Laurea’ Degree in Computer Engineering and currently he is PhD student in Information and Communication Engineering at University of Rome ‘La Sapienza’, Rome, Italy. His research interests are in the areas of digital signal processing, optimisation theory, array processing and multi-sensor data fusion.

Davide Renzi received the ‘Laurea’ Degree in Computer Engineering and currently he is PhD student in Information and Communication Engineering at University of Rome ‘La Sapienza’, Rome, Italy. His research interests are in the areas of neural networks and signal processing, array processing and multi-sensor data fusion.

Raffaele Parisi received the ‘Laurea’ Degree in Electrical Engineering with Honours and the PhD Degree in Information and Communication Engineering from the University of Rome ‘La Sapienza’, Rome, Italy, in 1991 and 1995, respectively. In 1994 and 1999, he was a visiting student and a visiting researcher with the Department of Electrical and Computer Engineering, University of California at San Diego, La Jolla. Since 1996, he has been with the University of Rome ‘La Sapienza’, where he is currently

Data Fusion Framework: concurrent architecture for real-time processing 425

an Associate Professor with the Information and Communication (INFOCOM) Department. His research interests are in the areas of neural networks, digital signal processing, optimisation theory, and array processing.

Aurelio Uncini received the ‘Laurea’ Degree in Electronic Engineering from the University of Ancona, Italy, on 1983 and the PhD Degree in Electrical Engineering in 1994 from University of Bologna, Italy. At present time he is Professor at the Department of the University of Rome ‘La Sapienza’, Italy, where he is teaching circuits theory, adaptive algorithm for signal processing and digital audio processing. He his author of several papers in the field of circuits theory, optimisation algorithms for circuits design, neural networks and signal processing. His present research interests include also adaptive filters, blind signal processing, adaptive audio and array processing, machine learning for signal processing and multi-sensors data fusion.

1 Introduction

Nowadays, array techniques and intelligent systems are frequently used in various contexts to monitor subjects of interest by using homogeneous/heterogeneous sensors. These systems often use sensor redundancy to produce robust and more informative data that can be processed with ad hoc algorithms to retrieve a unique information flow (Hall and Llinas, 1997, 2001; Estrin et al., 2001; Li et al., 2002; Akita, 2002).

This method is similar to the biological inference capacity: human brain elaborates concurrent information captured by the five (heterogeneous) senses from the environment.

DF can be interpreted as a new way of combining and processing information through which data collected by a large set of heterogeneous/homogeneous sensors are fused, mixed and elaborated at different levels to guarantee high parallelisation and inference comparison.

The concept of DF is simple to describe, but its exact meaning and its implementation vary from one scientist to another: in 1991, the US Department of Defense formalised multi-sensor DF approach as

“a multilevel process that combines data from multiple sensors, with related information stored in databases, to solve automatic detection, association, correlation, estimation and combination of data to achieve refined position and identity estimates, timely assessments of situations and threats.”

The process is characterised by continuous refinements of the results, thanks to the additional information retrieved by sensors (Hall and Linas, 1997; White, 1987). This kind of approach achieves improved accuracy when compared with single sensor system. Many other definitions can be found in literature such as Hall and Llinas (1997), Klein (1993) and Mangolini (1994). A discussion on these can be found in Wald (1998).

In this paper, we propose a programming structure formalisation to supply a useful framework for DF algorithms based on the definitions and terms proposed by Wald (1999).

The first section is a short introduction on DF historical background. Section 2 is divided into two parts: the first one shows the importance of defining a common framework in developing DF applications and introduces basic definitions for elements

426 S. Laoreti et al.

that will be used in the rest of the paper; the second section describes C++ language implementation of Data Fusion Framework (DFF) and analyses in depth basic procedures. Sections 3 and 4 deal with multi-thread programming context and synchrony issues, respectively. The last section is dedicated to future steps and final considerations on the proposed DFF.

2 Data Fusion Framework (DFF)

A DF system works as data collector that operates on numeric flow (symbolic flow can be represented as numeric flow too). It acquires, exchanges and elaborates data that come from sensors, algorithms or databases. At each step, elaboration output can be proposed to other processing block and so on (see Figures 1–3).

A system architecture that maintains DF algorithms and interlayer communications was defined in Carvalho et al. (2003) and Besada et al. (2002).

The proposed generic DF architecture explained in these papers can be applied in network-based distributed systems or conventional stand-alone systems. Important DF characteristics can be highlighted: each single block of the system can be inserted or excluded without interfering with the entire system; the environment changes with time; each component can be used again depending on changes in the environment, according to it.

Thus, before developing a DF application, we must divide the main application in asynchronous basic operating blocks as pointed out by Joint Directors of Laboratories (JDL). Each block can be modelled as a time-dependent processing node that acquires inputs, elaborates data and sends out its results.

Figure 1 Example of fusion at data level

Source: Hall and Linad (2001)


Figure 2 Example of fusion at feature level


Figure 3 Example of fusion at identity level


Proposals of DF architectures based on this concept can be found in literature (Carvalho et al., 2003; Bossé et al., 2000; Besada et al., 2002; Hall and Kasmala, 1996). We describe a real implementation of a DFF that takes these as guidelines.

A generic DFF must claim an elevated abstraction capability to permit implementation of various DF applications. Obviously DFF total performances must be evaluated in a new way: algorithm’s quickness against error ratio (external performance), multi-tasking operability against fusion layers (internal performance). Nowadays, there is no way to connect isolated/independent systems with high external performance without a loss in internal performance. For example, we can use different systems (each based on different sensor typologies) such as distinct modular elements, but we cannot apply any type of fusion at any level because of lack of common context; on the other hand, we cannot fuse each system output in a serial way without a loss in execution time owing to the lack of concurrent elaboration.


In our research, we focus on the possibility to develop a framework that can be used as a common platform to develop and benchmark concurrent algorithms that are based on DF.

We focus our attention on different key points: abstraction (sensor, algorithms and fusing procedure must be considered as black box), data independence (data flow can be seen as an input/output raw flow, even if its nature is unknown), task independence (each single element can execute its own task without interfering with other elements), parallelisation (each block executes its task in a concurrent context), synchronisation (each block executes its task in a well-defined temporal slot), modularity (the system can be expanded at any time replacing blocks, adding blocks or increasing the number of elements).

2.1 Architecture definition

The proposed DFF is used to define a canonical methodology that allows one to plan a generic DF system proceeding block-by-block to allow quick software coding. Coded software must allow (depending on CPU calculus capabilities) one to execute real-time or offline applications in the same way.

Proposed DFF is built on three different macro-elements: Data (D), Processing Node (P), Device Block (B). Each element has a specific role and well-defined rules. Connections between elements define an oriented-graph. Thus, DF application can be clearly formalised by a simple representation (Figure 4).

Figure 4 Processing node architecture

Defined elements can be easily translated with any object-oriented language: each element corresponds to an object. To maintain elevated abstraction without any loss of generality, we defined objects as a template that can be extended and adapted to a wide range of situations. For this reason, each object has fixed attributes and extendable attributes.


2.1.1 Data element

Data element is the most important because DF system architecture is built on information flows. In Carvalho et al. (2003), two different data types have been declared: data, variable. Data defines a low-level information type, typically raw numeric data coming out from sensors. Variable defines a high-level information type, typically the result of some elaboration on data.

In our work, we drop this distinction in order to maintain an elevated independence of data from their specific content. We consider each numeric flow as a data element.

Each data flow is sampled, thus it depends on time. We use a generic structure to manage different data flows at different sampling frequencies and sample resolutions: stream – frame – sample.

The stream is a collection of consecutive samples acquired in precise time slots (see Figure 5). Obviously, a stream can be also the output of a sensor; because of hardware implementation, data flow does not consist of bursts. Data bursts have well-defined dimension and frequency. Therefore, the stream consists of consecutive frames (bursts) that consist of consecutive samples in their turn.

This structure can be adapted to all kinds of data elements. Raw data, features, elaboration results, etc., can be modelled as a numeric time-dependent flow just varying sample resolution and frame length.

Figure 5 Stream structure

2.1.1.1 Data stream attributes

Referring to Figure 6, we have:

• Stream. Frame Length – number of frames in the stream (maintained in the memory heap).

• Stream. Sample Rate – sample per second from which data are collected.

• Stream. Channels – number of tracks (i.e., audio channels acquired through the microphonics array).

• Frame. Index – self-increment numeric value to identify univocally the frame into the stream.

• Frame. Sample Length – number of samples in the frame (maintained in the memory heap). Each frame in the same stream can have different user-defined lengths.

• Frame. Timestamp – absolute time slot to which first sample within frame is referred. Subsequent frames can retrieve their timestamp by calculating their offset from this.

• Sample. Data Type – definition of sample types (numeric, string, etc.).


Figure 6 Data stream model using object-oriented paradigm

2.1.2 Processing Node element

Processing Node element is the only active element: it receives n data elements, applies some kind of elaboration and retrieves m data elements (where n ≥ 0 can be different from m ≥ 0). In Carvalho et al. (2003), authors make distinctions between output data types elaborated by some process: inputs (data, variable) become a mixture of outputs when they are a fusion of different data types. In our DFF, we do not differentiate them. We pointed our attention to the dynamic evolution of each data flow instead of to their contents. Fusion levels described in Carvalho et al. (2003) and Hall and Linas (2001) can be simply derived by inspecting the dependence relationships between outputs of processing nodes, in other words, by analysing cascade relationships in the hierarchy tree. If a processing node stays on an upper level, we can claim that the lower processing node applies a different kind of fusion on a different level (Figure 7).

Figure 7 Example of element interconnection and dependencies

A Processing node can implement a wide range of tasks: basic functions (FFT, CC, etc.) or more complex ones (smart sensor, neural network, etc.).

Beyond these examples, owing to the modularity of the framework, it is possible to collect different processing nodes in another logical complex one, performing a multi-part type node. In this manner, the construction of a system is easy, reusable and modular. Mathematical formulation of a processing node can be defined as an operator


that maps data from n-dimensional to another m-dimensional domain, where n can differ from m.

To guarantee data integrity during the execution time, different processing elements cannot write their results on the same data element. Let d0, d1, …, dm and c0, c1, …, cq represent data element outputs (with m and q generally not equal), and i0, i1, …, in and y0, y1, …, yp represent data element inputs (with n and p generally not equal). We define a constraint:

0 1 0 1

0 1 0 1

[ , , , ] ([ , , , ])[ , , , ] ([ , , , ])

m n

q p

d d d P i i ic c c P y y y

==

… …… …

(1.1)

where dk ≠ cj when k = 0, 1, …, m, j = 0, 1, …, q and k ≠ j must be verified.

2.1.3 Device Block element

The Device Block element has been defined to manage DFF communication with the external world. Device Block element is a particular Processing Node element that has no data elements in input or output.

Thanks to this kind of element, it is possible to model elements that acquire data directly from external input hardware (such as sensors or saved data files that store previous measures) or elements that show results on human-friendly devices (such as monitor, printer, etc.). The latter sort is close to data source defined in Carvalho et al. (2003), and our Device Block element can be intended just as an input/output device interface.

To avoid the programming of a specific device, we encourage the use of a standard interface drivers such as Microsoft DirectX®, Steinberg ASIO, OpenGL, and so on.

2.2 Object-Oriented (OO) implementation

Before translating DFF elements into some coded structure, a few working hypothesis must be made. First of all, each element must be replicable and suited to different working contexts; second, processing node elements must execute their tasks without interfering with other functioning elements; third, different processing node elements can share the same data at the same time. These hypotheses claim to use Object-Oriented (OO) programming paradigm (modularity, abstraction, reusability, etc.).

Data, Processing Node and Device Block elements are coded by using C++ class construct.

As said before, it is easy to understand that DFF is based on multi-thread programming paradigm, and therefore, every active element such as Device Block and Processing Node functions like this.

Each process node is coded with its own thread; each thread has its priority in the global process.

Calculation power is shared among all active threads according to the scheduling strategies of the Operating System (OS), so each process node has its time slice to process data. Time slice of a thread is subject to its execution priority, and each one may differ from another. Every process node behaves the same, as depicted by the State Machine Graph (Figure 8).


Figure 8 State machine for processing node

After the main program has set up variable and data structures, all threads are created. The starting state of a node is the WAITING FOR START. Below are described all states and their behaviour.

This kind of choice makes methodologies modular, and leads to a decomposition of greater problems into many independent smaller problems.

This implementation makes it easier to use all of the calculation power of multi-processor computer; consequently, it is possible to increase the number of sensors without losing project specification.

On the other way, it is important to avoid making many working threads because this requires time owing to several context switches operated by OS. To avoid this side–effect, it is important to find the best trade-off making Process Node. A guideline is to make a Node when the data produced by it can be used from at least other two different Nodes. This implementation makes it easier to build up a complex DF system without creating a big program with many if–then–else conditions. So, the program has many flows and it is easier to understand and manage.

Device Block is also implemented by a C++ class. The object provides functionality of specific device: parameters set-up, hardware instantiation and resource release. After all settings have been prepared (using a specific method), a thread that manages data transfer between device buffer and Stream is launched.

Three different device blocks can be used: for input, for output and input/output. If an input Device Block is used, it is also responsible for data Stream management

according to parameters. To guarantee system correctness and functionality, it is very important to pay

attention to the data management. In particular, when a processing node starts to read a frame, it marks the frame as ‘in use’. If that frame remains in use for too much time, it may happen that, after subsequent insertions, it becomes the last, and the data manager cannot delete it because it is marked as in use. In that case, there is no guarantee that the stream remains in the proper size. A device or a processing node needs to add new frames coming on, but it cannot delete the last one because it is in use. If this frame is deleted, it may cause some system error owing to the impossibility for the user to find data again.


This situation suggests that the stream size is out of control. To avoid this situation, an extra stream parameter called ‘KillOver’ has been introduced. It defines the biggest size the stream can reach, so if it happens that a stream reaches his maximum size (Stream Length) when the last frame is still in use, the data manager does not delete it, maintaining the proper system status. While the last frame remains in use, there is still space to continue frame insertions until KillOver size is reached. After that, the system deletes the frame to avoid an improper growth of stream size. The system deletes the frame because it interprets this situation as a processing node error during data release. These parameters are defined according to the context of the developed application. Every application needs its configuration parameters, coming from the estimation of processing node time.

In doing this, some elements of robustness are introduced into the software system. The structure of this framework is also suitable for testing system fault-tolerance

properties. It is possible to simulate some node error or broken device in order to understand what happens inside the system.

The software approach to this class of problems may be used for smart sensor simulations as well. In fact, the definition of Processing Node is absolutely general and it is suitable for a simple function as well as for a complex one like a sensor.

3 DFF multi-thread programming

Multi-threading (MT) (Li et al., 2002) is a programming paradigm for implementing application concurrency.

MT can be seen as a form of multi-tasking and its basic element, the thread, is a dispatch able unit of executable code. The traditional ‘single threaded’ process could be seen as a unique flow of control (thread) associated with a stack to keep track of local variables, of an address space and a set of resources. MT programming allows one program to execute multiple tasks concurrently, by dividing a single one into multiple threads. The main aspect is that different threads can execute their instructions independently and concurrently. This implies that the overlap of input, output and other operations with computational ones is allowed. Moreover, when an MT program is executed on a multi-processor machine, the different threads can run simultaneously (in parallel) on separate processors, exploiting the parallelism of the hardware (Negri et al., 2001).

4 DFF synchronisation

Synchronisation is one the hardest problems when more devices and sensors are involved. Synchrony is a key point in DFF: hardware peripherals acquire data at different rates, thus each frame has to be attached to the correct time slot to ensure the alignment of every sensor.

Different types of synchronisations are needed to ensure high performance results of the system. The synchrony issue can be examined from two points of view: an external perspective, regarding natural time flow against DFF system time, and an internal one, regarding DFF peripherals and temporal relations between nodes. We are not going to argue about the former.


We divide internal synchrony into two different classes: one called Hardware Sync and the other System Sync. The first type concerns hardware devices, in fact some devices have an external clock sync signal, so it is very easy to guarantee timing synchronisation. This synchrony is absolutely needed when algorithms are based on time differences, like audio array processing or generic critical time application. A basic solution consists of adding a Universal Time Clock (UTC) device that gives a common timeline to each hardware element (i.e., UTC device). Nowadays, not all of the devices are equipped with an external clock source, or they cannot be connected to UTC device. If it cannot be used, we have to use an internal clock. Global time refers to approximate measure of time generated by internally synchronised local clocks.

In Ross Bencina (2003), it was argued about sample rate differences from nominal value. It is well known that different hardware devices have different clocks, so it is not practicable to fix a reference time and then using the nominal clock to calculate the timestamp frame taking into account the number of samples captured.

Because of these limitations, we introduced in our framework a software mechanism of synchronisation.

In the DFF has been defined a global variable called Start, which is a signal variable used to indicate the start or stop action to every system device and node. So, after the setting-up environment, all threads remain in waiting for Start signal; when the signal comes back, all of the threads come out from the working state and set themselves on the exit state (Table 1). Start ensures that all of the threads start at the same time and, in lack of a hardware sync clock, data collected from threads are synchronised in some sense. So, we are sure that all threads receive data from the same time point, but we cannot assure a perfect synchronisation. Every data received has been marked with a time stamp from internal CPU clock, and written onto frame attribute. Synchrony from data cannot be assured because of some OS tasks scheduling. When some data are recorded from hardware, it is impossible to know when OS attaches some CPU time to a DFF thread. That means that the lag time between hardware and software calls is unpredictable; yet it is possible, under some conditions, to affirm that this lag time does not exceed a fixed value. It is reasonable if we consider that global time has a granularity larger than the precision of local clocks. In our implementation, we use CPU clock ticks to retrieve a monotonic timeline that is used to mark every data frame. Synchrony between threads, elements, etc., is assured, thanks to a universal time reference.

Table 1 Description of processing node’s states

State Description

WAIT FOR START

This is the first state where the thread starts. It stays here until the Start signal (see definition in Section 4) becomes ON, then it goes to the WAITING FOR DATA state The thread stays here waiting for data to process. When data become ready for processing, it goes on DATA PROCESSING

WAITING FOR DATA

It is also possible to come out of this state when Start signal becomes OFF It stays here until data processing is over and the output is ready DATA

PROCESSING From this state it goes back to WAITING FOR DATA EXIT The thread exits when it is in the EXIT state. It can be reached only

when Start signal is OFF


5 Conclusion

This paper proposes an operative framework implementation for modelling, developing and supporting DF applications. We point our attention to an efficient and reusable coding model that can be adapted to a wide range of different contexts. The proposed framework guarantees an optimum utilisation of CPU resources by making use of multi-thread programming.

Framework elements are defined to support an oriented-graph representation of the entire DF structure. Time alignment is supplied among the entire process by using common clock communication at data level. Dedicated real-time applications with automatic management of tasks synchrony can be developed, thanks to inter-task communication based on common driven signals. Interaction and communication with different input/output hardware and human operators have been based on established interfacing driver such as OpenGL, Steinberg ASIO, Microsoft DirectX®, etc.

The proposed DFF can be a valid instrument to test and benchmark different DF algorithms. External performances can be measured without taking into account internal performances owing to hardware specifications. Our research will focus on differences between classical AI approach and DF approach in solving the same kind of task. In particular, we will study performance variations depending on the number, type, and position of sensors.

Acknowledgement

The DFF project was sponsored by “Autostrade per l’Italia S.p.a.” who furnished their experience and collaboration during the development of the operative blocks.

References Akita, R.M. (2002) ‘User based data fusion approaches, information fusion’, Proceedings of the

Fifth International Conference on Information Fusion, Vol. 2, 8–11 July, pp.1457–1462. Besada, J., de Diego, J., de Miguel, G., Casar, J. and Garcia, J. (2002) ‘Generic software

architecture for development of data fusion systems’, Proceedings of the Fifth International Conference on Information Fusion, Vol. 1, pp.432–439.

Bossé, E., Roy, J. and Paradis, S. (2000) ‘Modeling and simulation in support of the design of a data fusion system’, Information Fusion, Elsevier, Vol. 1, pp.77–87.

Carvalho, H., Heinzelman, W. and Coelho, C. (2003) ‘A general data fusion architecture’, Proceedings of the Sixth International Conference on Info. Fusion, Vol. 2, pp.1465–1472.

Estrin, D., Girod, L., Pottie, G. and Srivastava, M. (2001) ‘Instrumenting the world with wireless sensor network’, Proc. ICASSP’2001, Salt Lake City, UT, pp.2675–2678.

Hall, D.L. and Kasmala, G.F. (1996) ‘Visual programming environment for multisensor data fusion’, Proceedings of the SPIE AeroSense 1996 Symposium, Orlando, FL, Vol. 2764, 8–12 April, pp.181–187.

Hall, D.L. and Linas, J. (1997) ‘An introduction to multisensor data fusion’, Proceedings of the IEEE, Vol. 85, No. 1, January, pp.6–23.

Hall, D.L. and Llinas, J. (2001) Handbook of Multisensor Data Fusion, CRC Press, June. Klein, L.A. (1993) Sensor and Data Fusion Concepts and Applications, SPIE Optical Engineering

Press, Tutorial Texts, Vol. 14, p.132.


Li, D., Wong, K.D., Hu, Y.H. and Sayeed, A.M. (2002) ‘Detection, classification, and tracking of targets’, IEEE Signal Processing Mag., Vol. 19, March, pp.17–29.

Mangolini, M. (1994) Apport de la Fusion d'images Satellitaires Multicapteurs au Niveau Pixel en Télédétection et Photointerprétation, Thèse de Doctorat, Université Nice – Sophia Antipolis, France, p.174.

Negri, A., Scannicchio, D.A., Touchard, F. and Vercesi, V. (2001) Multi Thread Programming, ATLAS DAQ, Available at atlas.web.cern.ch/Atlas/GROUPS/DAQTRIG/EF/documents/ MTNote.pdf

Ross Bencina (2003) ‘PortAudio and media synchronisation – it’s all in the timing’, ACMC2003, Perth, Western Australia.

Wald, L. (1998) ‘A European proposal for terms of reference in data fusion’, Proceedings of the 2nd Conference ‘Fusion of Earth Data: Merging Point Measurements, Raster Maps and Remotely Sensed Images’, SEE/EURISCA, Nice, France, pp.17–23.

Wald, L. (1999) ‘Definitions and terms of reference in data fusion’, International Archives of Photogrammetry and Remote Sensing, Part 7-4-3 W6, Valladolid, Spain, Vol. 32, 3–4 June, pp.1190–1193.

White, F. (1987) Data Fusion Lexicon, Joint Directorsof Laboratories, Technical Panel for C3, Data Fusion Sub-panel, Naval Ocean System Center, San Diego.

Website Sun Microsystems’ Multi-threading Programming Seminar, Available at http://www.sun.com/

smcc/solaris-migration/docs/courses/threadsHTML/cover.html

data fusion framework: concurrent architecture for real ... fusion... · data fusion framework:...

Documents