an open source framework for tracking and state...

CAN UNCLASSIFIED

Defence Research and Development Canada External Literature (N) DRDC-RDDC-2017-N031P00 September 2017

CAN UNCLASSIFIED

An Open Source Framework for Tracking and State Estimation ('Stone Soup') Paul A. Thomas, Jordi Barr Defence Science and Technology Laboratory (UK)

Bhashyam Balaji DRDC – Ottawa Research Centre

Kruger White Defence Science and Technology Group (Australia)

Society of Photo-Optical Instrumentation Engineers (SPIE), Vol. 10200, 1020008

Date of Publication from Ext Publisher: May 2017

Terms of Release: This document is approved for Public release.

CAN UNCLASSIFIED

Template in use: (2012) CR EL1 Advanced Template_EN 2017-09_21-V02_WW.dotm

© Her Majesty the Queen in Right of Canada (Department of National Defence), 2017 © Sa Majesté la Reine en droit du Canada (Ministère de la Défense nationale), 2017

CAN UNCLASSIFIED

IMPORTANT INFORMATIVE STATEMENTS

Disclaimer: This document is not published by the Editorial Office of Defence Research and Development Canada, an agency of the Department of National Defence of Canada, but is to be catalogued in the Canadian Defence Information System (CANDIS), the national repository for Defence S&T documents. Her Majesty the Queen in Right of Canada (Department of National Defence) makes no representations or warranties, express or implied, of any kind whatsoever, and assumes no liability for the accuracy, reliability, completeness, currency or usefulness of any information, product, process or material included in this document. Nothing in this document should be interpreted as an endorsement for the specific use of any tool, technique or process examined in it. Any reliance on, or use of, any information, product, process or material included in this document is at the sole risk of the person so using it or relying on it. Canada does not assume any liability in respect of any damages or losses arising out of or in connection with the use of, or reliance on, any information, product, process or material included in this document.

This document was reviewed for Controlled Goods by Defence Research and Development Canada (DRDC) using the Schedule to the Defence Production Act.

An Open Source framework for Tracking and State Estimation (‘Stone Soup’)

Paul A. Thomas ([email protected])a, Jordi Barra, Bhashyam Balajib, Kruger Whitec aDefence Science and Technology Laboratory (United Kingdom),

bDefence Research and Development Canada (Canada), cDefence Science and Technology Group (Australia).

ABSTRACT

The ability to detect and unambiguously follow all moving entities in a state-space is important in multiple domains both in defence (e.g. air surveillance, maritime situational awareness, ground moving target indication) and the civil sphere (e.g. astronomy, biology, epidemiology, dispersion modelling). However, tracking and state estimation researchers and practitioners have difficulties recreating state-of-the-art algorithms in order to benchmark their own work. Furthermore, system developers need to assess which algorithms meet operational requirements objectively and exhaustively rather than intuitively or driven by personal favourites.

We have therefore commenced the development of a collaborative initiative to create an open source framework for production, demonstration and evaluation of Tracking and State Estimation algorithms. The initiative will develop a (MIT-licensed) software platform for researchers and practitioners to test, verify and benchmark a variety of multi-sensor and multi-object state estimation algorithms. The initiative is supported by four defence laboratories, who will contribute to the development effort for the framework.

The tracking and state estimation community will derive significant benefits from this work, including: access to repositories of verified and validated tracking and state estimation algorithms, a framework for the evaluation of multiple algorithms, standardisation of interfaces and access to challenging data sets.

Keywords: Tracking, state estimation, open source, software development

Content includes material subject to

© Crown copyright (2017), Dstl. This material is licensed under the terms of the Open Government Licence except where otherwise stated. To view this licence, visit http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3 or write to the Information Policy Team, The National Archives, Kew, London TW9 4DU, or email: [email protected].

1. INTRODUCTION

Tracking & state estimation is a critical capability for military Situational Awareness (SA). The ability to detect and follow unambiguously all moving entities in a state-space is important for SA in multiple domains ranging from the traditional (e.g. air surveillance, maritime SA, ground moving target indication) to the less conventional (e.g. tracking internet transactions, monitoring changes in sentiment, CBRN dispersion prediction or following disease epidemics).

However, researchers and practitioners find difficulties recreating state-of-the-art algorithms in order to benchmark their own work1. Comparison of new algorithms with alternative solutions is time-consuming, involving recoding of the alternative algorithms from the academic literature. Industrial exploiters of algorithms also need to assess which algorithms meet operational requirements objectively and exhaustively rather than intuitively and driven by personal favourites.

The best solution to these issues is to develop an open source tracking and state estimation framework and to populate it with open source components. This is in line with wider open source and open data government initiatives2,3. The project described in this paper aims to provide a flexible and unified software platform for researchers and engineers to

Signal Processing, Sensor/Information Fusion, and Target Recognition XXVI, edited by Ivan Kadar, Proc. of SPIE Vol. 10200, 1020008 · © 2017 SPIE · CCC code: 0277-786X/17/$18 · doi: 10.1117/12.2266249

Proc. of SPIE Vol. 10200 1020008-1

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 9/27/2017 Terms of Use: https://spiedigitallibrary.spie.org/ss/TermsOfUse.aspx

3. ACCESSIBILITY The key to wide adoption of this framework is to make it as accessible as possible.

3.1 Licence

Most of the code developed in the Consortium phase of the Stone Soup project will be subject to copyright. All such code will be licensed under the MIT licence4, which is one of the freest forms of open source licence. Work done by US Government employees as part of their official duties is not subject to copyright and shall contain a disclaimer stating that it is not subject to copyright protection. All source code files shall contain a reproduction of the licence as a prominent boilerplate in the file.

During the open phase of the project external contributions may either be subject to copyright or not subject to copyright protection. Contributions subject to copyright protection will be permissible if they are released under any of the following licences:

The MIT License4;

The BSD 3-Clause license5;

The BSD 2-clause license6;

The International Astronomical Union's Standards of Fundamental Astronomy License7.

Contributions not subject to copyright shall contain a disclaimer stating that they are not subject to copyright protection.

3.2 Protective marking

The open source version of the framework will not be protectively marked, though users are at liberty to download a copy and run it on protected data/scenarios as they wish.

3.3 Releases

Ideally, in an open source project, the development team would “code in the open”11. However, due to the legacy requirement for release permission from each government lab before each released software version, this would require a culture change. Such a culture change is long overdue and this project should prove the driver for this, but until this happens, Stone Soup will work under the model of shared coding in a private Git repository, coupled with frequent authorised releases.

4. REQUIREMENTS Open source software, unlike the traditional software development model, tends to develop amorphously according to the will and enthusiasm of members of the user/developer community. Therefore, upfront definition of requirements for the open phase of Stone Soup is less valid. However, in order to rapidly build the initial framework in the Consortium phase it is useful to have a set of requirements. These allow the funding stakeholders in the Consortium phase to shape the initial project to suit their objectives, which gives them an initial return on their investment.

Of course, the Stone Soup consortium phase is more concerned with building a framework that is compatible with future algorithms, data, metrics, simulators and sensor models, than actually populating these elements. The requirements, therefore, relate to compatibility with specified classes of algorithm, types of data, types of model, etc., that might be added in the open phase; rather than requiring specific functions to be added immediately. These types of requirement will drive the data model inside the Stone Soup framework.

For example, the funding stakeholders for the consortium phase might decide that, although (of course) they would prefer compatibility with all types of algorithm, compatibility with certain algorithm classes have a higher priority for the initial work on the framework. This would drive the data model to ensure compatibility with these classes first.

Proc. of SPIE Vol. 10200 1020008-4


Algorithm

Provides algo

developer

thm

Tracking

Evaluates

Evaluator

Downloads relevant metricsI

1

Uses standard test dataset

Stone Soup

In order to frame the requirements, the project team constructed three Use Cases. As Stone Soup is open source software, where the community is characterised as “user/developers” rather than simply “users”, the use cases contain activities associated with code development.

The use cases are:

4.1.1 Analyst

A common analysis use case involves a task to evaluate the efficacy of a recently-developed novel state-estimation algorithm by a ‘Tracking Evaluator’. The evaluator has a technical knowledge of the field, but not a deep understanding of the algorithm, or its theoretical underpinnings. The evaluator is able to receive a Stone Soup compatible algorithm from the developer and download ‘standard’ verified benchmarking algorithms, test data, and some relevant metrics. In this way they are able to form a judgement on the algorithm, impartially, and with pertinence to their problem.

4.1.2 Component developer

An algorithm developer is at work on a bleeding-edge multi-sensor, multi-target tracking (MTT) algorithm. The developer codes up the algorithm in Python and is busy writing up their results for publication in a top journal. They are able to access standard, well-used, MTT algorithms in the Stone Soup repository. At the same time they can access appropriate data and some useful metrics. With this repository, the developer is able to reproduce the same experiment that others have run and to demonstrate that the new algorithm does indeed outperform the current state of the art by an impressive margin. The results are published – no doubt to much acclaim. The developer may wish to upload their algorithm to the Stone Soup repository in order to benefit the wider tracking community.

4.1 Use Cases

Proc. of SPIE Vol. 10200 1020008-5


Writes up re:yl

1(e..goh,Pap

Alpo Elm De

loads novel algorithm

Downloads standard algorithm

wnloads relavant me rICS

Produces novel otonmm

Tests aM renne

Stone Soup

New_Requiremenl

include»User/Developer

Upload to Stone Soup Repository

Update Software documentation

«include»

«extend >4

Authorise New Release

Mod erator

4.1.3 Framework developer

The Framework developer, perhaps prompted by an external requirement, or their own requirement, makes a modification to the code base of the Stone Soup framework. This modification must be associated with an update to the software documentation. Both these elements are uploaded to the Stone Soup (Git) repository. Perhaps prompted by the upload or perhaps during a scheduled release, the moderator authorises the new release.

The release process involves retrieval of code from version control and running a script that pulls dependencies, builds and tests the latest code. If the build is for release, the script will also collate the relevant information to populate a Release Note, including the commit ID, the test results, the changes to the code since last release and any outstanding and fixed issues.

The master branch will be used and tagged in Git to ensure every release can be replicated.

Proc. of SPIE Vol. 10200 1020008-6


4.2 Algorithms

Priority for the initial phases of Stone Soup will be to ensure compatibility with the following generic algorithm types:

Filtering algorithms: Discrete-Time State and Measurement Models o Standard Kalman Filter for the Linear State and Measurement Model o Extended Kalman Filter o Derivative-free Kalman Filters

Particle Filter Class of algorithms o Random Particle Filters o Deterministic Particle Filters

Multiple Model filtering algorithms for Kalman filter class of algorithms

4.3 Data Model

These different algorithms, at the most basic level, are attempting to represent a dynamic system. A general dynamic system consists of state and measurement models, either of which may be linear or non-linear (noting that the system might be time-varying; i.e. the state model may change with time, the measurement model may change with time or the state and measurement models may explicitly depend on time).

The fundamental requirement of Stone Soup is to enable the comparison of different algorithmic approaches against the same data or simulated scenario. Thus algorithms must be “swappable” at the state and measurement model level. Therefore, key to the success of the framework is the creation of an underlying data model which enables “algorithm swappability” across all the supported algorithm classes and can cope with all the supported data types. This is more complex than it seems because the algorithms represent uncertainty in different ways, e.g. algorithms based around the Kalman filter model a target state distribution as a mixture of one or more Gaussian distributions, versus particle filters which represent the distribution by a set of weighted samples.

Furthermore, the data model must support manipulation of a measurement model via data from a variety of data sources, in effect “data swappability”, across range and bearing, bearings only, range only, and/or categorical (identity) data.

4.4 Data

Priority for the initial phases of Stone Soup will be to enable the use of the following data types. This involves creation of an appropriate data model in the framework for handling the data and making available representative data sets in the repository. The initial data types will be:

Airborne radar detection data. This could be in 2D or 3D, from a rotating radar or from a planar array. Coincident AIS data EO/IR data (which typically is reasonably accurate in bearing but has poor accuracy or no information about

range) The intention is to build wider compatibility with further data types and provide more example data sets, as the project progresses.

4.5 Metrics

Assessment of alternative Tracking and State Estimation algorithms requires a means of objective evaluation using a set of metrics which correspond to characteristics of interest to operational systems16. In certain circumstances knowledge of the actual number of entities and their states may be available such as when simulations include the state of entities, sensor data and implementation of tracking and state estimation algorithms. In other circumstances knowledge of the actual number of entities and their states may not be available.

Priority for the initial phases of Stone Soup will be to enable assessments in the presence of entity “truth” state information. In this case, an absolute evaluation of each alternative algorithm can be conducted by comparing the tracking and state estimation output against the truth data. Typically, a data association approach is first used to associate track data with entity truth data16 Subsequently, performance metrics can be applied to quantify tracking characteristics such as the following:

Proc. of SPIE Vol. 10200 1020008-7


Initiation of a track following the first appearance of an entity Accuracy of a track estimate compared to the state of an entity Continuity of a track during the time evolution of the state of an entity Existence of false tracks which are tracks that do not correspond to entities of interest Credibility of a track estimate and its estimation uncertainty compared to the state of an entity17

As with other aspects of Stone Soup, the intention is to develop further performance metrics as the project progresses.

5. DEVELOPER GUIDE The Stone Soup project aims to provide an environment which is as open and inclusive as possible to contributions from the broad tracking community. However, some broad development guidance is required in order to ensure a minimum quality standard.

The Stone Soup Consortium phase will implement a standard body of developer guidance that is defined in the Stone Soup Requirements Document10. This provides guidance on the following:

Documentation / comments Code Quality Interfaces to and dependencies on external code Extensibility and reuse of algorithms Unit Tests

6. JOINING IN The Stone Soup project would welcome additional commitments of developer resources during the Consortium Phase. The minimum meaningful commitment is 0.5 x developer year per year. This is open to all types of contributing agency; e.g. Government, Industry or Academia, in both the Defence and Civil sectors. Agencies willing to make this level of commitment would gain membership of the Requirements Panel, shaping the software to their needs. Benefits to being part of the Consortium Phase are:

Influence on the development of what (the authors hope) will become an academic standard open architecture o This supports the most governments’ visions of making greater use of open systems.

Significant gearing on algorithm and testbed development o Several pledges of support of manpower have so far been received from government agencies.

Ability to influence the international tracking and state estimation community o Leading the community towards next-generation tracking and state estimation approaches, scenarios

and solutions. Any interested party should contact the corresponding author of this paper.

7. WHY THE NAME? The name ‘Stone Soup’ refers to an old folk story13, 14, 15, and many more, in which a traveller persuades the local people of a village into collaborating to share their food.

The story: A traveller arrives in a village, carrying nothing more than an empty cooking pot. Upon arrival, the discovers the village is undergoing a famine and everyone is hungry. So the traveller goes to a stream and fills the pot with water, drops a large stone in it, and places the pot over a fire. One of the villagers becomes curious and asks what is happening. The traveller answers that "stone soup" is being made and that, it is very nutritious, although it still needs a little bit of garnish to improve the flavour. The villager does not mind parting with an old cabbage so that is added to the soup. Another villager walks by, inquiring about the pot, and the traveller again mentions their stone soup which has not reached its full potential yet. The villager hands them some scraps to help them out. More and more villagers walk by,

Proc. of SPIE Vol. 10200 1020008-8


each adding meat, potatoes, etc, etc. Finally, the stone (being inedible) is removed from the pot, and a delicious and nourishing pot of soup is enjoyed by all.

Moral: By working together, with everyone contributing what they can, a greater good is achieved

Of course, in the case of the original story the soup is consumed. However in the case of a software project, the output from the collaboration will remain to benefit everyone into the future.

8. SUMMARY Stone Soup is a collaborative initiative to create an open source framework for comparison of tracking and state estimation algorithms, data, metrics, simulators and sensor models. The initiative will develop a software platform for researchers and practitioners to test, verify and benchmark multi-sensor and multi-object state estimation algorithms.

The initiative is split into two phases, Consortium and Open, with the Consortium phase being supported by a number of government defence laboratories, who will contribute to the development effort for the framework. The Open phase will build upon and existing repository which will be drawn from The Tracker Component Library5, published on Git with the addition of algorithms from the academic community. The authors hope this will become an academic standard open architecture.

Code developed in the Consortium phase of the Stone Soup project will be not protectively marked and released under the MIT license, where possible. Requirements are being developed currently and are driven by three use cases; Analyst, Component developer and Framework developer. Developer guidance is also being written to ensure a minimum software quality standard.

We predict that the tracking and state estimation community will derive significant benefits from this work, including: access to a repository of tracking and state estimation algorithms, framework for evaluation of multiple algorithms and access to challenging data sets. The authors encourage contributions of developer time to assist in building the framework. Any interested party should contact the corresponding author.

REFERENCES

[1] P. Thomas, “Open letter on the “stone soup” tracking framework,” in Proceedings of the 10th Data Fusion and Target Tracking Conference, University of Liverpool, United Kingdom, 30 Apr. 2014. [Online]. Available: http://conferences.theiet.org/target/past-presentations/2014/-documents/framework-presentation.cfm

[2] Defence White Paper: National Security Through Technology. pp. 21-22. [Online] Available https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/27390/cm8278.pdf

[3] B. Obama. “Presidential memorandum – building a 21st century digital government”. The White House, Office of the Press Secretary. (2012) [Online]. Available: http://www.whitehouse.gov/the-press-office/2012/05/23/presidential-memorandum-building-21st-century-digital-government

[4] Raymond, Eric S. “The Cathedral and the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary”. O'Reilly Media. ISBN 1-56592-724-9. (1999).

[5] David Frederic Crouse. The Tracker Component Library: Free Routines for Rapid Prototyping. Submitted for publication to IEEE Aerospace and Electronic Systems Magazine.

[6] https://opensource.org/licenses/MIT [7] https://opensource.org/licenses/BSD-3-Clause

[8] https://opensource.org/licenses/BSD-2-Clause

[9] http://www.iausofa.org/tandc.html

[10] Open Source Tracking and State Estimation "Stone Soup" Requirements Document. Thomas, PA, Crouse, DF, Balaji, B, White, K, Tagg, J, Barr, J, Brown, G, Green, R, Ablett, S, McKinlay, R. DSTL/PUB099273. (2017).

[11] “Coding in the open”. Blog - Government Digital Service. [online] Available https://gds.blog.gov.uk/2012/10/12/coding-in-the-open/

Proc. of SPIE Vol. 10200 1020008-9


[12] “Open Source Development Guidelines”. Civic Commons. [Online] Available http://wiki.civiccommons.org/Open_Source_Development_Guidelines/

[13] “Stone Soup”. DLTK's Educational Activities: Fables. [Online] Available http://www.dltk-teach.com/fables/stonesoup/mtale.htm

[14] Stone Soup: An Old Tale. Marcia Brown. Aladdin Picture Books. ISBN: 0689711034 [15] Stone Soup. Jon J Muth. Scholastic Press; 1st edition 2003. ISBN-10: 043933909X [16] A. Blackman, S. and Popoli, R. Design and Analysis of Modern Tracking Systems. Artech House, 1999. [17] B. Li, X.R., Zhao, Z., Li, X-B. “Evaluation of Estimation Algorithms – Credibility Tests”. IEEE Trans on

Systems, Man, and Cybernetics – Part A: Systems and Humans, Vol. 42, No. 1, January 2012.

Proc. of SPIE Vol. 10200 1020008-10


CAN UNCLASSIFIED

CAN UNCLASSIFIED

DOCUMENT CONTROL DATA (Security markings for the title, abstract and indexing annotation must be entered when the document is Classified or Designated)

1. ORIGINATOR (The name and address of the organization preparing the document.Organizations for whom the document was prepared, e.g., Centre sponsoring a contractor's report, or tasking agency, are entered in Section 8.)

DRDC – Ottawa Research CentreDefence Research and Development Canada3701 Carling AvenueOttawa, Ontario K1A 0Z4Canada

2a. SECURITY MARKING (Overall security marking of the document including special supplemental markings if applicable.)

CAN UNCLASSIFIED

2b. CONTROLLED GOODS

NON-CONTROLLED GOODS DMC A

3. TITLE (The complete document title as indicated on the title page. Its classification should be indicated by the appropriate abbreviation (S, C or U) in parentheses after the title.)

An Open Source Framework for Tracking and State Estimation ('Stone Soup')

4. AUTHORS (last name, followed by initials – ranks, titles, etc., not to be used)

Thomas, P.A.; Barr, J.; Balaji, B.; White, K.

5. DATE OF PUBLICATION (Month and year of publication of document.)

September 2017

6a. NO. OF PAGES (Total containing information, including Annexes, Appendices, etc.)

11

6b. NO. OF REFS (Total cited in document.)

17 7. DESCRIPTIVE NOTES (The category of the document, e.g., technical report, technical note or memorandum. If appropriate, enter the type of report,

e.g., interim, progress, summary, annual or final. Give the inclusive dates when a specific reporting period is covered.)

External Literature (P)

8. SPONSORING ACTIVITY (The name of the department project office or laboratory sponsoring the research and development – include address.)

DRDC – Ottawa Research CentreDefence Research and Development Canada3701 Carling AvenueOttawa, Ontario K1A 0Z4Canada

9a. PROJECT OR GRANT NO. (If appropriate, the applicable research and development project or grant number under which the document was written. Please specify whether project or grant.)

9b. CONTRACT NO. (If appropriate, the applicable number under which the document was written.)

10a. ORIGINATOR’S DOCUMENT NUMBER (The official document number by which the document is identified by the originating activity. This number must be unique to this document.)

DRDC-RDDC-2017-N031

10b. OTHER DOCUMENT NO(s). (Any other numbers which may be assigned this document either by the originator or by the sponsor.)

11a. FUTURE DISTRIBUTION (Any limitations on further dissemination of the document, other than those imposed by security classification.)

Public release

11b. FUTURE DISTRIBUTION OUTSIDE CANADA (Any limitations on further dissemination of the document, other than those imposed by security classification.)

Public release

CAN UNCLASSIFIED

CAN UNCLASSIFIED

12. ABSTRACT (A brief and factual summary of the document. It may also appear elsewhere in the body of the document itself. It is highly desirable that the abstract of classified documents be unclassified. Each paragraph of the abstract shall begin with an indication of the security classification of the information in the paragraph (unless the document itself is unclassified) represented as (S), (C), (R), or (U). It is not necessary to include here abstracts in both official languages unless the text is bilingual.) The ability to detect and unambiguously follow all moving entities in a state-space is important in multiple domains both in defence (e.g. air surveillance, maritime situational awareness, groundmoving target indication) and the civil sphere (e.g. astronomy, biology, epidemiology, dispersionmodelling). However, tracking and state estimation researchers and practitioners havedifficulties recreating state-of-the-art algorithms in order to benchmark their own work.Furthermore, system developers need to assess which algorithms meet operationalrequirements objectively and exhaustively rather than intuitively or driven by personal favourites.

We have therefore commenced the development of a collaborative initiative to create an opensource framework for production, demonstration and evaluation of Tracking and StateEstimation algorithms. The initiative will develop a (MIT-licensed) software platform forresearchers and practitioners to test, verify and benchmark a variety of multi-sensor and multi-object state estimation algorithms. The initiative is supported by four defence laboratories, whowill contribute to the development effort for the framework.

The tracking and state estimation community will derive significant benefits from this work,including: access to repositories of verified and validated tracking and state estimationalgorithms, a framework for the evaluation of multiple algorithms, standardisation of interfacesand access to challenging data sets.

___________________________________________________________________________

13. KEYWORDS, DESCRIPTORS or IDENTIFIERS (Technically meaningful terms or short phrases that characterize a document and could be helpful in cataloguing the document. They should be selected so that no security classification is required. Identifiers, such as equipment model designation, trade name, military project code name, geographic location may also be included. If possible keywords should be selected from a published thesaurus, e.g., Thesaurus of Engineering and Scientific Terms (TEST) and that thesaurus identified. If it is not possible to select indexing terms which areUnclassified, the classification of each should be indicated as with the title.)

Fusion; Tracking; state estimation; open source; software development

an open source framework for tracking and state...

Documents