apidis public annual report apidis ... - europa

APIDIS-FP7-216023

APIDIS Public Annual Report 2010 1

APIDIS Public Annual Report

APIDIS www.apidis.org

Autonomous production of images based on distributed sensing

APIDIS project description

APIDIS’s overall objective is cost-effective autonomous production, so as to make the creation of audiovisual reports profitable, even in case of small- or medium-size audience.

To meet this objective, APIDIS first investigated the automatic extraction of intelligent content from networks of multi-modal sensors. Intelligence refers here to the identification of salient segments within the audiovisual content, using distributed scene analysis algorithms.

Then, second, APIDIS exploited that knowledge about segments of interest to produce prototypes that automate the production of video content for controlled scenarios, e.g. most notably sports events or surveillance. It considers personalized and potentially interactive content summarization mechanisms to address heterogeneous user needs and access conditions.

The potential applications of the integrated technology developed within APIDIS are numerous, ranging from personalized access to digital media related to local sport events through a web portal or a mobile hand-set; cost-effective and fully automated production of content dedicated to small-audience, e.g. souvenirs DVD, university lectures, etc; but also browsing assistance for video surveillance.

Summary of Activities

APIDIS started in January 2008 and in December 2010 the project has successfully completed.

Project completed and objectives met The APIDIS system was successfully implemented and significant technological and scientific advances were achieved in fields related to:

• The extraction of meaningful content from the video sequences captured by a set of complementary cameras.

APIDIS-FP7-216023


• The automated production of video content.

• The personalized summarization of the content.

Scientific and technical solutions were created for:

• Cost-efficient and robust deployment of networked and bio-inspired omnidirectional sensors for the acquisition of visual inputs.

• Distributed tracking and interpretation of visual content.

• Autonomous or interactive production of summarized content.

• Flexible and personalized access to the content.

Prototypes were developed, tested and demonstrated for:

1. Automatic and personalized summarization of a video sequence, knowing its salient segments

2. Generation of high resolution images in any viewing direction based on an array of omnicams

3. Distributed feature extraction and analysis for people detection&tracking, and autonomous production

4. Interactive and semantically–driven access to video surveillance content 5. Sharing and remixing content 6. Automatic generation of content for Internet portals

APIDIS has advanced the state of the art in:

• User-centred design • Omnivision • Scene analysis and event detection • Production of content • Flexible and interactive access to content • Autonomous production through optimal decision process

Promotion of APIDIS results The APIDIS prototypes and results have been promoted to a community that has steadily grown over the 3 years of the project. The community’s interest in the APIDIS results ranges from scientific/technical for research purposes, industry professional for potential commercialization, to personal for non-professional end-users. The APIDIS partners have promoted results to the APIDIS scientific and technical community through invited talks, over 30 conference publications/presentations and 14 journal articles and book chapter. The partners have promoted the commercial application of the system to the broadcast and production and video surveillance communities with demonstrations at 10 major industry events, numerous consultations with professionals, collaborations with industry, possible transfer of technology to a spin-off company and finally with the installation of the system in a Belgian basketball court. A Showcase video of the prototypes and results is available.

APIDIS-FP7-216023


Promotion of the APIDIS Dataset The APIDIS dataset, acquired by the partners in the project’s first two years and used for testing and demonstrating in the sports and surveillance scenarios, has been heavily publicized and distributed to the APIDIS scientific community for further research. TrecVid participation APIDIS results were used in the prestigious TrecVid 2010 competition. Using the APIDIS dataset, partner QMUL developed the feature extraction part of the algorithm they successfully used in TrecVid. The summary of the results of the evaluation was generated by the National Institute of Science and Technology (NIST) and can be downloaded from http://www-nlpir.nist.gov/projects/tvpubs/tv10.slides/tv10.sed.slides.pdf. Testbed finalized In 2010, the final versions of the two main prototypes that have been developed to demonstrate the services offered by APIDIS have been released. The prototypes address: The autonomous production of sport events summaries, and the interactive browsing of video surveillance content. End of project trials for the APIDIS prototypes The partners ran the end-of-project trials in summer/fall2010. They successfully proved the three key concepts underlying APIDIS. The trials demonstrated and tested these three scenarios:

• interactive and semantically–driven access to video surveillance content, • sharing and remixing content, • and automatic generation of content for Internet portals.

Permanent prototype in the Spiroudome basketball arena A real-time prototype has been implemented within a C/C++ SDK and deployed as a permanent infrastructure in the Spiroudome of Charleroi, Belgium. It supports player detection, image stitching for panoramic reconstruction, and fully autonomous production of video content. By February 2011, a ball detection method was close to be implemented in the SDK. Graph-based player tracking should be supported in the SDK by summer 2011. Exploitation planned The APIDIS technologies are positioned for their application to the video surveillance and sports production and broadcast industries. In 2010, the final assessment of the commercial value of the APIDIS technologies was made. The project has ended on a high note with the deployment of system in the Spiroudrome, a major basketball arena in Belgium and with a Belgian national project initiated to continue the APIDIS work in the area of sport analytics solutions for coaching assistance, and automatic generation of content (basketball matches) for internet portals. Technical advances in 2010

• Software for event detection advanced. Implementation issues related to the high level interpretation mechanisms that have been considered by APIDIS to personalize the summarization of basket ball games have been explored.

• Software for distributed feature extraction and visual target tracking advanced and results include: The various single-camera and multi-camera feature

APIDIS-FP7-216023


extraction and tracking modules developed for local feature extraction, player detection and tracking developed in APIDIS project were compared. Ball detection and tracking solutions were finalized.

• The software for the autonomous and personalized summarization of basket ball games, and the software for the interactive browsing of video surveillance streams were developed. A permanent infrastructure has been deployed in a major basket-ball arena, based on a C/C++ efficient implementation of the APIDIS algorithms.

User Consultations In 2010, the existing prototypes were improved and new prototypes developed based on developed steady feedback from the APIDIS user community:

• Contact were strengthened with companies (ESPN, Cisco, NeuroTV, BeTV, …) • A collaboration with Telefonica Innovation Division has yielded results

including a publication at ICME, dealing with fully automatic –meaning without any manual annotation- summarization of broadcasted soccer games, e.g. for distribution through mobile handsets.

• Partners presented results and demonstrated prototypes at academic conferences and industrial tradeshows, the following are highlighted:

o NAB 2010 - Sports production and broadcast professionals were engaged in consultations.

o IBC 2010 - APIDIS has been invited to the New Technology Campus. Demos of the automatic production and summarization prototypes were given to industry professionals. Scientific contributions were presented during IBC conference sessions.

o ICT 2010 in Brussels. o 3D Stereo Media in Liège. o IFSEC 2010 and Expoprotection - Security/Videosurveillance

professionals • Users were engaged in consultations for the end-of-project trials.

Market Analysis and Final Business Plan A business case has been developed. The American and European demand for basketball video and the demand from sport clubs were assessed. Professionals in the sports industry were consulted at the NAB conference and in the end of project trials to understand if an APIDIS potential product would meet the demand. It seems that there is a niche market for the tools and technologies developed within the APIDIS project. On the one hand, TV producers are particularly interested in the low cost production facilities offered by APIDIS. On the other hand, sport professionals (coaches, managers, etc) appear to be interested in the game analysis part of the system, as well as in the personalized content production tools. A multi-year strategy has been developed to commercialize the APIDIS results. IPR protection is in place, as well, with filed patent applications. Beyond APIDIS, life after the project APIDIS ended in December 2010. Activities planned to further use and exploit results include:

• Continued collaborations with partners. Additional research funding to be sought.

APIDIS-FP7-216023


• Dataset distribution. • Additional functionality for the SDK to be implemented in 2011. • Work on the prototype deployed in major Belgian basketball arena, the

Spiroudome, funded by the SPORTIC project (Walloon region)

APIDIS Images:

Figure 1: APIDIS workflow.

Figure 2: APIDIS services for team-sport coverage.

APIDIS-FP7-216023


Figure 3: Autonomous production development interface.

Figure 4: Automatic production demo.

APIDIS-FP7-216023


Figure 5: Automatic&personalized summarization demo.

Figure 6: Panoramic view reconstruction for continuous panning support during automatic production.

APIDIS-FP7-216023


APIDIS important work areas The APIDIS Dataset

The APIDIS sports video content collection system The partners have collected two basketball datasets and three video surveillance datasets. These data consist in video datasets, audio recordings, external metadata as well as manual annotations produced within the project. Partner MP has provided several football matches. In 2010, datasets have been used for testing, validations and demonstration of the APIDIS final prototypes. The APIDIS basketball dataset has been heavily promoted to the scientific community and distributed for further research.

Trecvid participation Partner QMUL developed the feature extraction part of the algorithm used in TrecVid competition in APIDIS, using the APIDIS dataset. More about the competition and QMUL results from this prestigious competition:

“The event detection approach is divided into two main steps, namely feature extraction and classification. Feature extraction involves the conversion of video data into set of representative features (motion vectors). To this end we base our method on local feature modeling within spatio-temporal cuboids. The dimensions of the cuboid in space and time are decided based on the expected size of the person in a portion of the image. In the classification motion vectors are compared against known labeled samples and each spatio-temporal region (cuboid) is assigned a confidence, which is the basis of the final classification. The spatial relationship across cuboids is also used to discriminate between simultaneously occurring events. The evaluation in TrecVid[1], for “Person Runs” had a dataset consisting of ~150 hours of recorded video from 5 cameras from London Gatwick Airport. It contains dense scenes, clutter, considerable variations in viewpoint and high variability among different instances of the same action making it a challenging dataset. Within the TrecVid evaluation campaign the performance of the

APIDIS-FP7-216023


systems are measured using Normalized Detection Cost Rate (NDCR), a weighted linear combination of the Missed Detection Probability and False Alarm Rate (per unit time). The smaller the NDCR, the better the performance of a system. The summary of the results of the evaluation was generated by the National Institute of Science and Technology (NIST) and can be downloaded from http://www-nlpir.nist.gov/projects/tvpubs/tv10.slides/tv10.sed.slides.pdf. [1] A. Smeaton, P. Over, and W. Kraaij, “Evaluation campaigns and Trecvid,” in Proc.

of ACM Int. Workshop on Multimedia Information Retrieval, Oct. 2006, pp. 321–330.”

Real Time Permanent Prototype A real-time prototype has been implemented within a C/C++ SDK and deployed as a permanent infrastructure in the Spiroudome of Charleroi, Belgium. It supports player detection, image stitching for panoramic reconstruction, and fully autonomous production of video content. In February 2011, scoreboard analysis software is available, and a ball detection method is close to be implemented in the SDK. Graph-based player tracking should be supported in the SDK by summer 2011. (The actual software code is not yet available, as a licensable software package, a library or any other adequate means goes beyond the project.) End of project trials for the APIDIS prototype The partners ran three end-of-project trials for new prototypes in summer/fall 2010. The trials proved the three key concepts underlying the APIDIS prototypes. The three prototypes demonstrated and tested were:

• interactive and semantically–driven access to video surveillance content (partners BM, ACIC)

• sharing and remixing content (partners UCL, MP)

• and automatic generation of content for Internet portals (partner UCL)

The trials allowed partners to collect valuable final feedback to feed the final business plan. Detailed information can be found in public deliverable D7.2 published on www.apidis.org/publications Technical innovations The following APIDIS technologies were advanced. Detailed information is available on http://www.apidis.org/publications.htm#PublicDeliverables: Software for event detection. Implementation issues related to the high level interpretation mechanisms that have been considered by APIDIS to personalize the summarization of basket ball games have been explored. This has resulted in the development of a software for scoreboard analysis. Software for distributed feature extraction and visual target tracking. The various single-camera and multi-camera feature extraction and tracking modules developed for

APIDIS-FP7-216023


local feature extraction, object detection and tracking developed in APIDIS project were compared. Final version of the test-bed for personalized access to content, including autonomous and interactive production facilities was produced. The software for the autonomous and personalized summarization of basket ball games and the software for the interactive browsing of video surveillance streams were produced. Technology and Market Analysis Partners have analyzed how APIDIS technologies can be applied to the video surveillance and broadcast environments through contacts with companies made at industry events (NAB 2010, IBC 2010, IFSEC 2010, Expoprotection, ESSEN Security, etc…), collaboration with industry, and the deployment of the system in a basketball arena. A final business plan has been produced.

APIDIS User Involvement, Promotion and Awareness

Scientific and professional end-users group consultations were held for six prototypes in 2010. Three APIDIS prototypes were developed in 2009 and demonstrated to end-user communities: 1. The autonomous production, 2. the automatic and personalized summarization of broadcasted sport video, and 3. The generation of high resolution images in any viewing direction based on omnicams. End-user consultations were also held for the end of project trials which demonstrated and tested three more, new prototypes developed in 2010: 4. interactive and semantically–driven access to video surveillance content, 5. sharing and remixing content, and 6. automatic generation of personalized content for Internet portals. Feedback from the end user consultations and trials served for final tuning of prototypes, and for collecting feedbacks from potential markets.

All partners made presentations at academic conferences and industrial tradeshows to promote the project. Meeting were conducted with potential end users. APIDIS maintains contact with the APIDIS user community through a newsletter and mailing list.

More information follows about each of the 6 APIDIS prototypes.

1. The Autonomous production prototype. In 2010 the prototype was further improved:

• To solve the few visual artifacts pointed out by viewers during the mid-project trials.

• To reduce the computational complexity of the process. • To define production profiles, matching different user needs. Here, a profile is

defined by a particular instance of the production parameters, e.g. to forbid some views, or to favor continuous production at the cost of reduced closeness.

APIDIS-FP7-216023


According to most recent user feedbacks, the main issues that have to be addressed in the future deal with the production of close views and replays, so as to enrich the local stories of summaries that will be created a posteriori

The following is a list of events and activities where user feedback was collected, commercial interest generated and the project publicized for the Autonomous production prototype (distributed feature extraction for video production):

• The NAB tradeshow in April 2010 • The Intermedia project invited APIDIS to give a tutorial on the APIDIS

technologies and prototypes on 28 May 2010. • ICIP in September 2010. • At the IBC 2010 tradeshow where the production and summarization prototypes

were demonstrated to professional broadcasters, system integrators, IPTV providers, media distributors, telecommunication and mobile operators. In particular, feedback from these professional sectors have revealed strong interests from the market, and helped shape realistic market prospects and user requirements.

• ICT and 3D Stereo Media, in Sept. and Dec. 2010, respectively. • Video report in the New Scientist, February 2010. • Paper has been published about APIDIS by Focus Online in March 2010 • Ercim news article in July 2010 • Entreprise globale video in December 2010 • Agence Wallonne des Télécommunications article in December 2010 • An article in Le vif/L�express� magazine in January, 2011 • Initiated meetings with several companies, targeted as potential end-users

APIDIS at ICT 2010

APIDIS-FP7-216023


2. The Automatic and personalized video summarization prototype. In 2010, the

automatic and personalized video summarization of broadcasted content was improved based on the exploitation of audio analysis (collaboration with Telefonica, publication to ICME2010). Subsequent versions were released and demo’d. The summarization prototype framework was then extended to autonomous production and summarization of basketball summaries.

The following is a list of events and activities where user feedback was collected, commercial interest generated, and the project publicized for the Automatic and personalized video summarization prototype (Automatic and personalized summarization of a video sequence, knowing its salient segments)

• At the IBC 2010 tradeshow, and through subsequent industrial consultations. • With Telefonica where a collaboration demonstrated and exploited the

complementarily between the APIDIS summarization framework, and the tool designed by Telefonica to extract salient segments from audio commentary analysis.

• IEEE International Conference on Multimedia & Expo, Singapore, July 2010 • IEEE International Conference on Image Processing, Hong-Kong, Sept. 2010 • E-letter, IEEE Communications Society Multimedia Communications Technical

Committee, Special Issue on Human-centric Multimedia Communications, Invited paper, January 2011.

3. Generation of planar images. The generation of planar images prototype was improved in 2010 in terms of reconstructed image quality. Work was focused on the TV regularized method to reconstruct planar images from several omnicameras. This offers increased quality and/or semiarbitrary viewpoint reconstruction. Work also focused on possible solutions to speed up the time needed to process and image.

User feedback was collected about the reconstruction of planar images from panoramic views, or in semiarbitrary viewpoints (Generation of high resolution images in any viewing direction based on an array of omnicams) at:

• IBC 2010

4. Video surveillance prototype (Interactive browsing)

User feedback was collected for the video surveillance prototype (Interactive browsing) at:

• International Conference on distributed Smart Cameras, Atlanta, August 2010.

• IFSEC 2010

• Expoprotection

• ESSEN Security.

APIDIS-FP7-216023


• End-of-project trials - The purpose of this trial was to test the video surveillance interface developed as part of APIDIS. The input was a set of video streams recorded in a network of cameras, augmented with a set of trajectories automatically extracted for each camera. The APIDIS solution automatically learns two types of relationships between camera views: connections and overlaps. These relationships are used to enhance the browsing capabilities of a user reviewing the video streams. The interface also provides a set of tools that facilitate the search of specific events in the camera network. During the trial, we demonstrated the following two aspects of the video surveillance interface:

a) That the learned relationships actually benefit the user during browsing; b) That the search tools enable the user to find events of interest.

We defined the following four different modes for each scenario:

a) Very basic: no metadata provided to the user. b) Basic: trajectory metadata displayed on each video, but relationships that

have been learned about camera topology are not used. c) Normal: learned components used to suggest alternative related views. d) Auto-switch: camera view automatically switched when the system infers

that the selected object appears in a different camera view. 28 users were involved in the video surveillance end of project trials.

Sample log file for one videosurveillance user in the end-of-projec trials

After having analyzed the logs and questionnaires of 28 users, the following main conclusions are drawn:

• First, the prototype was well received and the users managed to take it in hand very quickly although the interface uses uncommon graphical user interface concepts to navigate in the video streams.

APIDIS-FP7-216023


• Then, all the proposed concepts have convinced people for their usefulness as well on the fact they could improve existing video surveillance applications.

• The analysis of both the objective and subjective parts of this assessment suggests that the best mode is the one with the capability to automatically switch the main view when the system has enough confidence on the matching of two objects in different views.

• When reliable inference is not reliable, the ‘normal’ mode appears to ba a valuable alternative to ‘basic’ solutions, but more efforts have to be devoted to the design of user-friendly and intuitive interfaces.

• Hence, taking into account users remarks to enhance the presentation of the interface and its usability and keeping the proposed version with the capability of automatically switching the main view would fit all users requests.

• The trial demonstrates the usefulness of the interactive and semantically driven video surveillance interface in a real case study. The (usefulness of the) system intelligence has been proven (which was the APIDIS goal); while the presentation aspects of the interface have to be enhanced.

5. Sharing and remixing content

User feedback was collected for the ‘sharing and remixing content’ trial.

• Collaboration with Telefonica, Spain. The outcomes of this collaboration has been presented during ICME2010 and have demonstrated that it is possible to summarize broadcasted soccer games strictly based on automatic audio-visual analysis tools.

• End-of-project trials. The purpose of this trial was to demonstrate that the automatic analysis capabilities offered by APIDIS can support novel approaches to re-use broadcasted content. The content considered as input to our trial corresponds to the content broadcasted by TV channels, augmented by low level metadata (e.g. type of shot, and kind of action, etc.) that are computed through automatic analysis tools. Those metadata primarily support the automatic segmentation of the game into a sequence of semantically meaningful actions. Our trial mainly aims at assessing the quality of the segmentation, and its usefulness when remixing content. Hence, we do not assess summarization, which has been extensively validated in previous tests (see publications in D7.2), but rather want to demonstrate how the automatic analysis capabilities of APIDIS can help the (professional) user to search for the segments to include in a remixed content. In practice, a list of pre-segmented actions is presented to the user, so that the user can select on the fly the segments (s) he wants to further process to produce his/her remixed content. 6 professional producers have run the test.

As a main conclusion, we have observed that the prototype was well received by the users, and the use of the tool was relevant to them. The automatic segmentation concept has convinced the users, and could facilitate the content reutilization. Improved segmentation accuracy would be a plus for a small

APIDIS-FP7-216023


fraction of users. Actions rating (for example based on automatic audio analysis tools) is also a desired feature expressed by some of the users.

6. Automatic generation of content for Internet portals.

User feedback was collected for the automatic generation of content for Internet portals

• ICT 2010

• IBC 2010

• 3D Stereo Media

• End-of-project trials: 17 participants responded to a questionnaire. They correspond to a representative panel of users since they include 4 content production professionals, 3 sport professionals, 6 basket-ball fans, and 4 computer vision experts. As a conclusion, our trial demonstrates the relevance and feasibility of the APIDIS sport production concept, but also reveals a number of challenges that need to be addressed before lucrative commercial exploitation of the concept. Specifically, even if the current solution appears to fulfill the expectations of basket-ball fans for (local) game coverage, it also appears that improved image quality or more accurate game analysis solutions are required to fully satisfy the requirements of content providers and sports professionals, respectively. Those questions are among the key concerns of the SPORTIC project, recently initiated by UCL under Walloon Region funding. Besides, the positive feedbacks collected during the trials contribute positively to the on-going initiative to create a spin-off.

APIDIS at 3D Stereo Media: Automatic production of content for internet portals

APIDIS Exploitation Prospects, beyond APIDIS…

APIDIS-FP7-216023


There is little doubt that there is a market demand for the type of product envisioned by APIDIS. However, work remains before the results can be commercialized. This work is ongoing in UCL through a two-year project funded by the Walloon Region. A prototype, integrating parts of the APIDIS solutions has already been deployed in a major Belgian basketball arena. It supports real-time autonomous production, and hopefully should be extended towards summarization in the coming months.

Further Information

Coordinator: Christophe De Vleeschouwer, [email protected]. tel.: +32 10 472543

Administration: [email protected] tel.: +34 93 238 1400

www.apidis.org

apidis public annual report apidis ... - europa

Documents