LREC 2006 workshop Genova 28 May 2006

LREC 2006 workshop Genova 28 May 2006

EU-funded research initiatives: EU-funded research initiatives:

- Interactive Systems- Interactive Systems

- New Interfaces- New Interfaces

- Language Technology - Language Technology

Xavier Gros Xavier Gros

DG INFSO, European Commission, LuxembourgDG INFSO, European Commission, Luxembourg

ContentsContents

• What has been done so far?

• Projects running so far under “FP6”

• 2nd batch of proposals under negotiation

• “FP7” info

• What is it?

• By when?

• What’s in it for you?

Our Mission Our Mission

The challenge is:

• To make interfaces as simple as possible, and interaction as easy as possible

• For citizens to be able to access, receive and use information in their own language

• We support the research, development and integration of:

• Advanced technologies for interfaces & interaction

• Integrated multilingual information systems and services

We focus on:

• Natural interaction between humans and physical or virtual environments

• Multilingual communication systems for unrestricted domains, including real-time understanding of spoken and gesture input

E1 Interfaces

So what happened? So what happened?

• Calls: 1st in late 2002, and 5th in mid-2005

• Proposals: 169 requesting a total funding of 804 M€

• High quality: 58 asking for a total funding of 296 M€

• Projects: 29 projects for a total funding of ~136 M€ (15 start 2006)

• Participants: ~320

• Type: 7 IP’s, 4 NoE’s, 17 STReP’s, 1 SSA

1st Call:

• CHIL - Computers In the Human Interaction Loop http://chil.server.de/

• AMI - Augmented Multi-party Interaction http://www.amiproject.org/

• TC-STAR - Technology and Corpora for Speech to Speech Translation http://www.tc-star.org/http://www.tc-star.org/

5th Call:

• Augmented Multi-party Interaction with Distance Access

• Intelligent, Persistent, Personalised Multimodal Interfaces to the Internet

• Conveying Affectiveness in Leading-edge Living Adaptive Systems

• Multimodal Interfaces for Capturing and Transferring of Skills

Integrated Projects Integrated Projects

1st Call:

• SIMILAR - Federating the communities of human-computer interaction and signal processing http://www.similar.cc http://www.similar.cc

• HUMAINE - Human-Machine Interaction Network on Emotion http://emotion-research.net http://emotion-research.net

• ENACTIVE - Enactive Interfaces http://www.enactivenetwork.orghttp://www.enactivenetwork.org

And an SSA called MWEB - Multimodal Web Interaction http://www.w3.org/2004/MWeb/http://www.w3.org/2004/MWeb/

Networks of Excellence Networks of Excellence

• MATRIS - Markerless Real-time Tracking for Augmented Reality Image Synthesis http://www.ist-matris.org/http://www.ist-matris.org/

• T’nD - Touch and Design http://www.kaemart.it/touch-and-http://www.kaemart.it/touch-and-design/design/

• TAI-CHI - Tangible Acoustic Interfaces for Computer-Human Interaction http://www.taichi.cf.ac.uk/ http://www.taichi.cf.ac.uk/

• DIVINES - Diagnostic and Intrinsic Variabilities in Natural Speech http://www.divines-project.org/http://www.divines-project.org/

• TALK - Talk and Look: Tools for Ambient Linguistic Knowledge http://www.talk-project.org/http://www.talk-project.org/

• HIWIRE - Human Input That Works In Real Environments http://www.cvsp.cs.ntua.gr/projects/bin/view/HIWIRE/WebHomhttp://www.cvsp.cs.ntua.gr/projects/bin/view/HIWIRE/WebHomee

Targeted Research 1st call Targeted Research 1st call

Statistical Multilingual Analysis for Retrieval and Translation

• New and more effective statistical approaches to machine translation and cross-lingual textual information access

Multi-modal and Multi-Sensor Zero-distraction Interaction Interface

• Develop a mobile human-computer interface for eye-busy and hand-busy applications in zero-distraction mode

Multimodal Interactions Analysis of Users within Controlled Environments

• Techniques for analysing eye gaze/fixation, eye blink and body movement in human-controlled environment interaction

3D and Speech Interfacing Technologies

• 3D information visualisation and manipulation based upon the fusion of holographic displays, 3D hand tracking and natural speech processing

Sound and Tangible Interfaces for Novel Product Shaping

• Develop a new type of interface for natural hands-free interaction with virtual shapes

Statistical and Hybrid Machine Translation between all European Language

• Combining statistical techniques with linguistic knowledge in machine translation for all EU language pairs

Targeted Research 5th call Targeted Research 5th call

subject to negotiation

OpenInterfaceOpenInterface

• OpenInterface is an open source platform for rapid development of multimodal OpenInterface is an open source platform for rapid development of multimodal interactive systems interactive systems

Question Answering Learning Technologies in a Multilingual and Multimodal Question Answering Learning Technologies in a Multilingual and Multimodal EnvironmentEnvironment

• Open domain question-answering (QA) as a new way to search, integrating Open domain question-answering (QA) as a new way to search, integrating multilingual open domain QA, context-aware QA, and machine learning multilingual open domain QA, context-aware QA, and machine learning technologies technologies

Distant-talking Interfaces for Control of Interactive TVDistant-talking Interfaces for Control of Interactive TV

• Develop hands-free voice interaction for use with a interactive-TV remote controlDevelop hands-free voice interaction for use with a interactive-TV remote control

Action Recognition and Tracking based on time-of-flight SensorsAction Recognition and Tracking based on time-of-flight Sensors

• Algorithms and prototype for a time-of-flight (ToF) camera that captures 3D Algorithms and prototype for a time-of-flight (ToF) camera that captures 3D information and can track head and eye movement information and can track head and eye movement

Spoken Language Understanding in Multilingual Communication SystemsSpoken Language Understanding in Multilingual Communication Systems

• New algorithms and tools for robust spoken language understanding in multilingual New algorithms and tools for robust spoken language understanding in multilingual telephone servicestelephone services

Targeted Research 5th call Targeted Research 5th call

subject to negotiation

E1 FP6 call-1 portfolioE1 FP6 call-1 portfolio

HIWIRE

SIMILAR

T’n D

MATRIS

TAI-CHI

DIVINES

ENACTIVE

TC-STAR

AMI

TALK

HUMAINECHIL

MULTILINGUALITY

SPEECH TECHNOLOGIES

MULTIMODALITY – NEW INNOVATIVE INTERFACES

~73 million EURO

INTERACTION

IP

STREP

NoE

MWEB

SSA

PASCAL

IST call 5 IST call 5

Closed: Closed: 21 September 2005 21 September 2005

Budget: Budget: 63 M€63 M€

Number of proposals:Number of proposals: 101101

Under negotiation:Under negotiation: 15 new projects15 new projects

• 4 IP:4 IP: 6-10 MEUR 6-10 MEUR 3-5 3-5 yearsyears

• 11 STREP:11 STREP: 2-3 MEUR2-3 MEUR 2.5-3 years2.5-3 years

Projects launch:Projects launch: around September 2006 around September 2006

subject to negotiation

FP7 (2007-2013)

4 main Specific Programmes

•Cooperation Implementation of ICT in the “Cooperation” partto gain leadership in key scientific and technological areas by supporting cooperation between universities, industry, research centres and public authorities across the European Union as well as the rest of the world.

•Ideas to support the truly creative scientists, engineers and scholars, whose curiosity and thirst for knowledge are most likely to make the unpredictable and spectacular discoveries that can change the course of human understanding and open up new vistas for technological progress and solving enduring social and environmental problems.

•Peopleto strengthen, qualitatively and quantitatively, human resources in R&D in EU

•Capacitiesenhance research and innovation capacity throughout Europe

http://cordis.europa.eu.int/ist/about/fp7.htm

Societal challenges

Business and industry

Content, creativity personal

development

Personal environments

Intelligent infrastructures

Robotic systems

Home environments

Nano-electronics, photonics and micro/nano-systems

CommunicationNetworks

Embedded systems,Computing andcontrol

Software, Grids,Security anddependability

Knowledge, cognitiveand learning systems

Simulation, visualisationInteraction and mixedRealities

New perspectives in ICTDrawing on other science and technology disciplines

FET

Trust and confidence

Specific Programme: ICT Collaborative Research – Activities

1st activity: Technology pillarspushing the limits of performance, usability, dependability, cost-efficiency

2nd activity: Integration Environments

integrating multi-technology sets that underlie new functionalities,

services and applications

3rd activity: Applications researchproviding the knowledge and the means

to develop a wide range of ICT-based services and applications and supporting

research at the frontiers of knowledge

Draft FP7 Texts

• FP7 Proposal by Commission to Council & FP7 Proposal by Commission to Council & Parliament Parliament „„BUILDING THE EUROPE OF KNOWLEDGE”BUILDING THE EUROPE OF KNOWLEDGE”

• Information & Communication Technologies • ICT Technology Pillar

Simulation, visualisation, interaction and mixed realities: tools for innovative design and creativity in products, services and digital media, and for natural, language-enabled and context-rich interaction and communication.

• Specific ProgrammesSpecific Programmes

• Co-operation – ICTSimulation, visualisation, interaction and mixed realities: tools for modelling, simulation, visualisation, interaction, virtual, augmented and mixed reality and their integration in end-to-end environments; tools for innovative design and for creativity in products, services and digital audio-visual media; more natural, intuitive and easy-to-use interfaces and new ways to interact with technology, machines, devices and other artefacts; multilingual and automatic machine translation systems.

“Cooperation” – Collaborative Research – Themes

“Cooperation” – Collaborative Research – Themes

Health: 5984 m€ (18%)

Food, agri, biotech: 1936 m€ (6%)

ICT: 9080 m€ (28.5%)

Nano, materials, production: 3467 m€ (11%)

Energy: 2236 m€ (7%)

Environment: 1886 m€ (6%)

Transport: 4150 m€ (13%)

Socio-econ research: 607 m€ (2%)

Space and security: 2858 m€ (9%)

Preparing for FP7Preparing for FP7

• Interactive Systems & New Interface Interactive Systems & New Interface TechnologiesTechnologies

• 30-31 January 2006, Luxembourg

• Industrial Workshop on Simulation, Industrial Workshop on Simulation, Visualisation, Interaction, and Mixed RealitiesVisualisation, Interaction, and Mixed Realities

• 14-15 February 2006, Luxembourg

• Machine Translation & Human Language Machine Translation & Human Language TechnologiesTechnologies

• 23 February 2006, Luxembourg

Some pointers …Some pointers …

Important issues highlighted in FP7 consultations:Important issues highlighted in FP7 consultations:

• Individual interaction modalities need to be made far more accurate and robust. One important application challenge is the lack of robust and cost-effective hands-free and eye-free interfaces (e.g. hand-busy and eye-busy situations).

• People want interaction and information access in their own language, and we must understand how to develop interfaces that are capable of dealing more efficiently and cost-effectively with different languages. A specific challenge is machine-assisted translation.

• A major research challenge involves the shift from explicit interaction to implicit human-computer interaction, where the computer adapts to the user, their language and context. In addition, we must develop ways to create adaptive interfaces that enable personalised interaction. The move towards personalised interaction requires us to address ethical and privacy issues.

under discussion

Some pointers …Some pointers …

Important issues highlighted in FP7 consultations:Important issues highlighted in FP7 consultations:

• We must be able to understand and model the behaviour of users in the way they interact with machines, with services, with their surrounding environment and with each other.

• We must demonstrate how effective multimodal interaction helps people and we must answer the question about the benefits that multimodal interaction brings to users. One way forward is to stress-test new interaction modalities in industrial prototypes. Standards and open source software platforms are an essential prerequisite for the rapid adoption of prototype applications.

• We need common evaluation and benchmarking schemes, test and training data, tools, and metrics. Open competitions are an effective way to mobilise the research community.

under discussion

Some pointers …Some pointers …

• Weak integration into real applications

• Language resources: lack of integration and sharing

• Speech understanding still difficult

• MT: lack of robustness, questionable quality

• No evaluation in specific real-life situations

Perspectives

• Resource sharing, integration of applications, Resource sharing, integration of applications, collaboration across scientific domainscollaboration across scientific domains

• Fully automatic end-to-end MT systemsFully automatic end-to-end MT systems

• Easy to use and robust MT systemsEasy to use and robust MT systems

• Text and speech recognitionText and speech recognition

• Linguistic infrastructureLinguistic infrastructure

Perspective 1: Machine Translation

• Fully automatic end-to-end systems Fully automatic end-to-end systems and semi-automatic machine and semi-automatic machine translation systems (spoken and translation systems (spoken and written languages) to support a human written languages) to support a human translator/interpreter. Automatic translator/interpreter. Automatic multilingual speech-recognition multilingual speech-recognition toolkits. Initial objective is “high toolkits. Initial objective is “high quality translation” of (semantically) quality translation” of (semantically) domain-limited but otherwise domain-limited but otherwise (syntactically) unrestricted languages.(syntactically) unrestricted languages.

Perspective 2: Robustness

• Easy to use machine assisted Easy to use machine assisted translation tools and portable and translation tools and portable and robust translation devices for text-to-robust translation devices for text-to-text and speech-to-speech translation text and speech-to-speech translation across multiple languages. Effective across multiple languages. Effective affordable, reusable systems. affordable, reusable systems. Interfaces and systems for on-line Interfaces and systems for on-line translation of web pages.translation of web pages.

Perspective 3: Resources

• Shared databases corpora and free Shared databases corpora and free distribution among research non-profit distribution among research non-profit institutesinstitutes

• Open source toolkits and modulesOpen source toolkits and modules

General FP6:General FP6:

http://europa.eu.int/comm/research/fp6/http://europa.eu.int/comm/research/fp6/

IST:IST:

http://www.cordis.lu/isthttp://www.cordis.lu/ist

FP7:FP7:

http://www.cordis.lu/ist/about/fp7.htmhttp://www.cordis.lu/ist/about/fp7.htm

Xavier GrosXavier Grose-mail: e-mail: xavier.gros@cec.eu.intxavier.gros@cec.eu.int

tel:tel: +352 4301 37193+352 4301 37193

Further informationFurther information