university of east anglia, uk dicta-sign partner 3

1Dicta-Sign Kick Off Meeting, Athens, 06 February 2009

University of East Anglia, UK

Dicta-Sign Partner 3

Prof John Glauert

Virtual Humans Group

School of Computing Sciences


Major Roles in Dicta-Sign• WP3Synthesis and Animation• Lead

• WP6Sign Language Corpora• BSL Lexicon and Corpus

• WP7 Bidirectional Integration• Lead


Partnerships


Virtual Signing at UEA• Dicta- Sign• John Glauert• Richard Kennaway, Post Doc• Vince Jennings, Engineer

• Face Tracking, Audio-Visual Speech • Barry Theobald

• Crowd Modelling, Urban Modelling• Prof Andy Day


Independent Television

CommissionTelevirtual

University of East Anglia

The Post Office

Royal Institute for Deaf People

Instituutvoor Doven

Hamburg University

Institut für Rundfunktechnik

Institut National des Télécommunications

The ViSiCAST Consortium


The eSIGN Consortium

Televirtual

University of East Anglia

Royal National Institute for Deaf People

ViataalHamburg University

NorfolkCounty Council


Sign Language Animation from Sign Gesture Notation

Prof John GlauertUniversity of East Angliawith: Norfolk County Council, RNID, Televirtual Ltd.


Some Signing Avatars


Virtual Characters


Virtual Characters that can Sign

•


Virtual Human Signing Approaches


Motion Capture• Very lifelike animation

• Time-consuming to set up• Blending of signs• Combining signs from different

signers


Hand-Crafted Animation• Define Key Frames• Interpolate between Key Frames

• Can give very lifelike animation

• Time consuming (maybe hours per sign)• Blending of signs still required


Synthetic Signing• Synthesis from Abstract Representation

• Quick to create lexicon• a few minutes to transcribe a sign

• Instantly retargettable to any avatar with humanoid topology

• Automatic Blending

• Low Bandwidth


Uses of Signing Avatars:Supporting Communication


Simon the Signer, 1997-1999

• Simon-the-Signer [Broadcast TV]• Generate signed accompaniment to

broadcast• Teletext stream as source• SSE – Sign Supported English• Funded by the ITC


Simon the Signer, 1997-1999

• Developing transmission technology

• virtual signer in set-top boxes• transmission of signing from text subtitles

TV Capture Card

Teletext Stream

Audio/Video Stream

Avatar Software Mixer

Computer


SiSi: Say it Sign it, 2007

• IBM with UEA and RNID• Uses Speech recogniser• Might help some meetings• Not pure BSL

• www.youtube.com/watch?v=RarMKnjqzZU


SiSi: Say it Sign it, 2007• “I’ve seen an exciting movie”


TESSA, 1998-2002

• TESSA [Retail, PO]• Using speech recognizer• Convert counter-clerk’s voice input to text• Generate sign stream from text• BSL – limited repertoire• Funded by the Post Office


Post Office Counter Services

• Post Offices transact business with almost all Deaf people

• Counter Clerk asks questions using speech - No ‘back channel’ yet

• Customer• Listens or• Reads or• Watches Virtual Human Signer

• Real trials at 5 sites


VANESSA, 2002 – 2005

• Council Information Centres (CIC’s)• Signing avatar conveys signed information• Text version also displayed• Signs created from special notation• Developed and enhanced

version of TESSA (Post Office)


VANESSA – Customer’s View


VANESSA – Clerk’s View


Uses of Signing AvatarsInformation Services?


Web ApplicationsWeather Forecast: NGT,BSL,DGS


Weather Forecasts

Today Tonight Sunday

Mostly cloudy and dry in the morning. Rain breaking out during the afternoon.Maximum temperature 13 °C.

Met Office for London and South East Met Office for London and South East EnglandEngland

Rain becoming heavy and persistent in the evening and continuing through the night.


Web ApplicationsWeather Forecasts

• Grammar for normal Weather Phrases• Sign Language version for each Phrase• Forecast is sequence of Phrases• Generate Common XML Weather Model• XSLT processing for each Sign Language• XSLT processing for Spoken Languages

• Content creation by non- signers


Uses of Signing AvatarsEducation?


Signing Avatars in Education• PC Passport• SQA computer skills qualification• Need to be fair to all deaf candidates• Not all interpreters have IT knowledge• Different signs used by different schools• Students will have a dictionary of IT signs


PC Passport Assessment•


PC Passport Dictionary


Performing Hands, 2006-2009


Performing Hands, 2006-2009• GameLab London, RNID, UEA

• Teach literacy for Deaf Children• BSL and English literacy• Mixes Video, 2D Animation, Avatars• Games and Stories• Shockwave Application

• BAFTA Nominated• www.youtube.com/user/blahdeblahdeblahdebl• www.performinghands.com


LinguaSign 2006-2009


LinguaSign 2006-2009• Primary Modern Foreign Language

Learning• EU Socrates Lingua 2• Vocabulary supported by iconic gestures

(“signs”)• Speech in French, Dutch, English• Mix of conventional and synthetic

animation


Thanks for your attention

university of east anglia, uk dicta-sign partner 3

Documents