Transcription System using Automatic Speech Recognition (ASR)

for the Japanese Parliament (Diet)

Tatsuya Kawahara

(Kyoto University, Japan)

Brief Biography

• 1995 Ph. D. (Information Science), Kyoto Univ.• 1995 Associate Professor, Kyoto Univ.• 1995-96 Visiting Researcher, Bell Labs., USA• 2003- Professor, Kyoto Univ.

• 2003-06 IEEE SPS Speech TC member• 2006- Technical Consultant,

The House of Representative, Japan

Published 150~ papers in automatic speech recognition (ASR) and its applications

Web http://www.ar.media.kyoto-u.ac.jp/~kawahara/

Contents

1. Review of ASR technology

2. ASR system for the Japanese Diet

3. Next-generation transcription system of the Japanese Diet

Trend of ASR

style

Informal

Formal

one multipleNumber ofspeakers

FormalFormalpresentationpresentation

Classroom Classroom lectureslectures

Phone Phone conversationconversation Business Business

meetingsmeetings

Reading/Reading/Re-speakingRe-speaking

Broadcast Broadcast newsnews

SpontaneousSpontaneousspeechspeech

ParliamentParliament

Review of ASR technology (1/2)

• Broadcast News [world-wide]– Professional anchors, mostly reading

manuscripts– Accuracy over 90%

• Public speaking, oral presentations [Japan]– Ordinary people making fluent speech– Accuracy ~80% (close-talking mic.)

• Classroom lectures [world-wide]– More informal speaking– Accuracy ~60% (pin mic.)

Review of ASR technology (2/2)

• Telephone conversations [US]– Ordinary people, speaking casually– Accuracy 60%85%

• Business meetings [Europe/US]– Ordinary people, speaking less formally– Accuracy 70% (close mic.), 60% (distant mic.)

• Parliamentary meetings [Europe/Japan] – Politicians speaking formally– EU: plenary sessions: 90%– Japan: committee meetings: 85%

Deployment of ASRin Parliaments & Courts

• Some countries– Steno-mask & Voice writing– Re-speaking Commercial dictation software

• Some local autonomies in Japan– Direct recognition of politicians’ speech

• Japanese Courts– ASR for efficient retrieval from recorded sessions

• Japanese Parliaments (=Diet)– to introduce ASR; direct recognition of politicians’

speech– Mostly in committee meetings

…interactive, spontaneous, sometimes excited

Language-specific Issuesin Japanese

• Need to convert kana (phonetic symbol) to kanji• Conversion ambiguous many homonym

(ex.) KAWAHARA ( カワハラ ) 　→　河原　 (not 川原 )

– Very hard to type-in real-time– Only limited stenographers using special keyboards

can

• Difference in verbatim-style and transcript-style(ex.) おききしたいのですが ききたい（のです）

– Re-speaking is not so simple– need to rephrase in many cases

ASR Architecture

Signal processing

Acoustic model

Language model

Dictionary

RecognitionEngine

(decoder)

P(W/X) ∝P(W) ・ P(X/W) P(W)

X

P(P/W)

P(W)

P(X/P)P(X/W)

/a, i, u, e, o…/

京都 ky o: t o

京都 + の + 天気

output:W=argmax P(W/X)

Depend on input condition

Depend on application

Current Status of ASR

• Problems unsolved– Spontaneous/conversational speech– Noisy environments

• Including distant microphones

• Solutions ad-hoc– Collect large-scale “matched” data (corpus)

• Same acoustic environment, speakers (10hours~)• Cover same topics, vocabulary (~M words)

– Prepare dedicated acoustic & language models• Huge cost in development & maintenance

Contents

1. Review of ASR technology

2. ASR system for the Japanese Diet

3. Next-generation transcription system of the Japanese Diet

ASR Research in Kyoto Univ.

• Since 1960s, one of the pioneers

• Development of free software Julius

• Research in spontaneous speech recognition– 1999- Oral presentations– 2001- TV discussions– 2004- Classroom lectures– 2003- Parliamentary meetings

Free ASR Software: Julius

• Developed since 1997 in Kyoto-U & other sites• Open-source

multi-platform (Linux, Mac, Windows, iPhone)

• Open architecture– Independent from acoustic & language models

Ported to many languages

Ported to many applications (telephony, robot…)

• Standard model for Japanese• Widely-used research platform

http://julius.sourceforge.jp

Corpus of Parliamentary Meetings

• Cover all major committees and plenary sessions• 200 hours, 2.4M words• Faithful transcripts of utterances including fillers,

which are aligned with official minutes

{ えー } それでは少し、今 { そのー } 最初に大臣からも、{ そのー } 貯蓄から投資へという流れの中に { ま } 資するんじゃないだろうかとかいうような話もありましたけれども、 { だけど / だけれども } 、 { まあ } あなたが言うと本当にうそらしくなる { んで / ので }{ ですね、えー } もう少し { ですね、あのー } これは { あー } 財務大臣に { えー } お尋ねをしたいんです { が } 。{ ま } その { あの } 見通しはどうかということでありますけれども、これについては、 { あのー } 委員御承知の{ その } 「改革と展望」の中で { ですね } 、我々の今 { あのー } 予測可能な範囲で { えー } 見通せるものについてはかなりはっきりと書かせていただいて ( い ) るつもりでございます。

Cover pronunciationvariations

Cover poor articulation

Cover disfluencies &colloquial expressions

ASR modules oriented forSpontaneous Speech

Signal processing

Acoustic model

Language model

Dictionary

RecognitionEngine

(decoder)

P(W/X) ∝P(W) ・ P(X/W) P(W)

XP(X/W)

Corpus

Innovative techniques

ASR Performance

• Accuracy– Word accuracy 85% (Character accuracy 8

7% ）• Plenary sessions 90%• Committee meetings 80 ～ 87%

– 90% seems almost perfect– No commercial software can achieve!!

• Real-time factor 1-3– Latency in 10 min.

Related Techniques

• Noise suppression & dereverberation– Not serious once matched training data available

• Speaker change detection– Preferred – Current technology level seems not sufficient

• Auto-edit– Filler removal easy– Colloquial expression replacement non-trivial– Period insertion still research stage

Contents

1. Review of ASR technology

2. ASR system for the Japanese Diet

3. Next-generation transcription system of the Japanese Diet

The House of Representatives in Japan

• 2005: terminated recruiting stenographers

• 2006: investigated ASR technology for the new transcription system

• 2007: developed a prototype system and made preliminary evaluations

• 2008: system design

• 2009: system implementation

• 2010: trial and deployment

ASR system: Kyoto Univ. model integrated to NTT engine

Signal processing

Acoustic model

Language model

Dictionary

RecognitionEngine

(decoder)

P(W/X) ∝P(W) ・ P(X/W) P(W)

X

P(P/W)

P(W)

P(X/P)P(X/W)

/a, i, u, e, o…/

京都 ky o: t o

京都 + の + 天気

NTTCorp.

Kyoto Univ. House

Issues in Post-Editor

• For efficient correction of ASR errors and cleaning transcript into document-style

• Easy reference to original speech (+video)– by time, by utterance, by character (cursor)– Can speed up & down speech-replay

• Word-processor interface (screen editor); not line editor– to concentrate on making correct sentences– Serious misunderstanding between system

developers and stenographers!!

System Evaluation (@Kyoto)

• Subjects ： 18 students• Post-editing ASR outputs is more efficient than typing

from scratch, regardless of the accuracy Those hard for ASR are also hard for human

3456

789

10

50 55 60 65 70 75 80 85 90 95ASR accuracy

edit

time

(min

)

Type from scratch

Post-edit ASR output

System Evaluation (@Kyoto)• Subjective evaluation correlates with ASR accuracy• Threshold in 75% to have ASR preferred

1

2

3

4

5

6

7

50 55 60 65 70 75 80 85 90 95

ASR accuracy

Usa

bilit

y sc

ore

of A

SR

System Evaluation (@House)

• Subjects: 8 stenographers

• System: proto-type

• ASR-based system reduced the edit time, compared with current short-hand system– 78 min. 68 min. (for 5 min. segment)

• Threshold in ASR accuracy of 80%– 75% degradation in edit time;

a half say negative in using ASR

Side effect of ASR-based system

• Everything (text/speech/video) digitized and hyper-linked Efficient search & retrieval

• Less burden? may work on longer segments??

• Significantly less special training needed compared with current short-hand system

Conclusions

• ASR of parliamentary meetings is feasible, given a large collection of data– ~100 hour speech– ~1G word text (minutes)– Accuracy 85-90%

• Effective post-processing is still under investigation

• Automatic translation research is also ongoing