Unlocking Audio/Video Content with Speech Recognition Behrooz Chitsaz Director, IP Strategy Microsoft Research Frank Seide Lead Researcher Microsoft Research Kit Thambiratnam Researcher Microsoft Research Microsoft Research Multimedia Research Speech Search Video summarization Semantic extraction Face identification Object recognition Visual search 3D Modeling Speech Applications Indexing Search Metadata extraction Advertisin g Transcription Meeting notes Closed caption Voic Translation Translating phone Speech as interface Speech as 1 st class content Mobile access Search Automation PC application Web service Text input Dictation Mobile access Search Automation PC application Web service Text input Dictation Indexing Search Metadata extraction Advertising Transcription Meeting notes Closed caption Voic Translation Translating phone meta-data surrounding & anchor text, URL top-N lists, collaborative filtering editorial meta-data file content itself keyword search in audio track using speech recognition Searching Media Today Demo Spectral Analysis Matching (Decoding) time alignment most likely hypothesis W=argmax (w 1..w N ) p(o t..o |w 1..w N ) P(w 1..w N ) Acoustic Models p(o t..o |phoneme) Dictionary P(phonemes|w) Grammar (Language Model) P(w 1..w N ) Hello World o 1..o T (w 1..w N )^ Speech recognition speech recognition in a nutshell Acoustic Models p(o t..o |phoneme) Dictionary P(phonemes|w) Grammar (Language Model) P(w 1..w N ) Speech recordings + full manual transcripts Speech recognition Acoustic Models p(o t..o |phoneme) Dictionary P(phonemes|w) Grammar (Language Model) P(w 1..w N )... microscopem:s ay:n k:n r:n ax:n s:n k:n ow:n p:e microsecond m:s ay:n k:n r:n ax:n s:n eh:n k:n ax:n n:n d:e microsecondm:s ay:n k:n r:n ow:n s:n eh:n k:n ax:n n:n d:e microsoftm:s ay:n k:n r:n ax:n s:n ao:n f:n t:e microsoftm:s ay:n k:n r:n ow:n s:n ao:n f:n t:e Speech recognition Acoustic Models p(o t..o |phoneme) Dictionary P(phonemes|w) Grammar (Language Model) P(w 1..w N ) this is a this is about this is absolutely this is accomplished this is actually is a barnyard is a barometer is a baseball is a baseless is a baseline Speech recognition Challenges Speaker accent Background noise Reverberation Vocabulary Language lattice-based indexing into this bank account lattice-based indexing into this bank account expected benefits from indexing lattices: alternative recognition candidates recall++ confidence scores precision++ (time information user experience) expected benefits from indexing lattices: alternative recognition candidates recall++ confidence scores precision++ (time information user experience) Speech Word statistics Metadata NP extraction Web query builder Recognizer Bing Search Docs Queries Docs Base Dict Base LM Adapt Dictionary Adapt Language Model Adapted Dict Adapted LM Vocabulary Adaptation from NLC group Architectural decisions SQL Server(s) 1. Submit audio/video to index 2. Get back AIB 3. Import AIB in SQL Web server(s)Media server(s) 4. Search/Retrieve results video RSS feed Azure integration Cloud computing made simple Windows Azure + Power shell = Cloud computing at your fingertips Demo media content submission Microsoft Research Tell us if you are interested Tell us if you are interested Visit us: Visit us: Thank you! Questions?