cmu 2011 watson event
DESCRIPTION
Background information on Watson shown to the CMU community viewing of the final Jeopardy game.TRANSCRIPT
![Page 1: CMU 2011 Watson Event](https://reader033.vdocuments.mx/reader033/viewer/2022052522/55490831b4c90553458b70b9/html5/thumbnails/1.jpg)
© 2011 IBM Corporation© 2011 IBM Corporation
Building WatsonA Brief Overview of the Jeopardy! Challenge
Dr. Mark ShermanIBM Software Group Strategy
![Page 2: CMU 2011 Watson Event](https://reader033.vdocuments.mx/reader033/viewer/2022052522/55490831b4c90553458b70b9/html5/thumbnails/2.jpg)
© 2011 IBM Corporation
Capture the imagination– The Next Deep Blue
Engage the scientific community– Envision new ways for computers to impact society & science– Drive important and measurable scientific advances
Be Relevant to IBM Customers– Enable better, faster decision making– Business Intelligence, Knowledge Discovery and Management, Government,
Compliance, Publishing, Legal, Healthcare, Business Integrity, Customer Relationship Management, Web Self-Service, Product Support, etc.
A Grand Challenge Opportunity
2
![Page 3: CMU 2011 Watson Event](https://reader033.vdocuments.mx/reader033/viewer/2022052522/55490831b4c90553458b70b9/html5/thumbnails/3.jpg)
© 2011 IBM Corporation
Informed Decision Making: Search vs. Expert Q&A
Decision Maker
Search EngineFinds Documents containing Keywords
Delivers Documents based on Popularity
Has Question
Distills to 2-3 Keywords
Reads Documents, Finds Answers
Finds & Analyzes Evidence
![Page 4: CMU 2011 Watson Event](https://reader033.vdocuments.mx/reader033/viewer/2022052522/55490831b4c90553458b70b9/html5/thumbnails/4.jpg)
© 2011 IBM Corporation
Informed Decision Making: Search vs. Expert Q&A
ExpertUnderstands Question
Produces Possible Answers & Evidence
Delivers Response, Evidence & Confidence
Analyzes Evidence, Computes Confidence
Asks NL Question
Considers Answer & Evidence
Decision Maker
![Page 5: CMU 2011 Watson Event](https://reader033.vdocuments.mx/reader033/viewer/2022052522/55490831b4c90553458b70b9/html5/thumbnails/5.jpg)
© 2011 IBM Corporation
Informed Decision Making: Search vs. Expert Q&A
Decision Maker
Search EngineFinds Documents containing Keywords
Delivers Documents based on Popularity
Has Question
Distills to 2-3 Keywords
Reads Documents, Finds Answers
Finds & Analyzes Evidence
ExpertUnderstands Question
Produces Possible Answers & Evidence
Delivers Response, Evidence & Confidence
Analyzes Evidence, Computes Confidence
Asks NL Question
Considers Answer & Evidence
Decision Maker
![Page 6: CMU 2011 Watson Event](https://reader033.vdocuments.mx/reader033/viewer/2022052522/55490831b4c90553458b70b9/html5/thumbnails/6.jpg)
© 2011 IBM Corporation6
Broad Domain
Our Focus is on reusable NLP technology for analyzing vast volumes of as-is text. Structured sources (DBs and KBs) provide background knowledge for interpreting the text.
We do NOT attempt to anticipate all questions and build databases.
In a random sample of 20,000 questions we found2,500 distinct types*. The most frequent occurring <3% of the time.
The distribution has a very long tail.
And for each these types 1000’s of different things may be asked.
*13% are non-distinct (e.g, it, this, these or NA)
Even going for the head of the tail willbarely make a dent
We do NOT try to build a formal model of the world
![Page 7: CMU 2011 Watson Event](https://reader033.vdocuments.mx/reader033/viewer/2022052522/55490831b4c90553458b70b9/html5/thumbnails/7.jpg)
© 2011 IBM Corporation7
What It Takes to compete against Top Human Jeopardy! PlayersOur Analysis Reveals the Winner’s Cloud
Winning Human Performance
Winning Human Performance
Grand Champion Human Performance
Grand Champion Human Performance
Each dot – actual historical human Jeopardy! games
More ConfidentMore Confident Less ConfidentLess Confident
![Page 8: CMU 2011 Watson Event](https://reader033.vdocuments.mx/reader033/viewer/2022052522/55490831b4c90553458b70b9/html5/thumbnails/8.jpg)
© 2011 IBM Corporation8
What It Takes to compete against Top Human Jeopardy! PlayersOur Analysis Reveals the Winner’s Cloud
Winning Human Performance
Winning Human Performance
2007 QA Computer System2007 QA Computer System
Grand Champion Human Performance
Grand Champion Human Performance
Each dot – actual historical human Jeopardy! games
More ConfidentMore Confident Less ConfidentLess Confident
Computers?Not So Good.
![Page 9: CMU 2011 Watson Event](https://reader033.vdocuments.mx/reader033/viewer/2022052522/55490831b4c90553458b70b9/html5/thumbnails/9.jpg)
© 2011 IBM Corporation
Baseline
v0.1 12/07
v0.3 08/08
v0.5 05/09
v0.6 10/09
v0.7 04/10
v0.4 12/08
DeepQA: Incremental Progress in Answering Precision: 6/2007-4/2010
v0.2 05/08
![Page 10: CMU 2011 Watson Event](https://reader033.vdocuments.mx/reader033/viewer/2022052522/55490831b4c90553458b70b9/html5/thumbnails/10.jpg)
© 2011 IBM Corporation
One Jeopardy! question can take 2 hours on a single 2.6Ghz CoreOptimized & Scaled out on 2880-Core IBM HPC using UIMA-AS,
Watson is answering in 2-6 seconds.
Question100s Possible
Answers
1000’s of Pieces of Evidence
Multiple Interpretations
100,000’s scores from many simultaneous Text Analysis Algorithms100s sources
. . .
HypothesisGeneration
Hypothesis and Evidence Scoring
Final Confidence Merging & Ranking
SynthesisQuestion &
Topic Analysis
QuestionDecomposition
HypothesisGeneration
Hypothesis and Evidence Scoring Answer &
Confidence
![Page 11: CMU 2011 Watson Event](https://reader033.vdocuments.mx/reader033/viewer/2022052522/55490831b4c90553458b70b9/html5/thumbnails/11.jpg)
© 2011 IBM Corporation
Potential Business Applications
Tech Support: Help-desk, Contact Centers
Healthcare / Life Sciences: Diagnostic Assistance, Evidenced-Based, Collaborative Medicine
Enterprise Knowledge Management and Business Intelligence
Government: Improved Information Sharing and Security
![Page 12: CMU 2011 Watson Event](https://reader033.vdocuments.mx/reader033/viewer/2022052522/55490831b4c90553458b70b9/html5/thumbnails/12.jpg)
© 2011 IBM Corporation
The Core Technical TeamResearchers and Engineers in NLP, ML, IR, KR&R and CL at
IBM Labs and a growing number of universities
![Page 13: CMU 2011 Watson Event](https://reader033.vdocuments.mx/reader033/viewer/2022052522/55490831b4c90553458b70b9/html5/thumbnails/13.jpg)
© 2011 IBM Corporation
THANK YOU