cas open api 2009 hack day
Post on 20-Jan-2015
908 Views
Preview:
DESCRIPTION
TRANSCRIPT
CAS Open APIChloris KuoY! Search Asia2009/10/17
CAS?
Content Analysis Service
成語新解
• 斷章取義 :
對一篇文 進行 詞,然後 出符合本文主要意 的關鍵字
斷章 取義
CAS Open API
• Functionality-Word Segmentation (WS)
- POS tagging - Language detection
-Keyword Extraction (KE)
Word Segmentation - Concept
• Word-A minimal-form meaning pair
• Word Segmentation-王建民首場熱身賽預定下星期二在客場出戰太空人隊
• Part-of-Speech (POS)-Noun, Verb, Adjective, etc.
王建民∕首場∕熱身賽∕預定∕下∕星期二∕在∕客場∕出戰∕太空人隊
Challenges
• Ambiguity-下雨天留客天天留我不留-不要把大便當飯吃
• Unknown Word-named entity (ex. 丁丁 )-acronym (ex. 貓纜 )-derivation (ex. 數位化 )-compound (ex. 台北火車站 )
Methodology
• Lexical rule-based-Maximum Matching
• Grammar-based
• Learning-based-HMM
Keyword Extraction
• Extract keywords representing the concept of a given context.
• Dictionary-based and statistical method
tf-idf
• A measurement widely used in text mining and information retrieval
• tf (term frequency)-The frequency of a specific term appears in a given context
• idf (inverse document frequency)-The general importance of a specific term
tf-idf (cont.)
tf-idf (cont.)
• 飛呀!飛呀!小飛俠!在那天空邊緣拚命的飛翔看看他多麼勇敢多麼堅強
為了正義他要消滅敵人為了公理他要奮鬥到底
飛呀!飛呀!飛呀!小飛俠!衝呀!衝呀!衝呀!小飛俠!
我愛科學小飛俠我愛科學小飛俠多勇敢呀!小飛俠!
tf-idf (cont.)
• 飛∕呀∕飛∕呀∕小飛俠在∕那∕天空∕邊緣∕拚命∕的∕飛翔看看∕他∕多麼∕勇敢∕多麼∕堅強
為了∕正義∕他∕要∕消滅∕敵人為了∕公理∕他∕要∕奮鬥∕到底
飛∕呀∕飛∕呀∕飛∕呀∕小飛俠衝∕呀∕衝∕呀∕衝∕呀∕小飛俠
我∕愛∕科學∕小飛俠我∕愛∕科學∕小飛俠多∕勇敢∕呀∕小飛俠
tf idf tf*idf (10-6)
呀 9 1/8394663 1.07
公理 1 1/110369 9.06
小飛俠 6 1/151774 39.53
Applications: Dynamic Content Match
Applications: Yahoo! Knowledge Plus
Applications: Yahoo! Sports
Applications: Related Search for Yahoo! News
Applications: Atlaspost
Resources
• CAS API Doc-http://tw.developer.yahoo.com/cas/
• Sample Code-PHP
- http://program.faryne.tw/2009/02/yahoo-apiphp.html-Ruby
- http://blog.xdite.net/?p=971-Java
- http://blog.cheyingwu.tw/index.php/2009/02/27/ckip-client-03/
top related