introduction to information retrievalleeck/ir/score2.pdf · 2017-03-12 · introduction to...
TRANSCRIPT
![Page 1: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/1.jpg)
Introduction to Information Retrieval
Introduction to
Information Retrieval
Scoring
Part 2
Park Cheon Eum 1
![Page 2: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/2.jpg)
Introduction to Information Retrieval
Index 첫 번째 : stemmer 맛보기
stemmer로 AP88s 문서셋을 postingList로 만들기까지
두 번째 : Sed/Ack 복습하기
Sed/Ack로 AP88s 문서셋을 postingList로 만들기
세 번째
Porter stemmer 추가 (Porter stemmer 모듈은 주어짐) 및 Sed/Awk 대신 Python을 이용한 Posting list 생성 모듈 구현
2
![Page 3: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/3.jpg)
Introduction to Information Retrieval
Introduction to
Information Retrieval
첫 번째 stemmer 맛보기
3
![Page 4: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/4.jpg)
Introduction to Information Retrieval
AP88.txt
Document ID
Text
필요한 것은
ID값과 Text 4
![Page 5: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/5.jpg)
Introduction to Information Retrieval
AP88s.txt
5
쉽게 하기 위해 AP88.txt 문서셋을
AP88s.txt로 작게 줄여봤다.
※AP88s.txt를
사용하시면
됩니다.
![Page 6: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/6.jpg)
Introduction to Information Retrieval
Stemmer Preview
6
Stemmer를 이용하여 AP88s.txt
문서셋을 stemming해봅시다.
![Page 7: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/7.jpg)
Introduction to Information Retrieval
AP88s.txt
7
위치에서 stemming 합시다.
stemming 명령어
결과
간단하게 : 특문 제거, 소문자화, 어미제거 됨.
![Page 8: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/8.jpg)
Introduction to Information Retrieval
stemmedAP88s를 minGW안으로
8
![Page 9: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/9.jpg)
Introduction to Information Retrieval
이 참에 tf.txt와 posting.txt까지
만들어보자
9
![Page 10: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/10.jpg)
Introduction to Information Retrieval
Stemmer로 만든 postingList
10
![Page 11: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/11.jpg)
Introduction to Information Retrieval
Stemmer로 만든 postingList
11
여기서 만든 postingList를 가지고
세 번째 section에서
scoring을 하면 됩니다.
![Page 12: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/12.jpg)
Introduction to Information Retrieval
Introduction to
Information Retrieval
두 번째 Sed/Ack 복습하기
12
![Page 13: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/13.jpg)
Introduction to Information Retrieval
Sed/Ack로 AP88.txt > posting
13
![Page 14: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/14.jpg)
Introduction to Information Retrieval
Text file을 python에서 사용하기
Text file들을 python 폴더 안으로 이동 시킵시다.
14
Sed/Ack로 AP88.txt > posting
![Page 15: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/15.jpg)
Introduction to Information Retrieval
Text file을 python에서 사용하기
15
Sed/Ack로 AP88.txt > posting
![Page 16: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/16.jpg)
Introduction to Information Retrieval
Sed/Ack부분 – Scoring (1)
16
![Page 17: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/17.jpg)
Introduction to Information Retrieval
Sed/Ack부분 – Scoring (2)
17
![Page 18: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/18.jpg)
Introduction to Information Retrieval
18
Sed/Ack부분 – Scoring (3)
![Page 19: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/19.jpg)
Introduction to Information Retrieval
Introduction to
Information Retrieval
세 번째 Sed/Awk 대신 Python을 이용한 Posting list 생성 모듈
구현
19
![Page 20: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/20.jpg)
Introduction to Information Retrieval
Stemmer를 이용한 posting list
20
![Page 21: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/21.jpg)
Introduction to Information Retrieval
Stemmer를 이용한 posting list
21
![Page 22: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/22.jpg)
Introduction to Information Retrieval
Stemmer를 이용한 posting list
22
결
과
posting list
![Page 23: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/23.jpg)
Introduction to Information Retrieval
완성된 파일들을 python폴더 안으로
23
![Page 24: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/24.jpg)
Introduction to Information Retrieval
Stemmer부분, Scoring (1)
24
![Page 25: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/25.jpg)
Introduction to Information Retrieval
Stemmer부분, Scoring (2)
25
![Page 26: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/26.jpg)
Introduction to Information Retrieval
Stemmer부분, Scoring (3)
26
![Page 27: Introduction to Information Retrievalleeck/IR/Score2.pdf · 2017-03-12 · Introduction to Information Retrieval Index 첫 번째 : stemmer 맛보기 stemmer로 AP88s 문서셋을](https://reader033.vdocuments.mx/reader033/viewer/2022043007/5f9485acc15f3146f92f44b4/html5/thumbnails/27.jpg)
Introduction to Information Retrieval
Stemmer부분, Scoring
27