social media & text analysissocialmedia-class.org/slides/lecture4_twitter_nlppipeline.pdftwitter...
TRANSCRIPT
![Page 1: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/1.jpg)
Social Media & Text Analysis lecture 7 - Twitter NLP Pipeline
Tokenization, Normalization, POS/NE Tagging
CSE 5539-0010 Ohio State UniversityInstructor: Alan Ritter
Website: socialmedia-class.org
![Page 2: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/2.jpg)
Alan Ritter ◦ socialmedia-class.org
LangID Tool: langid.py• main techniques:
- Multinominal Naïve Bayes
- diverse training data from multiple domains (Wikipedia, Reuters, Debian, etc.)
- plus feature selection using Information Gain (IG) to choose features that are informative about language, but not informative about domain
Source: Lui and Baldwin “langid.py: An Off-the-shelf Language Identification Tool" ACL 2012
![Page 3: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/3.jpg)
Alan Ritter ◦ socialmedia-class.org
Naïve Bayes• For a document d, find the most probable class c:
cMAP = argmaxc∈C
P(t1,t2,...,tn | c)P(c)
cNB = argmaxc∈C
P(c) P(ti | c)ti∈d∏
Source: adapted from Dan jurafsky
![Page 4: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/4.jpg)
Alan Ritter ◦ socialmedia-class.org
LangID features• n-grams features:
- 1-gram: “the” “following” “Wikipedia” “en” “español” …
- 2-gram:“the following” “following is” “Wikipedia en” “en español” …
- 3-gram:….
The following is a list of words that occur in both
Modern English and Modern Spanish, but
which are pronounced differently and may have
different meanings in each language.
…
Wikipedia en español es la edición en idioma
español de Wikipedia. Actualmente cuenta con
1 185 590 páginas válidas de contenido y
ocupa el décimo puesto en esta estadística entre
…
English
Spanish
![Page 5: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/5.jpg)
Alan Ritter ◦ socialmedia-class.org
Correlated Features• For example, for spam email classification, word
“win” often occurs together with “free”, “prize”.
• Solution: - feature selection - or other models (e.g. logistic/softmax regression)
![Page 6: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/6.jpg)
Alan Ritter ◦ socialmedia-class.org
A Heuristic from Information Theory
• Let X be a random variable
• The Surprise of each valve of X is defined as:
P(X=0) P(X=1)
0.3 0.7
S(X = x) = � logP (X = x)
• Notes:
• An event with probability 1 has 0 surprise
• An event with probability 0 has infinite surprise
![Page 7: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/7.jpg)
Alan Ritter ◦ socialmedia-class.org
Entropy & Information Gain• Entropy is a measure of disorder in a dataset
(expected surprise)
H (X) = − P(xi )logP(xi )i∑
H(X) = 0
![Page 8: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/8.jpg)
Alan Ritter ◦ socialmedia-class.org
Entropy & Information Gain• Entropy is a measure of disorder in a dataset
(expected surprise)
H (X) = − P(xi )logP(xi )i∑
H(X) = 0
![Page 9: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/9.jpg)
Alan Ritter ◦ socialmedia-class.org
Entropy & Information Gain• Entropy is a measure of disorder in a dataset
(expected surprise)
H (X) = − P(xi )logP(xi )i∑
H(X) = 0
• Conditional Entropy quantifies the amount of information needed to describe the outcome of Y given that X is known.
H(Y |X) =X
i
P (xi)H(Y |X = xi)
![Page 10: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/10.jpg)
Alan Ritter ◦ socialmedia-class.org
Entropy & Information Gain• Entropy is a measure of disorder in a dataset
• Information Gain is a measure of the decrease in disorder achieved by partitioning the original data set.
IG(Y | X) = H (Y )− H (Y | X)
H (X) = − P(xi )logP(xi )i∑
H(X) = 0
![Page 11: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/11.jpg)
Alan Ritter ◦ socialmedia-class.org
Information Gain
Source: Andrew Moore
H (X) = − P(xi )logP(xi )i∑ IG(Y | X) = H (Y )− H (Y | X)
![Page 12: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/12.jpg)
Alan Ritter ◦ socialmedia-class.org
Information Gain
Source: Andrew Moore
![Page 13: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/13.jpg)
Alan Ritter ◦ socialmedia-class.org
Information Gain used for?• choose features that are informative (most useful)
for discriminating between the classes.
Wealth Longevity
IG(LongLife | HairColor) = 0.01
IG(LongLife | Smoker) = 0.2 IG(LongLife | Gender) = 0.25
IG(LongLife | LastDigitOfSSN) = 0.00001
![Page 14: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/14.jpg)
Alan Ritter ◦ socialmedia-class.org
LangID Tool: langid.py• feature selection using Information Gain (IG)
Source: Lui and Baldwin “langid.py: An Off-the-shelf Language Identification Tool" ACL 2012
correlate
domain independent
![Page 15: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/15.jpg)
Alan Ritter ◦ socialmedia-class.org
LangID Tool: langid.py• main advantages:
- cross-domain (works on all kinds of texts) - works for Twitter (accuracy = 0.89) - fast (300 tweets/second — 24G RAM) - currently supports 97 language - retrainable
Source: Lui and Baldwin “langid.py: An Off-the-shelf Language Identification Tool" ACL 2012
![Page 16: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/16.jpg)
Alan Ritter ◦ socialmedia-class.org
Language Identification Tokenization
Part-of-Speech (POS)
Tagging
Shallow Parsing
(Chunking)
Named Entity
Recognition (NER)
Summary
Stemming
Normalization
classification(Naïve Bayes)
![Page 17: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/17.jpg)
Alan Ritter ◦ socialmedia-class.org
Language Identification Tokenization
Part-of-Speech (POS)
Tagging
Shallow Parsing
(Chunking)
Named Entity
Recognition (NER)
NLP Pipeline
Stemming
Normalization
![Page 18: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/18.jpg)
Alan Ritter ◦ socialmedia-class.org
Tokenization• breaks up the string into words and punctuation • need to handle:
- abbreviations (“jr.”), number (“5,000”) …
inputoutput
![Page 19: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/19.jpg)
Alan Ritter ◦ socialmedia-class.org
Tokenization• for Twitter, additionally need to handle:
- emoticons, urls, #hashtags, @mentions …
input
output
![Page 22: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/22.jpg)
Alan Ritter ◦ socialmedia-class.org
Tokenization• main techniques:
- hand-crafted rules as regular expressions
![Page 23: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/23.jpg)
Alan Ritter ◦ socialmedia-class.org
Regular Expression• a pattern matching language
• invented by American Mathematician Stephen Kleene in the 1950s
• used for search, find, replace, validation … (very frequently used when dealing with strings)
• supported by most programming languages
• easy to learn, but hard to master
![Page 24: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/24.jpg)
Alan Ritter ◦ socialmedia-class.org
Regular Expression
• [] indicates a set of characters: - [amk] will match ‘a’, ‘m’, or ‘k’ - [a-z] will match any lowercase letter
(‘abcdefghijklmnopqrstuvwxyz’) - [a-zA-Z0-9_] will match any letter or digit or ‘_’
• + matches 1 or more repetitions of preceding RE
![Page 25: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/25.jpg)
Alan Ritter ◦ socialmedia-class.org
Regular Expression
• will match strings that: - start with a ‘#’ - follow with one or more letters/digits/‘_’
![Page 27: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/27.jpg)
Alan Ritter ◦ socialmedia-class.org
Regular Expression
• will match strings that: - start with one or more ‘<‘ - then maybe a ‘/’ - then one or more ‘3’ - and maybe repetitions of the above
![Page 28: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/28.jpg)
Alan Ritter ◦ socialmedia-class.org
Regular Expression
• ‘+’ matches 1 or more repetitions of the preceding RE - ‘<+’ matches ‘<’, ‘<<’, ‘<<<’ … - ‘3+’ matches ‘3’, ‘33’, ‘333’ …
• ‘?’ matches 0 or 1 repetitions of the preceding RE - ‘/?’ matches ‘/’ or nothing (so handles ‘</3’)
• (?: …) is a non-capturing version of ( … ) • ( … ) matches whatever RE is inside the parentheses
![Page 31: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/31.jpg)
Alan Ritter ◦ socialmedia-class.org
Regular Expression• learn more (https://docs.python.org/2/library/re.html)
![Page 32: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/32.jpg)
Alan Ritter ◦ socialmedia-class.org
Tokenization• for Twitter, additionally need to handle:
- emoticons, urls, #hashtags, @mentions …
input
output
![Page 33: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/33.jpg)
Alan Ritter ◦ socialmedia-class.org
Emoticons
Dirk Hovy, Anders Johannsen, and Anders Søgaard. User review sites as a resource for large-scale sociolinguistic studies. WWW, 2015
![Page 34: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/34.jpg)
Alan Ritter ◦ socialmedia-class.org
Emoticons
Dirk Hovy, Anders Johannsen, and Anders Søgaard. User review sites as a resource for large-scale sociolinguistic studies. WWW, 2015
![Page 35: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/35.jpg)
Alan Ritter ◦ socialmedia-class.org
Tokenization• language dependent
Source: http://what-when-how.com
![Page 36: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/36.jpg)
Alan Ritter ◦ socialmedia-class.org
Language Identification Tokenization
Part-of-Speech (POS)
Tagging
Shallow Parsing
(Chunking)
Named Entity
Recognition (NER)
NLP Pipeline
Stemming
Normalization
![Page 37: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/37.jpg)
Alan Ritter ◦ socialmedia-class.org
Language Identification Tokenization
Part-of-Speech (POS)
Tagging
Shallow Parsing
(Chunking)
Named Entity
Recognition (NER)
NLP Pipeline
Stemming
Normalization
![Page 38: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/38.jpg)
Alan Ritter ◦ socialmedia-class.org
Stemming• reduce inflected words to their word stem, base or
root form (not necessarily the morphological root)
• studied since the 1960s
![Page 39: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/39.jpg)
Alan Ritter ◦ socialmedia-class.org
Stemming• different steamers: Porter, Snowball, Lancaster …
• WordNet’s built-in lemmatized (dictionary-based)
![Page 40: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/40.jpg)
Alan Ritter ◦ socialmedia-class.org
Stemming• language dependent
Source: All Things Linguistic
![Page 41: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/41.jpg)
Alan Ritter ◦ socialmedia-class.org
Text Normalization• convert non-standard words to standard
Source: Tim Baldwin, Marie de Marneffe, Han Bo, Young-Bum Kim, Alan Ritter, Wei Xu Shared Tasks of the 2015 Workshop on Noisy User-generated Text:
Twitter Lexical Normalization and Named Entity Recognition
![Page 42: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/42.jpg)
Alan Ritter ◦ socialmedia-class.org
Text Normalization• types of non-standard words in 449 English tweets:
Category Ratio Exampleletter&numer 2.36% b4 → before
letter 72.44% shuld → shouldnumber substitution 2.76% 4 → for
slang 12.20 lol → laugh out loudother 10.24% sucha → such a
most non-standard words are morphophonemic “errors”
Source: Bo Han and Timothy Baldwin “Lexical normalisation of short text messages: Makn sens a #twitter” ACL 2011
![Page 43: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/43.jpg)
Alan Ritter ◦ socialmedia-class.org
A Normalization Lexicon• automatically derived from Twitter data + dictionary
Source: Bo Han, Paul Cook and Timothy Baldwin “Automatically Constructing a Normalisation Dictionary for Microblogs” EMNLP-CoNLL 2012
Performance Precision = 0.847
Recall = 0.630 F1-Score = 0.723
![Page 44: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/44.jpg)
Alan Ritter ◦ socialmedia-class.org
Phrase-level Normalization• word-level normalization is insufficient for many
cases:
Category Example
1-to-many everytime → every time
incorrect IVs can’t want for → can’t wait for
grammar I’m going a movie → I’m going to a movie
ambiguities 4 → 4 / 4th / for / four
Source: Wei Xu, Alan Ritter, Ralph Grishman “Gathering and Generating Paraphrases from Twitter with Application to Normalization” BUCC 2013
in-vocabulary words
![Page 45: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/45.jpg)
Alan Ritter ◦ socialmedia-class.org
Language Identification Tokenization
Part-of-Speech (POS)
Tagging
Shallow Parsing
(Chunking)
Named Entity
Recognition (NER)
NLP Pipeline (summary so far)
Stemming
Normalization
classification(Naïve Bayes)
RegularExpression
![Page 46: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/46.jpg)
Alan Ritter ◦ socialmedia-class.org
Language Identification Tokenization
Part-of-Speech (POS)
Tagging
Shallow Parsing
(Chunking)
Named Entity
Recognition (NER)
NLP Pipeline (next)
Sequential TaggingStemming
Normalization
![Page 47: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/47.jpg)
Alan Ritter ◦ socialmedia-class.org
Part-of-Speech (POS) Tagging
Cant MDwait VBfor INthe DT
ravens NNPgame NN
tomorrow NN… :go VBray NNPrice NNP
!!!!!!! .
![Page 49: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/49.jpg)
Alan Ritter ◦ socialmedia-class.org
Part-of-Speech (POS) Tagging
• Words often have more than one POS: - The back door = JJ - On my back = NN - Win the voters back = RB - Promised to back the bill = VB
• POS tagging problem is to determine the POS tag for a particular instance of a word.
Source: adapted from Chris Manning
![Page 50: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/50.jpg)
Alan Ritter ◦ socialmedia-class.org
Twitter-specific Tags• #hashtag
• @metion
• url
• email address
• emoticon
• discourse marker
• symbols
• …Source: Gimpel et al.
“Part-of-Speech Tagging for Twitter : Annotation, Features, and Experiments” ACL 2011
![Page 51: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/51.jpg)
Alan Ritter ◦ socialmedia-class.org
Notable Twitter POS Taggers• Gimpel et al., 2011 • Ritter et al., 2011
• Derczynski et al, 2013 • Owoputi et al. 2013
Source: Derczynski, Ritter, Clark, Bontcheva “Twitter Part-of-Speech Tagging for All: Overcoming Sparse and Noisy Data" RANLP 2013
State-of-the-art: Token Accuracy: ~ 88% Sentence Accuracy ~20%
(97% on news text)
![Page 52: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/52.jpg)
Alan Ritter ◦ socialmedia-class.org
ChunkingCant
VPwaitfor PPthe
NPravensgame
tomorrow NP…go VPray
NPrice
!!!!!!!
![Page 53: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/53.jpg)
Alan Ritter ◦ socialmedia-class.org
Chunking• recovering phrases constructed by the part-of-speech
tags
• a.k.a shallow (partial) parsing:
- full parsing is expensive, and is not very robust
- partial parsing can be much faster, more robust, yet sufficient for many applications
- useful as input (features) for named entity recognition or full parser
![Page 54: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/54.jpg)
Alan Ritter ◦ socialmedia-class.org
Named Entity Recognition(NER)
Cantwaitforthe
ravens ORGgame
tomorrow…goray
PERrice
!!!!!!! .
ORG: organization PER: person LOC: location
![Page 55: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/55.jpg)
Alan Ritter ◦ socialmedia-class.org
Cantwaitforthe
ravens ORGgame
tomorrow…goray
PERrice
!!!!!!! .
ORG: organization PER: person LOC: location
NER: Basic Classes
![Page 56: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/56.jpg)
Alan Ritter ◦ socialmedia-class.org
NER: Rich Classes
Source: Strauss, Toma, Ritter, de Marneffe, Xu Results of the WNUT16 Named Entity Recognition Shared Task (WNUT@COLING 2016)
![Page 57: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/57.jpg)
Alan Ritter ◦ socialmedia-class.org
NER: Genre DifferencesNews Tweets
PER Politicians, business leaders, journalists, celebrities
Sportsmen, actors, TV personalities, celebrities, names of friends
LOC Countries, cities, rivers, and other places related to current affairs
Restaurants, bars, local landmarks/areas, cities, rarely countries
ORG Public and private companies, government organisations
Bands, internet companies, sports clubs
Source: Kalina Bontcheva and Leon Derczynski “Tutorial on Natural Language Processing for Social Media” EACL 2014
![Page 58: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/58.jpg)
Alan Ritter ◦ socialmedia-class.org
Notable Twitter NE Research• Liu et al., 2011 • Ritter et al., 2011
• Owoputi et al. 2013 • Plank et al, 2014 • Cherry & Guo, 2015
![Page 61: Social Media & Text Analysissocialmedia-class.org/slides/lecture4_Twitter_NLPpipeline.pdfTwitter Lexical Normalization and Named Entity Recognition. ... “Lexical normalisation of](https://reader033.vdocuments.mx/reader033/viewer/2022042322/5f0c98e57e708231d43631ee/html5/thumbnails/61.jpg)
Alan Ritter ◦ socialmedia-class.org
Language Identification Tokenization
Part-of-Speech (POS)
Tagging
Shallow Parsing
(Chunking)
Named Entity
Recognition (NER)
Summary
Stemming
Normalization
classification(Naïve Bayes)
RegularExpression
Sequential Tagging