named entity recognitionsuwangcompling.com/wp-content/uploads/2016/11/ner_overview_ver… ·...

NAMED ENTITY RECOGNITION

JACOB SU WANG OJO LABS INC.

WE EXPLORE …

IN THIS TALK

• WHAT IS NER, WHAT ARE ITS APPLICATIONS

• WHAT ARE THE METHODS USED IN VARIOUS CONDITIONS

• WHAT MODEL TO USE WHEN?

• HOW DO THE MODELS WORK?

WHAT IS A NAMED ENTITY?

WORDS/PHRASES OF INTEREST IN TEXT

NAMED ENTITIES

• NATURAL NAMED ENTITIES

• PROPER NOUNS

• E.G. PERSON NAME (Steve Jobs), ORGANIZATION (OJO Labs), LOCATION (Austin), ETC.

• DEFINED NAMED ENTITIES

• NON-PRON WORDS/PHRASES WE DEFINE TO BE INFORMATIVE

• E.G. TIME (9pm, 1945), INDICATORS (which, where, etc.), CONTEXTUALLY-SIGNIFICANT TERMS (buffalo hunter, horse trading in Lonesome Dove).

NAMED ENTITY RECOGNITION TASKpic1: https://www.semanticengine.ws/namedentityrecognitionpic2: https://www.ravn.co.uk/named-entity-recognition-ravn-part-1/

IDENTIFICATION OF WORD/PHRASES OF INTEREST

IN TEXT

APPLICATIONS

QUESTION-ANSWERING: LOCATE INFORMATIVE TEXT

WHY?

Q: WHEN DID ADOLF HITLER COME TO POWER?

WHY?

Q: WHEN DID ADOLF HITLER COME TO POWER?

QUESTION-ANSWERING: LOCATE INFORMATIVE TEXT

MEDICAL DIAGNOSIS: LEVERAGE OPEN DATA SOURCE

WHY?

ARE THERE ANY EVIDENCE MEDICINE X IS A GOOD TREATMENT FOR DISEASE Y?

pic: http://www.slideshare.net/larsjuhljensen/one-tagger-many-uses-illustrating-the-power-of-ontologies-in-named-entity-recognition

http://www.slideshare.net/larsjuhljensen/one-tagger-many-uses-illustrating-the-power-of-ontologies-in-named-entity-recognition




INFO SUPPLEMENTATION: AMAZON X -RAY

WHY?pic1: http://www.blogher.com/kindle-paperwhite-smart-bitches-reviewpic2: https://adelightfulspace.wordpress.com/2015/09/14/review-amazon-kindle-

INFO SUPPLEMENTATION: AMAZON X -RAY

WHY?

TEXT NORMALIZATION: PROPER LEVEL OF ABSTRACTION

WHY?

WHAT DO PEOPLE CARE ABOUT?

WHAT DO COMPANIES CARE ABOUT?

COOL! HOW?

LOOK UP A GAZETTEER?

HOW? http://www.bph-postcodes.co.uk/licenced_products/postcode-sector-town.cgi

http://www.bph-postcodes.co.uk/licenced_products/postcode-sector-town.cgi



LOOK UP A GAZETTEER?

HOW?

STRING MATCHING ONLY IS NOT GOING TO CUT! MAINLY BECAUSE OF WEAK GENERALIZATION!

LOOKING-UP AIN’T GONNA CUT!

HOW?

ENGLAND: COUNTRY_NAME OR LOCATION?

1945: NUMBER OR TIME?

CHASE: PERSON_NAME OR ORGANIZATION?

ISSUE 1: AMBIGUITY


HOW?

STREET, ST., STRT, …

UNIVERSITY OF TEXAS, UNIV TX, UT, …

NAMED ENTITY RECOGNITION, NER, …

ISSUE 2: VARIANTS


HOW?

MOST ITEMS WILL BE OOVS!

ISSUE 3: OUT-OF-VOCAB ITEMS

ZIPF’S LAW

SOLUTION: FEATURIZATION

HOW?

… W W W NAMED ENTITY W W W …MUCH RICHER INFORMATION THAN HAVING ONY THE ITEM ITSELF!!

SEMANTIC

SYNTACTIC

LEXICAL

MORPHOLOGICAL

FEATURE VECTOR OF WORD 1

HOW?SOLUTION: FEATURIZATION

• E.G. DISAMBIGUATION

1945 (NUMBER): - PARSER LIKELY TO GIVE NUM/ADJ POS TAG. - IN MORE SIMILAR CONTEXT AS ARBITRARY-LENGTH DIGIT SEQUENCES AS “YYYY” FORMAT SEQUENCES.

1945 (TIME): - MORE LIKELY TO BE THE LAST ITEM BEFORE DELIMITERS (COMMA OR PERIOD). - MORE LIKELY TO HAVE PRECEDING ‘IN’.


• E.G. VARIANTS

UT, UNIVERSITY OF TEXAS, UNIV TX, …

- SIMILAR COOCCURRENCE VECTORS (OVER VOCAB). - MORE SIMILAR TO EACH OTHER THAN TO OTHER NON-WORD ABBREVIATIONS (UT, UNIV TX)


• E.G. OUT-OF-VOCAB ITEMS

BARFKNECHT, THORUP, PECKENPAUGH, … (RARE SURNAMES, LESS THAN 0.15% IN 100,000 PEOPLE)

- SIMILAR CONTEXTUAL DISTRIBUTION TO “JOHNSON” OR “SMITH” THAN TO RANDOM WORDS. - LIKELY TO BE TAGGED AS “PRON” BY PARSER.

FEATURIZATION

METHODS

FEATURIZATION

• FEATURE ENGINEERING

• DOMAIN EXPERT KNOWLEDGE

• LINGUISTIC KNOWLEDGE

• FEATURE ABSTRACTION

• “ALMOST FROM SCRATCH” WITH AUTOMATIC FEATURE DISCOVERY

FEATURE ENGINEERING

… W W W NAMED ENTITY W W W …

• MORPHOLOGY: PREFIX, SUFFIX, STEM, ETC. IN A ENGLISH • anti-, con-, dis-, re-, …, -ly, -ness, …

• LEXICAL: GAZETTEER, SPELLING • {city_names}, …, capitalized, …

• SEMANTICS: COOCCURRENCE PATTERN, HEARST PATTERNS • cooccurrence counts, CITIES such as …

• SYNTAX: DEPENDENCY CONTEXT, SYNTACTIC PATH • (NE, dobj, kill), …, V->NP->N, …

FEATURE ABSTRACTION

… W W W NAMED ENTITY W W W …

• MORPHOLOGY: CHARACTER EMBEDDINGS • from sequences of characters in words (CNN)

• LEXICAL & SEMANTICS: WORD EMBEDDINGS • from sequences of words in sentences (word2vec with RNN/FNN)

• SYNTAX: WORD EMBEDDING + PHRASE EMBEDDINGS • from sequences of words in sentences (vectors with RecNN)

FEATURE ENGINEERING VS. FEATURE ABSTRACTIONA BIG DIFFERENCE WE CARE ABOUT: INTERPRETABILITY

THEY WORK IN THE CLASSIFICATION TASK BUT I DON’T KNOW WHAT THEY MEAN!!

WHAT MODEL TO USE?

NERWHAT TO USE?

ALL DEPENDS ON DATA AVAILABILITY & KNOWLEDGE STATE

WHAT MODEL TO USE?

SUPERVISED SEMI-SUPERVISED UNSUPERVISED

ANNOTATION

LABELING SCHEME

WHAT MODEL TO USE?OUR GENERAL OPINION

PURPOSEPERFORMANCE EXPECTATION

(ACCURACY/F1)

SUPERVISEDWEAPONIZED,

PRODUCTION-LEVEL MODELS

95%+

SEMI-SUPERVISED

EXPLORATORY MODELS PATTERN DISCOVERY

~75%

UNSUPERVISED ~65%

OVERVIEW

FEATURE ENGINEERING FEATURE ABSTRACTION

SUPERVISED• CONDITIONAL RANDOM

FIELDS (CRF)• RECURRENT NEURAL

NETS (RNN)

SEMI-SUPERVISED• BOOTSTRAPPING • LABEL PROPAGATION

-

UNSUPERVISED• HEARST PATTERN • EXTERNAL TAXONOMY

LINEAR-CHAIN GRAPHICAL MODEL

CRF

STATES: LATENT VARIABLES THAT TAKE LABELS AS VALUES

OBSERVATION: WORDS

EXAMPLE

ADOLF HITLER CAME TO POWER IN 1933

CRF

EXAMPLE

CRF


OBJECTIVE: FINDING THE BEST PATH!

COMPUTING FOR THE BEST SEQUENCE OF LABELS (Y’s)? EXPENSIVE!!

CRF

CRFOBJECTIVE: FINDING THE BEST PATH!

COMPLEXITY OF BRUTE-FORTH COMPUTATION

E.G. ATIS DATASET

- TYPICAL SENTENCE: ~15 WORDS - SIZE OF LABEL SET: 127 - POSSIBLE PATHS / LABEL SEQUENCES: 12715

CRFBRUTE FORTH?

WE WANT TO BREAK THE GRAPH INTO SUBCOMPONENTS (FACTORS)

CRFFACTORIZATION

: FACTOR AT TIME T�(Xt, Yt)

HITLER CAME TO POWER IN

CRFFACTORIZATION

: FACTOR AT TIME T�(Xt, Yt)


<0, 1, 1, 0, 0, 0, … >

FEATURE INDICATOR FUNCTION

f1 f2 f3 f4 f5 f6, …

CRFFEATURE FUNCTIONS (INDICATOR FUNCTIONS)

fk(Xt, Yt) =

(1 if �(Xt, Yt) has feature k

0 otherwise

E.G. - X_t-1 IS WORD “TO” - X_t+1 HAS POS TAG “PREP” - Y_t+1 IS LABEL “B-PER” - e.t.

�(Xt, Yt)


CRFBEST SEQUENCE

Y = argmax

Y

QTt exp

⇣PKk wkfk(Xt, Yt)

⌘

PY 0

QTt exp

⇣PKk wkfk(Xt, Y

0t )⌘

w_k: WEIGHT OF FEATURE FUNCTION f_k

THE PROBABILITY OF THE ENTIRE SEQUENCE PRODUCT OF “WEIGHTED FEATURE SCORE” OF FACTORS

THIS COULD BE FORMULATED INTO AN OPTIMIZATION PROBLEM AND SOLVED WITH VARIOUS ALGORITHMS!

FACTORIZATION: AS YOU LIKE IT

Sutton & McCallum (2011)

CRF

WE FOUND THESE MODELS LESS APPROPRIATE

CRF

HIDDEN MARKOV MODEL (HMM)

• EMPIRICALLY WEAKER PERFORMANCE • TROUBLE CAPTURING LONG-DISTANCE DEPENDENCY

MAXIMUM ENTROPY MARKOV MODEL

(MEMM)• CRF IS AN IMPROVED VERSION OF MEMM

SUPPORT VECTOR MACHINE (SVM)

• SVM, AS A BINARY CLASSIFIER, AGGREGATES ERROR!

Lafferty et al. (2001) Sutton & McCallum (2011)

RECAP

SUPERVISED + FEATURE ENGINEERING

IN: ADOLF HITLER CAME TO POWER IN 1933.

OUT: B-PER I-PER O O O O B-TIME. NER

LEARNING A MAPPING

TOOLBOX

SUPERVISED + FEATURE ENGINEERING

LIBRARIES

CRF• pycrfsuite: https://python-crfsuite.readthedocs.io/en/latest/ • crf++: https://taku910.github.io/crfpp/

HMM• seqlearn: https://github.com/larsmans/seqlearn • hmmlearn: https://github.com/hmmlearn/hmmlearn

MEMM • nltk: http://www.nltk.org/_modules/nltk/classify/maxent.html

SVM • sklearn: http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html

https://python-crfsuite.readthedocs.io/en/latest/

https://taku910.github.io/crfpp/

https://github.com/larsmans/seqlearn

https://github.com/hmmlearn/hmmlearn

http://www.nltk.org/_modules/nltk/classify/maxent.html

http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html




NETS (RNN)


-


LEARNING TYPE & OBJECTIVE

SUPERVISED VS. SEMI-SUPERVISED NER

LEARNING TYPE

SUPERVISEDINDUCTIVE LEARNING

LEARNING CLASSIFIER THAT INCORPORATES GENERAL RULES

SEMI-SUPERVISEDTRANSDUCTIVE LEARNING

CLASSIFY UNLABELED DATA AS SPECIFIC CASES BY THEIR SIMILARITY TO LABELED DATA

SUPERVISED LEARNING: INDUCTIVE


WE ARE LEARNING THESE PARAMETERS!

classifier(label sequences|word sequences;⇥)

SEMI-SUPERVISED LEARNING: TRANSDUCTIVE


SIMILAR WORDS SHOULD HAVE SAME LABELS

BOOTSTRAPPINGALTERNATING/MUTUAL BOOTSTRAPPING

BY LEXICAL FEATURES

EXTRACTION

• AUTHOR

{[A-Z][A-Za-z .,&]; [A-Za-z.]; ...}

• TITLE

{[A-Z0-9][A-Za-z0-9 .,:’#!?;&]; [A-Za-z0-9?!]}

• ...

E.G. REGEX CHARACTERIZATION

E.G. LEXICAL RULES

Brin (1999)

Collins & Singer (1999)

BY CONTEXTUAL FEATURES

EXTRACTION

Riloff & Jones (1999)

BY DISTRIBUTIONAL SIMILARITY

EXTRACTION

Pasca et al. (2006)

BOOTSTRAPPINGALTERNATING/MUTUAL BOOTSTRAPPING

THE SET OF NAMED ENTITIES!

LABEL PROPAGATION

ITEMS

SIMILARITIES

(TOKENS)

*THE ACTUAL GRAPH IS USUALLY FULLY CONNECTED, BUT NOT NECESSARILY SO IN SOME VARIANTS OF LP

LABEL PROPAGATION

L_1

L_2

<0,1,1,0,0, …><1,1,0,1,1, …>

<0,1,1,1,0, …>

LABELED NODES: LABEL + FEATURE VEC UNLABELED NODES: FEATURE VEC ONLY

LABEL PROPAGATION

Network graph from Mejova (2015), interpretation differs here.

LABELED ITEMS

UNLABELED ITEMS

ITEMS

SIMILARITIES

PROPAGATION

FULLY LABELED!

PROCEDURE

LABEL PROPAGATION

l

l l

u

u u

Xl+u

Xl+u Xl+u

C

SIMILARITY MATRIX SOFT LABEL ASSIGNMENT DISTRIBUTION

Zhu & Ghahramani (2002)

STEP 1

LABEL PROPAGATION

NEW DISTRIBUTION MATRIX <= SIMILARITY MATRIX * DISTRIBUTION MATRIX

X =

C

Xl+u


STEP 1

LABEL PROPAGATION

AN INTUITIVE EXPLANATION WHY THE UPDATE WORKS

X =

C

Xl+u


INFLUENCE OF LABELED DATA ON UNLABELED DATA PROPORTIONAL TO THEIR SIMILARITY!

WORD X_i

LABELED WORD

X_jx =

STEP 2

LABEL PROPAGATION

ROW NORMALIZE SIMILARITY MATRIX

EACH ROW IS A PROBABILITY DISTRIBUTION OVER CLASSES!

Xl+u

C


STEP 3

LABEL PROPAGATION

ROW NORMALIZE SIMILARITY MATRIX

CLAMP/ “REPLENISH” THE LABELED DATA

C

Xl+u


CONVERGENCE

LABEL PROPAGATION

UNTIL ALL THE ITEMS ARE LABELED …

C

Xl+u


PROS & CONS

BOOTSTRAPPING VS. LABEL PROPAGATION

PROS CONS

BOOTSTRAPPING• EASY IMPLEMENTATION • FAST EXTRACTION

• LABOR INTENSIVE (HEAVY HUMAN SUPERVISION TO

GUARANTEE QUALITY • PRONE TO INTRODUCING

NOISE

LABEL PROPAGATION• CONVERGENCE

GUARANTEED • MORE AUTOMATED

• PARAMETERS DIFFICULT TO TUNE (PERFORMANCE DEPENDS HEAVILY ON PARAMETER SETTING)

• SLOW WITH LARGE GRAPH (SOPHISTICATED VARIANTS)

RECAP

SEMI-SUPERVISED + FEATURE ENGINEERING

SIMILAR WORDS SHOULD HAVE SAME LABELS

TOOLBOX

SEMI-SUPERVISED + FEATURE ENGINEERING

LIBRARIES

BOOTSTRAPPING NONE NEEDED

LABEL PROPAGATION

• sklearn: http://scikit-learn.org/stable/modules/label_propagation.html • MAD: https://github.com/psorianom/modified_adsorption

Taludkar & Crammer (2009)

http://scikit-learn.org/stable/modules/label_propagation.html

https://github.com/psorianom/modified_adsorption




NETS (RNN)


-


OBJECTIVE

UNSUPERVISED LEARNING VS. OTHER

SUPERVISED SEMI-SUPERVISED UNSUPERVISED

ANNOTATION

LABELING SCHEME

LEARNING OBJECTIVE LABELING STUFF ALSO INDUCE A

LABELING SCHEME

GENERAL IDEA

UNSUPERVISED + FEATURE ENGINEERING

INDUCING LABELING SCHEME


UNSUPERVISED LEARNING

HYPERNYMS

HYPONYMS

UNSUPERVISED LEARNINGINDUCING LABELING SCHEME

WHAT IS A HEARST PATTERN?

HEARST PATTERN BASED EXTRACTION

PARADIGM

Y such as X (LABEL CANDIDATE, NE CANDIDATE)

E.G.

CITIES such as Austin, Dallas, and Houston

Hearst (1992)

HEARST PATTERN BASED EXTRACTIONSTEP 1

STEP 2


LABEL(X) = argmax

YSCORE(X,Y)

Evans (2003)

THIS IS A SELECTION PROCESS FOR LABELS

STEP 3


Etzioni et al. (2005)

PMI(X,Yi[X]) = PMI(Austin, CITY such as Austin)where

X = Austin

Y = CITY

Yi[X] = CITY such as Austin

Y_1 Y_2 Y_3 …. Y_k (Austin,CITY)

Y such as X

GROUNDING

EXTERNAL TAXONOMY BASED EXTRACTIONEXAMPLE: WORDNET

HYPERNYM (MORE GENERAL)

HYPONYM (MORE SPECIFIC)

EXTERNAL TAXONOMY BASED EXTRACTIONOBJECTIVE: FIND THE SWEET SPOT

FIND ALL CAPITALIZED WORDS / PHRASES

(OPTION: BOOTSTRAP FROM GAZETEER)

MANUALLY FIND A SET OF HYPERNYMS THAT

COVERS ALL THE NE CANDIDATES!


TOPIC SIGNATURE

SIG(X) = {(word, freqword

) | word in X

0s context}


argmax

Y(SIM(SIG(X), SIG(Y ))) = LOCATION

SIM=230

SIM=410SIM=140

CURRENT NODE = ENTITY


argmax

Y(SIM(SIG(X), SIG(Y ))) = VILLAGE

SIM=410

SIM=251

SIM=533

CURRENT NODE = LOCATION


SWEET SPOT FOUND!!

argmax

Y(SIM(SIG(X), SIG(Y ))) = VILLAGE

SIM=533CURRENT NODE = VILLAGE


NE CANDIDATES: MORDOR HOBBITON, HOBBIT, WIZARD, DWARF, FAIRY

Alphonseca & Manandhar (2002)

RECAP

UNSUPERVISED + FEATURE ENGINEERING





NETS (RNN)


-


FEATURE ENGINEERING VS. FEATURE ABSTRACTION

FEATURE ENGINEERING

AUTOMATIC FEATURE ABSTRACTION THROUGH JOINT OPTIMIZATION

• WHAT ARE EMBEDDINGS?

FEATURE ABSTRACTIONREPRESENTATION: EMBEDDINGS

REPRESENTATIONS WHICH LIVE IN A HIGH DIMENSIONAL SPACE WHERE THE DISTANCE AMONG ITEMS IS DEFINED WITH SIMILARITY OF SORTS…

FEATURE ABSTRACTIONE.G. WORD EMBEDDINGS

FEATURE ABSTRACTIONPIPELINE: HOW ARE EMBEDDINGS LEARNED?

ONE-HOT

PREDICTION RECALIBERATION

ONE-HOT

FEATURE ABSTRACTIONJOINT OPTIMIZATION

FEATURE ABSTRACTION

ONE-HOT

JOINT OPTIMIZATION

MULTICHANNEL EMBEDDINGS

EMBEDDINGS COULD DRAW ON INFORMATION FROM MULTIPLE SOURCES!

MORPHOLOGICAL LEXICAL SEMANTIC SYNTACTIC

dos Santos & Guimaraes (2014a)

MULTICHANNEL EMBEDDINGSEXAMPLE: CHAR-WORD JOINT FEATURIZATION

EXAMPLE: CHAR-WORD JOINT FEATURIZATION

dos Santos & Guimaraes (2015)

PROJECTION

PROJECTION

EMBEDDING LV1

EMBEDDING LV2

MULTICHANNEL EMBEDDINGS

ARCHITECTURE

RECURRENT NEURAL NETS


B-PER I-PER O O O O B-TIME

TIME DISTRIBUTED PREDICTION

OUTPUT SEQUENCE: LABELS

INPUT SEQUENCE: WORDSTHE MODEL “REMEBERS” WHAT HAPPENED IN THE 3 PREVIOUS TIME STEPS

PROCESS


ADOLF

B-PER

PROJECTION TO EMBEDDING SPACE

PROJECTION TO LABEL SPACE

PROCESS


ADOLF HITLER

B-PER I-PER

THE PARAMETERS “REMEMBER”

ITS TRANSITIONAL HISTORY!

THE SAME HIDDEN LAYER AT DIFFERENT TIME POINTS

PROCESS


ADOLF HITLER CAME

B-PER I-PER O

THE PARAMETERS “REMEMBER”

ITS TRANSITIONAL HISTORY!

RESULT



B-PER I-PER O O O O B-TIME

AT EACH TIME POINT, THE PREVIOUS HISTORY IS ENCODED IN PARAMETERS

STATE-OF -THE-ART: BIDIRECTIONAL LSTM-CRF

RECURRENT NEURAL NETSClassifier

EncoderInput

Join

t Tra

inin

g

Lample et al. (2016)

ClassifierEncoder

InputJo

int T

rain

ing

Lample et al. (2016)CAN ALSO BE MULTICHANNEL EMBEDDINGS!

RECURRENT NEURAL NETSSTATE-OF -THE-ART: BIDIRECTIONAL LSTM-CRF

TOOLBOX

SUPERVISED NER + FEATURE ABSTRACTION

LIBRARIES

RNN

• word embeddings - pre-trained: spacy (https://spacy.io/) - create new: https://radimrehurek.com/gensim/models/word2vec.html • neural nets - Keras: https://keras.io/layers/recurrent/ - Tensorflow: https://www.tensorflow.org/tutorials/recurrent/ - Theano: http://deeplearning.net/tutorial/rnnslu.html

https://spacy.io/

https://radimrehurek.com/gensim/models/word2vec.html

https://keras.io/layers/recurrent/

https://www.tensorflow.org/tutorials/recurrent/

http://deeplearning.net/tutorial/rnnslu.html

COMPARISON


FEATURIZATION INTERPRETABILITY

FEATURE ENGINEERING MANUAL INTERPRETABLE

FEATURE ABSTRACTION AUTOMATIC NOT

INTERPRETABLE

ARE DEEP LEARNING BASED MODELS NECESSARILY BETTER?


• CRF CONVERGES FAST • CRF IS GOOD IN LOW DATA • CRF IS MORE INTERPRETABLE • PERFORMANCE DIFFERENCE ~1%

LIME (LOCAL INTERPRETABLE MODEL-AGNOSTIC EXPLANATIONS) https://github.com/marcotcr/lime

https://github.com/marcotcr/lime

• OPTION 1 (ABSENT DOMAIN KNOWLEDGE)

• 1) UNSUPERVISED EXPLORATION

• 2) PAID LABELING, THEN SUPERVISED MODEL

• OPTION 2 (EXPERT DOMAIN KNOWLEDGE AVAILABLE)

• 1) PAID LABELING ON SMALL SET

• 2) SEMI-SUPERVISED EXPLORATION

• 3) PAID LABELING, THEN SUPERVISED MODEL

SUGGESTIONS ON MODELINGNEW DOMAIN + UNLABELED DATA

PAID LABELINGTOOLS

CONFIDENT IN DOMAIN

KNOWLEDGE

MECHANICAL TURK (https://www.mturk.com/mturk)

LESS CONFIDENT IN DOMAIN

KNOWLEDGE

• CROWDFLOWER (https://www.crowdflower.com/)

• SPARE5 (https://app.spare5.com/fives)

https://www.mturk.com/mturk

https://www.crowdflower.com/

https://app.spare5.com/fives

THANK YOU!

named entity recognitionsuwangcompling.com/wp-content/uploads/2016/11/ner_overview_ver… ·...

Documents