webinar on deep learning's impact on nlp - amazon s3deep+learning+-+dec...why deep learning is...

Webinar on Deep Learning's Impact on NLP

Kfir BarChief Scientist

Who is Kfir Bar and what does Basis Tech do?

AI, Machine Learning, Deep Learning, Text Analytics...

"With industry leading NLP tools we enable Smart Search, Advanced Name Matching and Identity Analytics.”

Visit our website

www.rosette.com

Deep Learning's Impact on NLP: In 30 Minutes

Kfir BarChief Scientist

Why deep learning is so hyped?

“deep learning”

Deep learning papers in top NLP conferences

https://arxiv.org/pdf/1708.02709.pdf

Why deep learning is so hyped?

1. Data

2. Computational power

3. Algorithms

4. Rebranding

Why deep learning is so hyped?

1. Data

2. Computational power

3. Algorithms

4. Rebranding

Deep learning algorithms get better with more data

Amount of data

Traditional Machine Learning

Perf

orm

ance

Deep Learning

Why deep learning is so hyped?

1. Data

2. Computational power

3. Algorithms

4. Rebranding

➔ Deep learning algorithms require significantly more computational power

➔ GPUs become available at reasonable prices

Why deep learning is so hyped?

1. Data

2. Computational power

3. Algorithms

4. Rebranding

➔ Recent advances in training procedures make deep learning a feasible device

Why deep learning is so hyped?

1. Data

2. Computational power

3. Algorithms

4. Rebranding

1955

Perceptron1970

Artificial Neural Network

2010

Deep learning

Advantages of using Deep Learning

1. Deep Learning models outperform nearly every other machine learning algorithms

2. They don’t require feature engineering

Deep Learning models outperform nearly every other machine learning algorithms

https://www.dsiac.org/resources/journals/dsiac/winter-2017-volume-4-number-1/real-time-situ-intelligent-video-analytics

Traditional ML vs. Deep LearningCongratulations to @Cristiano for winning the "Best International Soccer Player" award at the 2018 @ESPYS!

words, part of speech tags, lemmas, brown clusters

[00010010110000101001…..001]

SPORTS

Feature extraction

Vectorization

Modeling

Embeddings lookup

[0.323, -0.3434, 0.901, …, -0.267][-0.4923, 0.554, 0.001, …, -0.365]

[1.58845, 0.478, 0.0901, …, -0.171]…

[-0.0592, 0.588, -0.01, …, -0.111]

Modeling

SPORTS

15

Congratulations to @Cristiano for winning the "Best International Soccer Player" award at the 2018 @ESPYS!

Word embeddings

- + BerlinJapan Germany

German

European

Europe

Africa

Tokyo =

Multilingual embeddings

Machine Learning

Eagleלמידה חישוביתPharmaceuticals Inc.

Eagle Drugs, Co.

Tesla

Energy Storage

טסלה

AI

تیسال موتورز

計算学習

אחסון אנרגיה

1. Explainability

2. Need more data

3. Computationally expensive

Disadvantages of using Deep Learning

Disadvantages of DL

1. Explainability

2. Need more data

3. Computationally expensive

➔ It’s difficult to understand why a DL model decided on something

1

1.2

3.2

-0.3

0.5

2

Google’s AI won the game Go

Google’s AI won the game Go

22

By Siddhartha Mukherjee

The dying algorithm - predicts death for oncological patients

“Here is the strange rub of such a deep learning system: It learns, but it cannot tell us why it has learned…

...the algorithm looks vacantly at us when we ask, Why? It is, like death, another black box.”

Jan 2018

Traditional algorithm for gender classification

1. Female writers use more pronouns (I, you, she, their, myself)

2. Males prefer words that identify or determine nouns (a, the, that) and words that quantify them (one, two, more)

Koppel et al., 2002, Automatically Categorizing Written Texts by Author Gender

Disadvantages of DL

1. Explainability

2. Need more data

3. Computationally expensive

➔ Neural Networks usually require more data than traditional algorithms

➔ Usually they need at least tens of thousand (if not millions) of labeled samples

Disadvantages of DL

1. Explainability

2. Need more data

3. Computationally expensive

➔ State of the art DL algorithms can take days and sometimes even weeks to train completely from scratch

➔ The complex structure and relatively large number of parameters result in a slower prediction process

➔ Usually DL algorithms require GPU to maintain a reasonable running time

26

Example: Named Entity Recognition

27

Automatically find names of people,

organizations, locations, and more in text

across many languages.

Named entity recognition (NER)

According to Elon Musk, Mars rocket will

fly ‘short flights’ next year.

28

?

30

Context is important

Edward AdelsonNeuroscientist, MIT

Checker shadow illusion

The squares represented by A and B are of the same color

31

Context is important

Edward AdelsonNeuroscientist, MIT

Checker shadow illusion

The squares represented by A and B are of the same color

Can't play Spain? Improve your playing via easy step-by-step video lessons!

32

But sometimes it gets ambiguous...

33

But sometimes it gets ambiguous...

Can't play Spain? Improve your playing via easy step-by-step video lessons!

34

Feed forward network for NER

listen

to

while

I

B-PER

B-LOC

...

...

Layer 1 Layer 2 Output

Spain I-PER...

+

35

Bidirectional LSTM for Sequence Labeling

LSTM

Washington

B-PER

LSTM

+

LSTM

said

OTHER

LSTM

+

LSTM

in

OTHER

LSTM

+

LSTM

Chicago

B-LOC

LSTM

+

LSTM

last

OTHER

LSTM

...

36

Overall: better accuracy in multiple languages for NER!

English Arabic Korean

Deep learning model 91.3 83.3 86.4

Traditional model 89.3 80.3 80.7

Some takeaways

➔ Deep Learning algorithms perform better for some NLP tasks

➔ They don’t require feature engineering

➔ For us it provides a more generic approach for NLP, so supporting new languages becomes easier

➔ They are slower than the traditional algorithms

Want to learn more?

Want to learn more?

Thank you!

40

Questions?Use the chatbox!

kfir@basistech.com

@kfirbar

www.rosette.com