webinar on deep learning's impact on nlp - amazon s3deep+learning+-+dec...why deep learning is...
TRANSCRIPT
Webinar on Deep Learning's Impact on NLP
Kfir BarChief Scientist
Who is Kfir Bar and what does Basis Tech do?
AI, Machine Learning, Deep Learning, Text Analytics...
"With industry leading NLP tools we enable Smart Search, Advanced Name Matching and Identity Analytics.”
Visit our website
www.rosette.com
Deep Learning's Impact on NLP: In 30 Minutes
Kfir BarChief Scientist
Why deep learning is so hyped?
“deep learning”
Deep learning papers in top NLP conferences
https://arxiv.org/pdf/1708.02709.pdf
Why deep learning is so hyped?
1. Data
2. Computational power
3. Algorithms
4. Rebranding
Why deep learning is so hyped?
1. Data
2. Computational power
3. Algorithms
4. Rebranding
Deep learning algorithms get better with more data
Amount of data
Traditional Machine Learning
Perf
orm
ance
Deep Learning
Why deep learning is so hyped?
1. Data
2. Computational power
3. Algorithms
4. Rebranding
➔ Deep learning algorithms require significantly more computational power
➔ GPUs become available at reasonable prices
Why deep learning is so hyped?
1. Data
2. Computational power
3. Algorithms
4. Rebranding
➔ Recent advances in training procedures make deep learning a feasible device
Why deep learning is so hyped?
1. Data
2. Computational power
3. Algorithms
4. Rebranding
1955
Perceptron1970
Artificial Neural Network
2010
Deep learning
Advantages of using Deep Learning
1. Deep Learning models outperform nearly every other machine learning algorithms
2. They don’t require feature engineering
Deep Learning models outperform nearly every other machine learning algorithms
https://www.dsiac.org/resources/journals/dsiac/winter-2017-volume-4-number-1/real-time-situ-intelligent-video-analytics
Traditional ML vs. Deep LearningCongratulations to @Cristiano for winning the "Best International Soccer Player" award at the 2018 @ESPYS!
words, part of speech tags, lemmas, brown clusters
[00010010110000101001…..001]
SPORTS
Feature extraction
Vectorization
Modeling
Embeddings lookup
[0.323, -0.3434, 0.901, …, -0.267][-0.4923, 0.554, 0.001, …, -0.365]
[1.58845, 0.478, 0.0901, …, -0.171]…
[-0.0592, 0.588, -0.01, …, -0.111]
Modeling
SPORTS
15
Congratulations to @Cristiano for winning the "Best International Soccer Player" award at the 2018 @ESPYS!
Word embeddings
- + BerlinJapan Germany
German
European
Europe
Africa
Tokyo =
Multilingual embeddings
Machine Learning
Eagleלמידה חישוביתPharmaceuticals Inc.
Eagle Drugs, Co.
Tesla
Energy Storage
טסלה
AI
تیسال موتورز
計算学習
אחסון אנרגיה
1. Explainability
2. Need more data
3. Computationally expensive
Disadvantages of using Deep Learning
Disadvantages of DL
1. Explainability
2. Need more data
3. Computationally expensive
➔ It’s difficult to understand why a DL model decided on something
1
1.2
3.2
-0.3
0.5
2
Google’s AI won the game Go
Google’s AI won the game Go
22
By Siddhartha Mukherjee
The dying algorithm - predicts death for oncological patients
“Here is the strange rub of such a deep learning system: It learns, but it cannot tell us why it has learned…
...the algorithm looks vacantly at us when we ask, Why? It is, like death, another black box.”
Jan 2018
Traditional algorithm for gender classification
1. Female writers use more pronouns (I, you, she, their, myself)
2. Males prefer words that identify or determine nouns (a, the, that) and words that quantify them (one, two, more)
Koppel et al., 2002, Automatically Categorizing Written Texts by Author Gender
Disadvantages of DL
1. Explainability
2. Need more data
3. Computationally expensive
➔ Neural Networks usually require more data than traditional algorithms
➔ Usually they need at least tens of thousand (if not millions) of labeled samples
Disadvantages of DL
1. Explainability
2. Need more data
3. Computationally expensive
➔ State of the art DL algorithms can take days and sometimes even weeks to train completely from scratch
➔ The complex structure and relatively large number of parameters result in a slower prediction process
➔ Usually DL algorithms require GPU to maintain a reasonable running time
26
Example: Named Entity Recognition
27
Automatically find names of people,
organizations, locations, and more in text
across many languages.
Named entity recognition (NER)
According to Elon Musk, Mars rocket will
fly ‘short flights’ next year.
28
?
30
Context is important
Edward AdelsonNeuroscientist, MIT
Checker shadow illusion
The squares represented by A and B are of the same color
31
Context is important
Edward AdelsonNeuroscientist, MIT
Checker shadow illusion
The squares represented by A and B are of the same color
Can't play Spain? Improve your playing via easy step-by-step video lessons!
32
But sometimes it gets ambiguous...
33
But sometimes it gets ambiguous...
Can't play Spain? Improve your playing via easy step-by-step video lessons!
34
Feed forward network for NER
listen
to
while
I
B-PER
B-LOC
...
...
Layer 1 Layer 2 Output
Spain I-PER...
+
35
Bidirectional LSTM for Sequence Labeling
LSTM
Washington
B-PER
LSTM
+
LSTM
said
OTHER
LSTM
+
LSTM
in
OTHER
LSTM
+
LSTM
Chicago
B-LOC
LSTM
+
LSTM
last
OTHER
LSTM
...
36
Overall: better accuracy in multiple languages for NER!
English Arabic Korean
Deep learning model 91.3 83.3 86.4
Traditional model 89.3 80.3 80.7
Some takeaways
➔ Deep Learning algorithms perform better for some NLP tasks
➔ They don’t require feature engineering
➔ For us it provides a more generic approach for NLP, so supporting new languages becomes easier
➔ They are slower than the traditional algorithms
Want to learn more?
Want to learn more?
Thank you!
40
Questions?Use the chatbox!
@kfirbar
www.rosette.com