how large are lions? dan roth tania bedrax-weiss iscol ... · iscol, september 11, 2019 yanai...
TRANSCRIPT
![Page 1: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/1.jpg)
How Large are Lions?Inducing Distributions over Quantitative AttributesISCOL, September 11, 2019
Yanai ElazarAbhijit Mahabal
Deepak RamachandranTania Bedrax-Weiss
Dan Roth
![Page 2: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/2.jpg)
Proprietary + ConfidentialProprietary + Confidential
Proprietary + Confidential
Quantitative Understanding
![Page 3: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/3.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Quantitative Understanding
● Understanding numerical properties and the way they relate to words.
Lionweight
speed
size
...Physical attributes
![Page 4: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/4.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Quantitative Understanding
● Understanding numerical properties and the way they relate to words.
Lion30 kg
150 kg
100 k
g ...Commonsense
quantization attributes
![Page 5: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/5.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Quantitative Understanding in Q&A
● “What is a fast but expensive way to send small cargo?”
○ Ship’s hold○ Boat○ Airplane
Talmor et. al 2019
![Page 6: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/6.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Quantitative Understanding in Q&A
● “What is a fast but expensive way to send small cargo?”
○ Ship’s hold○ Boat○ Airplane
Talmor et. al 2019
Slow
Fast and Expensive
Slow
![Page 7: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/7.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Other Quantitative Work
Since the summer of '99 I haven't eaten apples like this one
Shameless plug
![Page 8: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/8.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Other Quantitative Work
I haven't eaten apples like this one since the summer of '99 YEAR
TACL 19’
Shameless plug
![Page 9: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/9.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Other Quantitative Work
TACL 19’
spaCy plugin
Shameless plug
![Page 10: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/10.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
● It is hard to generalize numerical quantization and common sense from datasets alone.
● Running End-to-End distributional solutions on these tasks is not enough to solve them.
Quantitative Understanding
![Page 11: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/11.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Quantitative Understanding
![Page 12: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/12.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Quantitative Understanding
![Page 13: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/13.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Quantitative Understanding
![Page 14: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/14.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Quantitative Understanding
![Page 15: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/15.jpg)
Proprietary + Confidential
Scalable Attributes of Objects
![Page 16: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/16.jpg)
Proprietary + Confidential
Let’s ground our “Measurable World”
We focus on…
● Items which can be measured objectively
![Page 17: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/17.jpg)
Proprietary + Confidential
How Big is a...
How big is Big?
https://en.wikipedia.org/wiki/Mouse
https://unsplash.com/photos/IPRFX7CVVoU
https://www.thisisinsider.com/homes-popular-style-us-2017-10
![Page 18: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/18.jpg)
Proprietary + Confidential
Let’s ground our “Measurable World”
● These can be object’s attributes, but also other things, like adjective, verbs, etc...
![Page 19: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/19.jpg)
Proprietary + Confidential
Proprietary + Confidential
Solution - Counting!
![Page 20: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/20.jpg)
Proprietary + Confidential
The Idea
● Count co-occurrences of measurements with the words that appear in their context
● By using a large text corpora
![Page 21: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/21.jpg)
Proprietary + Confidential
Counting can be useful!
● Google Books NGram
![Page 22: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/22.jpg)
Proprietary + Confidential
Counting can be useful!
● Google Books NGram● Google Syntactic NGram
![Page 23: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/23.jpg)
Proprietary + Confidential
Counting can be useful!
● Google Books NGram● Google Syntactic NGram
Coincidence?
![Page 24: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/24.jpg)
Proprietary + Confidential
Example - Walk Through The Process
![Page 25: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/25.jpg)
Proprietary + Confidential
Example - Input Sentence
“These breeds can vary in size and weight from a
0.46 kg teacup poodle ...”
Source: Wikipedia
![Page 26: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/26.jpg)
Proprietary + Confidential
Example - Measurement Detection
We detect numerical measurements using a set of rules:kg/kgs/kilogram -> MASS
“These breeds can vary in size and weight from a
0.46 kg teacup poodle ...”
Source: Wikipedia
![Page 27: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/27.jpg)
Proprietary + Confidential
Example - Measure Normalization
460 gram
“These breeds can vary in size and weight from a
0.46 kg teacup poodle ...”
Source: Wikipedia
Using the units and the measurement type to normalize the number
![Page 28: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/28.jpg)
Proprietary + Confidential
Example - Co-Occurring objects
“These breeds can vary in size and weight from a
0.46 kg teacup poodle ...”
Source: Wikipedia
We detect objects of interest (Nouns, Adjectives and Verbs) using a POS tagger.
460 gram
Noun Noun NounVerb
NP
![Page 29: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/29.jpg)
Proprietary + Confidential
Example - Aggregating Measurements
“These breeds can vary in size and weight from a
0.46 kg teacup poodle ...”
Source: Wikipedia
460 gram
Noun Noun NounVerb
NP
![Page 30: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/30.jpg)
Proprietary + Confidential
Example - Aggregating Measurements
![Page 31: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/31.jpg)
Proprietary + Confidential
![Page 32: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/32.jpg)
Proprietary + Confidential
Example - Aggregating Measurements
![Page 33: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/33.jpg)
Proprietary + ConfidentialProprietary + Confidential
Proprietary + Confidential
Underlying Resource
![Page 34: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/34.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Resource Statistics - Origin
Not enough data
Billion of web pages!
(in English)
![Page 35: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/35.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Resource Statistics - DoQ
● We present: Distributions over Quantities (DoQ)● A very large and diverse resource
● ~120M Unique tuples (object, measurement)○ ~350K with >= 1000 occurrences
![Page 36: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/36.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
● Measurement types:○ Length○ Mass○ Currency○ Temperature○ …
● 27 In total (But not all are useful)
Resource Statistics - DoQ
![Page 37: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/37.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
Using DoQ
● We collected a bunch of numbers for each key● Which in turn creates: Distributions!
![Page 38: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/38.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Proprietary + Confidential
● Given two objects and a scale, we can compare them using their corresponding distributions
● By:○ Comparing the Mean - Noisy○ Comparing the Median - Better○ Comparing a Statist - Doesn’t make much difference,
but returns a probability
Using DoQ
![Page 39: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/39.jpg)
Proprietary + Confidential
Quantitative Evaluation
![Page 40: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/40.jpg)
Proprietary + Confidential
Comparable Objects
● Comparing 2 objects on a given dimension● Nouns
○ 3 different datasets (including a new one we created)● Adjectives
○ 2 different datasets
![Page 41: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/41.jpg)
Proprietary + Confidential
Comparable Objects
● A dataset of ~3.6K object pairs, compared on 5 dimension (e.g. speed, weight, size)
![Page 42: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/42.jpg)
Proprietary + Confidential
Comparable Objects
● Learning a transformation over pre-trained word embedding to infer relations
![Page 43: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/43.jpg)
Proprietary + Confidential
Comparable Objects
● Dataset for size comparison● A combination of Images and texts to infer sizes
![Page 44: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/44.jpg)
Proprietary + Confidential
Comparable Objects
● In this work, we introduce a new dataset for object comparison
● 4 dimension (including Currency, which wasn’t evaluated on before)
● High agreement score (77.1 Kappa)
![Page 45: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/45.jpg)
Proprietary + Confidential
Comparable Objects - Results
![Page 46: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/46.jpg)
Proprietary + Confidential
Comparable Objects - Results
![Page 47: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/47.jpg)
Proprietary + Confidential
Comparable Adjectives - Intensifiers
Freezing < Cold < Warm < Hot
Freezing < Cold Warm < Hot
![Page 48: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/48.jpg)
Proprietary + Confidential
Comparable Adjectives - Intensifiers
● Previous work used Open-IE style methods to infer relations between two objects○ E.g “hot and almost scorching” X < y
![Page 49: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/49.jpg)
Proprietary + Confidential
Comparable Adjectives - Intensifiers
● Previous work used Open-IE style methods to infer relations between two objects○ E.g “hot and almost scorching”
● We have concrete individual distributions for each term, so we don’t rely on specific comparisons
X < y
X y<
![Page 50: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/50.jpg)
Proprietary + Confidential
Comparable Adjectives Inference
![Page 51: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/51.jpg)
Proprietary + Confidential
Comparable Adjectives - Polarities
Freezing, Cold < Warm, Hot
“hot and almost freezing”
“hot and almost scorching”
![Page 52: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/52.jpg)
Proprietary + Confidential
Proprietary + Confidential
Intrinsic Evaluation
![Page 53: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/53.jpg)
Proprietary + Confidential
Intrinsic Evaluation
● Extract the median of “popular” noun distributions● Expand to a range
○ 20 mm 10-100 mm
![Page 54: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/54.jpg)
Proprietary + Confidential
Intrinsic Evaluation
● Extract the median of “popular” noun distributions● Expand to a range
○ 20 mm 10-100 mm● Ask annotators if the item fits the range
![Page 55: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/55.jpg)
Proprietary + Confidential
Intrinsic Evaluation
● Extract the median of “popular” noun distributions● Expand to a range
○ 20 mm 10-100 mm● Ask annotators if the item fits the range
○ “Is the usual length of a screw between 10-100mm?”
![Page 56: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/56.jpg)
Proprietary + Confidential
Intrinsic Evaluation
● 69% agreement with our predictions● Not perfect, but a good start for acquiring such knowledge
![Page 57: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/57.jpg)
Proprietary + Confidential
Qualitative Analysis
![Page 58: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/58.jpg)
Proprietary + Confidential
Comparable Objects - Cool Results
● Many (many) cool and accurate examples
![Page 59: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/59.jpg)
Proprietary + Confidential
Comparable Objects - Cool Results
![Page 60: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/60.jpg)
Proprietary + Confidential
Comparable Objects - Cool Results
We will focus on the errors
![Page 61: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/61.jpg)
Proprietary + Confidential
Comparable Objects - Some Issues
That’s a small sea!
“Elevation ranges from 3,000 feet ... above sea level.”
![Page 62: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/62.jpg)
Proprietary + Confidential
Comparable Objects - Some Issues
That’s a heavy alfalfa
“Alfalfa is the most cultivated legume ... reaching around 454 million tons ...”
https://alivebynature.com/the-right-way-to-eat-alfalfa-sprouts/
![Page 63: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/63.jpg)
Proprietary + Confidential
Comparable Objects - Case StudyCollected temperatures of US States
“Real” average Predicted median
![Page 64: How Large are Lions? Dan Roth Tania Bedrax-Weiss ISCOL ... · ISCOL, September 11, 2019 Yanai Elazar Abhijit Mahabal Deepak Ramachandran Tania Bedrax-Weiss Dan Roth. Proprietary +](https://reader035.vdocuments.mx/reader035/viewer/2022071509/612d24771ecc515869420167/html5/thumbnails/64.jpg)
Proprietary + Confidential
Summary
● A simple method for collecting measure attribution● Obtaining distribution for a various of objects● Releasing a big, new and unique resource● Releasing a refined annotation for an existing dataset and
a new one.
Thanks
Try Me!