outsourcing framenet to the crowd
TRANSCRIPT
Filling
“A trolley was heaped with beer cans”
ThemeGoal
Frame annotation
2
Lexical unit
Frame Frame elements
to heap Filling [Goal, Theme]
Frame annotation
3
Lexical unit
to heap
Frame 1
Filling
Frame 2
Placing
Goal CauseTheme Agent
CHALLENGEFull frame annotation by the man in the street
4
2-step methodology
5
1Frame
discrimination
2FE recognition
Word sense disambiguationWhich is the sense of heaped?
Filling
Semantic role assignmentThe Theme is...
Which is the Theme?with beer cans
“A trolley was heaped with beer cans”
Critical issuesin crowdsourcing
6
“The element Theme is generally an NP object”
???Definition by experts for
experts
Critical issuesin crowdsourcing
7
1Frame
discrimination
2FE recognition
!!!Error propagation
Frame emersion
Alternative methodology
8
1-step workflow (bottom-up)
1FE recognition
Implementation
9
“A trolley was heaped with beer cans”
to heap
A trolley
with beer cans None None
PlacingFilling
Theme?
Goal?
Agent?
Cause?
Manual simplification
a. Replace the FE name with the semantic type
b. Simplify complex syntax
c. Avoid variability
d. Reformulate technical concepts using common words
10
Simplification impact
11
LU Frame FE Gain
to throwCause motion
Theme + 44%to throw
Cause motionGoal + 19%to throw
Body movement Body part + 31%
to guide Influence of event on cognizer
Cognizer + 25%
Average gainAverage gainAverage gain + 30%
Simplification examples
The element Theme is generally an NP object
The Theme is the element that undergoes the motion
12
ThemeCause motion
Simplification examples
13
With some verbs in this frame, the Body part involved in the action is specified by the meaning of the verb and cannot be
expressed separately
This element describes the Body part that is involved in the action
Body part
Body Movement
EXPERIMENTSwith the CrowdFlower platform
14
SettingsLexical unit Frames
to disappearCeasing to be
Departing
to guideCotheme
Influence of event on cognizer
to heapFillingPlacing
to throwBody movementCause motion
JudgmentsCost per sentence
51.83 $ cents
15
2-STEPTop-down standard annotation workflow
1. Frame discrimination
2. FE recognition
16
“A trolley was heaped with beer cans”Which is the correct sense?
Filling Placing
“A trolley was heaped with beer cans”Theme: The Theme is the object which changes location
A trolley with beer cans
Goal: The Goal is...
Choose the right sense of a word
Find the participants in the event
Simplified FE
definitions
Theme: The Theme is the object which changes location A trolley with beer cans None
Goal: The Goal is...
1-STEPBottom-up workflow
17
“A trolley was heaped with beer cans”
Filling
correct frame
Agent: The Agent is the person that cause the theme to move A trolley with beer cans None
Cause: The Cause is...
1-STEPBottom-up workflow
“A trolley was heaped with beer cans”
Placing
wrong frame
18
Results
2-step 1-step
Majority vote accuracy
Execution time (h)
Cost per sentence ($ cents)
.687 .792
171 130
4.57 8.41
19
986 judgments collected so far
Lessons learnt
Difficult gold = low agreement
Automatic task takedown = time increase
Contested gold is useful
Signal for tricky FE definitions
Negation and modality are problematic
20
Negation
21
“On their way to the station she would not throw her coin into the Trevi Fountain”
the Goal is the place where the element ends up at the end of the motionGoal
A worker said
“If she WOULD NOT THROW her coin, it did NOT end up in the fountain. Therefore this answer is wrong. She still
has the coin.”
Conclusion
22
We can crowdsource frame annotation
FE definitions simplification
Bottom-up approach
Research directions
Larger scale experiments
Get rid of FE definitions
Entity linking techniques
Semantic type information from structured knowledge bases
23
CROWDCRAFTING.ORGFree crowdsourcing platform
Our task
25
FE recognition pilots
26
Original Automatic Simplified
Majority Accuracy
Untrusted judgments
.777 .666 .750
99 222 36