probabilistic context free grammars for representing action song mao november 14, 2000
TRANSCRIPT
![Page 1: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/1.jpg)
Probabilistic Context Free Grammars for Representing Action
Song Mao
November 14, 2000
![Page 2: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/2.jpg)
Action Understanding in Computer Vision
Interpretation of basic movements Sitting, walking , running …
Description of motion of several objects Recognition of gestures High-level event
![Page 3: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/3.jpg)
High-level Events
Consists of primitives For car drop-off event: car-enter, car-stop,
person-enter, person-exit, etc.
Spatio-temporal structure & constraint Semantically defined activities Span extended periods of time Multi-object interactions
![Page 4: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/4.jpg)
Approaches
Statistical techniques Syntactic techniques Methods that combine the two techniques
![Page 5: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/5.jpg)
Statistical Techniques
Classifying pattern by assuming an statistical model Tennis stroke recognition Gesture recognition Visual language recognition
Advantages Real world data are noisy in nature (signal noise) Uncertainty in observation (sensor noise)
![Page 6: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/6.jpg)
Statistical Techniques (cont.)
Disadvantages Insufficient data Semantic ambiguity Temporal ambiguity Known structure
![Page 7: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/7.jpg)
Syntactic Techniques
Describe pattern structure Formal grammar Context free grammar (CFG) Stochastic context free grammar (SCFGs) Parsing
![Page 8: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/8.jpg)
Combine the Two Techniques
Independent primitives detection using statistical techniques
Actions (structured primitives) recognition by syntactic techniques Parsing primitives by SCFGs Removing ambiguity by parsing SCFGs Correcting errors (substitution, insertion, deletion) by
adding SKIP rules and penalty function to SCFGs
![Page 9: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/9.jpg)
Decoupling Primitive Detection and Primitive Structuring
![Page 10: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/10.jpg)
Parsing
What is Parsing?
The process of taking an input and producing some sort of structure for it.(Jurafsky & Martin)
Structure assigned by Context Free Grammar (CFG) / Stochastic Context Free Grammar (SCFG)
![Page 11: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/11.jpg)
Parsing Approaches
Top-down approach Bottom-up approach Dynamic programming approach
Cocke-Younger-Kasami parser (CYK) Graham-Harrison-Ruzzo parser (GHR) Earley parser
![Page 12: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/12.jpg)
Context Free Grammars (CFG)
A set of non-terminal symbols A set of terminal symbols A set of productions P of form
Start symbol Directly derivation:
if ,
A
N
)(, NNA S
A ,)(, N A
![Page 13: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/13.jpg)
Context Free Grammar (Cont.)
Derivation: . . .,
A language generated by a grammar
,, 3221 ,1 mm
m
1
},|{ wSwwLG
G
![Page 14: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/14.jpg)
Stochastic Context Free Grammar (SCFG)
Modify production as: Where is the rule
probability of the production from a Context-Free Grammars (CFGs)
Rules are conditionally independent
pA
)|( AAPp A
)|()|()|(
),,|,,(
CCPBBPAAP
CBACBAP
![Page 15: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/15.jpg)
Earley Parsing Algorithm
A set of states for each position in the input Dot denotes the current input position A state with the dot at the right most position is a
complete state A state produced by prediction is a predicted state A state produced by completion is a completed state
![Page 16: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/16.jpg)
Earley Parsing Algorithm (cont.)
A State:
Prediction:
:
,
, :
ik Yi
Y
YXi
YXi k :
![Page 17: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/17.jpg)
Earley Parser (cont.)
Scanning: Completion:
:1 : aXiaXi kk
:
, :
, :
YXiYi
YXjk
j
k
![Page 18: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/18.jpg)
An Example
VP VerbVP VerbVP
ProperNounNP Noun Det NP
flightNoun VPS
thatDet VP NPAux S
bookVerb VP NPS
state set (0) (1) Book (2) that (3) flight
NP Verb VP:0
Verb VP:0
VP S:0
VP NPAux S:0
ProperNoun NP:0
NounDet NP:0
VP NP S:0
predicted
S 0 :0
0
0
0
0
0
0
0
0
ProperNoun NP:1
NounDet NP:1
predicted
NP VerbVP:1
VPS:1
VerbVP:1
completed
bookVerb:1
scanned
1
1
0
0
0
0
Noun Det NP:2
completed
thatDet:2
scanned
1
1
VPS:3
NP VerbVP:3
NounDet NP:3
completed
flightNoun:3
scanned
0
0
1
2
![Page 19: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/19.jpg)
Earley-Stolcke Parser (1)
A state Forward probability
Inner probability
Earley path: a sequence of states needed to reach the current state
Length of path: number of scanning states
],[ : YXi k : state in start that length of path ( Y X k k i Pk
path
path
k YXiP ) : statein ends that ilength ofpath (
) : statein end and YXi k
![Page 20: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/20.jpg)
Earley-Stolcke Parser (2)
Prediction
where
],[ : ,
],[ :
ik Yi
Y
ZXi
)(
)(),() :(,
YP
YPYZRZXi Lk
![Page 21: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/21.jpg)
Compute
Left-recursion in grammar Possibly infinite prediction loop that accumulate probability
computation Example: Left Corner relation:
aAAaA ,YXYX L iff ,
...)()(),(10
YZPYZPYZR LLL
LR
)...()( 121 YYYYZPYZP kL
k
L
![Page 22: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/22.jpg)
Compute(cont.)
LR
Matrix form
Computed once for the grammar, and used at each iteration of the prediction step
1
0
10 )( ...
k
LLLL PIPPR
![Page 23: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/23.jpg)
Earley-Stolcke Parser (3)
Scanning
Completion
where
],[ :1 ],[ : aXiaXi kk
],[ : , ],[ :
, ],[ :
ZXi
Yi
ZXjk
j
k
):(),() :(
):(),() :(
jUk
jUk
YiYZRZXj
YiYZRZXj
![Page 24: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/24.jpg)
Compute
Unit production: Infinite completion by unit production
e. g.
Unit production relation matrix Similarly as computing in prediction step
UR
YXYX U iff ,
ABaABA , ,
UP
1
0
10 )( ...
k
UUUU PIPPR
LR
![Page 25: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/25.jpg)
Uncertainty in the Input
Source of the input symbols is probabilistic Modify scanning of the Earley-Stolcke parser
Address the substitution error
0)( t.s. ,
],[ :1 ],[ :
aPa
aXiaXi kk
)() :(
)() :(
aPXi
aPXi
k
k
![Page 26: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/26.jpg)
An Example for a Grammar acbabcA |
![Page 27: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/27.jpg)
Insertion and Deletion
Use a robust form grammar of
Includes all repetitions of all terminals Set small Penalize derivation consuming less terminals
... | | | ... | | SKIPcSKIPbcbSKIP
G G
) ( ruleSKIPP
CBA bCA
G :G
SKIP b | b SKIP | bB
![Page 28: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/28.jpg)
Enforcing Consistency (1)
Types of consistency Temporal consistency Spatial consistency Object identity consistency
Add 2 vector valued state variables low mark high mark
Containing the data for computing distance penalty between two joining states
:l:h
![Page 29: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/29.jpg)
Enforcing Consistency (2)
Prediction
Scanning
],[ : ,
],[ :tti
k SSYiY
hlZXi
else ],[ :1
if ],[ :1
],[ :
ak
aak
k
hlaXi
hlaXi
hlaXi
![Page 30: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/30.jpg)
Enforcing Consistency (3)
Completion
: distance penalty function Computed based on high mark of completed
state and low mark of completing state
):(),() :()(
):(),() :()(
jUk
jUk
YiYZRZXjdf
YiYZRZXjdf
)(df
![Page 31: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/31.jpg)
Choice of
Sever penalty: step function e. g.
Softer penalty: exponential function e. g.
)(df
else ,1
0 ,0 )(
ddf
2
)(d
Cedf
![Page 32: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/32.jpg)
Application:Vedio Surveillance of Parking Lot
Outdoor environment – occlusions and lighting change
Static cameras Real-time performance Labeling activities and person-vehicle
interactions in a parking lot Handling simultaneous events
![Page 33: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/33.jpg)
Known Structure, Uncertain Elements
Activities as sequences of primitives represented by SCFG Car drop-off, car pick-up Dancing
Input primitives are uncertain Uncertain observation of primitive Noisy symbols
![Page 34: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/34.jpg)
Approach
First detect primitives using statistical method Tracker Event generator
Then Recognize activity by parsing input stream of uncertain primitives (partial tracks) by an SCFG parser
![Page 35: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/35.jpg)
System Overview
![Page 36: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/36.jpg)
System Overview (cont.)
Tracker Assign identity to the moving objects Collects the trajectory data into partial tracks
Event generator Maps partial tracks onto predetermined set of events
Parser Labels sequences of events by parsing using a SCFG Enforce consistency constraint
![Page 37: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/37.jpg)
Tracker
Object found Assign a unique ID Track changes in objects’ appearance, position,
velocity Based on the data, assign each object a class label
(e.g. a car or a person) Object lost Object exit
![Page 38: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/38.jpg)
Event Generator
Based on data from tracker Object-enter Object-found Object-exit Object-lost Object-stopped
Initially, tracker can not figure out class label, When object exit, tracker has enough information to
assign a class label to the object
![Page 39: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/39.jpg)
An Example of Generating Events
![Page 40: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/40.jpg)
Parsing Events
![Page 41: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/41.jpg)
Sample Stochastic Context-Free Grammar
![Page 42: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/42.jpg)
Tracker and Event Generator Data for Parser States
Tracker event generator provides data for “low mark” and “high mark” of parser states
f: frame number t: timing stamp (x,y): location (dx,dy): velocity
h
h
h
h
h
h
l
l
l
l
l
l
dy
dx
y
x
t
f
h
dy
dx
y
x
t
f
l
![Page 43: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/43.jpg)
Distance Penalty Function
: high mark data of state being completed :low mark of the completing state
Where is the predicted position of the object at time
)(
else ,)()(
exp
0)( if ,0
),,,(
1211
22
12
221
ttdrrr
rrrr
tt
rrttf
p
pT
pp
2t
111 ,, drrt22 , rt
![Page 44: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/44.jpg)
An Example
![Page 45: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/45.jpg)
Events Data for Drive-In and Drop-Off Activities
![Page 46: Probabilistic Context Free Grammars for Representing Action Song Mao November 14, 2000](https://reader034.vdocuments.mx/reader034/viewer/2022051820/56649e845503460f94b855a9/html5/thumbnails/46.jpg)
Vedio Frame Illustration
Person passed through Person drove in Person drop off Car passed through