local affine feature tracking in films/sitcoms chunhui gu cs 294-6 final presentation dec. 13, 2006
Post on 15-Jan-2016
218 views
TRANSCRIPT
![Page 1: Local Affine Feature Tracking in Films/Sitcoms Chunhui Gu CS 294-6 Final Presentation Dec. 13, 2006](https://reader036.vdocuments.mx/reader036/viewer/2022062409/56649d4a5503460f94a27684/html5/thumbnails/1.jpg)
Local Affine Feature Tracking in Films/Sitcoms
Chunhui GuCS 294-6
Final PresentationDec. 13, 2006
![Page 2: Local Affine Feature Tracking in Films/Sitcoms Chunhui Gu CS 294-6 Final Presentation Dec. 13, 2006](https://reader036.vdocuments.mx/reader036/viewer/2022062409/56649d4a5503460f94a27684/html5/thumbnails/2.jpg)
Objective
• Automatically detect and track local affine features in film/sitcom frame sequences.
– Current Dataset: Sex and the City– Why sitcom?
• Simple daily environment• Few or no special effects• Repeated scenes
![Page 3: Local Affine Feature Tracking in Films/Sitcoms Chunhui Gu CS 294-6 Final Presentation Dec. 13, 2006](https://reader036.vdocuments.mx/reader036/viewer/2022062409/56649d4a5503460f94a27684/html5/thumbnails/3.jpg)
Outline
• Preprocessing• Tracking Algorithm
– Pairwise local matching– Robust features
• Feature Matching across Shots• Results
– Feature matching vs baseline color histogram– Time complexity– When does tracking fail
![Page 4: Local Affine Feature Tracking in Films/Sitcoms Chunhui Gu CS 294-6 Final Presentation Dec. 13, 2006](https://reader036.vdocuments.mx/reader036/viewer/2022062409/56649d4a5503460f94a27684/html5/thumbnails/4.jpg)
Preprocessing
FrameExtraction
(i-1)’th shot i’th shot
ShotDetection
MSER Interest PointDetection
SIFT FeatureExtraction
![Page 5: Local Affine Feature Tracking in Films/Sitcoms Chunhui Gu CS 294-6 Final Presentation Dec. 13, 2006](https://reader036.vdocuments.mx/reader036/viewer/2022062409/56649d4a5503460f94a27684/html5/thumbnails/5.jpg)
Tracking Algorithm
• Basic: Pairwise Matching
Frame i Frame j=i+1
imf
,i im mx y
jnf
![Page 6: Local Affine Feature Tracking in Films/Sitcoms Chunhui Gu CS 294-6 Final Presentation Dec. 13, 2006](https://reader036.vdocuments.mx/reader036/viewer/2022062409/56649d4a5503460f94a27684/html5/thumbnails/6.jpg)
Tracking Algorithm
• Basic: Pairwise Matching
Frame i Frame j=i+1
imf
,i im mx y
jnf
![Page 7: Local Affine Feature Tracking in Films/Sitcoms Chunhui Gu CS 294-6 Final Presentation Dec. 13, 2006](https://reader036.vdocuments.mx/reader036/viewer/2022062409/56649d4a5503460f94a27684/html5/thumbnails/7.jpg)
Tracking Algorithm
• Basic: Pairwise Matching
Frame i Frame j=i+1
imf
,i im mx y
jnf min fd
Thresholding on both minimum distance and ratio
![Page 8: Local Affine Feature Tracking in Films/Sitcoms Chunhui Gu CS 294-6 Final Presentation Dec. 13, 2006](https://reader036.vdocuments.mx/reader036/viewer/2022062409/56649d4a5503460f94a27684/html5/thumbnails/8.jpg)
Tracking Algorithm
• Basic: Pairwise Matching
Frame i Frame j=i+1
imf
,i im mx y
jnf
![Page 9: Local Affine Feature Tracking in Films/Sitcoms Chunhui Gu CS 294-6 Final Presentation Dec. 13, 2006](https://reader036.vdocuments.mx/reader036/viewer/2022062409/56649d4a5503460f94a27684/html5/thumbnails/9.jpg)
Tracking Algorithm
• Basic: Pairwise Matching
Frame i Frame j=i+1
imf
,i im mx y
jnf
![Page 10: Local Affine Feature Tracking in Films/Sitcoms Chunhui Gu CS 294-6 Final Presentation Dec. 13, 2006](https://reader036.vdocuments.mx/reader036/viewer/2022062409/56649d4a5503460f94a27684/html5/thumbnails/10.jpg)
Tracking Algorithm
• Problem of Pairwise Matching– Sensitive to occlusion and feature misdetection
• Solutions:– Use multiple overlapping windows– Backward Matching
• Match features in current frame to features in all previous frames within the shot
• Pruning process (reduce computation time)
• Select a proportion of features that have longer tracking length as robust features
![Page 11: Local Affine Feature Tracking in Films/Sitcoms Chunhui Gu CS 294-6 Final Presentation Dec. 13, 2006](https://reader036.vdocuments.mx/reader036/viewer/2022062409/56649d4a5503460f94a27684/html5/thumbnails/11.jpg)
Shot grouping/Scene Retrieval
60
601 2, ,...rf mf x x x
56
561 2, ,...rf mf x x x
10746 10747 10772
Shot 49
10933 10934 10968
Shot 53
11393 11394 11435
Shot 56
Shot 60
11533 11534 11560
Scene 5
49
491 2, ,...rf mf x x x
53
531 2, ,...rf mf x x x
![Page 12: Local Affine Feature Tracking in Films/Sitcoms Chunhui Gu CS 294-6 Final Presentation Dec. 13, 2006](https://reader036.vdocuments.mx/reader036/viewer/2022062409/56649d4a5503460f94a27684/html5/thumbnails/12.jpg)
Inter-Shot Matching
Shot I Shot J
1 11 2, ,...I mf x x x
2 21 2, ,...I mf x x x
1 11 2, ,...J nf x x x
2 21 2, ,...J nf x x x
1 2, ,...q qJ nf x x x 1 2, ,...
p pI mf x x x
D
![Page 13: Local Affine Feature Tracking in Films/Sitcoms Chunhui Gu CS 294-6 Final Presentation Dec. 13, 2006](https://reader036.vdocuments.mx/reader036/viewer/2022062409/56649d4a5503460f94a27684/html5/thumbnails/13.jpg)
“Confusion Table”
Ground Truth50 55 60 65 70 75
50
55
60
65
70
75
Color Histograms50 55 60 65 70 75
50
55
60
65
70
75
Feature Matching50 55 60 65 70 75
50
55
60
65
70
75
![Page 14: Local Affine Feature Tracking in Films/Sitcoms Chunhui Gu CS 294-6 Final Presentation Dec. 13, 2006](https://reader036.vdocuments.mx/reader036/viewer/2022062409/56649d4a5503460f94a27684/html5/thumbnails/14.jpg)
ROC
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
False Alarm
Tru
e D
etec
tion
ROC curve of Feature Matching
![Page 15: Local Affine Feature Tracking in Films/Sitcoms Chunhui Gu CS 294-6 Final Presentation Dec. 13, 2006](https://reader036.vdocuments.mx/reader036/viewer/2022062409/56649d4a5503460f94a27684/html5/thumbnails/15.jpg)
When Does Tracking Fail?• Tracking feature outside local window
– Rare when continuous tracking– Happens when occlusion occurs
• Same feature splitting to two or more groups– Long occlusion– Multiple matching in a single frame
Frame i Frame j=i+1
imf
,i im mx y
jnf
![Page 16: Local Affine Feature Tracking in Films/Sitcoms Chunhui Gu CS 294-6 Final Presentation Dec. 13, 2006](https://reader036.vdocuments.mx/reader036/viewer/2022062409/56649d4a5503460f94a27684/html5/thumbnails/16.jpg)
Computation Complexity• Everything except for MSER and SIFT algorithms are
implemented in Matlab (slow…)
Complexity Time
Frame Extraction O(N) ~0.3s/frame
Shot Detection O(N*f(B)) ~0.07s/frame (B=16)
MSER Detection O(N) ~0.3s/frame
SIFT Detection O(N) ~0.9s/frame
Feature Tracking O(N*F*W*L) ~0.5s/frame
Matching across shots
O(S2*T2) ~1s/shot pair
N: # of frames; (30,000) B: # of bins for color hist (16) F: ave. # of features per frame; (400) W: Local window size; (15)L: tracking length; (20) T: ave. # of robust trackers per shot; (300)S: # of shots; (35)
![Page 17: Local Affine Feature Tracking in Films/Sitcoms Chunhui Gu CS 294-6 Final Presentation Dec. 13, 2006](https://reader036.vdocuments.mx/reader036/viewer/2022062409/56649d4a5503460f94a27684/html5/thumbnails/17.jpg)
Conclusion
• We successfully implemented local affine feature tracking in sitcom “sex and the city”. The tracking method is robust to occlusion and feature misdetection.
• Although no quantitative precision/recall curve (hard to find ground truth), the demonstration shows that precision is almost perfect with good recall performance.
• We show one successful application of using robust features to associate similar shots together for scene retrieval.
![Page 18: Local Affine Feature Tracking in Films/Sitcoms Chunhui Gu CS 294-6 Final Presentation Dec. 13, 2006](https://reader036.vdocuments.mx/reader036/viewer/2022062409/56649d4a5503460f94a27684/html5/thumbnails/18.jpg)
Future Work
• Implement algorithm in real-time (C/C++)
• Search unique shots in films/sitcoms
• Separate indoor scenes from outdoor scenes
• Determine context of the scene
![Page 19: Local Affine Feature Tracking in Films/Sitcoms Chunhui Gu CS 294-6 Final Presentation Dec. 13, 2006](https://reader036.vdocuments.mx/reader036/viewer/2022062409/56649d4a5503460f94a27684/html5/thumbnails/19.jpg)
Acknowledgement