© leo burnett. 2 “moe”: april 23 “moe” : june 6 3 population size estimation behavioral...
TRANSCRIPT
![Page 1: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/1.jpg)
Automatic Zebra Identification from PhotographsMayank Lahiri
University of Illinois at Chicago
© Leo Burnett
![Page 2: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/2.jpg)
2
The goal, in two pictures
“Moe”: April 23
“Moe” : June 6
![Page 3: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/3.jpg)
3
Identifying zebras by their stripes:Why identify zebras?
Population size estimation
Behavioral studies
Phenotype analysis
![Page 4: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/4.jpg)
4
Identifying zebras by their stripes:And why use stripes?
Tracking devices:
Require anesthetization
Expensive
Unreliable
Broad coverage all but impossible
![Page 5: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/5.jpg)
5
The algorithm: ZebraGoogle
1.Moe 93%2.Alice 5%3.Bob 2%
The “ALGORITHM
”
Database of zebra pictures
Input – “the query”
Output – “zebra ranking”
![Page 6: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/6.jpg)
6
Attempts #1 and #2
Face recognition algorithms▪ Early algorithms too rigid▪ Modern algorithm could work, but are black
boxes
Fingerprint recognition▪ Generally look for well-defined features▪ Rarely deal with occlusion, perspective skew,
varying distance to camera
![Page 7: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/7.jpg)
7
Attempt #3: Shape matching
© Colchester Zoo
y
x
1. Shape contour tracing2. Spline function fitting3. Query and retrieve splines
![Page 8: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/8.jpg)
8
Attempt #3: Shape matching
![Page 9: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/9.jpg)
9
Is there a simpler way?
How much information is there in the data?
![Page 10: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/10.jpg)
10
Is there a simpler way?
How much information is there in the data?
![Page 11: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/11.jpg)
11
Attempt #4: ZEBRA BARCODE Hypothesis: width and spacing of
stripes are distinctive when measured finely
![Page 12: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/12.jpg)
12
Zebras vs. Barcodes (seriously)Zebras are/have:
seldom two-dimensional
frequently obscured
non uniform stripes
high tendency towards pregnancy and violence
afraid of barcode scanners
© Barcodeman
![Page 13: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/13.jpg)
13
The approach
Build a solution by eliminating the problems!
![Page 14: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/14.jpg)
14
PROBLEM #1:“Width” of stripes change with distance
Closer to camera, more pixels for the body
Further away, fewer pixels for the body
10 Megapixel camera = 3648 pixels across, 2736 pixels down1080p HD TV = 1920 pixels across, 1080 pixels down15” MacBook Pro screen = 1440 pixels across, 900 pixels down
![Page 15: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/15.jpg)
15
PROBLEM #1:“Width” of stripes change with distance
Solution: measure widths relative to the previous stripe
52 46 64 66 63 88 75 104
![Page 16: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/16.jpg)
16
PROBLEM #1:“Width” of stripes change with distance
Solution: measure widths relative to the previous stripe
52 46 64 66 63 88 75 1.38
![Page 17: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/17.jpg)
17
PROBLEM #1:“Width” of stripes change with distance
Solution: measure widths relative to the previous stripe
52 46 64 66 63 88 0.85 1.38
![Page 18: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/18.jpg)
18
PROBLEM #1:“Width” of stripes change with distance
Solution: measure widths relative to the previous stripe
52 .8 1.4 1.0 .95 1.39 0.85 1.38
![Page 19: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/19.jpg)
19
PROBLEM #2:Camera is rarely perpendicular to zebra
![Page 20: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/20.jpg)
20
PROBLEM #2:Camera is rarely perpendicular to zebra
Shear transformation flattens small amount of perspective skew [1].
Original image With shear transformation
![Page 21: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/21.jpg)
21
PROBLEM #2:Camera is rarely perpendicular to zebra
Shear transformation flattens small amount of perspective skew [1].
Shear transformation is a special case of affine transformation.
Affine transformation: Ratios of distances along a line are
preserved
![Page 22: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/22.jpg)
22
PROBLEM #2: Camera is rarely perpendicular to zebra
Solution: measure widths relative to the previous stripe
52 .8 1.4 1.0 .95 1.39 0.85 1.38
![Page 23: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/23.jpg)
23
PROBLEM #2: Camera is rarely perpendicular to zebra
Solution: measure widths relative to the previous stripe
52 .8 1.4 1.0 .95 1.39 0.85 1.38
0.8 1.4 1.0 0.95 1.39 0.85 1.38
A “strip” of stripes
![Page 24: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/24.jpg)
24
PROBLEM #3:Zebras move, and computers are stupid
1.4 1.0 0.95 1.39 0.85 1.38 0.7
From original photograph in database:
0.83 1.39 0.85 1.38 0.7
Zebra occluded from the left side:
Missed the rightmost black stripe:
1.4 1.0 0.95 1.39 0.85 1.98
Extremely oblique viewing angle:
1.5 1.1 1.2 1.7 1.01 1.66 1.3
![Page 25: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/25.jpg)
25
PROBLEM #3:Zebras move, and computers are stupid
Solution: Dynamic programming Align two strips to minimize “errors” a.k.a. Spell-check, DNA sequence alignment, Needleman-Wunsch
algorithm, Smith-Waterman algorithm, edit distance, dynamic time warping, etc.
Stripe-alignment! occlusion = indel cost image processing errors = indel + matching cost stripe distortion = matching cost strong perspective skew = matching cost
Low alignment “cost” = fewer differences in strips = zebras are very similar
![Page 26: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/26.jpg)
27
PROBLEM #3:Zebras move, and computers are stupid
For a new picture (the “query”):
Read a strip off the body at a known location
Align against all the zebra strips in the database, also from the same location
Rank zebras in the database by the alignment cost of their strips
![Page 27: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/27.jpg)
28
In Theory...
One click per zebra
Analogous to a barcode scanner
Handles occlusion, minor perspective skew
Can be applied to any part of the body
Computationally efficient
![Page 28: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/28.jpg)
29
In Practice. 20 zebras, ~6 photos per zebra =
109 pictures.
“Transcription” errors
![Page 29: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/29.jpg)
30
Evaluating accuracy
Photos were manually identified by Rosemary at Ol’Pejeta Conservancy. Manually coded stripes along the
shoulder
For each photo, rank the closest matches using dynamic programming.
Metric: rank of the correct zebra in the list of closest matches.
![Page 30: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/30.jpg)
31
Results: Rank of correct zebraPro
port
ion
of
qu
eri
es
at
or
belo
w
rank
Average rank = 1.5
![Page 31: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/31.jpg)
32
Results: Where does it fail? No love from Zebra #3 – “01_700”
Photo 87 Zebra 3 Flank R PicID 8159 CORRECT_RANK 4 Photo 65 Zebra 3 Flank L PicID 8142 CORRECT_RANK 9 Photo 63 Zebra 3 Flank R PicID 8170 CORRECT_RANK 5 Photo 52 Zebra 3 Flank R PicID 8166 CORRECT_RANK 2
8159
8142
8170
8166
![Page 32: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/32.jpg)
33
Results: Where does it fail? Worst performance on this picture:
![Page 33: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/33.jpg)
34
Performance
Time to search database of 108 pictures: 0.023 seconds -- my ageing 2006 laptop
If the number of stripes on a zebra is O(1), then the time complexity of a single search is linear in the number of photographs. 1,000 pictures ~ 0.2 seconds 10,000 pictures ~ 3 seconds
![Page 34: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/34.jpg)
35
Applicability
![Page 35: © Leo Burnett. 2 “Moe”: April 23 “Moe” : June 6 3 Population size estimation Behavioral studies Phenotype analysis](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cda5503460f949a4086/html5/thumbnails/35.jpg)
36
Questions?Future work:1. Build an effective user interface2. Get field biologists to discover better ways to
use it!3. Run tests on 3,000+ pictures from January
Kenya trip4. Tweak image processing algorithms
“So you see! There’s no end to the thing you might know, depending how far beyond Zebra
you go.”- Dr. Seuss, Beyond Zebra
References:[1] Hutchison and Barrett . Fourier-Mellin registration of line-detained tabular document images. Intl. J. Doc. Analysis 8(2):87-110, 2006.