3d safari: learning to estimate zebra pose, shape, and ... · prediction input image + params...
TRANSCRIPT
![Page 1: 3D Safari: Learning to Estimate Zebra Pose, Shape, and ... · prediction input image + Params predictor Stitching Network. Encoder Shape predictor features dv uv-flow SMAL horse template](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8ce985dd432e064432be32/html5/thumbnails/1.jpg)
3D Safari: Learning to Estimate Zebra Pose, Shape, and Texture
from Images “In the Wild”Silvia Zuffi, Angjoo Kanazawa,
Tanya Berger-Wolf, Michael J. Black
![Page 2: 3D Safari: Learning to Estimate Zebra Pose, Shape, and ... · prediction input image + Params predictor Stitching Network. Encoder Shape predictor features dv uv-flow SMAL horse template](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8ce985dd432e064432be32/html5/thumbnails/2.jpg)
The Grevy’s zebra
![Page 3: 3D Safari: Learning to Estimate Zebra Pose, Shape, and ... · prediction input image + Params predictor Stitching Network. Encoder Shape predictor features dv uv-flow SMAL horse template](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8ce985dd432e064432be32/html5/thumbnails/3.jpg)
The Grevy’s zebra
https://zebra.wildbook.org/First census of the Grevy’s zebra with photographs of ordinary citizens
![Page 4: 3D Safari: Learning to Estimate Zebra Pose, Shape, and ... · prediction input image + Params predictor Stitching Network. Encoder Shape predictor features dv uv-flow SMAL horse template](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8ce985dd432e064432be32/html5/thumbnails/4.jpg)
• Skinned Multi-Animal Linear model• A 3D shape model representing articulation
and shape variation across different species
• From 3D data, fast to computeS. Zuffi, A. Kanazawa, D. Jacobs, M. J. Black, 3D Menagerie: Modeling the 3D Shape and Pose of Animals, CVPR 2017
Examples from the training set
SMAL
![Page 5: 3D Safari: Learning to Estimate Zebra Pose, Shape, and ... · prediction input image + Params predictor Stitching Network. Encoder Shape predictor features dv uv-flow SMAL horse template](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8ce985dd432e064432be32/html5/thumbnails/5.jpg)
vshape(�) = vtemplate +Bs�
Training set: Toys scans in correspondence and in reference pose
SMAL shape space
![Page 6: 3D Safari: Learning to Estimate Zebra Pose, Shape, and ... · prediction input image + Params predictor Stitching Network. Encoder Shape predictor features dv uv-flow SMAL horse template](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8ce985dd432e064432be32/html5/thumbnails/6.jpg)
Manual segmentation and manually annotated keypoints
Applications of SMAL
![Page 7: 3D Safari: Learning to Estimate Zebra Pose, Shape, and ... · prediction input image + Params predictor Stitching Network. Encoder Shape predictor features dv uv-flow SMAL horse template](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8ce985dd432e064432be32/html5/thumbnails/7.jpg)
B. Biggs, T. Roddick, A. Fitzgibbon, R. Cipolla, Creatures great and SMAL: Recovering the shape and motion of animals from video, ACCV2019
Applications of SMALAutomatic segmentation and keypoints detection from silhouette
![Page 8: 3D Safari: Learning to Estimate Zebra Pose, Shape, and ... · prediction input image + Params predictor Stitching Network. Encoder Shape predictor features dv uv-flow SMAL horse template](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8ce985dd432e064432be32/html5/thumbnails/8.jpg)
• GOAL: Estimate 3D shape and pose as a direct regression from RGB
• APPROACH: Supervised, training based only on synthetic data
Our work
![Page 9: 3D Safari: Learning to Estimate Zebra Pose, Shape, and ... · prediction input image + Params predictor Stitching Network. Encoder Shape predictor features dv uv-flow SMAL horse template](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8ce985dd432e064432be32/html5/thumbnails/9.jpg)
S. Zuffi, A. Kanazawa, M. J. Black, Lions and Tigers and Bears: Capturing Non-Rigid, 3D, Articulated Shape from Images, CVPR2018
1. SMAL model fitting 2. Model-free shape Refinement
SMAL with Refinement (SMALR)
![Page 10: 3D Safari: Learning to Estimate Zebra Pose, Shape, and ... · prediction input image + Params predictor Stitching Network. Encoder Shape predictor features dv uv-flow SMAL horse template](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8ce985dd432e064432be32/html5/thumbnails/10.jpg)
S. Zuffi, A. Kanazawa, M. J. Black, Lions and Tigers and Bears: Capturing Non-Rigid, 3D, Articulated Shape from Images, CVPR2018
Animals avatars with SMALR
![Page 11: 3D Safari: Learning to Estimate Zebra Pose, Shape, and ... · prediction input image + Params predictor Stitching Network. Encoder Shape predictor features dv uv-flow SMAL horse template](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8ce985dd432e064432be32/html5/thumbnails/11.jpg)
Grevy’s zebra avatarsMultiple images of the same subject
3D model
Texture map
![Page 12: 3D Safari: Learning to Estimate Zebra Pose, Shape, and ... · prediction input image + Params predictor Stitching Network. Encoder Shape predictor features dv uv-flow SMAL horse template](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8ce985dd432e064432be32/html5/thumbnails/12.jpg)
Synthetic
Real
Synthetic dataset from avatars
![Page 13: 3D Safari: Learning to Estimate Zebra Pose, Shape, and ... · prediction input image + Params predictor Stitching Network. Encoder Shape predictor features dv uv-flow SMAL horse template](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8ce985dd432e064432be32/html5/thumbnails/13.jpg)
vshape(fs) = vtemplate + dvdv = Wfs + b
vshape(�) = vtemplate +Bs�SMAL model:
Shape predictor:
Encoder Shapepredictor
features
dv
uv-flow
SMALhorsetemplate
translationfocallength3Dpose
uv-flowpredictor
+zebraT-pose
texturemap
prediction
inputimage
+
Paramspredictor
Stitching
Network
![Page 14: 3D Safari: Learning to Estimate Zebra Pose, Shape, and ... · prediction input image + Params predictor Stitching Network. Encoder Shape predictor features dv uv-flow SMAL horse template](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8ce985dd432e064432be32/html5/thumbnails/14.jpg)
Encoder Shapepredictor
features
dv
uv-flow
SMALhorsetemplate
translationfocallength3Dpose
uv-flowpredictor
+zebraT-pose
texturemap
prediction
inputimage
+
Paramspredictor
Stitching
Ltrain = Lmask(Sgt, S) + Lkp2D (K2D,gt,K2D) +Lcam(fgt, f) + Limg(Iinput, I, Sgt) + Lpose(✓gt, ✓) +Ltrans(�gt, �) + Lshape(dvgt,dv) + Luv(uvgt,uv) +Ltex(Tgt, T ) + Ldt(uv, Sgt)
Network
![Page 15: 3D Safari: Learning to Estimate Zebra Pose, Shape, and ... · prediction input image + Params predictor Stitching Network. Encoder Shape predictor features dv uv-flow SMAL horse template](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8ce985dd432e064432be32/html5/thumbnails/15.jpg)
Results on test set
![Page 16: 3D Safari: Learning to Estimate Zebra Pose, Shape, and ... · prediction input image + Params predictor Stitching Network. Encoder Shape predictor features dv uv-flow SMAL horse template](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8ce985dd432e064432be32/html5/thumbnails/16.jpg)
Encoder Shapepredictor
features
dv
uv-flow
SMALhorsetemplate
translationfocallength3Dpose
uv-flowpredictor
+zebraT-pose
texturemap
prediction
inputimage
+
Paramspredictor
Stitching
backgroundmodel
+ -
@
RegressionnetworkPer-instanceoptimization
Unsupervised optimization
Lopt = Lphoto(Iinput, I) + Lcam(f̂ , f) + Ltrans(�̂, �)
![Page 17: 3D Safari: Learning to Estimate Zebra Pose, Shape, and ... · prediction input image + Params predictor Stitching Network. Encoder Shape predictor features dv uv-flow SMAL horse template](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8ce985dd432e064432be32/html5/thumbnails/17.jpg)
Unsupervised optimization
inputprediction
overlap predicted
image optimization
result
![Page 18: 3D Safari: Learning to Estimate Zebra Pose, Shape, and ... · prediction input image + Params predictor Stitching Network. Encoder Shape predictor features dv uv-flow SMAL horse template](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8ce985dd432e064432be32/html5/thumbnails/18.jpg)
Results
Texture prediction helps!
Better to optimize over features
![Page 19: 3D Safari: Learning to Estimate Zebra Pose, Shape, and ... · prediction input image + Params predictor Stitching Network. Encoder Shape predictor features dv uv-flow SMAL horse template](https://reader030.vdocuments.mx/reader030/viewer/2022040915/5e8ce985dd432e064432be32/html5/thumbnails/19.jpg)
S. Zuffi, A. Kanazawa, T. Berger-Wolf, M.J. Black, 3D Safari: Learning to Estimate Zebra Pose, Shape, and Texture from Images “In the Wild”, ICCV 2019
Poster n.93, 31st Oct 10:30