p20131108 acpr2013 invited - osaka university€¦ · mr. akira tsuji dr. hidetoshi mannami ms....

Yasushi MakiharaThe Institute of Scientific and Industrial Research,

Osaka University, Japan

The 2nd Asian Conf. on Pattern Recognition (ACPR 2013), Nov. 8, 2013

Acknowledgements-Yagi lab., ISIR, Osaka University- Staff

Prof. Yasushi Yagi Prof. Tomio Echigo Prof. Yasuhiro Mukaigawa Prof. Ryusuke Sagawa Prof. Ikuhisa Mitsugami Dr. Junqiu Wang Dr. Md Altab Hossain Dr. Chunsheng Hua Dr. Al Mansur Dr. Daigo Muramatsu Dr. Haruyuki Iwama Dr. Rasyid Aqmar

Students Mr. Kazushige Sugiura Mr. Akira Tsuji Dr. Hidetoshi Mannami Ms. Koko Cho Mr. Atsushi Mori Ms. Mayu Okumura Mr. Akira Shiraishi Mr. Naoki Akae Mr. Yusuke Fujihara Ms. Betria Silvana Rossa Mr. Ryo Kawai Mr. Yuma Higashiyama Mr. Ruochen Liao Mr. Takuya Tanoue Mr. Takuhiro Kimura Mr. Jaemin Son 2 / 50

Pattern of movement of the limbs of animalsduring locomotion over a solid substrate[1]

[1] Wikipedia. Gait — Wikipedia, the free encyclopedia, 2013.[2] http://youtu.be/uG4NNaRttUE

FastSlow

Horse gait [2]

3 / 50

-Biological motion- BMLwalker[3]

[3] BML walker, http://www.biomotionlab.ca/Demos/BMLwalker.html 4 / 50

-Age-[Makihara et al. IJCB 2011]

Quiz: How old are they?Person 1 2 3

Gait

Age A. 4 years oldB. 14 years oldC. 24 years old

A. 4 years oldB. 14 years oldC. 24 years old





5 / 50

-Identity-

6 / 50

-Personality-

A

B

C

StoopStride Arm swingAsymmetry

Gait recognition: Person authentication from gait

7 / 50

Distance from sensor

DNA FaceFingerprint

IrisVein Gait

Near Far

Authentication at a distanceCriminal investigation

CCTV images

8 / 50

[4][5]

[4] I. Bouchrika et al., `` On Using Gait in Forensic Biometrics,’’ J. of Forensic Sciences, 56(4), 882-889, 2011.[5] http://news.bbc.co.uk/2/hi/programmes/click online/7702065.stm, ”How biometrics could change security,” BBC News, 31 Oct. 2008.9 / 50

8 expert evidences requested by police

First packaged gait verification system[Iwama et al. BTAS 2012, CVA 2013] [Muramatsu et al. ACPR 2013 Demo]

10 / 50

11 / 50

Basic approachBasic approach

Low frame-rate

View invariance

Speed invariance

12 / 50

Input

Background

Silhouette

Background subtraction-based graph-cut segmentation

[Makihara and Yagi 2008]

Normalized silhouette

Height normalizationRegistration by gravity center

13 / 50

Normalizedsilhouette ・・・

Time [frame]

Period 1 Period 2 Period 3Period 1 Period 2

Period detection

Fourier

FDF

0-timesBody shape

1-timeAsymmetry

2-timesSymmetry

Fourier

FDF FDF

Fourier

Period 3

14 / 50

-Dissimilarity measure-Gallery

(Database)

Probe(Query)

),,(1 kyxAG

),,(1 kyxAP

kyx

PG kyxAkyxAt,,

2111,1 ),,(),,(

),,(2 kyxAG ),,(3 kyxAG

・・・

),,(2 kyxAP ),,(3 kyxAP

kyx

PG kyxAkyxAt,,

2212,1 ),,(),,(

kyx

PG kyxAkyxAt,,

2313,1 ),,(),,(

kyx

PG kyxAkyxAt,,

2121,2 ),,(),,(

kyx

PG kyxAkyxAt,,

2222,2 ),,(),,(・・・jiji

tt ,,min

15 / 50

Performance evaluation: Identification[Iwama et al. IFS 2012]

94% rank-1 identification rate (N = 3,141)

FDF

Bad

Good

16 / 50

17 / 50

Basic approach

Low frame-rate

View invariance

Speed invariance

18 / 50

-View differences-Probe

(E.g., perpetrator)

Direct matchingdoes not work

Gallery(E.g., suspect)

Matching withthe same view

Difficult to collect multi-view gaitfeatures for uncooperative subject

19 / 50

View transformation model (VTM)[Makihara et al. ECCV 2006]

Training subjects(Cooperative)

Subjects

・・・

・・・

・・・

・・・

・・・

・・・

Test subject(Uncooperative)

Train Apply

VTM

・・・

Gallery

ProbeGallery

Match

Same view

20 / 50

Featurefrom view i

Feature fromview j

Training subj. 1

Trainingsubj. 1

Trainingsubj. 2

Training subj. 2

Testsubj.

Test subj.

Point injoint subspace

Project feature from view iinto joint subspace

Back-project the pointinto feature from view j

Joint subspace(VTM)

21 / 50

Formulation-Training VTM-

Individuals

M

M

M

KKK

aaa

aaaaaa

21

21

21

222

111

Joint subspace bases(Projection matrix to each view feature)

Individual pointsin joint subspace

Training matrix

・・・

・・・

・・・

・・・

・・・

・・・

M

KP

PP

vvv

212

1

i

j

Joint subspace(VTM)

22 / 50

Formulation-Transformation from i to j - Project feature

to point in joint subspace Least square

Back-project point in joint subspaceinto feature from view

iiiiPP aavv

v

minargˆ

iijjjPPP ava ˆˆ

Joint subspace(VTM)

i

j

Transformation matrix

23 / 50

Gallery

0 deg

15 deg

30 deg

45 deg

60 deg

75 deg

90 deg

Ground truth

24 / 50

Verification

Direct Perspective projection [Kale+ 2003]VTM VTM+

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0 45 90 135 180 225 270 315 360Probe [deg]Gallery

Bad

Good

25 / 50

26 / 50

Basic approach

Low frame-rate

View invariance

Speed invariance

27 / 50

-Speed difference-

2 km/h 3 km/h 4 km/h 5 km/h 6 km/h

7 km/h 8 km/h 9 km/h 10 km/h

28 / 50

Energy minimization Data term: Penalize diff. between model and silhouette Smoothness term: Penalize abrupt joint angle changes

l low

wknee

wtoe

wwaist

upt

lowt

Link leg model Fitting result

29 / 50

Speed transformation model (STM)[Tsuji et al. CVPR 2010]

Body shape

Motion

Decouple Couple

Training subjects

Subjects・・・

・・・

・・・

・・・

・・・

・・・

STM

30 / 50

Transformed

3 km/h Transformfrom 3 km/hto 7 km/h

Transformed

2 km/h

Transformfrom 6 km/hto 2 km/h

6 km/h

7 km/h

31 / 50

Verification

0

5

10

15

20

25

30

35

2 3 4 5 6 7

変換元の速度 (km/h)

EER (%)

STMDirect

Bad

Good

Gallery speed [km/h]Probe4

32 / 50

33 / 50

Basic approach

Low frame-rate

View invariance

Speed invariance

34 / 50

-Low frame-rate-

15 fps 5 fps1 fps

15-times error!

35 / 50

Periodic Temporal Super Resolution (PTSR)

Low frame-rate video High frame-rate video

PTSR

Improve accuracy36 / 50

Period

Phase (gait stance)

Silhouette sequenceInput(Low frame-rate)

Output(High frame-rate)

Eigen space

Reconstruction-based PTSR-Overview- [Makihara et al. ACCV 2010]

1

2

3

・・・

・・・

Silhouette:point in eigen space

Period of silhouette sequence:Closed curve in eigen space

Various gait stances among periods

37 / 50

Reconstruction-based PTSR-Stroboscopic effect-

Sampling interval = period

It looks as if it is static!

38 / 50

Output(High frame-rate)

Input(1fps)

Reconstruction-based PTSR-Failure mode-

1

2

3

・・・

・・・

PeriodSilhouette sequence Eigen space

Fail under stroboscopic effect

Similar gait stances among periods

39 / 50

Eigen space

Exemplars(Cooperative subj.: High frame-rate)

Example-based PTSR-Overview-

1

2

3

321

Input(Uncooperative subj.: Low frame-rate)

X

Output(Uncooperative subj.: High frame-rate)

X + + +・・・X 1 2 3 40 / 50

Eigen space

Exemplars(Cooperative subj.: High frame-rate)

Example-based PTSR-Failure mode-

1

2

3

321

Input(Uncooperative subj.: Low frame-rate)

Output(Uncooperative subj.: High frame-rate)

Estimate

+ + +・・・X 1 2 3

XLarge error

Groundtruth

Fail when input isfar from exemplars

41 / 50

Unified approach to PTSR[Akae et al. CVPR 2012]

Input（Low frame-rate） Reconstruction-based cue

Output（High frame-rate）

Eigen space

1

2

3

・・・

・・・

Period

Multi-period observation

Example-based cue

Training set (High frame-rate)

#001

#002

TrainEigen gait

42 / 50

Reconstruction-based

Example-based

Reconstruction+ Example

Reconstruction+ Example+ Prior (smoothness)

Error from inputDiscontinuity

Solve by energy minimization framework43 / 50

-2 fps-

Frame-rateof input Input [Al-Huseiny et al.

2010] [Akae et al. 2011] Proposed

2 fps

44 / 50

-1 fps-

Frame-rateof input Input [Al-Huseiny et al.

2010] [Akae et al. 2011] Proposed

1 fps

45 / 50

Verification

Good

Bad

Frame-rate [fps]

Proposed

[Akae et al. 2011]

[Al-Huseiny et al. 2010]

Direct

46 / 50

47 / 50

Towards robust gait recognition

Cooperativesubject

Uncooperativesubject

VTM

・・・

InputReconstruction

Output

1

2

3・・・

・・・

Example

Cooperative subject

…

Eigen gaitCooperativesubject

Cooperative subjectEigen gait#1

#2

Train

・・・

・・・

・・・

・・・

・・・

・・・

Make the most of cooperative training subjects’ variationsto cope with uncooperative test subject’s variations

Key idea

48 / 50

Mainstream of gait recognition

Silhouette-based(Movement + body shape)

Definition of gait

Pattern of movementof the limbs of animals …

Gap!

Human model fitting?

High computational costError-prone

49 / 50

S. Lombardi, K. Nishino (Drexel Univ.), Y. Makihara, Y. Yagi (Osaka Univ.),``Two-Point Gait: Decoupling Gait from Body Shape,’’ ICCV 2013.

50 / 50

p20131108 acpr2013 invited - osaka university€¦ · mr. akira tsuji dr. hidetoshi mannami ms....

Documents