1

玉木徹（広島大）

牛山幸彦（新潟大学教育人間科学部）

Bisser Raytchev（広島大）, 金田和文（広島大）

Inverse Composite Alignment of a sphere under orthogonal

projection for ball spin estimation

Compositional

2

研究の背景

●  バイオメカニクス ●  スキル判定 ●  チームプレイ情勢判断 ●  放送映像可視化技術

●  講習会・体育授業 n  ビデオ撮影・上映 n  指導者の直接指導

動画像処理技術

多数の生徒に対する教師や指導者による対面指導 少数の生徒に対する指導

生徒の自習

3

スポーツ指導とスキル獲得

1.  全身協応動作の発現n  動作を教え込まず、明確な課

題を与え発現を待つこと

n  適切な時点に適切な熟練者の指導言葉を与えること

2.  練習による洗練n  新しい情報を与えすぎないこ

と（過度の依存を避ける）

n  内在的フィードバックを重視し、付加的フィードバックは補助的に用いること

3.  自動化

スキル獲得における3つのプロセス（工藤, 2000）

分析過程が見えない 情報をフィードバック しない

適切な指導のための資料の提示

練習者の課題となる目標の提示

システムとして目指すもの

4

Table Tennis■  Factors of table tennis

n  ball speed and spin ■  Ball spin (rotation)

n  One of major factors that affects performance

l  Average 8000 [rpm] for Chinese national team (Wu, 1992)

n  Great effect on a rally l  Large angle change after the impact to a

racket ■  Conventional measurement

n  Counting by manual n  Official ball was changed in radius

l  In 2000, 38mm to 40mm l  Effect investigated (Tang 2002, Kawazoe

2004)

●  Method: motion estimation from a video ●  Goal: feedback spin to a player

玉木徹, 牛山幸彦, 八坂剛史 : 「スポーツ選手の技能向上のための動画像処理とその実用化」, 電子情報通信学会技術報告 パターン認識・メディア理解研究会 PRMU2005-116, No.116, pp.13-18, 広島市立大学, 広島 (2005 11). ,

Wu Huan Qun, Qin Zhifeng, Xu Shaofa, Xi Enting: Experimental Research in Table Tennis Spin,“ International Journal of Table Tennis Science, The International Table Tennis Federation, No. 1, pp.73-79 (1992 08). Hai-peng Tang, Masato Mizoguchi, Shintaro Toyoshima: Speed and spin characteristics of the 40mm table tennis ball," Table Tennis Sciences, No. 4&5, ITTF Sports Science Committee, pp.278{284 (2002) Y. Kawazoe, D. Suzuki, \Comparison of the 40 and 38 mm table tennis balls in terms of impact with a racket based on predicted impact phenomena," in Science and Racket Sports III, pp.140{145 (2004).

5

Related work

Alexander Szep, Quantifying Rotations of Spheric Objects, The Twelfth IAPR Conference on Machine Vision Applications (MVA2010), 2010.

Giacomo Boracchi and Vincenzo Caglioti and Alessandro Giusti. Estimation of 3d instantaneous motion of a ball from a single motion-blurred image. In VISIGRAPP, Computer Vision and Computer Graphics. Theory and Applications, Communications in Computer and Information Science, 2009, Volume 24, Part 3, Part 6, 225-237, DOI: 10.1007/978-3-642-10226-4_18

Christian Theobalt, Irene Albrecht, Jörg Haber Haber, Marcus Magnor, and Hans-Peter Seidel. 2004. Pitching a baseball: tracking high-speed motion with multi-exposure images. In ACM SIGGRAPH 2004 Papers (SIGGRAPH '04), Joe Marks (Ed.). ACM, New York, NY, USA, 540-547. DOI=10.1145/1186562.1015758

B&W corner tracking (2D rotation) color corner tracking, stereo

One shot estimation from motion blur

6

Related work

井上 卓也, 植松 裕子, 斎藤 英雄: “高速カメラ映像を用いた硬式野球ボールの回転速度推定システム”, 電学論Ｄ, Vol. 131, No. 4, pp.608-615 (2011) .

Shum, H.; Komura, T.; , "Tracking the translational and rotational movement of the ball using high-speed camera movies," Image Processing, 2005. ICIP 2005. IEEE International Conference on , vol.3, no., pp. III- 1084-7, 11-14 Sept. 2005

Parametric Eigenspace method with CG

Motion parameter estimation by minimizing intensity error

M. Yamamoto, "A General Aperture Problem for Direct Estimation of 3-D Motion Parameters," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 11, no. 5, pp. 528-536, May 1989, doi:10.1109/34.24785

Shape and motion from optical flow

Ken-ichi Kanatani, Structure and motion from optical flow under perspective projection, Computer Vision, Graphics, and Image Processing, Volume 38, Issue 2, May 1987, Pages 122-146, ISSN 0734-189X, DOI: 10.1016/S0734-189X(87)80133-0. Ken-ichi Kanatani, Structure and motion from optical flow under orthographic projection, Computer Vision, Graphics, and Image Processing, Volume 35, Issue 2, August 1986, Pages 181-199, ISSN 0734-189X, DOI: 10.1016/0734-189X(86)90026-5.

7

An experimental system

■  Motion parameter estimation of a textured rigid object n  Rotation: angles about each axis

n  Translation

X axis

Y axis

Z axis

ωx ωy

ωz

Zoom capture High speed camera 500[fps]

Image processing

8

Spin estimationExpert Beginner

6000

5000

4000

3000

2000

1000

0

-1000

-2000

-3000 31 32 33 34 35 36 37

Frame number

rpm

6000

5000

4000

3000

2000

1000

0

-1000

-2000

-3000 15 16 17 18 19 20 21

Frame number

rpm

xω

yω

zωω

Toru Tamaki, Takahiko Sugino, Masanobu Yamamoto: "Measuring Ball Spin by Image Registration," Proc. of FCV2004 ; the 10th Korea-Japan Joint Workshop on Frontiers of Computer Vision, pp.269-274 (2004 2), Kyushu University, Fukuoka, Japan, 2004/2/3-4.

9

Motion model of a ball in 2 frames

Optimized by Gauss-Newton method with 1D line search / coase-to-fine

10

Depth value from a sphere model

11

Improvements

■  Use orthographic projection n  Perspective projection is not necessary n  Ball is 40mm in diameter n  Distance between a ball and a camera is 3 to 5 m

■  Ignore Z translation n  Z value variation is at most 152.5cm (table width)

■  Use ball centers and radius n  Provided by manual or circle detection methods

■  Inverse Compositional Alignment (ICA) n  Precomputing Hessian and steepest decent images

(Baker and Matthews, 2004)

12

Improvements

■  Use orthographic projection n  Perspective projection is not necessary n  Ball is 40mm in diameter n  Distance between a ball and a camera is 3 to 5 m

■  Ignore Z translation n  Z value variation is at most 152.5cm (table width)

■  Use ball centers and radius n  Provided by manual or circle detection methods

■  Inverse Compositional Alignment (ICA) n  Precomputing Hessian and steepest decent images

(Baker and Matthews, 2004)

13

Orthographic projection

■  A ball is too small n  Distance: 3 to 5 m n  Appearance difference is

less than a pixel. n  Hence, ignore perspective!

3m

4cm

camera

ball

0.76[deg]

179.23[deg]

3.999911110cm

Real scale

camera

ball

3m

14

Improvements

■  Use orthographic projection n  Perspective projection is not necessary n  Ball is 40mm in diameter n  Distance between a ball and a camera is 3 to 5 m

■  Ignore Z translation n  Z value variation is at most 152.5cm (table width)

■  Use ball centers and radius n  Provided by manual or circle detection methods

■  Inverse Compositional Alignment (ICA) n  Precomputing Hessian and steepest decent images

(Baker and Matthews, 2004)

15

Z variation

■  Table width is too small n  Z variation is at most ± 1m n  Hence, give up for Z!

camera

ball

3m (about)

1.575m

2.74m

Official table size 3m

4cm

camera

ball

0.76[deg]

179.23[deg]

3.999911110cm

16

Z variation

■  Table width is too small n  Z variation is at most ± 1m n  Hence, give up for Z!

camera

ball

3m (about)

1.575m

2.74m

2m

4cm

camera

ball

1.15[deg]

178.85[deg]

3.999799995cm ±

1m (about) Official table size

17

Z variation

■  Table width is too small n  Z variation is at most ± 1m n  Hence, give up for Z!

camera

ball

3m (about)

1.575m

2.74m

Official table size 3m

4cm

camera

ball

0.76[deg]

179.23[deg]

3.999911110cm ±

1m (about)

18

Z variation

■  Table width is too small n  Z variation is at most ± 1m n  Hence, give up for Z!

camera

ball

3m (about)

1.575m

2.74m

4m

4cm

camera

ball

0.57[deg]

179.43[deg]

3.999950000cm

Official table size

± 1m

(about)

19

Computing area

25.84[deg] =2584rpm

3.6cm (x0.9)

4cm

Smaller area (x0.9 radius) is used for computing intensity differences

Maximum rpm (round per minute): 10,000 rpm = 0.2777… round per 1/600sec = 100 degree / frame (1/600 sec)

20

Computing area

45.57[deg] =4557rpm

2.8cm (x0.7)

4cm

Smaller area (x0.7 radius) is used for computing intensity differences

Maximum rpm (round per minute): 10,000 rpm = 0.2777… round per 1/600sec = 100 degree / frame (1/600 sec)

21

Improvements

■  Use orthographic projection n  Perspective projection is not necessary n  Ball is 40mm in diameter n  Distance between a ball and a camera is 3 to 5 m

■  Ignore Z translation n  Z value variation is at most 152.5cm (table width)

■  Use ball centers and radius n  Provided by manual or circle detection methods

■  Inverse Compositional Alignment (ICA) n  Precomputing Hessian and steepest decent images

(Baker and Matthews, 2004)

22

Ball centers and radius

■  Provided by other methods n  Circle detection by Hough

transform n  Manual operations by

mouse clicking n  Tracking frame by frame

23

Improvements

■  Use orthographic projection n  Perspective projection is not necessary n  Ball is 40mm in diameter n  Distance between a ball and a camera is 3 to 5 m

■  Ignore Z translation n  Z value variation is at most 152.5cm (table width)

■  Use ball centers and radius n  Provided by manual or circle detection methods

■  Inverse Compositional Alignment (ICA) n  Precomputing Hessian and steepest decent images

(Baker and Matthews, 2004)

24

ICA

■  Previous approach (Tamaki et. al 2004)

n  Forward additive approach (Baker and Matthews, 2004)

n  Minimizing a cost function f n  Find parameter updates ∆ p n  Update p+∆ p→p

Previous approach

p+∆ p

25

ICA

■  Previous approach (Tamaki et. al 2004)

n  Forward additive approach (Baker and Matthews, 2004)

n  Minimizing a cost function f n  Find parameter updates ∆ p n  Update p+∆ p→p

■  ICA approach n  Inverse Compositional

approach (Baker and Matthews, 2004)

n  Minimizing a cost function f n  Find parameter updates ∆ p n  Update

Previous approach

ICA approach

p+∆ p

26

Advantages of ICA

■  Pre-comutation n  Hessian, Steepest decent images

■  ICA approach n  Inverse Compositional

approach (Baker and Matthews, 2004)

n  Minimizing a cost function f n  Find parameter updates ∆ p n  Update

Previous approach

ICA approach

p+∆ p

27

Warp update ruleX1=¢R(X—C)+¢T+C

From 2D to 3D :

Sphere model

Ball center

28

Warp update ruleX1=¢R(X—C)+¢T+C X2=R(X—C)+T+C

From 2D to 3D :

Sphere model

Ball center

The updated motion parameters

29

Experiments with real sequences

432x192, 600fps

(radius: 16 to 17 pixel)

30

Experiments

31

Simulation

■  Synthetic image sequence n  with a 3D CG software n  Orthographic projection

■  Background: gray ■  Texture: text ■  Radius: 35 pixels ■  Rotation

n  5 degree about each of X, Y, and Z axes

■  Translation: no

-­‐6  

-­‐5  

-­‐4  

-­‐3  

-­‐2  

-­‐1  

0  

1  

2  

3  

4  

5  

6  

Est

imat

ed a

ngle

[deg

ree]

Y rotation X rotation Z rotation

32

Visual inspectionI1 I2

I1(x)— I2(w(x;p))

Difference between images (the brighter the closer)

Cost function

Radius: 61 pixels

Warped image

33

Conclusion

■  Proposed an ICA-based ball spin estimation method n  Orthogonal projection n  Centers and radius given n  ICA approach

■  Future work n  Easy-to-use system design n  Accuracy/error analysis n  Fast implementation n  Ball detection/tracking