magic book: enhancing natural feature tracking with the user´s movement context

Magic Book: Enhancing Natural Feature Tracking with the User´s Movement Context

Felix Loew

Master Thesis (TU München)

21.01.2005

• Context-Awareness and Augmented Reality

• Vision Based Tracking: Natural Feature Tracking

• A „Magic Book“ User Study

• Software Architecture (DWARF )

• Expected results and conclusions

Outline

Context-Awareness and Augmented Reality

• A system has to adopt to the user´s behaviour!

• A system has to learn how it is used!

• New intuitive methods of interaction are required!

• The system has to disappear (Marc Weiser)!

The „Magic Book“ in this context

• User has to „learn“ how it is used

• Very error-prone to movement– Movement of the handheld device is not

considered in the tracking routine– Vision based tracking loses orientation– User has to reinitialize the tracking (looking on

the marker again)– Immersion decreases!

My Idea: Enhancing the Magic Book

Inertial Tracker: Intersense, Inertial Cube3 (~120FPS)

Vision Based Tracking: NFT (~20 FPS)

Movement

• Context-Awareness and Augmented Reality

• Vision Based Tracking: Natural Feature Tracking

• A „Magic Book“ User Study

• Software Architecture (DWARF )

• Summary, Conclusions & Limitations

Outline

Basics: Natural Feature Tracking

Frame 1 Frame 2

Frame 3 Frame 4

• NFTARToolkit (Kato): 2D planar scenes• Estimation of feature position in next frame

– It is not realistic that no movement occurs– Linear estimation– Kalman Filter

• Within a window the “best match” with the template is searched (Template Matching)

• For every pixel within window the similarity with the template is calculated (Normalized Cross Correlation)

Basics: Natural Feature Tracking

Template Matching

• Calculates similarity between template and area around search point– values [-1..1]

• Best match is considered as feature point in current frame (threshold ~0.7)!

• Computational expensive!– O (templateSize2 * searchSize2)

Normalized Cross Correlation

My approach

• Consider movement as well!

• Use additional tracking device (Inertial Tracker / 3DOF)

• Naive approach:much movement large search window

low movement small search window

• Dynamic configuration during runtime

What is much movement? Which movements actually occur?How does a user use such an application?

USER STUDY ON MAGIC BOOK

My approach

• Context-Awareness and Augmented Reality

• Vision Based Tracking: Natural Feature Tracking

• A „Magic Book“ User Study

• Software Architecture (DWARF )

• Expected results and conclusions

Outline

• Goals– Detecting specific movement patterns– Derive Movement model– Estimation of a correlation between changes in

orientation and feature points coordinates

User Study Setup

Orientation (Handheld, 3D)

Position (FP-Coords, 2D)

User Study Setup

• Tracking Orientation and Position (6DOF)– Magnetic Tracker:

Flock of Birds

– Bird1: Magic Book

– Bird2: Handheld device

– Calibration to Magic Book coordinate system (Bird1)

• Magnetic Tracker– Timestamp

– Position P = (px,py,pz)

– Orientation (quaternion) Q = (v,s)

• NFT– Timestamp

– 4 feature points per frame• Id

• Screen-Position S = (sx,sy)

Logging

• Magnetic Tracker

pics\2_20_1.Case2.Flock.xls

• NFT

pics\2_20_1.Case2.Coords.xls

Logging

User Tasks

• Comparable Tasks (Navigation Tasks)– User has to answer certain questions (Overview,

Focus and Search Tasks)• „What is the girls hair color?“

• „How many people do you see?“

• Free Tasks– User can use Magic Book freely

Evaluation

• Ideas for evaluation (Correlation)– Which angular offset (FOB) causes which

coordinate offset (NFT)?– Which angular velocities occur?– Which rotations occur?– Which changes of position occur?– Is there a common movement profile or are

there serious differences between different users?

Evaluation

• Find mapping between:

• Configure Software Architecture with this information

orientation search window

• Context-Awareness and Augmented Reality

• Vision Based Tracking: Natural Feature Tracking

• A „Magic Book“ User Study

• Software Architecture (DWARF)

• Expected results and conclusions

Outline

Software Architecture (DWARF)

NFT ARToolkit

Dynamic Configuration

• Context-Awareness and Augmented Reality

• Vision Based Tracking: Natural Feature Tracking

• A „Magic Book“ User Study

• Software Architecture (DWARF)

• Expected results and conclusions

Outline

Conclusions & Summary

• Hybrid Tracking approach to consider context information as well– Movement context– 2DO: What about user context?

• User Study on Magic Book (work in progress)– Expected conclusions on usage of Magic Book– Find correlation between movement and NFT properties– Process a configuration out of the study– Derive Movement model of the Magic Book– Limitation: Very task specific (features are not moving, only 2D)!

• Software Architecture

• Heaps of work– I won’t cover everything I want to do

• Other “sensor-fusion” approaches might be more suited (Kalman-Filtering)

• Evaluation of User Study will be hard

• Is there an abstraction for NFT based applications?

Limitations & 2DO´s

Thanks

• I would be glad if some of you want to join the study!