making robust computer vision in games presented by: diarmid campbell
TRANSCRIPT
![Page 1: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/1.jpg)
Making robust computer vision in gamesPresented by: Diarmid Campbell
![Page 2: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/2.jpg)
Introduction
Who I am: Diarmid Campbell
What I do: Run the Vision R&D group
Where we do it : Sony’s London development studio
What we do: Research computer vision for camera based games
This talk: Making robust computer vision in games
![Page 3: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/3.jpg)
Contents
What we do and why The development process Testing and videos Computer Vision Concepts A robust head tracker Marker based Augmented Reality The problems we faced A demo of EyePet
![Page 4: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/4.jpg)
Camera based games
Camera mounted on the TV You see yourself on the TV Game is overlaid on top of you
![Page 5: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/5.jpg)
Past games on PS2
![Page 6: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/6.jpg)
Computer Vision is hard
“Computer vision makes you want to kill yourself”
-Dr Nick Lord 2009
![Page 7: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/7.jpg)
Why is it hard?
Humans mange it effortlessly Image is a 2D array of numbers Take 5 images and plot them as a
height map
![Page 8: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/8.jpg)
Pick the odd one out
![Page 9: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/9.jpg)
Pick the odd one out
Odd one out
![Page 10: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/10.jpg)
Pick the odd one out
Odd one out
![Page 11: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/11.jpg)
Pick the odd one out
Odd one out
![Page 12: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/12.jpg)
Factors affecting the pixels
Background objects in scene Orientation/position of objects Lighting/Shadows Occlusion
![Page 13: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/13.jpg)
George is in the pixels
Not interested in those George was hidden in the pixels “Here is an image, what is it of?” The general computer vision
problem is hard If we constrain the problem, it is
much easier (but still hard)
![Page 14: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/14.jpg)
Robust Inputs
We can use computer vision as an input mechanism Motion detection in EyeToy games
Robustness is how consistently an input mechanism does what the player is expecting
An input mechanism must be robust
![Page 15: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/15.jpg)
Importance of robustness
If your fire button only worked 9 times out of 10, you would chuck your controller out.
![Page 16: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/16.jpg)
Importance of robustness
There are ways around it
![Page 17: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/17.jpg)
Importance of robustness
Imagine your gun is a champagne bottle
![Page 18: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/18.jpg)
Importance of robustness
Each button click shakes it Eventually the top blows off The lack of robustness is hidden
![Page 19: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/19.jpg)
Importance of robustness
Perhaps you need to now fight tortoises instead of warriors
![Page 20: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/20.jpg)
Importance of robustness
The mechanic is now “robust” But it is laggy and unresponsive Cannot rely on split-second timing
![Page 21: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/21.jpg)
Importance of robustness
Illustrates a general point If the game copes well with non-
robust inputs It will also cope well with someone
not playing it well It creates a skill ceiling Manifests itself as lack of game-
play depth
![Page 22: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/22.jpg)
Importance of robustness
If you want a deep skill base game mechanics
Robust input is essential
![Page 23: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/23.jpg)
The Development Process
Computer Vision Researcher Game designer
“Tell me what the game mechanic is and I’ll make you a state of the art solution “
“Give me something that works and I’ll see what we can make that’s fun”
![Page 24: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/24.jpg)
The chicken and the egg
You cannot do one before the other Both development timelines happen in
parallel We are still figuring it out Here are some guidelines
![Page 25: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/25.jpg)
Months Game development Computer Vision research1st pass prototype
1 Concepting2 Survey state of the art3 Prototyping4 Extend methods/analyse56789 Production Production
101112131415 Alpha1617 Beta1819 Alpha Master2021 Beta2223 Master24
Research timeline
Convinced we can create the technology
Something up and running
Time
Vision tech beta before game reaches alpha
![Page 26: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/26.jpg)
Required infrastructure
Prototyping environment Matlab Octave
Be able to capture videos Runtime algorithms
Open CV VXL
![Page 27: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/27.jpg)
Videos and testing
![Page 28: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/28.jpg)
Videos and testing
Computer vision is hard because many variables affect the images The lighting The player’s clothes The wallpaper Spectators
3D cameras have their own pros and cons
![Page 29: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/29.jpg)
Representative videos
Videos allow us to capture these variables and test
Videos MUST be representative Works in 99% of cases Useless if that 1% appear in 50% of
living rooms Make videos early in development Demo: head tracker capturing
![Page 30: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/30.jpg)
Head detection videos
We run it through different algorithms Cell SDK face detector Show failure modes
When it fails we can find the frame it failed in and debug
![Page 31: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/31.jpg)
Regression testing
Automated testing Run through load of videos Compare with expected results Expected results could be is head
visible?
![Page 32: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/32.jpg)
When videos aren’t enough
SCEA R&D labs invented the forthcoming PlayStation®Move controller
Uses a camera and other sensors to track the controller
Videos were good early on But cannot change a video:
Lighting Backgrounds Camera settings
![Page 33: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/33.jpg)
Solution
Video1 Video2
![Page 34: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/34.jpg)
Reasons to buy a robot arm(as if you really need persuading)
Can test the same motion under many different conditions
Can try special hard cases
![Page 35: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/35.jpg)
Computer Vision Concepts
![Page 36: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/36.jpg)
Computer Vision Concepts
Videos tell us when it fails How do we fix it? This is the field of computer vision I cannot go into details of techniques Instead I will explain:
The common concepts How they link together
This should help if you: Read papers Talk to experts
![Page 37: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/37.jpg)
Feature extraction
Images contain a lot of information This one is 900K
![Page 38: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/38.jpg)
Feature extraction
Instead of using pixels directly extract high level properties of groups of pixels
Result in less data which is more relevant to the problem at hand
Image Feature Extraction Features
![Page 39: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/39.jpg)
Feature extraction
PS3 Demo: Basic image PS3 Demo: Canny edge detector
Invariant to lighting changes Store additional gradient info
PS3 Demo: Motion Used in all our camera games
PS3 Demo: Feature points Store image patch for each one Can match them frame to frame
![Page 40: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/40.jpg)
Likelihood functions
“Given that we have observed these features, what is the probability that we are observing what we modelled“
Conditional probability
Bayesian statistics underpins most vision algorithm
Model
Features
Likelihood Function P(F | M)
![Page 41: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/41.jpg)
Cost functions = Likelihood functions Some terminology Sometimes you will here about
“Cost functions” They are the same concept:
Likelihood goes up with a good match Cost goes down
One is (conceptually) the inverse of the other
![Page 42: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/42.jpg)
Cost functions
Sum of Squared Differences (SSD)
SSD
SSD
1532
12
High cost = bad match
Low cost = good match
![Page 43: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/43.jpg)
Cost functions
Sum of Squared Differences (SSD)
![Page 44: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/44.jpg)
Model(1) . . .
Classifier Most likely model
Model(2) Model(n)
Features
Classifiers
Compares observed features to a number of models Tells you which model fits the features best
Which model fits best
![Page 45: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/45.jpg)
Classifiers: Face example
Is this a face?
![Page 46: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/46.jpg)
Classifiers: Face example
Classic detector (Voila-Jones)
Image
Feature Extraction
Haar Wavelet Features
Boosted cascade Classifier
Face Model
Non-Face model
Is it a face?
Models are trained on example images
Classifier
![Page 47: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/47.jpg)
Classifiers: Face example
PS3 demo
![Page 48: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/48.jpg)
Detectors
We have a model (with associated state) Given some observed features Detector returns:
Is the object present? What it’s state? It’s state (X,Y position/rotation/Human pose)
Model
Features Detector
State
Is object present?
![Page 49: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/49.jpg)
Detectors: Faces again
Viola-Jones face detector: Scans a box over the image Different positions and sizes Runs the classifier and returns any
positives Recall face detection demo
![Page 50: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/50.jpg)
Trackers
We have a model, some observed features and the previous state
Tracker returns the next state
Model
Features TrackerNext State
Previous State
![Page 51: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/51.jpg)
Trackers: Face example
PS3 Demo: SSD tracker PS3 Demo: Wand game If we move quickly the tracker gets stuck in a
local minimum
![Page 52: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/52.jpg)
Learning more
Computer Vision Conferences ICCV CVPR ECCV
Read papers accepted by conferences
Get friendly with an academic Or hire one!
![Page 53: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/53.jpg)
Robust Head Tracking
![Page 54: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/54.jpg)
Track rotation and scale
The SSD based tracker did not track rotation and scale
Next iteration of tracker does: X, Y position Scale θ : in plane rotation
PS demo: Hager tracker (swap demo)
![Page 55: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/55.jpg)
Track rotation and scale
Tracked more types of movement But very fragile Problem:
A 2D image patch is not a good model of a head
![Page 56: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/56.jpg)
Track rotation and scale
Does not deal with out-of-plane rotation
![Page 57: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/57.jpg)
Track rotation and scale
Even in-plane rotation is not right
![Page 58: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/58.jpg)
Colour histograms
Lets move away from comparing pixels and think about features
Consider these images of the same objects
![Page 59: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/59.jpg)
Colour histograms
If we compared them pixel for pixel they would seem very different
But look at a histogram of the colours that appear in them and they look the same
![Page 60: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/60.jpg)
Colour histograms
Histograms are a feature that throw away all spatial information
![Page 61: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/61.jpg)
Where we are now
Current system uses: Colour histograms Keeps approximate spatial information
![Page 62: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/62.jpg)
Where we are now
It has a foreground and a background model – each with its own histograms
![Page 63: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/63.jpg)
Where we are now
PS3 Demo
![Page 64: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/64.jpg)
Marker based Augment Reality (AR)
![Page 65: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/65.jpg)
Marker based AR
Marker based AR is in a published game: EyePet
![Page 66: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/66.jpg)
Camera setup
Topics to discuss
Camera based gamesWhat is EyePet?Improving the techFuture research
![Page 67: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/67.jpg)
What the player sees on the TV
Virtual
RealTopics to discuss
Camera based gamesWhat is EyePet?Improving the techFuture research
![Page 68: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/68.jpg)
Marker based AR
We shipped a “magic card” with the game
Allows the players to manipulate virtual objects in 3D
![Page 69: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/69.jpg)
Finding the marker Input image
Topics to discuss
Camera based gamesWhat is EyePet?Improving the techFuture research
![Page 70: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/70.jpg)
Finding the marker Threshold
Topics to discuss
Camera based gamesWhat is EyePet?Improving the techFuture research
![Page 71: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/71.jpg)
Finding the marker Trace outlines
Topics to discuss
Camera based gamesWhat is EyePet?Improving the techFuture research
![Page 72: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/72.jpg)
Finding the marker Test for quad shapes
Topics to discuss
Camera based gamesWhat is EyePet?Improving the techFuture research
![Page 73: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/73.jpg)
Finding the marker Actually, just keep pairs of quads
Topics to discuss
Camera based gamesWhat is EyePet?Improving the techFuture research
![Page 74: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/74.jpg)
Finding the marker
Take corner positions Calculate a 2D transform
Topics to discuss
Camera based gamesWhat is EyePet?Improving the techFuture research
![Page 75: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/75.jpg)
Finding the marker Match the pattern
Topics to discuss
Camera based gamesWhat is EyePet?Improving the techFuture research
![Page 76: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/76.jpg)
Finding the marker Match the pattern
Topics to discuss
Camera based gamesWhat is EyePet?Improving the techFuture research
![Page 77: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/77.jpg)
Finding the marker Match the pattern
Topics to discuss
Camera based gamesWhat is EyePet?Improving the techFuture research
![Page 78: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/78.jpg)
Finding the marker Match the pattern
Topics to discuss
Camera based gamesWhat is EyePet?Improving the techFuture research
![Page 79: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/79.jpg)
Finding the marker Match the pattern
Topics to discuss
Camera based gamesWhat is EyePet?Improving the techFuture research
![Page 80: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/80.jpg)
Finding the marker Match the pattern (Yes!)
Topics to discuss
Camera based gamesWhat is EyePet?Improving the techFuture research
![Page 81: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/81.jpg)
Finding the marker
Decompose the 2D transform Camera projection Model view matrix
Use a Kalman filter
Topics to discuss
Camera based gamesWhat is EyePet?Improving the techFuture research
![Page 82: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/82.jpg)
Finding the marker And we’re done…..
Topics to discuss
Camera based gamesWhat is EyePet?Improving the techFuture research
![Page 83: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/83.jpg)
Problems we faced
![Page 84: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/84.jpg)
Picking the right threshold
Threshold to find black and white regions
But which one? Many clever solutions – didn’t work Brute force approach Try lots (around 60) thresholds
![Page 85: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/85.jpg)
Picking the right threshold
PS3 Demo: Thresholds PS3 Demo: AR Thresholds
![Page 86: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/86.jpg)
Light sensitive matching
Pattern matching used Sum of Square Differences (SSD)
SSD = 2242 SSD = 14 SSD = 874
Brightness of image affected the score
![Page 87: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/87.jpg)
Light sensitive matching
Use Normalised Cross Correlation (NCC) instead
SSD = 0.8 SSD = 0.9 SSD = 0.9
![Page 88: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/88.jpg)
Light sensitive matching
New way to look at images An image is an array of numbers We can list out every number and it becomes a
vector
12 0 34 23 123 63 23143 7 34 23 23 34 51
156 34 34 51 4 234 2313 34 234 23 63 1 23
0 2 1 14 2 4 2456 52 4 254 24 132 13232 34 132 23 23 4 35
2 4 4 54 3 34 4523 2 34 231 35 23 0
1 32 23 143 34 254 23243 0 1 32 234 23 234
232 34 45 65 4 54 14232 4 45 54 132 254231 2 1 143 234 23 45
100
100
1243
15613
05632
223
143
23207
3434
25234
42
320
34…
34
10,000
![Page 89: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/89.jpg)
Light sensitive matching
This is a co-ordinate vector in “image space” Every 100X100 image corresponds to a single
unique point in image space
![Page 90: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/90.jpg)
Light sensitive matching
This is a co-ordinate vector in “image space” Every 100X100 image corresponds to a single
unique point in image space Brightening an image corresponds to scaling the
position vector
![Page 91: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/91.jpg)
Light sensitive matching
When comparing two images SSD corresponds to the distance between them
in image space NCC corresponds to their angle
SSD
θ
θ
![Page 92: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/92.jpg)
Light sensitive matching
Linear algebra is the other pillar of computer vision
Feature extracting is just a transformation from one space to another
Image space -> Feature space Classifiers are often just planes
which divide up the space (e.g. into a region that contains faces and a region that doesn’t)
![Page 93: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/93.jpg)
Occlusion
It is easy to occlude the marker with your fingers
![Page 94: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/94.jpg)
Occlusion
Put big red handle on and instruct the player to hold it
Also put handle on the back
![Page 95: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/95.jpg)
Occlusion: Another approach(still in research phase)
Edge based tracking Uses AR Marker to initialise Then tracks using edge features PS3Demo (load EyePet)
![Page 96: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/96.jpg)
False positives
When not occluded, we find the marker (almost) all the time
Our home videos showed this False positives were a problem Not represented in our videos Added some Hollywood films to the
video tests We knew that no markers were
present
![Page 97: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/97.jpg)
False positives
Saved out all spurious frames
![Page 98: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/98.jpg)
False positives
Made a number of tweaks to algorithm E.g. Pattern matching whole marker,
not just the centre pattern 20 times less false detections
![Page 99: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/99.jpg)
EyePet Demo
![Page 100: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/100.jpg)
EyePet Demo
Use motion detection for normal interaction Call Jump Stroke
Use AR card for health monitor Screen-facing case
Needs stimulation Trampoline
Finally Give him a shower
![Page 101: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/101.jpg)
Summary
What we do and why The development process Testing and videos Computer Vision Concepts A robust head tracker Marker based Augmented Reality The problems we faced A demo of EyePet
![Page 102: Making robust computer vision in games Presented by: Diarmid Campbell](https://reader036.vdocuments.mx/reader036/viewer/2022062619/55199d0655034643068b4a48/html5/thumbnails/102.jpg)
The End(please fill out your questionnaires)