Matthias Wimmer, Bernd Radig, Michael Beetz
Chair for Image UnderstandingComputer Science
TU München, Germany
A Person and Context Specific Approach for
Skin Colour Classification
2005-12-18 2/9Technische Universität MünchenMatthias Wimmer
high-level vision module
low-level vision module
Output Parameters
Input Parameters
Calculation Rules
Motivation
Face LocatorSkin Colour Classifier
motivation our approach results outlook
see: A.E. Broadhurst, S. Baker: Setting Low-Level Vision Parameters, CMU-RI-TR-04-20, Robotics Institute, Carnegie Mellon University, 2004.
Our scenario: Adaptive Skin Colour Classification Classifier adapts to person and context
low-level vision module
Input Parameters
simple way:
promote parameters: high-level to low-level vision module mathematically transform parameters
2005-12-18 3/9Technische Universität MünchenMatthias Wimmer
Motivationmotivation our approach results outlook
2005-12-18 4/9Technische Universität MünchenMatthias Wimmer
Observations Skin colour depends on image conditions:
illumination: light source, light colour, shadow, shading,… camera: type, settings,… visible person: ethnic group, tan,…
Skin colour occupies a large area within colour space Skin colour varies greatly between images. Skin colour varies slightly within an image.
motivation our approach results outlook
gree
n
red
skin colour pixels (red) and other pixels (blue), static skin colour clusters (white), adaptive skin colour clusters (yellow)
gree
n
red
image 1 image 2
2005-12-18 5/9Technische Universität MünchenMatthias Wimmer
Our ApproachOffline step: learn a mask that extracts skin colour pixels
specific for the face detector
motivation our approach results outlook
Online steps: Step 1: detect the image specific skin colour
using the face detector using the skin colour mask
Step 2: calculate the input parameters Step 3: adapt the skin colour classifier
Face Locator(Step 1)
Skin Colour Classifier(Step 3)
Output Parameters
Input ParametersCalculation Rules
(Step 2)
2005-12-18 6/9Technische Universität MünchenMatthias Wimmer
Learn the Calculation Rules
Gather many training images Manually annotate images with ground truth Learn calculation rules via machine learning techniques
e.g. linear regression, neural networks, model trees, …
motivation our approach results outlook
Face LocatorSkin Colour Classifier
Output Parameters
Input Parameters
Calculation Rules
specify these (ground truth)
specify these (ground truth)
learn those
2005-12-18 7/9Technische Universität MünchenMatthias Wimmer
Results good robustness for
coloured persons exact shape outline detection of facial parts:
eyes, lips, brows,…
correctly detected pixels: fixed parameters: 90.4% 74.8%
40.2% adaptive parameters: 97.5% 87.5%
97.0% improvement: 0.08 0.17 1.41
motivation our approach results outlooka
da
ptive
pa
ram
ete
rs
fixed
pa
ram
ete
rs
orig
ina
l
ima
ge
2005-12-18 8/9Technische Universität MünchenMatthias Wimmer
Outlook We will create further
adaptive colour classifiers lip teeth eyes brows, hair …
Preliminary results for lip colour classifier:
motivation our approach results outlook
original fixed adaptive
2005-12-18 9/9Technische Universität MünchenMatthias Wimmer
Thank you!
2005-12-18 10/9Technische Universität MünchenMatthias Wimmer
Motivation
Skin colour detection supports… face model fitting
mimic recognition person identification gaze estimation fatigue detection (e.g. vehicle)
hand tracking gesture recognition action recognition supervising work
challenge our approach results outlook
2005-12-18 11/9Technische Universität MünchenMatthias Wimmer
Challenge Skin colour depends on image conditions:
illumination: light source, light colour, shadow, shading,… camera: type, settings,… visible person: ethnic group, tan,…
Skin colour occupies a large area within colour space
challenge our approach results outlook
2005-12-18 12/9Technische Universität MünchenMatthias Wimmer
Challenge (2): non-skin colour pixels Skin colour pixels have to be separated from non-
skin colour pixels. Areas of skin colour and
non-skin colour overlap. Colour can not make a
distinctive separation.
challenge our approach results outlook
2005-12-18 13/9Technische Universität MünchenMatthias Wimmer
Our approach
Offline step: learn the skin colour mask
specific for the face detector
Online steps: Step 1: detect the image specific skin colour model
using the face detector using the skin colour mask
Step 2: adapt a skin colour classifier Step 3: calculate the skin colour image
challenge our approach results outlook
2005-12-18 14/9Technische Universität MünchenMatthias Wimmer
Offline: Learn the skin colour mask face image database with labeled skin colour pixels skin colour mask: array with 24 x 24 cells
Computational steps:
1. detect the face in every image
2. every cell is assigned the relative number of labeled skin colour pixels at its position
3. apply threshold
1. 2. 3.
challenge our approach results outlook
2005-12-18 15/9Technische Universität MünchenMatthias Wimmer
Step 1: Detect the image specific skin colour model
detect the face extract the skin colour pixels normalized RGB colour space:
base = R + G + B
r = R / base
g = G / base
skin colour model: mean values: μr, μg, μbase
standard deviations: σr, σg, σbase
challenge our approach results outlook
2005-12-18 16/9Technische Universität MünchenMatthias Wimmer
Step 2: Adapt a skin colour classifier non-adaptive skin colour classifier:
skin := 0.35 ≤ r ≤ 0.5 0.2 ≤ g ≤ 0.7 200 ≤ base ≤ 740
adaptive skin colour classifier:skin := lowr ≤ r ≤ highr
lowg ≤ g ≤ highg lowbase ≤ base ≤ highbase
learn the bounds via the skin colour model mean value and standard deviationlowr := μr – 2σr
highr := μr + 2σr . . . . . . . . .
linear function:lowr := aμr + bμg + cμbase + dσr + eσg + fσbase + g
. . .
challenge our approach results outlook
2005-12-18 17/9Technische Universität MünchenMatthias Wimmer
Related work Feedback of information from
high level vision components to low level vision components
challenge our approach results outlook
2005-12-18 18/9Technische Universität MünchenMatthias Wimmer
Conclusion Challenge: much variation within skin colour
illumination, camera, visible person skin colour occupies a large area within colour space
We propose a way to reduce those variations exploit an image specific skin colour model adapt a skin colour classifier to that skin colour model
We proved our approach using a simple but real-time capable skin colour classifier comparison: non-adaptive ↔ adaptive
challenge our approach results outlook
2005-12-18 19/9Technische Universität MünchenMatthias Wimmer
Ongoing research Learn skin colour mask for other face detectors Specialize more powerful skin colour classifiers Recognize other feature images/colour images
lip colour image tooth colour image eye colour image hair colour image eye brow colour image
example: lip colour detection
challenge our approach results outlook
2005-12-18 20/9Technische Universität MünchenMatthias Wimmer
Adaptive skin colour classifier non adaptive skin colour classifier:
skin := 0.35 ≤ r ≤ 0.5 0.2 ≤ g ≤ 0.7 200 ≤ base ≤ 740
adaptive skin colour classifier:skin := lowr ≤ r ≤ highr lowg ≤ g ≤ highg
lowbase ≤ base ≤ highbase
learn the bounds out of the skin colour model