automatic photo selection for media and entertainment applications

Automatic Photo Selection for Media and Entertainment

ApplicationsEkaterina Potapova,

Marta Egorova, Ilia Safonov

National Nuclear Research University MEPhI Moscow, Russia

GraphiCon 20095-9 October

Applications

Automatic Photo Selection for Media and Entertainment Applications

GraphiCon 2009 2

Applications – photo book

Images are taken from printbook.ru, ehow.com, snapfish.com.au, smilebooks.co.ukGraphiCon 2009 3


Applications – slide show

Photos from ITaS’2008GraphiCon 2009 4


General workflow

GraphiCon 2009 5


Detection of low-quality photos

GraphiCon 2009 5


General workflow

General workflow


Adaptive quantization on

time-camera plane

GraphiCon 2009 5


Selection of appealing photos


Adaptive quantization on

time-camera plane

General workflow

GraphiCon 2009 5



GraphiCon 2009 6


Estimation of JPEG qualityA.Foi et al.,2007

3

1

3

1,9

1

i jjiqK

Images are taken from en.wikipedia.org

Quantization Table

GraphiCon 2009 7


Detection of backlit, low-contrast & blurred photos

+

Good photo

Bad photo

True

False

…

…

11 TF

ii TF

NN TF

}{ iF

1w

iw

Nw

N

ii Tw

1

Two Ada Boost classifiers committee: -for detection of low-contrast and backlit photos-for detection of blurred photos

GraphiCon 2009 8


Detection of backlit and low-contrast photos - 1 S1/S2 - ratio of tones in shadows to midtones

)/()(]85,0[

1 NMiHS )/()(]170,85(

2 NMiHS

GraphiCon 2009 9


S11/S12 - ratio of tones in first to second part of shadows

)(/)(]42,0[

11 NMiHS )(/)(]85,42(

12 NMiHS

Detection of backlit and low-contrast photos - 1

GraphiCon 2009 9


M1/M2 - ratio of the histogram maximum in shadows to the maximum in midtones

]255,0[]85,0[1 ))(max(/))(max( iHiHM

]255,0[]170,85(2 ))(max(/))(max( iHiHM


GraphiCon 2009 9


P1 - location of the histogram maximum in shadows

]85,0[1 ))(max()(| iHlHlP

]85,0[))(max( iH

P1


GraphiCon 2009 9


C – global contrastlowhighC

})][|min{},][|min(min{0

00

i

k

CkHiHiHilow })][|max{},][|max(max{1

11

ik

RR CkHiHiHihigh

H0

C0

C1

H1


GraphiCon 2009 9


Training set: 480 photos

Error rate on cross-validation test : ~0.055

Testing set: 1830 with 2% affected by backlit and low-contrast photos

The number of False Positives (FP) is 10 The number of False Negatives (FN) is 3

Low-contrast photoBacklit photo


GraphiCon 2009 10


Image

Intensity image

Z1=[-1 1]Z2=[-1 0 1]

Z3=[-1 0 0 1]

Z10=[-1 0 0 0 0 0 0 0 0 0 1]

I.Safonov et al.,2008

…

Edge image

ii ZIE

3

bgrI

Histogram

iHe

Normalized entropy

k

kiHeiA )1)(log(

Entropy to [0, 1]

iAn

121 An An F

10

12

iiAnF

23 A F

?

?

?

?

An An

GraphiCon 2009 11

Detection of blurred photosAutomatic Photo Selection for Media and Entertainment Applications

SDI

SVSDI F h )(

4

Crete et al., 2007

cr

crDISDI,

),(

crcrDBcrDIhh

hecrDBcrDISV

,)),(),((1001

1)),(),((

F.Crete et al.,2007

LPFIBh

HPFIDI

HPFBDB hh

231 An An F

11

22

iiAnF

23 A F

?

Image

Blurred image Edge image

Edge image

Comparison of the images

HPF=[1 -1]LPF=[1 1 1 1 1 1 1 1 1]/9

Detection of blurred photos

GraphiCon 2009 11


Training set: 416 photos

Error rate on cross-validation test : ~0.07

Testing set: 1830 with 171 blurred photos

The number of False Positives (FP) is 34

The number of False Negatives (FN) is 10SDI

SVSDI F h )(

4

231 An An F

11

22

iiAnF

23 A F

Detection of blurred photos

GraphiCon 2009 11


Time and camera-based quantization

Photo creation time

Photo source

1

243

L

H

evenisiiNpsH

oddisiiHYpi :)2/1(

:2/)1(

i is an index of source

L is time between the least and the most time for the largest source

Nps is a number of sources

H = L/M

M is count of images

Nregion < M

Calculation of bounding boxes

Partition into 2 app. equal subregions

Seeking for the biggest region

1200

3600

2400

72000 36000 T, s21600GraphiCon 2009 11


Nregion < Ngroup

GraphiCon 2009 12


Salient Photo SelectionThe most appealing photo is the most salient photo

L.Itti, C.Koch et al.

Images are taken from the Internet

Conspicuity maps

Gaussian pyramids

Image

Intensity image

r-channel g-channel b-channel

R-channel G-channel B-channel Y-channel

Orientation map

Intensity map

Color map

Saliency map

Feature maps

Gabor pyramids

GraphiCon 2009 13


Salient Photo Selection

original image

saliency map

intensity map

color map

orientation map

ROI



GraphiCon 2009 14Image is taken from the Internet



GraphiCon 2009 15

Saliency Index

4),( maxS

yxS 124

88

11

100

81 9262

83105 70

83

11124



GraphiCon 2009 15

Saliency Index

4),( maxS

yxS

81

88 62 92

105 70

100

Main Disadvantages:

average number of FP increases a lot with picture size

0

1

2

3

4

5

6

7

0 500 1000 1500 2000 2500

S, pixelav

gn

FP

0

5

10

15

20

25

30

0 500 1000 1500 2000 2500 3000

S, pixelt,

s.

0

1

2

3

4

5

6

7

0 500 1000 1500 2000 2500S, pixel

avg

nF

PBefore skin tone detection After skin tone detection

0

5

10

15

20

25

30

0 1000 2000 3000S, pixel

t, s.

After modification Before modification

We consider, that images of people attracts more attention

processing time also increases a lot with picture size

Six places were detected erroneously

Modifications: image down sampling is applied at preprocessing step

optimization of search using color information – skin tone detection

P.Viola, M.Jones, 2001


Face Detection

GraphiCon 2009 16

Viola-Jones, Intel OpenCV

Before modifications After modifications

Photos ranking

Heuristic formula, experiments have shown that value w=25 gives the best result


GraphiCon 2009 17

124

88

11

116 92

118148 95

62

100

118

62

Photos ranking

Heuristic formula, experiments have shown that value w=25 gives the best result


GraphiCon 2009 17

124

88

11

100

116

92

148 95


Results and discussion

GraphiCon 2009 18



GraphiCon 2009 18

Autocollage choiceOur choice



GraphiCon 2009 18



GraphiCon 2009 19

Set 1Set 1 Set 2Set 2 Set 3Set 3 Set 4Set 4 Set 5Set 5 SumSum

Agree with experts 6 5 6 5 7 29

Acceptable 3 4 4 4 2 17

Unacceptable 1 1 0 1 1 4







Prop

osed

Auto

Colla

geRa

ndom

?Automatic Photo Selection for Media and Entertainment Applications

Questions & Answers

GraphiCon 2009 8


GraphiCon 2009 9

Thank you for your attention

=)

automatic photo selection for media and entertainment applications

Education