Chang et. al.: SUBJECT DETECTION AND MANIPULATION 1

Subject Detection and Manipulation

Eric Chang, Austin Lee and Samuel Yang

Abstract— We implement and extend ex-

isting methods to detect salient objects and

generate a segmentation map which is used to

artistically desaturate the background. We

evaluate the performance using a Jaccard

similarity metric against the ground truth

bounding box.

I.. Introduction

Many interesting artistic manipulations can be doneon images and video if the subject can be reliablydetected, including background blurring, automaticrefocusing, exposure and white balance, and objectimage search. Here we implement and extend exist-ing methods for detecting salient objects [1], [2], [3],[4]. We then use the result to demonstrate an in-teresting artistic manipulation, whereby we desatu-rate the background and generate a simulated stereoview of the subject, viewable as an animated GIF.[2]

II.. Methods

In this section, we outline the entire process flowon a high level, and we also discuss the evaluationmetrics. Detailed implementations will be describedin the next section.

To detect the salient object, We first computeseveral saliency maps from the original image. Asaliency map is a probability function that tells ushow likely a pixel belongs to the salient object, usingcertain features of the image as the criterion. Then,we generate a segmentation map that optimizes theobjective function described in [3]. Then, since werestrict our input to images that contain only onesalient object, if there are multiple connected re-gions in the segmentation map, we pick the regionthat has reasonable area and has the most color va-riety. Finally, this segmentation map can be usedto perform desaturation and the 3D gif animation.

Since our images come with hand-labeled rect-angles that enclose the salient object, we evalu-ate the correctness of our algorithm by comparingthe smallest rectangle enclosing our segmentationmap to the ground-truth rectangle. The results areshown in Section IV..

III.. Saliency Maps

In this section, we describe in detail the 7 saliencymaps implemented, along with the algorithm we useto optimize the objective function. The entire pro-cessing pipeline is shown in Figure 1.

A.. Multiple-Scale Contrast

This saliency map takes advantage of the fact thatthe salient object is often in focus, and also “inter-esting” objects tend to contain a lot of edges or highcontrast features. Therefore, a lot of times comput-ing the contrast will gives us a reasonable indicationof the location of the salient object. By computingthe contrast across multiple scales of the image, wemake this technique more robust to object size. Ourimplementation follows the description in [3] closely.Figure 1a shows the result of this map.

B.. Revised Multiple-Scale Contrast

This saliency map is a modification of the multiple-scale contrast map. As discussed above, we noticedthat multiple-scale contrast is a very effective edgedetector, but it is poor at detecting a homogeneousregion. Therefore, in a simple example of a blackdot on a white background, multiple-scale contrastwill correctly outline the dot, but fail to realize thatthe interior of the dot is part of the salient object.

To get around this issue, we introduce the Re-vised Multiple-Scale Contrast map, shown in Fig-ure 1e. This map takes the multiple-scale contrastmap, performs local adaptive thresholding to iden-tify all strong edges in the figure, then removes allregions completely surrounded by edges. To con-vert this binary mask into a probability map, weassign to each connected region the maximum prob-ability in the corresponding region of the originalMultiple-Scale Contrast map. The reason we usethe maximum instead of some averaging is becauseour segmentation approach (discussed in subsectionH.) is insensitive to probability values around 0.5, sousing average values will make the map ineffective.Since we use many different maps to perform seg-mentation, the use of extreme values do not swaythe result as much.

2 EE 368 TRANSACTIONS

Subject Detection and Manipulation Eric Chang, Austin Lee and Samuel Yang

Department of Electrical Engineering, Stanford University

Motivation

color1 contrast1 center surround1

color2

content aware3

contrast2 content aware2

Saliency maps

segmentation input

poor examples

hist

ogra

m c

ount

output

a b c d e f

a b c d

e f g

Figure 1. Image processing pipeline. 7 saliency maps are combined to generate a segmentation map that identifies the

salient object. superscript 1 comes from [3], superscript 3 comes from [2], and superscript 2 are our own modifications of

other saliency maps.

C.. Content-Aware Saliency

This method comes from [2], and was the onlymethod for which we used existing code for. Themethod attempts to optimize an objective functionwhich factors in how interesting an object is as per-ceived by a human, and includes various psycholog-ical factors such as the impact of regions with highcontrast, suppression of repeating patterns, and theuse of some prior information. This map is shownin Figure 1c.

D.. Revised Content-Aware Saliency

From empirical observations, we noticed thatContent-Aware Saliency map works very much likea smart edge detector, where it only picks up theedges that belong to the salient object. Therefore,we perform the same processing described in sub-section B.on Content-Aware Saliency map to obtainthe Revised Content-Aware Saliency map. Figure1g shows the result of this map.

E.. Center-Surround Histogram.

This saliency map takes advantage of the fact thatthe colors belonging to a salient object is often dif-ferent than its surroundings. Therefore, if we draw arectangle around the salient object, and compare thecolors inside the rectangle to the colors surroundingthe rectangle (in a region having the same area), weshould expect to see a large difference.

With this observation, we can simply draw nu-merous different kinds of rectangles around a chosenpixel, and perform the color comparison. If the col-ors are very different, then the chosen pixel is mostlikely near the salient object. The reverse shouldalso hold. Now, if we iterate through all pixels, wecan get an idea of where the salient object shouldbe located.

The detailed implementation is documented in [3].However, we noticed that since we perform exhaus-tive search through all pixels, the algorithm takesvery long to run. Therefore, we downsample theimage by a factor of 2, and after computation weupsample the result by a factor 2. This makes thealgorithm 4 times faster, and we find little lost inaccuracy. Figure 1d shows the result of this map.

F.. Color Spatial Distribution

This saliency map is in some sense the dualof Center-Surround Histogram. While Center-Surround Histogram looks at local color differences,Color Spatial Distribution studies how each coloris distributed across the image. If a certain colorspreads throughout a large region in the image, thenchances are this color does not belong to the salientobject. Whereas if a color is highly localized inspace, there’s a higher chance that this color belongsto the salient object. The very complex mathemati-cal details of color distribution modeling is describedin [3], and we follow closely to his implementation.Figure 1c shows the result of this map.

G.. Revised Color Spatial Distribution

Since Color Spatial Distribution picks out colors,it is good at picking out globs instead of just theedges, compared with Multiple-Scale Contrast andContent-Aware Saliency. However, since we knowthat we will only have one salient object in the im-age, we know that ideally we should only get oneglob in the saliency map. With this insight, wetake the Color Spatial Distribution map, performglobal Otsu thresholding, remove any completelysurrounded black regions, then pick one connectedregion that’s both large in size and rich in color con-tent. We use the volume of the convex hull sur-

Chang et. al.: SUBJECT DETECTION AND MANIPULATION 3

rounding all colors in RGB vector space as a metricfor color content. Finally, we set the probabilityof this connected region to the average of the con-nected region in the original saliency map. Figure1e shows the result of this map.

H.. Segmentation Map GenerationFinally, to generate the segmentation map, we useEquation 3 in [3] as our objective function, repro-duced here:

E(A|I) =X

x

KX

k=1

�

k

F

k

(ax

, I) +X

x,x’

S(ax

, a

x

0, I)

E(A|I) is the “energy”, a measure of penalty, ofthe segmentation map A for the image I. The goalis to minimize E(A|I) as much as possible. x is oneof the pixels of the image, whereas x0 is an adjacentpixel of x. K is the number of salient maps used.�

k

is a weight assigned to the kth salient map, andF

k

(ax

, I) is a value close to 0 if the segmentationmap pixel matches the salient map pixels, and avalue close to 1 otherwise. S(a

x

, a

x

0, I) is an expo-

nential decay function of color that has low value ifx and x

0 has the same label and the same color, andhigh value otherwise.

In summary, the first double summations mea-sures how far off the segmentation map is fromthe salient maps. The second summation measureswhether our assignment gives the same label to sim-ilar adjacent color.

In [3], the weights for each salient map is com-puted based on machine learning from the imagedatabase. Since we choose to simplify our project,we assign roughly equal weights for each salientmap, with some skews to favor maps that shownto be more effective on randomly selected images.

To generate the segmentation map, we first com-pute a first-order map that optimizes only the firstsummation in the equation (in other words, we ig-nore the color relationship of adjacent pixels). Thisis easily achieved because there are no inter-pixeldependence. Then, we check each pixel of the seg-mentation map, and invert its value if that improvesthe objective function. This process is iterated untilno such pixels exist, and we use the final result asthe segmentation map.

IV.. Results

We used a subset of 200 images out of the 20,000 in[1], in which each image has exactly one salient ob-ject, and for which a ground truth bounding box ofthe salient object is known. The Jaccard similarityis given by:

J(A,B) =|A \B||A [B|

We Using Jaccard similarity of the boundingboxes as our performance metric, we increased ac-curacy from 58% to 68% over 200 images by usingthe three derived saliency maps and our new seg-mentation scheme.

Subject Detection and Manipulation Eric Chang, Austin Lee and Samuel Yang

Department of Electrical Engineering, Stanford University

Motivation

color1 contrast1 center surround1

color2

content aware3

contrast2 content aware2

Saliency maps

segmentation input

poor examples

hist

ogra

m c

ount

output

a b c d e f

a b c d

e f g

Figure 3. Performance of our segmentation method by com-

paring against the ground truth bounding boxes using Jac-

card similarity for 200 test images. The average Jaccard sim-

ilarity index over all images was 0.68.

V.. Conclusion

Salient object detection is a hard image processingproblem, and remains one of the popular area ofresearch. In this project, we studied several recentpapers describing potential solutions, and replicatedtheir algorithms and added some of our own modifi-cations. We demonstrated that even without robustmachine learning, we were able to achieve satisfac-tory results on a reasonable number of images, andachieve 68% accuracy based on Jaccard’s Index on200 images.

VI.. Future work

One area which could be improved significantly isthe performance metric. Because our ground truthdata set contained as labels only bounding boxescontaining the object, we were unable to use a morespecific performance criteria, such as a clean seg-mentation boundary of the object. For example, theimage of the hot air balloon nearly maximizes ourJaccard similarity criteria, and yet it’s still quitefar from getting a perfect segmentation. In fact,there appears to be systematic error in overestimat-ing the size of the segmentation masks, an errorwhich would better be accounted for using a moreprecisely labeled ground truth labeling set and bet-ter performance metric.

Furthermore, more interesting manipulationscould be explored, including depth-dependent de-

4 EE 368 TRANSACTIONS

Subject Detection and Manipulation Eric Chang, Austin Lee and Samuel Yang

Department of Electrical Engineering, Stanford University

Motivation

color1 contrast1 center surround1

color2

content aware3

contrast2 content aware2

Saliency maps

segmentation input

poor examples

hist

ogra

m c

ount

output

a b c d e f

a b c d

e f g

Figure 2. Top row shows input images, bottom row shows output images that have been desaturated using the segmentation

map generated from the saliency maps. Results a-c are successes whereas results d-f show failure cases.

blurring, along with additional saliency maps. Fi-nally, accuracy of any of these methods could be im-proved even further by using actual machine learn-ing techniques to build more sophisticated classi-fiers for integrating the data from all of the saliencymaps, using the entire database of 20,000 images fortraining and validation (compared with the 200 weused).

VII.. Acknowledgements

We thank Andre Araujo, Peter Vajda, and the EE368 Course Staff for assistance with the project.

VIII.. Appendix

The work was divided equally amongst the threestudents. Eric extended the saliency maps, im-plemented Jaccard similarity evaluation metricand implemented the map combination algorithm.Austin implemented the existing methods for con-trast, color spatial distribution and center surroundsaliency maps. Samuel implemented desaturationand animated .gif effects, wrote wrapper for the callto the external content-aware saliency code base,set up a testing infrastructure, attended on-campusmeetings and made poster.

References

[1] V. Ordonez A. Dhar and T. Berg. High level describableattributes for predicting aesthetics and interestingness. Proc.

IEEE Conf. CVPR, 2011.[2] L. Zelnik-Manor S. Goferman and A. Tal. Context-aware

saliency detection. Proc. IEEE Conf. CVPR, pages 2376–2383, June 2010.

[3] J. Sun J. Wang N. Zheng X. Tang T. Liu, Z. Yuan and H.-Y. Shum. Learning to detect a salient object. IEEE Trans.

Pattern Anal. Mach. Intell., 33(2):353–367, February 2011.[4] C. T. Vu and D. M. Chandler. Main subject detection via

adaptive feature selection. Int. Conf. Image Proc., pages3101–3104, 2009.