a hybrid image retargeting approach via combining … hybrid image retargeting approach via...

A Hybrid Image Retargeting Approach via

Combining Seam Carving and Grid Warping

Lifang Wu 1, Lianchao Cao

1, Min Xu

2, and Jinqiao Wang

3

1. School of Electronic Information and Control Engineering, Beijing University of Technology, Beijing, China

2. iNEXT, School of Computing and Communications, University of Technology, Sydney, Australia

3. National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Science, Beijing, China

Email: [email protected], [email protected], [email protected], [email protected]

Abstract—Image retargeting is a critical technique for

browsing images in diversified terminals. In this paper, we

propose a hybrid image resizing approach by jointly using

seam carving and warping. Firstly, based on the importance

partition with the saliency map, we apply a weighted seam

carving approach to make the seams distributed dispersedly

in the important regions. Then we propose Content Aware

Image Distance (CAID) to assess the deformation caused by

removing seams. The weighted seam carving will stop with a

fixed threshold to assure little visual image quality

degradation. Finally, the grid based warping is utilized to

achieve the final size with a global optimization model, since

warping tends to avoid discontinuity artifacts of important

region and typically make the distortion distribution of

unimportant region more coherently. Experiments and

comparison in the public RetargetMe dataset [1] with Dong

[2], Energy-based deformation [3], Multi-operator [4],

SeamCarving [5], Simple scaling operator, Shift-maps [6],

Scale and Stretch [7], Streaming Video [8], Non-

homogeneous warping [9], show the superiority of the

proposed approach.

Index Terms—Image Resizing; Weight Transfer; Content

Aware Image Distance

I. INTRODUCTION

With the rapid development of mobile multimedia

techniques, it is possible that users browse images on

devices of different sizes such as cellular phones and high

resolution TV. This requires image retargeting techniques

[10], which adapt the images of various aspect ratios to

the target screen, maximizing the viewer experience.

Many content-aware retargeting methods have been

proposed such as seam carving [4], [5], [11], [12], mesh-

based retargeting [7], [9], [13], [14] and hybrid

approaches [2], [15], [16].

Seam carving (SC) [4], [5], [11], [12] removes or

inserts 1D seam that passes through the less important

regions to preserve media content. But it usually brings

discontinuity artifacts when the removed/inserted seams

pass through some important objects or regions. To

restrain the noise brought by seam carving, Achanta [17]

computed the saliency map only once, independent of the

number of seams inserted or removed. Toony [18]

assumed that the object edges were removed during

carving seams, and they proposed a modified saliency

map combining the traditional saliency map with local

edges for seam carving. Domingues [19] proposed stream

carving by utilizing an adaptive importance map to merge

several features like gradient magnitude, saliency, face,

edge and straight line detection. Chen [20] proposed

balanced seam carving (BSC) with a criterion to evaluate

the diagonal artifacts in addition to the previous

horizontal and vertical artifacts. Zhang [21] defined

handles to describe both local regions and image edges.

They assigned a weight for each handle based on an

importance map for the source image. They constructed a

quadratic distortion energy to measure the shape

distortion for each handle. Huang [22] presented a fast

seam based image resizing approach. They searched

seams through establishing the matching relation between

adjacent rows or columns. And a linear algorithm was

proposed to find the optimal matches, which could save

about 99% time compared to [12].

In addition to seam carving, mesh based approaches

[7], [9], [13], [14] are kinds of continuous approaches. It

optimizes an image from the source media size to some

target size using several types of constraints to protect

media content. Warping tends to avoid discontinuity arti-

facts and typically preserves the overall shapes of image

objects more coherently.

The others are hybrid approaches. Dong [2] proposed

an approach to combining seam carving and scaling

based on an image distance function. Their strategy was

interesting but it needed to scale the image to target size

and compute the distance once a seam is removed, which

was a time-consuming process. Also, scaling may cause

deformation of important regions. A similar idea was

proposed by Hwang [16], which exploited the important

map weighted combining saliency map, gradient and face

regions. The switching scheme (switching from seam

carving to warping) is that the energy of a seam is larger

than a threshold. In [15], a scheme to jointly use seam

carving with warping was proposed. The deformation of

semantic edges (DSE) was defined to measure

deformation of resized images. This approach was

computationally more efficient than [2], but the semantic

edges needed to be detected beforehand and DSE can

only be applied to straight line edges. To compare image

quality by different retargeting methods reliably, an

objective metric was presented from global to local

viewpoints to access the quality of retargeted images [23].

JOURNAL OF MULTIMEDIA, VOL. 9, NO. 4, APRIL 2014 483

© 2014 ACADEMY PUBLISHERdoi:10.4304/jmm.9.4.483-492

Generally speaking, removing excessive seams can

easily cause image quality degradation for seam carving.

Cho [24] proposed an importance diffusion scheme which

propagated importance of removed pixels to their neigh-

bors for preserving visual contexts and avoiding over-

shrinkage of unimportant parts. It can be seen that the

artifacts brought by seam carving in unimportant regions

cause less visual quality degradation than important

regions. Based on our observation, if the same number of

removed seams is relatively far from each other, the

subjective image quality degrades not obviously, as

shown in Figure 1. In this paper, we propose a weighted

seam carving approach, in which the weighted forward

energy function [5] is computed in seam carving instead

of energy diffusion [24]. And the weight of removed

seams falling in important regions is propagated to the

pixels in its neighborhood. It will increase the cost of

removing these neighboring pixels and decrease the

possibility of these neighboring pixels involved in a seam.

Therefore, the seams fall into important regions non-

adjacently.

Figure 1. Seam carving under different distribution for seams. (a) The original image. (b) Seams distribute uniformly. (c) Seams distribute

consecutively. (d) Result of b. (e) Result of c

The above scheme can reduce image degradation when

the same number of seams is removed. However, it can

not essentially resolve the problem of artifacts. After

more seams are removed, the above scheme also causes

visual quality degradation. While the warping approach is

a continuous way, which tends to avoid discontinuity

artifacts of important region and typically makes the

distortion distribution on unimportant regions more

coherently. Therefore an optimal scheme to jointly using

seam carving and grid warping is proposed in this paper,

as illustrated in Figure 2. Motivated by the objective

quality assessment model [25], which showed that SSIM

(Structural Similarity) is consistent with the subjective

mean opinion score (MOS) by Logarithm function [25],

we propose Content Aware Image Distance (CAID) to

assess image quality of different retargeting size. With

the CAID model, we can assess the structural damage

caused by removing seams so as to optimally combine the

advantage of the weighted seam carving and warping.

The weighted seam carving will stop when visual image

quality degradation is greater. Then, the grid based

warping is utilized to achieve the target size with a global

optimization model, which could effectively keep the

whole structure through the distort the unimportant

regions coherently.

The contributions of this paper can be summarized as

follows:

For the structural deformation caused by neighboring

seams removal, we propose a weighted seam carving

approach. By transferring the weights of removed seams

to its neighboring pixels, the seams fall in important

regions are removed more uniformly. Therefore, the

weighted seam carving causes less visual quality

degradation than seam carving [5].

To optimally combine seam carving with warping, we

introduce a CAID to give an objective measure from the

view of image structural quality assessment. The

structure components of sub images are selected for

effectively measuring the deformation brought by

removed seams, which is more consistent with human

perception than BDS [26] and Liu [23].

With the CAID model for quality assessment, we can

effectively control the structural deformation caused by

removing seams and optimally combine the advantage of

the weighted seam carving and grid warping. Our

approach is superior in terms of removing un-important

regions, and keeping the aspect ratio of important objects.

II. CONTENT AWARE STRUCTURAL SIMILARITY

In this paper, we introduce a CAID to give an objective

measure for seam carving from the view of image quality

assessment. The CAID is used to measure the structural

quality loss of the important regions and judge the

switching point from the seam carving to the grid

warping.

A. Region Importance Determination

We calculate the visual saliency with [27] since it is

easy to implement and the performance is acceptable. The

saliency value of each pixel is normalized to [0-1] as the

weight ( , )w x y for the pixel in ( , )x y . The threshold r0 is

determined by a condition that half number of ( , )w x y is

smaller than0r . Then we obtain the important regions

from the saliency map by binaryzation. The binary image

( , )b x y is computed as follows,

1 ,0

,0 ,

0

w x y rb x y

w x y r

(1)

Figure 3 shows an example of region importance

determination, including the original image, the saliency

map and the important regions respectively. In Figure

3(c), the important regions are marked as 1.

484 JOURNAL OF MULTIMEDIA, VOL. 9, NO. 4, APRIL 2014

© 2014 ACADEMY PUBLISHER

http://dict.youdao.com/search?q=binaryzation&keyfrom=E2Ctranslation

Figure 2. Image retargeting by combination of seam carving and grid warping

Figure 3. An example of region importance determination. (a) the original image, (b) the saliency map, (c) the important regions

B. Statistic Structural Similarity

Structural similarity measure (SSIM) [25] is classic

approach used for image quality assessment, and it is

superior to apply the SSIM index locally rather than

globally for effective quality assessment. In SSIM, three

comparison functions are calculated from the aspects of

luminance, contrast and structure. As the statistical results

in our previous work [28], the removed seam brings large

intensity discontinuity and it causes more structural

variation. Therefore, the structural comparison function is

more effective to measure the quality degradation in seam

carving. For two image f and g , the structural

comparison function is,

( , )fg

f g

Cs f g

C

(2)

where C is small constant in both denominator and

numerator to avoid instability when the remaining part of

the denominator is very close to zero. The local statistic

features including mean intensity, standard variation and

correlation variation , , ,f g f g and fg are as

follows:

1 1

1 1,

N N

f i g i

i i

f gN N

(3)

1 1

2 22 2

1 1

1 1( ( ) ) , ( ( ) )

1 1

N N

f i f g i g

i i

f gN N

(4)

1

1( )( )

1

N

fg i f i g

i

f gN

(5)

where N is the total pixel numbers of each image, and

if and ig are the gray value for pixel i . By Cauchy-

Schwarz inequality ( , ) 1. ( , ) 1s f g s f g if and only if

f and g are linearly dependent. With the analysis above,

we define the distance between two images f and g as

follows:

( , ) 1.0 ( , )Dis f g s f g (6)

When the two image are the same, the value of ( , )s f g is

Otherwise, the value of ( , )Dis f g is 0.

C. Content Aware Image Distance

For the original ( , )f x y and the target image ( , )g u v ,

in order to calculate the structural similarity, we

uniformly split the important regions into sub images

with 9×9 pixels in the original image. Each sub image is

represented by its center coordinate,

,_ ( , ), 1,2,...,sub n n n subI x y n N and subN is the total

number of sub images. For a sub image in ( , )f x y , we

will find the corresponding sub image in ( , )g u v . During

the process of seam carving, if the seam path pass

through the center point of the sub image, the center of

the sub image should be updated.

Let us take removing vertical seams in the image as an

example. When the removed seams don't pass through the

center of the sub images, Dis is calculated as Equation 6.

When a seam passes through the center of the thi sub

image, and its coordinate is ( , )i ip x y . Its left and right

neighboring pixels are ( , 1)i ip x y and ( , 1)i ip x y . The

centers of sub images on its nearest left/right are

_( , )i i leftp x y and _( , )i i rightp x y respectively. The

distances of the left and right neighboring pixels to the

center of the left and right sub images are computed as

Equation 7, respectively.

_ _

_ _

_ ( 1)

_ ( 1)

i left i i left

i right i right i

Dis Ave y y

Dis Ave y y

(7)

The one with the larger distance is set as the center of

the thi sub image. If the neighboring pixels of the pixel

( , )i ip x y include the centers of some other important sub

images, the thi sub image is deleted and the

corresponding Dis is set to 1.0.

Similarly, if horizontal seams are removed, the up and

down neighboring pixels are used for calculating the

distances. Figure 4(b) gives the corresponding sub images

of Figure 4(a) after 150 seams are removed.

After we build the correspondence between the sub

images in ( , )f x y and ( , )g u v , we compute the distance

_ _ _( , )sub n sub n sub nDis f g between each pair of important sub

images as Equation 6. CAID is the average distance of all

the sub images as follows,

_

1

1 subN

sub n

nsub

CAID DisN

(8)



Figure 4. The corresponding sub images between source image and target image. (a) the important sub images of Figure 3 (a), (b) the

corresponding sub images of Figure 3(a) after 150 seams removed

III. WEIGHTED SEAM CARVING

In order to reduce the degradation of visual image

quality, the strategy for removing seams is critical. But

based on the different strategies of removing seams

illustrated in Figure 1, there is visible quality degradation

when the seams are consecutive especially in the

important regions, while the visual image degradation is

not obvious when the seams are dispersed. Therefore, we

can remove seams by the following rules:

Seams in the unimportant regions are removed as

many as possible.

The distribution of seams is as uniform as possible.

Based on the two rules, we adopt a weighted strategy

to improve the performance of seam carving. If a seam to

be removed lies in the important regions, the pixels

around the seam will be penalized with a weight. So the

pixels around its neighborhood are less likely to be

removed. This could ensure that seams in important

regions are non-adjacently removed. Moreover, this

approach also results that more seams from the

unimportant regions are removed with less degradation.

A. Weighted Gradient Energy

As for the weighted seam carving, we firstly compute

the weighted gradient energy. With an image ( , )I x y , we

compute three possible step costs to the left, up or right

respectively.

( , ) | ( , 1) ( , 1) ( , 1) ( , 1) | | ( 1, ) ( 1, ) ( , 1) ( , 1) |

( , ) | ( , 1) ( , 1) ( , 1) ( , 1) |

( , ) | ( , 1) ( , 1) ( , 1) ( , 1) | | ( 1, ) ( 1, ) ( , 1)

Left

up

Right

C x y w x y I x y w x y I x y w x y I x y w x y I x y

C x y w x y I x y w x y I x y

C x y w x y I x y w x y I x y w x y I x y w x y

( , 1) |I x y

(9)

where the initial weight ( , )w x y is the normalized

saliency value in Section II-A. Then we compute the

forward-cumulative cost matrix ( , )M i j . For vertical

seams, the cost ( , )M i j is updated as follows,

( 1, 1) ( , )

( , ) min ( 1, ) ( , )

( 1, 1) ( , )

left

up

right

M x y C x y

M x y M x y C x y

M x y C x y

(10)

The minimal ( , )M i j is the minimal cost RLCost . The

( , )ypath x y corresponding to RLCost is the optimal seam

path. In another word, removing ( , )ypath x y will insert

the minimal amount of weighted gradient energy.

B. Weight Transfer

When the removed seam pass through the important

regions, the weights of pixels in the neighborhood are to

be updated correspondingly. The main idea is to transfer

the weight of removed pixels on the seam to its

neighboring pixels. Using removing a vertical seam as an

example, the seam to be removed is marked in red in

Figure 5. We will update the weights of pixels in the

horizontal neighborhood of this seam. For an pixel

0 0( , )x y on there moved seam, the neighboring pixel set is

denoted as 0 0( , ), 1, 2,...,C x p y p

( 1)Bandwidth . For each pixel 0 0( , )C x p y in

set C , the corresponding weight will be updated using

Equation 11.

0 0 0 0 0 0( , ) ( , ) (1.0 ) ( , )

pw x p y w x p y w x y

Bandwidth (11)

where Bandwidth is the half width of the neighborhood,

and we set Bandwidth = 5 in our experiment.

Figure 5. A seam and corresponding neighboring regions for weight transfer

IV. GRID WARPING

After a seam is removed, the CAID of the target image

to the original image is computed. If the CAID is larger

than or equal to 0.2 (based on our extensive observation),

we think the distortion is unacceptable, and the procedure

is switched to warping. The optimization model [29] is

employed to optimally allocate image aspect distortion

based on grid model. The constraints of rectangular grids

are employed to avoid serious shape transformation in



resizing. All grids' aspect ratio changes are summed up to

measure distortion energy in retargeting. For grid

construction, an image is divided into N K grids, and

the grids are denoted by ( , )M V E in which V is the

2D grid coordinate, E are the edges of grids. Each grid is

denoted by 11,... ,...ij NKG g g g with its location ,i j .

Owing to the constraint of rectangular grids, all the grids

in each row have the same height while the grids in each

column have the same width. So the edge is simply

denoted by 1 1( , ),...( , )N KE w h w h , and ,i jw h is the

width and the height of the grid ijg respectively. Each

grid's importance ijs is computed by averaging the pixel-

wise importance values in that grid .ijg With the

definition above, The objective functions as well as

boundary constraints are formulated as follows:

The Objective Function A nonlinear objective

function is employed to reallocate distortion to a large

proportion of (all) unimportant regions to avoid

discontinuity. To minimize the grid distortion energy, the

objective function is defined as:

min ( ) ( )m

n

i s j ijy t ar x t s (12)

m is a even number and 2, 1m n . As to the choice of

object function, we do not prefer a linear form. For

example, when ( ) ( )i s jy t ar x t is used as the distort-

ion energy of grids, the optimization always chooses one

integral row or column of grids with a minimum

importance value to shrink or stretch. When the width or

height of those grids is reduced to zero, the spatial

continuity of a whole image would be destroyed.

Distortion Energy We use the edges of grids rather

than the coordinates of vertices to measure the distortion

energy of each grid. For an image t , the distortion energy

of each grid ijg is defined as:

( ) ( )ij i s jg y t ar x t (13)

sar is the aspect ratio of the original grid respectively,

,i jx y is the width and height of the target grid ijg ,

respectively.

Boundary Constraints We introduce the constraints

as follows:

1

1

( )

( ) , , 1,2,...

( ) 0

( ) 0

N

i Ti

N

j Ti

i

j

y t H

x t W i j n

y t

x t

(14)

,T TH W are the height and width of a target grid,

respectively. Note that the minimum height or length of a

grid is set to one pixel, as adjacent grids should not

overlap each other.

Global Solution To get a global solution, we employ

an active-set method to solve this optimization problem.

This nonlinear program is convex programming, and any

local solution of convex programming is actually a global

solution. ( ) ( )m

i s jy t ar x t is a convex function, so our

objective function ( ) ( )m

n

i s j ijy t ar x t s is a

convex one.

Moreover, the equality constraints are linear functions

and the inequality constraints can be seen as a concave

function. The solutions satisfying equality and inequality

constraints finally form a convex set. When a local

solution is resolved, the global solution is yielded.

An active-set starts by making a guess of the optimal

active set which satisfies equalities. With the width and

height of a target image ,T TH W , the initial guess is

,... ,...T TH W

N N

satisfying the equality constraint. Then,

the nonlinear program can be solved iteratively to get

global solutions in feasible region.

For the convex programming, the Hessian matrix of

the objective function is positive semi-definite. The

complexity is similar to a linear programming that

depends on the number of the model variables ( )O N K

(i.e. the division of width and height).

Figure 6. Results of grid based image warping

V. EXPERIMENTAL RESULTS

A. Experimental Setting

To evaluate the effectiveness and efficiency of our

retargeting method, we use the RetargetMe dataset [1]

and our collected images datasets, to conduct our

experiments and comparison. Our experiments involve

three parts:

Evaluate the effects of our weighted seam carving;

Evaluate the measure of CAID;

Evaluate the effectiveness of our combination

approach.



B. Effects of Weighted Seam Carving

In order to evaluate the performance of weighted seam

carving, we compare our weighted seam carving with

Rubinstein's SC [5] in Figure 7, which shows the cor-

responding images when seam number is 100, 200,

300,400, 500 respectively. Figure 3 (a) is the original

image of size 500 500 , we resize it to the image of size

250 250. We use CAID to assess the loss of the visual

quality. We also indicate the center of important sub

images on the corresponding images.

Figure 7. Comparison of Rubinstein's SC [5] and weighted seam carving. (a1)- (a5) are results of Rubinstein's SC [5] and the CAID

values. (b1)-(b5) are the results of ours

From Figure 7, our approach can achieve more

uniform distribution of important sub images than

Rubinstein's SC [5], especially for 300, 400 and 500

removed seams. It demonstrates that our approach causes

less deformation (lower CAID) in important regions than

Rubinstein's SC [5]. Moreover, in terms of the overall

visual effect our approach is much better after the seams

are removed. The most important is that, as the seam

number increases, Rubinstein's SC [5] causes faster

degradation of image quality than ours. Further-more,

from Figure 7, the important sub images distribute more

uniformly from our approach than from [5].

C. Experimental Results of CAID Measure

To show the performance of the CAID measure, we

add comparison with the objective measures: BDS [26],

Liu [23]. We chose ten images in TargetMe dataset

whose width or height is bigger than 1000 pixels. Each

image is resized using our weighted seam carving to

reduce 100, 200, 300, 400, 500, 600 and 700 seams

respectively. Both subjective and objective measures are

used to assess each resized image. For subjective

evaluation, we collected the subjective measure results of

40 participators based on how similar each retargeted

image and its original image were. The participators are

asked to give a dissimilarity score in range of [0,1], and

not influencing the participators aesthetic concept, no

hints and prior-knowledge are delivered to them in

advance. We average all the scores as the result of

subjective measure. The measures Liu [23] and BDS [26]

are used to compare with our approach. All the results are

normalized to dissimilarity in the range of [0-1], and the

experiment results are shown in Figure 8. From Figure 8,

we could see that the curve shape of our results are more

similar with the results of user study than other methods.

And the values of CAID and user study are different duo

to different quantitative method. The results of CAID and

the user study increase dramatically with the increasing

of removed seams, whereas Liu [23] increases linearly

and BDS [26] is fluctuant. The results could explain that

the CAID measure is more consistent with human

perception.

Figure 8. Comparison results between CAID, BDS [26], Liu [23], and the User study

D. Performance of Weighted Seam Carving + Warping

Our approach is compared with warping [29],

Rubinstein's SC [5], our weighted SC and Dong [2], our

weighted SC + scaling, and our weighted SC + warping.

An comparison example is given in Figure 9. From

Figure 9, we can see that our SC is much better than

Rubinstein's SC [5]. Compared with other approaches,

our SC + warping can remove more unimportant regions

(such as sky in the left images) than Dong [2], and it

causes less deformation of the key object (Tower) than

our weighted SC + scaling. But the tower in Figure 9(g)

is a little smaller than those in Figure 9(d)-(f).

In order to further show the effectiveness of our results,

we also compute the area and aspect ratio of the

important object in Figure 10.The key object lotus is

marked using a bounding box in Figure 10(a). The area

and aspect ratio of key objects in different images are

listed in table I. The results of comparison in Table I

show that our weighted SC + warping can preserve more

important regions than warping, and it keeps the aspect



ratio of the important object much close to the original

image than other approaches. Our weighted SC causes

less deformation of details than Rubinstein's SC [5]. In a

word, our weighted SC+warping shows the better

performance in terms of removing more unimportant

pixels, while preserving more details and aspect ratio of

the important objects. Our approach achieves an optimal

trade off between preserving the aspect ratio of important

regions and the loss of deformation.

Figure 9. Comparison of different image resizing approaches

E. User Study

A subjective evaluation is further performed to com-

pare with the other methods in RetargetMe dataset [1].

By means of user preference and scoring evaluation, the

effectiveness is measured quantitatively. Totally 40

students and teachers participate in the user study. Each

participant is showed an original image and a randomly

ordered sequence of retargeting results with different

methods including Cropping, Energy-based deformation

[3], Multi-operator [4], Rubinsteins SC [5], Simple

scaling operator, Shift-maps [6], Scale-and-Stretch [7],

Streaming Video [8], Non-homogeneous warping [9],

Wang's method [9] and our weighted SC + warping. The

participators are asked to vote the most favorite one and

least favorite one for all the retargeting methods. Not

influencing the participators' aesthetic concept, no hints

and prior-knowledge are delivered to them in advance.

The statistical results are shown in Table II. As listed in

Table II, our approach rank 2th for "most favorite", and

rank 9th for "least favorite". This interesting results show

that most participants prefer our results. Figure 11 gives

some comparison results from the RetargetMe database.

Figure 10. Comparison of different image resizing approaches. (Image is resized from 333 × 500 to 333 × 250.) (a) Original image (b) warping

(c) Rubinstein's SC (d) Our weighted SC (e) Dong's with DCD [2] (f) Dong's Without DCD [2] (g) Our weighted SC+scaling (h) our weighted

SC + warping

TABLE I. THE AREA AND ASPECT RATIO OF IMAGES IN FIGURE. 10.

Area (Pixels) Aspect Ratio

(a) 35.6k 1.76

(b) 17.1k 1.47

(c) 29.7k 1.44

(d) 27.5k 1.37

(e) 23.3k 1.29

(f) 26.8k 1.50

(g) 24.8k 1.22

(h) 20.4k 1.57

F. Failure Case

Figure 12 give a failure case of our approach. The

original image (in Figure 12(a)) is resized from 683×1024

to 684×768. There is obvious deformation in the shelf as

shown in Figure 12(e). Since we give more constraints on

the important regions, and we have not any operation for

the unimportant regions. When the removed seams pass

through the region of "shelf", the serious deformation



TABLE II. RESULTS OF SUBJECTIVE EVALUATION. CR: CROPPING, LG: ENERGY-BASED DEFORMATION [3], SC: RUBINSTEINS SC [5], SCL: SCALE, SM:SHIFT-MAPS [6], SNS: SCALE-AND-STRETCH [7], SV: STREAMING VIDEO [8]

Cr Lg Multioperator Sc Scl Sm Sns Sv Warp Ours

Number of images 78 75 77 77 78 73 71 78 77 78

Most favorite (%) 18.28 5.20 14.42 7.03 4.58 12.52 6.13 9.07 5.97 16.79

Least favorite (%) 16.67 8.84 3.51 14.40 17.07 13.41 7.40 4.42 7.24 7.04

Figure 11. Comparison results in RetargetMe images

Figure 12. A failure case. (a) Original image. (b) The saliency map. (c) the important regions. (d) The important regions with structural

constraints. (e) The resized results of our approach. (f) the improved resized result with structural constraints

happens. Next step, we will add some structural

constraints for the unimportant regions like Figure 12(d),

where is not obvious deformation in the resized image as

shown in Figure 12(f).

Our approach is implemented in C++, and it is run on a

PC with duo CPU 2.10 GHZ, 1G memory. Average 40

seconds are needed to resize a 500×500 image to half size.

VI. CONCLUSIONS

In this paper, we have proposed a hybrid image

resizing approach by jointly using seam carving and

warping. We apply CAID to monitor degradation of

visual quality when removing seams in the resizing

process. The weighted seam carving will stop with a

fixed threshold to assure little visual image quality

degradation. Then the current image is warped to the

target size. In this way, we remove the unimportant pixels

as possible and make the deformation of the important

regions as small as possible. Experiments and comparison

in Retarget Me database shows the superiority of the

propose approach.



ACKNOWLEDGMENT

The authors are grateful to the anonymous referees for

their valuable comments and suggestions to improve the

presentation of this paper. This work was supported by

973 Program (2010CB327905) and National Natural

Science Foundation of China (61273034, 61040052).

REFERENCES

[1] M. Rubinstein, D. Gutierrez, O. Sorkine, and A. Shamir,

"A comparative study of image retargeting," ACM Trans-

action Graphics, vol. 29, no. 6, Dec. 2010.

[2] W. Dong, N. Zhou, J. Paul, and X. Zhang, "Optimized

image resizing using seam carving and scaling," ACM

SIGGRAPH Asia 2009 papers, vol. 28, no. 5, pp. 1–10,

2009.

[3] Z. Karni, D. Freedman, and C. Gotsman, "Energybased

image deformation," CGF, vol. 28, no. 5, pp. 1257–1268,

2009.

[4] M. Rubinstein, A. Shamir, and S. Avidan, "Multi-operator

media retargeting," ACM Transaction Graphics, vol. 28,

no. 3, pp. 1–11, 2009.

[5] M. Rubinstein and A. Shamir, "Improved seam carving for

video retargeting," ACM Transactions on Graphics (TOG),

vol. 27, no. 3, pp. 1–9, 2008.

[6] Y. Pritch, E. Kav-venaki, and S. Peleg, "Shift-map image

editing," ICCV, pp. 151–158, 2009.

[7] [7] Y. Wang, C. L. Tai, O. Sorkine, and T. Y. Lee,

"Optimized scale-and-stretch for image resizing," ACM

Trans. Graph, vol. 27, no. 5, pp. 1–8, 2008.

[8] P. Krahenbuhl, M. Lang, A. Hornung, and M. Gross, "A

system for retargeting of streaming video," ACM TOG, vol.

28, no. 5, 2009.

[9] L. Wolf, M. Guttmann, and D. Cohen-Or, "Non-

homogeneous content driven video retargeting," IEEE 11th

International Conference on ICCV, pp. 1–6, 2007.

[10] L. Chen, X. Xie, X. Fan, W. Ma, H. Zhang, and H. Zhou,

"A visual attention model for adapting images on small

displays," ACM Multimedia Systems Journal, vol. 9, no. 4,

pp. 353–364, 2003.

[11] A. Shamir and O. Sorkine, "Visual media retargeting,"

ACM SIGGRAPH ASIA 2009 Courses, pp. 1–13, 2009.

[12] S. Avidan and A. Shamir, "Seam carving for content-aware

image resizing," ACM Trans. Graph, vol. 26, no. 3, p. 10,

2007.

[13] R. Gal, O. Sorkine, and D. Cohen-Or, "Feature aware

texturing," In EGSR, pp. 297–303, 2009.

[14] Y. Zhang, S. Hu, and R. Martin, "Shrinkability maps for

content-aware video resizing," Computer Graphics Forum,

vol. 27, no. 7, pp. 1797–1804, 2008.

[15] Y. Gong, L. Wu, and X. Zhang, "A semantic aware image

retargeting scheme combining seam carving and non-

homogeneous warping," Proceedings of the Second

International Conference on Internet Multimedia

Computing and Service, pp. 53–56, 2010.

[16] D.-S. Hwang and S.-Y. Chien, "Content-aware image re-

sizing using perceptual seam carving with human attention

model," Multimedia and Expo, 2008 IEEE International

Conference, pp. 1029–1032, 2008.

[17] R. Achanta and S. Susstrunk, "Saliency detection for

content-aware image resizing," Image Processing (ICIP),

2009 16th IEEE International Conference, pp. 1005–1008,

2009.

[18] Z. Toony and M. Jamzad, "A modified saliency detection

for content-aware image resizing using cellular automata,"

Signal and Image Processing (ICSIP), 2010 International

Conference, pp. 175–179, 2010.

[19] D. Domingues, A. Alahi, and P. Vandergheynst, "Stream

carving: An adaptive seam carving algorithm," Image Pro-

cessing (ICIP), 2010 17th IEEE International Conference,

pp. 901–904, 2010.

[20] J. Chen, L. Miao, and X. Liu, "Balanced energy for

content-aware image resizing," Ubi-media Computing (U-

Media), 2010 3rd IEEE International Conference, pp. 24–

29, 2010.

[21] G. Zhang, M. Cheng, S. Hu, and R. R. Martin, "A shape-

preserving approach to image resizing," Computer

Graphics Forum, vol. 28, no. 7, pp. 1897–1906, 2009.

[22] H. Huang, T. Fu, R. Paul, and L. Chun, "Real-time

content-aware image resizing," Science in China Series F:

Information Sciences, vol. 52, no. 2, pp. 172–182, 2009.

[23] Y. Liu, X. Luo, Y. Xuan, W. Chen, and X. Fu, "Image

retargeting quality assessment," In Eurographics, vol. 30,

no. 2, 2011.

[24] S. Cho, H. Choi, Y. Matsushita, and S. Lee, "Image

retargeting using importance diffusion," Image Processing

(ICIP), 2009 16th IEEE International Conference, pp.

977–980, Nov. 2009.

[25] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli,

"Image quality assessment: From error visibility to struc-

tural similarity," IEEE Transaction on Image Processing,

vol. 13, no. 4, pp. 600–612, 2004.

[26] D. Simakov, Y. Caspi, E. Shechtman, and M. Irani,

"Summarizing visual data using bidirectional similarity,"

CVPR, pp. 1–8, 2008.

[27] S. Goferman, L. Zelnik-Manor, and A. Tal, "Context-

aware saliency detection," IEEE Computer Vision and

Pattern Recognition (CVPR) 2010 (ORAL), pp. 2376–2383,

2010.

[28] L. Wu, L. Cao, J. Wang, and S. Liu, "Content aware metric

for image resizing assessment," 12th Pacific-Rim

Conference on Multimedia (PCM), pp. 203–212, 2011.

[29] B. Li, Y. chen, and J. Wang, "Fast retargeting with

adaptive grid optimization," Proc. ICME'11, 2011.

Lifang Wu received her B.E. and M.E.

degree from Beijing University of

Technology (BJUT), Beijing, China, in

1991 and 1994, respectively. She

received her Ph.D. degree of pattern

recognition and intelligent system from

BJUT in 2003. She is now the faculty of

School of Electronic Information and

Control Engineering, Beijing University

of Technology, where she currently serves as a professor. She

has published over 50 referred technical papers in international

journals and conferences of image/video processing, pattern

recognition. Her research interests include image/video analysis

and understanding, face detection and recognition, face

encryption. She is a senior member of Chinese Institute of

Electronics.

Lianchao Cao was born in China in

1987. He received the Bachelor Degree

in Software Engineering in 2010 from

China University of Geosciences in

Beijing. He is currently a postgraduate

student in Beijing University of

Technology. His research activity mainly

focuses on Digital Image Retargeting.



Min Xu received the B.E. degree from

University of Science and Technology of

China, in 2000, M.S. degree from

National University of Singapore in 2004

and Ph.D. degree from University of

Newcastle, Australia in 2010. Currently,

she is a lecturer in School of Computing

and Communications, Faculty of

Engineering and IT, University of

Technology, Sydney. Her research interests include multimedia

content analysis, video adaptation, interactive multimedia,

pattern recognition and computer vision.

Jinqiao Wang received the B.E. degree

in 2001 from Hebei University of

Technology, China, and the M.S. degree

in 2004 from Tianjin University, China.

He received the Ph.D. degree in pattern

recognition and intelligence systems

from the National Laboratory of Pattern

Recognition, Chinese Academy of

Sciences, in 2008. He is currently an

Associate Professor with Chinese Academy of Sciences. His

research interests include pattern recognition and machine

learning, image and video processing, mobile multimedia, and

intelligent video surveillance.



a hybrid image retargeting approach via combining … hybrid image retargeting approach via...

Documents