ying cao antoni b. chanrynson w.h. lau city university of hong kong

Ying Cao Antoni B. Chan Rynson W.H. Lau

City University of Hong Kong

Automatic StylisticLayout

Background

• Manga layout is crucial for manga production, with unique styles

©AYOYAMA Gosho / Shogakukan Inc.

Manga pages Their layouts

Background

• Effective manga layout can benefit– Storytelling– Attention guidance– Visual attractiveness

• It is a difficult task

Goal

• To create high-quality manga layout with ease

Resulting layout

1

2

2

3

3

3

SemanticsArtworks

Challenge

• Not a well-studied problem

• Our solution: data-driven strategy to learn stylistic aspects from existing manga pages

• No explicit rules

Related Work

• General layout problem: global optimization

[ Yu et al. 2011] [Merrell et al. 2011]

Related Work

• Comic layout: heuristic rules or templates

[Kurlander et al. 1996] [Shamir et al. 2006] [Preu et al. 2007]

Related Work

• Computational Manga

[Qu et al. 2006]

[Qu et al. 2008]

Overview

Overview

Overview

Overview

Manga Database

• 4,000 scanned manga pages from two manga series

Panel annotation Page clustering

One manga series

3-panel pages 10-panel pages4-panel pages …

Overview

Style Models

• Represent stylistic aspects of manga layout

• Learned from manga examples

3) Panel shape

…2) Panel importance (size)

1 2 3

1) Layout structure (i.e., spatial arrangement of panels)

…

• A probabilistic generative model:pdf(gm,[x,y])

• Synthesize novel plausible layout structures

Layout structure Model

Root

©AYOYAMA Gosho / Shogakukan Inc.

Layout structure Model

• Generative process: recursive spatial division

R1 R2 R3

C1C2 C1

R2R1

C3 C2 C1

Layout structure Model

• Parameterization: spatial division instance

: Instance label [Root-R]

: Number of rows [3]

: Splitting configuration [()]

X 1

X 2

X 3

©AYOYAMA Gosho / Shogakukan Inc.

Layout structure Model

: Instance label

: Number of rows

: Splitting configuration

𝓛𝑵

𝐗Probabilistic graphical model

• Parameterization: spatial division instance

Layout structure Model

• Sample splitting configuration

: Instance label

: Number of rows

: Splitting configuration

𝓛𝑵

𝐗Probabilistic graphical model

Layout structure Model

• Sample splitting configuration

: Instance label

: Number of rows

: Splitting configuration

𝓛𝑵

𝐗Probabilistic graphical model

Sample𝐗

Layout structure Model

Layout structures sampled from our modelTraining example

• Sampling: recursive splitting using sampled

• Panel clustering

Width

Hei

ght

Panel Importance

3 12

Size Importance Shape?

A shape-to-importance classifier𝑚= 𝒇 (𝒙 )

Panel Shape Variation Model

• Captures panel shape variability‒ Active Shape Model [Cootes et al. 1995]

…{𝒔2 ,𝐔2}

…{𝒔1 ,𝐔1 }

{𝒔3 ,𝐔3 }…

Overview

Semantic SpecificationSingle-panel semantics

Inter-panel semanticsImage geometry

Group of related panels

3

Importance

Overview

Initial Layout Generation

• A layout structure

• Maximum a posteriori (MAP) inference

Fitness between and Our generative model

Input semantics

Existing ones

matches

resembles

Initial Layout Generation

• Likelihood term

Penalize panel-wise mismatch in aspect ratio & importance

Single-panel Likelihood

𝑃 𝑗

𝐼 𝑗

Image geometry

panel geometry

Initial Layout Generation

• Likelihood termInter-panel Likelihood

Initial Layout Generation

• Likelihood termInter-panel Likelihood

Measure the smoothness of path through panels

Initial Layout Generation

• Likelihood termInter-panel Likelihood

Align group boundary with layout boundary

Initial Layout Generation

• Estimate optimal initial layout‒ Exact MAP inference is computationally expensive

…

pdf(gm,[x,y])

Generative Model Maximum Posteriori

Layout Optimization

Unoptimized

• Fit to and reproduce panel shape irregularity

Layout Optimization

• Energy function

Collinearity constraint

𝐕

Boundary constraint

𝐕

𝑃 𝑖

𝐓𝑖 ∙ 𝐼𝑖

Regularization term

Layout Optimization

• Minimize via an alternating solver

Results

(1)

(2)

(2)

(3)

(2)

(2)

(1)

(2)

(2)

(2)

(3)

(3)

(1)

Comparison with existing manga page

Input Our result Existing manga page

(3)

(3)

(1)

(3)

(3)

(2)

(3)

©AYOYAMA Gosho / Shogakukan Inc.

Layouts of different styles

(1)

(2)

(3)

(1)

(3)

(2)

Input Style of “Fairy Tail” Style of “Detective Conan”

Layouts of Western comic style

User Study

• 10 participants: manual tool + our tool

• 10 Evaluators: pairwise comparison

Manual tool Our tool012345 4.09

1.36Aver

age

time(

min

)

Functionality Visual Appeal Style0%

20%

40%

60%

Manual placement Our toolTie

Summary

• First attempt to computationally reproduce layout styles of manga

• A data-driven approach for automatic generation of stylistic manga layout

• Easy and quick production of professional-looking and stylistically rich manga layouts

Limitations & Future Work

• Story pacing

• Art composition & balloon placement

• Generic framework for other layout problems

Thanks