1

Automatic Storytelling in Automatic Storytelling in ComicsComics

Sheng-Wei (Kuan-Ta) ChenSheng-Wei (Kuan-Ta) ChenInst. of Information ScienceInst. of Information Science

Academia SinicaAcademia Sinica

2

Intrinsic Needs of Human Being!Intrinsic Needs of Human Being!

• Communication

• Storytelling– Today’s world is information-saturated– Computers are becoming a larger and larger part of

people’s lives (Intel 2005)– More and more people are documenting their lives

with digital images, video, music, and recordings (Imation 2003)

– As storage space gets cheaper, and people record more and more and more, they need some way to automate the categorization of all of these digital memories

3

• Recall, share and preserve (gaming) experience– Virtual community such as forums, blogs– Video, screenshots, comics or other art formats– Narrative, summary, story

Search “Lineage Story” or “Lineage Comics”…– Around 26,600 files in YouTube and 724,000 items in Google– http://www.lineage2.com/community/fanart.html– http://himewikia.blogspot.com/– …

4

• Example:

5

• Example:

6

Examples

7

More examples

8

Narrative• In short,

People Like Stories. Or, put more accurately,

People Remember Stories

• So we provide a summarization tool to aid storytelling of end-users

• But why Comics…

9

Comic• A storytelling medium

• Vocabulary of comics– Panel layout

• Border shape• Panel size• Bleed, splash page

– Speech balloon and sound effects– Motion lines

• Visual language– Draw attention to important events

Reference: http://scottmccloud.com/2-print/index.html (McCloud, Scott)

10

Comic Generation ToolsComic Generation Tools

• Automation support– The computer is charged with a greater role

in the creation of the comic

• Authoring support– Tools offer the most agency in creating panels

and layouts of any size and shape

Reference: http://dm.lcc.gatech.edu/~jalderman/comics/

11

Comic ChatComic Chat (1996)

Comic Chat visually represents conversations as sequences of

comic panels.

Automation support:• Changing posture and facial expression

of users’ avatars in chat room• Placing word balloons in natural

reading order• Zooming “camera” to frame the people

talking • Parsing text into panels procedurally.

Panels are all in the same size

Reference: http://kurlander.net/DJ/Projects/ComicChat/resources.html

12

• Microsoft Chat 2.5

Chat room:

Massage

Authoring support:Emotion Wheelallows users to choose the facial

expression and body language

Comic mode

13

• Microsoft Chat Character Editor

Character IconCharacter Contents• New Pose• New Pose

14

• Example:

15

Video MangaVideo Manga (1999-2003)

It provides a summary of video clips that allows users to quickly browse a longvideo in comic book format.

• Manga is named after a Japanese word for "comic book"

• A video is represented with different-sized keyframes that packed in a comic book format

• Video Manga allows users to quickly browse a long video

• The visual summaries support printing function

Reference:http://www.fxpal.com/?p=manga

16

Automation support:• A video is segmented based on the

color features of each frame• The segments are clustered

according to their similarities• We have introduced an importance

score to rank the segments• A segment is considered to be

important if it is long and rare• The keyframes are extracted from

highly ranked segments• The keyframes are sized according to

their scores• The frame-packing algorithm puts the

keyframes in a comic book format

Authoring support:• Handling captions to its frames

manually typed in by users

17

• Video Manga : Summary of staff meeting

18

Generating Comics From Generating Comics From 3D Interactive Computer 3D Interactive Computer Graphics Graphics (2006)(2006)

Automation support:• Recognizing scenes• Converting to visual depiction• Comic layout• Image processing

Reference:http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=1626186

19

• Recognizing scenes– Time and space– Separating scenes when

• The interaction level is low• The location changes

• Converting to visual depiction– Transforming each scene

into a sequence of images depicting the main event

– Using idioms to depict interactions• Change of scene• Shooting• conversation

• Automation support

20

• Comic layout– B (Big):

peak interaction images and change-of-scene

– S (small): action-to-action pair or FP viewpoint shots

– F (fixed): images with speech balloon

– N (neutral): all others

• Image processing– Mean shift– Laplacian– K-means

21

• Example:

22

Comic Life Comic Life (2005)

Application for creating comics:

loading images into comic panels

Authoring support:• Simple and Easy-to-use interface

– Drag in your pictures, captions, sound effects (‘ka-blam!') and speech balloons and it’s done!

– Layout templates– Filters like hand-drawn, painted

and night vision

• More freedom to create comic but still some restrictions

Reference:http://plasq.com/comiclife-win

23

• Comic Life 1.3

24

• Example:

25

Manga StudioManga StudioApplication for creating comics: drawing, laying out comic and styling comics

Authoring support:• High versatility and flexibility to

create comics – A large collection of special effects – Sketch, ink, and color artworks

without the need for any other graphics applications

• Interface similar to Photoshop

Reference:http://my.smithmicro.com/mac/manga/index.html

26

OurOur ProposalProposal1. Automation support

• Frame selection• Layout computing

2. Authoring support• Comic layout• Graphic Effects• Internet support

3. User interface

27

Comic GeneComic Gene (2009)

It automatically summarize players’ interactions (logs and screenshots)

in virtual world into comics.

Automation support:• Recording log and shots (Addon)• Frame selection• Comic layout computing• Rendering comics

Authoring support:• Editing log files and screenshots• Comic layout

– Option setting– Changing size and position of the shots

Reference:http://mmnet.iis.sinica.edu.tw/publication_detail.html?key=chan09_comics

28…How to improve!

• Prototype

29

1. Automation Support1. Automation Support

• Automatically summarize logs and frames into comics– Log: events– Frame: video, photos, screenshots and so on

• Definition of important events– Frequency, weight

– Significance score

30

ChallengesChallenges• Technical side

– Choose significant frames using machine learning– Image understanding– Special effect placement (font size, color, location)

• Chat balloons• Onomatopoeic sounds

• Design side– Comic-like style

• Page layout– Essential factors of good comic books

• Shooting• Zoom in/out• Level of detail • Timing control

31

• Frame selection– How many images does a comic strip need?– Is the criterion of significance score enough?

• Layout computing– Algorithm– Detection

32

2. Authoring Support2. Authoring Support

• Page properties– How many images in a comic/page/row/… ?

• Important event/frames

• Comic layout– Panel layout

• Border shape, bleed• Panel size

– Speech balloon– Sound effects

33

• Graphic effects– Templates– Cartoon-like design

• Cartoon-like filter• Motion line

• Internet support– Uploading comics to individual album or blogUploading comics to individual album or blog

34

3. User Interface Design3. User Interface Design

• User-friendly Features– Navigation bar– Easy-to-use

• Click, wheel• Drag-n-drop

35

UI Example: VideolyzerUI Example: Videolyzer

Navigation barNavigation bar

36

UI Example: Comic LifeUI Example: Comic Life

Drag-n-dropDrag-n-drop

37

Current PrototypeCurrent Prototype

38

Sheng-Wei (Kuan-Ta) ChenSheng-Wei (Kuan-Ta) Chen

swc@iis.sinica.edu.twswc@iis.sinica.edu.twhttp://www.iis.sinica.edu.tw/~swchttp://www.iis.sinica.edu.tw/~swc

Contact infoContact info

39

Thank youThank you