Color Based Comic Strip Character Recognition!Joe Landry, Philip Lee, Jessica Maxey!

Department of Electrical Engineering, Stanford University

Introduction Recognition Methods – CHAR and SIFT

Related Work

Experimental Results

Today’s widespread access to smartphones provides an outlet for the enjoyment of comics. Automatically recognizing both the characters and text within a comic strip could allow everyone to experience comics through text-to-speech (regardless of their ability to physically see the comic strip). To achieve this, tens of thousands of comics need to be analyzed. Our project describes the first steps to accomplish this goal, by segmenting frames and recognizing characters in comic strips. !!We focused on the comic strip Garfield. We implemented both a new algorithm, Color Histogram Area Recognition (CHAR), and a standard SIFT matching approach.!

- Detection results computed using 100 three-frame comic strips!- Number of characters in test set:!

!- Overview of detection results:!

Rectification and Frame Isolation 1. Original Image! - Image taken by Droid camera! - Angle of rotation varies ± 20 degrees!

2. Image Rotation! - Image is locally thresholded to accentuate dark edges in the image! - Hough transform used to rotate image from the strongest peak!

3. Frame Isolation and Cropping! - Regions labeled and filtered according to bounding box area and aspect ratio. These frames are then cropped and passed to SIFT or CHAR.!

1. Frame Color Clustering! - Frames converted to LAB color space! - Prominent colors clustered into regions!

CHAR!

2. Isolation of distinct cluster regions! - Color clusters are isolated! - Open-Close applied to isolate blobs!

Future Work

3. Recognition of Garfield Characters! - Color histogram of region found! - Histogram compared with those known from training set to determine character!

1. Build a Vocabulary Tree ! - Find SIFT keypoints for a database of comics and generate a vocabulary tree !

SIFT!

2. Find Set of Best Matches! - Use vocabulary tree to find the database images that most likely match the input comic strip!

[1] Comaniciu, D.; Meer, P., "Robust analysis of feature spaces: color image segmentation," Computer Vision and Pattern Recognition, 1997. Proceedings., 1997 IEEE Computer Society Conference on , vol., no., pp.750,755, 17-19 Jun 1997 [2] Chung Ho Chan , Howard Leung , Taku Komura, “Automatic panel extraction of color comic images,” Proceedings of the multimedia 8th Pacific Rim conference on Advances in multimedia information processing, December 11-14, 2007, Hong Kong, China [3] Wenjing Jia; Huaifeng Zhang; Xiangjian He; Qiang Wu, "A Comparison on Histogram Based Image Matching Methods," Video and Signal Based Surveillance, 2006. AVSS '06. IEEE International Conference on , vol., no., pp.97,97, Nov. 2006

Garfield   Jon   Odie   Liz   Arlene   Nermal   Pooky  

273   165   20   13   3   0   0  

Correct  Detec:ons  

(SIFT)  

Correct  Detec:ons  (CHAR)  

Incorrect  Detec:ons  

(SIFT)  

Incorrect  Detec:ons  (CHAR)  

Missed  Detec:ons  

(SIFT)  

Missed  Detec:ons  (CHAR)  

163   289   71   414   278   78  

3. Locate Characters !- Find best transform to map database images to input comic!- Only keep matches that produce a high number of matches after the application of the transform!

- Implement edge-based character detection and determine any improvements in functionality - Test other classification methods (such as svm) - Implement an algorithm that combines features of both the CHAR and SIFT methods for character recognition