detection of comic characters in comic books and manga

Click here to load reader

Post on 27-Jun-2015

248 views

Category:

Science

3 download

Embed Size (px)

DESCRIPTION

State of the art methods for comic character (people) localization in comic book images and manga.

TRANSCRIPT

  • 1. Detection of comic characters in comic booksand manga17/10/14 - christophe-rigaud.com

2. 217/10/14 - christophe-rigaud.comPresentationAdvisers Jean-Christophe Burie & Jean-Marc Ogier (L3i, La Rochelle, France) Dimosthenis Karatzas (CVC, Barcelona, Spain)eBDtheque project Comic book image processing and understanding Cultural heritage in many countries Paper digital comic book conversion enhancement New usages using the new technologiesApplications Text/image search, augmented reading, reflowing...FranceSpain 3. 3What researchers can do for the paper to digital transition?ResearchersPaper worldDigital world Digital world17/10/14 - christophe-rigaud.com 4. 4ResearchersImage credits: contributors of the eBDtheque dataset17/10/14 - christophe-rigaud.com 5. 517/10/14 - christophe-rigaud.comBalloon classificationSpeaker localizationText timestampsTone and pitchScene understandingLayout analysisPanel retrievalReflowingDe-hyphenationSpeech synthesisLanguage analysisTranslation assistancePose retrievalFacial expressionRelationship retrievalCharacter localization and recognitionResearchersPPAANNEELLSS 6. 617/10/14 - christophe-rigaud.comBalloon classificationSpeaker localizationText timestampsTone and pitchScene understandingLayout analysisPanel retrievalReflowingDe-hyphenationSpeech synthesisLanguage analysisTranslation assistancePose retrievalFacial expressionRelationship retrievalCharacter localization and recognitionResearchersTTEEXXTT 7. 717/10/14 - christophe-rigaud.comBalloon classificationSpeaker localizationText timestampsTone and pitchScene understandingLayout analysisPanel retrievalReflowingDe-hyphenationSpeech synthesisLanguage analysisTranslation assistancePose retrievalFacial expressionRelationship retrievalCharacter localization and recognitionResearchersBBAALLLLOOOONNSS 8. 8CCHHAARRAACCTTEERRSS17/10/14 - christophe-rigaud.comBalloon classificationSpeaker localizationText timestampsTone and pitchScene understandingLayout analysisPanel retrievalReflowingDe-hyphenationSpeech synthesisLanguage analysisTranslation assistancePose retrievalFacial expressionRelationship retrievalCharacter localization and recognitionResearchers 9. 9Challenges of comics analysis Variation of comic book styles Printing & scanning method changes over time (press, ink-jet, computer-based) Diversity of information to extract A lot of semantic relations to retrieve (element's relationships, text17/10/14 - christophe-rigaud.commeaning) 10. 11Summary of comic character extraction17/10/14 - christophe-rigaud.com Supervised approaches Character spotting given an example Unsupervised approaches Hypothesis from speech balloon tail position and orientation Region of interest from simple object removal Graphbasedapproach (Nam) 11. 12Summary of comic character extraction17/10/14 - christophe-rigaud.com Supervised approaches Character spotting given an example Unsupervised approaches Hypothesis from speech balloon tail position and orientation Region of interest from simple object removal Graphbasedapproach (Nam) 12. 13Character spotting given an example [5] Aim: retrieve all the instances (apparitions) of a given character in all thepages of a comic book title/album/collection17/10/14 - christophe-rigaud.com Method summary:1) Reduction of the number of colors2) Get the colors of the query example (user interaction)3) Compute the user query descriptor4) Spot similar character instances[5] C. Rigaud, D. Karatzas, J-C Burie, J-M Ogier Color descriptor for content-based drawing retrieval DAS'14 13. 141) Color reduction/quantizationN>10000 N=256 N=3217/10/14 - christophe-rigaud.com 14. 1517/10/14 - christophe-rigaud.com2) User query 15. 163) Color selection: occurrence level17/10/14 - christophe-rigaud.com 16. 173) Color selection: occurrence + discriminality levels17/10/14 - christophe-rigaud.comTop 3 colors 17. 184) Character spotting: example17/10/14 - christophe-rigaud.comUser inputImage credits: (Prunelle, la fille du cyclope, Ankama, 2010Descriptor 18. 194) Character spotting: examplecorrect detections wrong detections17/10/14 - christophe-rigaud.com 19. 2017/10/14 - christophe-rigaud.comConclusions Contributions Content Based Drawing Retrieval Color descriptor robust to scale,rotation, translation, deformation andocclusion Benefit of redundant information Limitations Colored comics Number of color (quantization) Requires one example of each characterQuery Correct WrongResult examples 20. 21Summary of comic character extraction17/10/14 - christophe-rigaud.com Supervised approaches Character spotting given an example Unsupervised approaches Hypothesis from speech balloon tail position and orientation Region of interest from simple object removal Graphbasedapproach (Nam) 21. 22Hypothesis from speech balloon tail position and direction Given balloons in a panel, where are the (speaking) characters?17/10/14 - christophe-rigaud.com 22. 2317/10/14 - christophe-rigaud.comExample 23. 2417/10/14 - christophe-rigaud.comExample 24. 2517/10/14 - christophe-rigaud.comConclusions Preliminary results Highly relies on the quality of thetail detection Only for speaking characters(others can be spotted) Implicitly retrieves therelationship between balloon andcharacter 25. 26Summary of comic character extraction17/10/14 - christophe-rigaud.com Supervised approaches Character spotting given an example Unsupervised approaches Hypothesis from speech balloon tail position and orientation Region of interest from simple object removal Graphbasedapproach (Nam) 26. 27Region of interest (ROI) from simple object removal Given an image from a double page of manga, can you predict where the17/10/14 - christophe-rigaud.comcharacters are? 27. 28Region of interest (ROI) from simple object removal Given an image from a double page of manga, can you predict where the? ?17/10/14 - christophe-rigaud.comcharacters are?? ?? ??? ??? ? 28. 29Region of interest (ROI) from simple object removal Given an image from a double page of manga, can you predict where the17/10/14 - christophe-rigaud.comcharacters are?? ?? ??? ??42%correct? ? ? ?Image credits: Yasuhisa Hara, "Kingdom", Shueisha, since 2006. 29. 30Region of interest (ROI) from simple object removal In the same image, knowing the position of panels and balloons, can youpredict where the comic characters are?17/10/14 - christophe-rigaud.com 30. 31Region of interest (ROI) from simple object removal Knowing the position of panels and balloons, can you predict where the?17/10/14 - christophe-rigaud.comcomic characters are?? ?????????? 31. 32Region of interest (ROI) from simple object removal?17/10/14 - christophe-rigaud.com Et voil !? ????????? ? 83%correct 32. 33A first test from automatic panels and balloonsextraction...17/10/14 - christophe-rigaud.com 33. 34Original image17/10/14 - christophe-rigaud.com 34. 35Image masked17/10/14 - christophe-rigaud.com 35. 36Image masked + distance transform17/10/14 - christophe-rigaud.com 36. 3917/10/14 - christophe-rigaud.comTo be continued... 37. 4017/10/14 - christophe-rigaud.comGraph-based approach Presented by Nam Le Thanh 38. 41Live demo: real time panels and balloons extraction17/10/14 - christophe-rigaud.com 39. 42https://github.com/crigaud/publication/tree/master/2014/TALK/intelligent_media_processing_group17/10/14 - christophe-rigaud.com