iiif curation platform: user-driven image sharing with machine...
TRANSCRIPT
![Page 1: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/1.jpg)
IIIF Curation Platform: User-Driven Image Sharing with Machine Learning-Based Image Annotation
Asanobu KITAMOTODirector, Center for Open Data in the Humanities (CODH) andNational Institute of Informaticshttp://codh.rois.ac.jp/@rois_codh
2019/6/27 2019 IIIF Conference 1
Co-authors:Jun Homma (Flx Style)Tarek Saier (NII Internship Student, Albert Ludwigs University of Freiburg)
![Page 2: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/2.jpg)
Important Messages
1. IIIF Curation Platform is developed by ROIS-DS Center for Open Data in the Humanities (Japan).
2. Treat “IIIF content as machine-actionable data,” not “IIIF content just for human viewing.”
3. “User-driven” IIIF content adds creativity and diversity to the “provider-driven” IIIF world.
4. Gateway to external services enriches possible actions on IIIF content, including machine learning.
2019/6/27 2019 IIIF Conference 2
![Page 3: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/3.jpg)
IIIF Curation Viewer (for Timeline)http://codh.rois.ac.jp/software/iiif-curation-viewer/
2019/6/27 2019 IIIF Conference 3
Developed on Leaflet-IIIF
![Page 4: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/4.jpg)
2019/6/27 2019 IIIF Conference 4
Facial expression data: Portrait http://codh.rois.ac.jp/curation/exhibition/2/index.html.enCredit: Yale Center For British Art
Sixteenth Century Seventeenth Century
![Page 5: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/5.jpg)
Good Old Analogue World
2019/6/27 2019 IIIF Conference 5
Scissors Paste
Source: いらすとや, http://www.irasutoya.com/
1 2
![Page 6: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/6.jpg)
Frictionless Digital World
2019/6/27 2019 IIIF Conference 6
1. Draw a rectangle.
2. Add to favorites. 1
2
![Page 7: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/7.jpg)
Crop and Curate Pattern
1. ‘■’ (13) is a “crop” button. You can draw a rectangle to specify a part of image.
2. ‘☆’ (6) is a “favorite” button. You can collect any images.
3. These buttons are the minimum requirement to create a new curation.
2019/6/27 2019 IIIF Conference 7
![Page 8: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/8.jpg)
Curathon!http://codh.rois.ac.jp/tutorial/dhsi-2019/
• “Curathon” is an event, like an “ideathon,” to enjoy creativity in an open environment.
• Participants are asked to create their own curations based on their motivation and interests.
• Present the result in front of other people.
2019/6/27 2019 IIIF Conference 8
At Digital Humanities Summer Institute 2019 (June 3-7 at Victoria, Canada), 11 participants, including scholars in Asian studies and subject librarians, are asked to create curations within two days as homework.
![Page 9: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/9.jpg)
2019/6/27 2019 IIIF Conference 9
Storytelling
Part-of-image research data
![Page 10: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/10.jpg)
2019/6/27 2019 IIIF Conference 10
Theme-based collection
Asking a new questionPersonal narrative
![Page 11: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/11.jpg)
User-Driven Interpretation
• Participants showed a different viewpoint on the interpretation of the content.
• Some participants found something new or started to ask a new research question.
• Explicit data structuring practice clarifies the interpretation of the content.
• Sharing user-driven interpretation enriches the world of IIIF.
2019/6/27 2019 IIIF Conference 11
![Page 12: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/12.jpg)
Providers vs. Users
Provider-Driven• Memory institutions
provide IIIF content. • Users access IIIF
content in the original structure of the content.
• Users cannot add new information to the original IIIF content.
User-Driven• Create derivative
content from the original content.
• Derivative content is itself self-contained.
• Curation is about user-generated content in the world of IIIF.
2019/6/27 2019 IIIF Conference 12
![Page 13: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/13.jpg)
Manifest and Curation
Manifest from providers Curation by users
2019/6/27 2019 IIIF Conference 13
1. Curation node is located outside of the structure of Manifest for the flexible grouping of canvases.
2. Redundant copying of the Manifest is not required, hence robust to changes. Physical containers such as books.
![Page 14: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/14.jpg)
Extension to Presentation API 2• Curation is another
upper element of Range.
• Range points to Manifest using “within” property.
• Curation is an array of ranges across multiple manifests.
2019/6/27 2019 IIIF Conference 14
Curation
withinselections
selections
![Page 15: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/15.jpg)
Client: User Tools
Server: IIIF resources
Client: Manager Tools
1
2 3
4
56
78
9
ICViewer
ICFinder
JSONkeeper
Canvas Indexer
ML Services
IIIF Server
ICManager
ICEditor
ICPlayer2019/6/27 2019 IIIF Conference 15
IIIF Curation Platformhttp://codh.rois.ac.jp/icp/
![Page 16: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/16.jpg)
Collection of Facial Expressionshttp://codh.rois.ac.jp/face/
Curation of the same person appearing in different Japanese Emaki handscroll across IIIF services.2019/6/27 2019 IIIF Conference 16
Tag: 「牛若丸」 (https://en.wikipedia.org/wiki/Minamoto_no_Yoshitsune)
![Page 17: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/17.jpg)
Facet Search by Human Metadatahttp://codh.rois.ac.jp/software/iiif-curation-viewer/
2019/6/27 2019 IIIF Conference 17
Click the tag, and get the list of a part of canvases annotated by the tag.
![Page 18: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/18.jpg)
Fine-to-coarse or from Content to Context
2019/6/27 2019 IIIF Conference 18
![Page 19: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/19.jpg)
Annotation by Machine Learning
2019/6/27 2019 IIIF Conference 19
1. Correct or not correct, that is not the question.
2. Serendipity: unexpected encounter with images.
Tag: Circle
![Page 20: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/20.jpg)
North China Railway Archivehttp://codh.rois.ac.jp/north-china-railway/
2019/6/27 2019 IIIF Conference 20
![Page 21: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/21.jpg)
Automatic Tagginghttp://codh.rois.ac.jp/north-china-railway/
2019/6/27 2019 IIIF Conference 21
![Page 22: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/22.jpg)
Automatic Colorizationhttp://codh.rois.ac.jp/north-china-railway/
2019/6/27 2019 IIIF Conference 22
![Page 23: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/23.jpg)
Choice of Imageshttp://codh.rois.ac.jp/north-china-railway/
2019/6/27 2019 IIIF Conference 23
Oa:Choice for switching multiple representations of the same object.
![Page 24: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/24.jpg)
Museum Exhibitionhttp://codh.rois.ac.jp/iiif-curation-platform/
2019/6/27 2019 IIIF Conference 24
Special exhibition at the musem of art, Kochi. Collaboration with Prof. Akihiko Takano (NII) and Rensou Shuppan.
![Page 25: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/25.jpg)
Curation and Offline to Online
2019/6/27 2019 IIIF Conference 25
Curation for a smartphone-based “museum guide” via the QR code in the museum, and IIIF Curation Player showing (curatorial) descriptions about the work.
いらすとや
![Page 26: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/26.jpg)
Gateway to External Services
2019/6/27 2019 IIIF Conference 26
After cropping, what shall we do?
1. Crop and Curate Pattern
2. Crop and Export Pattern
![Page 27: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/27.jpg)
ICP Service Repositoryhttp://codh.rois.ac.jp/icp/service-repository/
1. AI Kuzushiji Recognition (Single Character) (ROIS-DS Center for Open Data in the Humanities)
2. Harimazejo Similar Image Search Prototype (UTokyo Digital Archives Project)
3. Ukiyo-e Search (John Resig, ukiyo-e.org)4. Google Images (Google)2019/6/27 2019 IIIF Conference 27
1. Share the endpoint setting (a JSON file).2. Select the endpoint to export the cropped
image to the external service.
![Page 28: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/28.jpg)
AI Kuzushiji OCR (Single)http://codh.rois.ac.jp/char-shape/app/single-mobilenet/
2019/6/27 2019 IIIF Conference 28
Character recognition is integrated into ICViewer using an
iframe pop-up window running Tensorflow.js (deep learning library).
![Page 29: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/29.jpg)
Harimazejo Similar Image Search Prototype (UTokyo Digital Archives Project)
2019/6/27 2019 IIIF Conference 29
IIIF Image from National Diet Library
Search similar images indexed in UTokyoDigital Archives.
![Page 30: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/30.jpg)
Ukiyo-e Search (John Resig, ukiyo-e.org)
2019/6/27 2019 IIIF Conference 30
IIIF Image from Keio University
Search similar prints indexed in ukiyo-e.org
![Page 31: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/31.jpg)
Google Images (Google)
2019/6/27 2019 IIIF Conference 31
![Page 32: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/32.jpg)
Data-Driven IIIF Viewers
• Suitable for human viewing (reading).
• Physical materials and their typical usage are transferred to the digital space.
• Treat content as machine-actionable data.
• Explore the new potential of data-driven research.
2019/6/27 2019 IIIF Conference 32
Most IIIF Image Viewers
![Page 33: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/33.jpg)
ICP Tutorial (Ja / En)http://www.ch-suzuki.com/icpt/
2019/6/27 2019 IIIF Conference 33
![Page 34: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/34.jpg)
Important Messages
1. IIIF Curation Platform is developed by ROIS-DS Center for Open Data in the Humanities (Japan).
2. Treat “IIIF content as machine-actionable data,” not “IIIF content just for human viewing.”
3. “User-driven” IIIF content adds creativity and diversity to the “provider-driven” IIIF world.
4. Gateway to external services enriches possible actions on IIIF content, including machine learning.
2019/6/27 2019 IIIF Conference 34
![Page 35: IIIF Curation Platform: User-Driven Image Sharing with Machine …agora.ex.nii.ac.jp/~kitamoto/research/publications/iiif19-ppt.pdf · Manifest and Curation. Manifest from providers](https://reader033.vdocuments.mx/reader033/viewer/2022052801/5f1225bd9356b0600e582e9f/html5/thumbnails/35.jpg)
Acknowledgment
• Mr. Jack Reed for the development of Leaflet-IIIF.
• Mr. Chikahiko Suzuki (CODH) for creating curations and ICP tutorials.
• DHSI 2019 participants for creating their own curations.
2019/6/27 2019 IIIF Conference 35
IIIF Curation Platformhttp://codh.rois.ac.jp/icp/