digital document retrieval - music.mcgill.ca€¦ · digital document retrieval using optical music...
TRANSCRIPT
Digital Document Retrieval using Optical Music Recognition
Andrew HankinsonJohn Ashley Burgoyne, Gabriel Vigliensoni, Alastair Porter, Jessica Thompson, Wendy Liu, Remi Chiu, Ichiro Fujinaga
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Retrieving digitized document images with content search
2
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 213
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
10.5 million total volumes5.5 million book titles3.6 billion pages~1 Trillion words472 terabytes
4
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
We need OMR tools built for large-scale music digitization
projects
5
Full-Text Search Systems
Full-Music Search Systems
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 218
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Flexible and robust OMR systems
8
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Flexible and robust OMR systems
Digital storage and representation of recognition results
8
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Flexible and robust OMR systems
Digital storage and representation of recognition results
Search systems for indexing and retrieval
8
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Liber Usualis Prototype
9
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Liber Usualis Prototype
◘ Roman Catholic liturgical service book
9
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Liber Usualis Prototype
◘ Roman Catholic liturgical service book
◘ Neume (square note) notation
9
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Liber Usualis Prototype
◘ Roman Catholic liturgical service book
◘ Neume (square note) notation
9
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Liber Usualis Prototype
◘ Roman Catholic liturgical service book
◘ Neume (square note) notation
9
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Liber Usualis Prototype
◘ Roman Catholic liturgical service book
◘ Neume (square note) notation
◘ Monophonic “Gregorian chant”
9
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Liber Usualis Prototype
◘ Roman Catholic liturgical service book
◘ Neume (square note) notation
◘ Monophonic “Gregorian chant”
◘ Modernized by the monks at Solesmes, France in the 19th C.
9
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Liber Usualis Prototype
◘ Roman Catholic liturgical service book
◘ Neume (square note) notation
◘ Monophonic “Gregorian chant”
◘ Modernized by the monks at Solesmes, France in the 19th C.
◘ 2340 page images
9
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 219
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
OMR Workflow
10
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
OMR Workflow
10
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
OMR Workflow
◘ Automatic layout analysis using Aruspix
10
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
OMR Workflow
◘ Automatic layout analysis using Aruspix
◘ Separate music and text layers
10
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 2111
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
OMR Workflow
13
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
OMR Workflow
13
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
OMR Workflow
◘ Music images were sent to an OMR system (Gamera)
13
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
OMR Workflow
◘ Music images were sent to an OMR system (Gamera)
◘ Text was sent through an OCR process (OCRopus)
13
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
OMR Workflow
◘ Music images were sent to an OMR system (Gamera)
◘ Text was sent through an OCR process (OCRopus)
◘ Automatic recognition; human correction (of music)
13
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
OMR Workflow
14
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
OMR Workflow
14
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
OMR Workflow
◘ Musical output encoded using the Music Encoding Initiative (MEI)
14
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
OMR Workflow
◘ Musical output encoded using the Music Encoding Initiative (MEI)
◘ Indexed with a search engine (Solr)
14
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
OMR Workflow
◘ Musical output encoded using the Music Encoding Initiative (MEI)
◘ Indexed with a search engine (Solr)
◘ n-gram (2–10) of pitch name sequences and image coordinates
14
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
OMR Workflow
◘ Musical output encoded using the Music Encoding Initiative (MEI)
◘ Indexed with a search engine (Solr)
◘ n-gram (2–10) of pitch name sequences and image coordinates
◘ Web application front-end
14
Document Image Retrieval
Document Image RetrievalOMR System
MusicNotation
Note: DLocation: X,Y } .mei
Document Image RetrievalOMR System
MusicNotation
Note: DLocation: X,Y
SearchEngine (Solr)
} .mei
Document Image RetrievalOMR System
MusicNotation
Note: DLocation: X,Y
SearchEngine (Solr)
Query: “DCDF”
} .mei
Document Image RetrievalOMR System
MusicNotation
Note: DLocation: X,Y
SearchEngine (Solr)
Query: “DCDF”
} .mei
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Demohttp://ddmal.music.mcgill.ca/liber
16
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Liber Usualis Outcomes
17
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Liber Usualis Outcomes
◘ Full-Music Search is possible.
17
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Liber Usualis Outcomes
◘ Full-Music Search is possible.
◘ Results correction is extremely time and labour intensive.
17
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Liber Usualis Outcomes
◘ Full-Music Search is possible.
◘ Results correction is extremely time and labour intensive.
◘ Desktop-based OMR so!ware does not scale to large projects.
17
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Liber Usualis Outcomes
◘ Full-Music Search is possible.
◘ Results correction is extremely time and labour intensive.
◘ Desktop-based OMR so!ware does not scale to large projects.
◘ What does “full-music search” actually look like?
17
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Future Developments
18
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Future Developments
◘ “Cloud”-based distributed OMR systems
18
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Future Developments
◘ “Cloud”-based distributed OMR systems
◘ “Pluggable” music notation recognition
18
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Future Developments
◘ “Cloud”-based distributed OMR systems
◘ “Pluggable” music notation recognition
◘ Crowdsourcing correction interfaces
18
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Future Developments
◘ “Cloud”-based distributed OMR systems
◘ “Pluggable” music notation recognition
◘ Crowdsourcing correction interfaces
◘ Search and analysis systems for music document image retrieval
18
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Conclusion
19
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Conclusion◘ OCR allows millions of books to be searched and
retrieved in an instant
19
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Conclusion◘ OCR allows millions of books to be searched and
retrieved in an instant
◘ Current OMR tools and techniques are insufficient for large-scale music document image retrieval
19
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Conclusion◘ OCR allows millions of books to be searched and
retrieved in an instant
◘ Current OMR tools and techniques are insufficient for large-scale music document image retrieval
◘ We demonstrated a full-music search prototype application
19
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Conclusion◘ OCR allows millions of books to be searched and
retrieved in an instant
◘ Current OMR tools and techniques are insufficient for large-scale music document image retrieval
◘ We demonstrated a full-music search prototype application
◘ We want to scale to millions of books, of all notation types.
19
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Our outstanding R&D team, past and present: Greg Burlet, Mahtab Ghamsari, Peter Henderson, Anton Khelou,
Jamie Klassen, Saining Li, Mikaela Miller, Catherine Motuz, Laura Osterlund, Laurent Pugin, Caylin Smith, Brian Stern
20
HANKINSON et al. Document Image Retrieval using Optical Music Recognition. ISMIR 2012, Porto, Portugal of 21
Thank you.http://ddmal.music.mcgill.ca
http://simssa.ca
21