melt - semantics 2020...data and web science group, university of mannheim...
TRANSCRIPT
MELTMatching EvaLuation Toolkit
SEMANTiCS 2019 – Karlsruhe – 2019/09/11
Sven Hertling; Jan Portisch; Heiko Paulheim
1
Joint Work
2
Heiko PaulheimData and Web Science Group, University of Mannheim
Sven HertlingData and Web Science Group, University of Mannheim
Jan PortischData and Web Science Group, University of Mannheim / SAP SE
Agenda
• Motivation
• What is MELT?
• Usage Example
• Q&A
3
MOTIVATION
4
Ontology Alignment Evaluation Initiative (OAEI)
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 5
Ontology Alignment Evaluation Initiative• running campaigns since 2005• structured in tracks (similar to task
sets)• researchers submit their
implementation• centrally evaluated by track
organizers• results published
Tooling
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 6
Semantic Evaluation at Large Scale• (among others) packaging and
evaluation• OAEI support since 2010
Holistic Benchmarking of Big Linked Data • (among others) packaging and
evaluation• OAEI support since 2017• OAEI 2018: 6/19 matchers support
HOBBIT
Pain Points
• Limited evaluation capabilities in SEALS, HOBBIT, and the Alignment API
• No easy-to-use parameter tuning
• Packaging process might be complicated for new entrants to the community
• Tooling Java-focused (no Python)
• Implementation of the Alignment API not maven-based
• Tool breaks
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 7
What is MELT?
8
What is MELT?
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE
9
• Facilitate matcher packaging• Facilitate matcher
submission
• Easy matcher development• Non-Java matcher
development• Maven support
• Advanced evaluation capabilities • Evaluation before packaging• Allow for interactive visualization
• Allow for parameteroptimization
• Streamlined development process• Integration with existing tooling• OAEI support• Extensibility
What is MELT?
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 10
Matcher Development
Matcher Fine-Tuning
Matcher Evaluation
Matcher Submission
What is MELT?
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 11
Matcher Development
Matcher Fine-Tuning
Matcher Evaluation
Matcher Submission
Yet Another Alignment API (YAAA)
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 12
Full SEALS/HOBBIT Support
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 13
External Matching
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 14
• Simple wrapping
• Packageable for HOBBIT and SEALS
• Matcher can still be evaluated in MELT
• Documentation and demo project available on GitHub
What is MELT?
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 15
Matcher Development
Matcher Fine-Tuning
Matcher Evaluation
Matcher Submission
Parameter Tuning
• Run matcher configurations in parallel (i.e., multi-threaded)
• Hand over ExecutionResultSet to Evaluator and pick best value according to what you want to optimize
• Out-of-the-box classes that assist you
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 16
What is MELT?
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 17
Matcher Development
Matcher Fine-Tuning
Matcher Evaluation
Matcher Submission
Evaluation
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 18
Track track = TrackRepository.Multifarm.getSpecificMultifarmTrack("ar", "cn");
ExecutionResultSet ers = new ExecutionResultSet();
ers.addAll(Executor.run(track.getTestCases(), new Matcher(), "Matcher"));
EvaluatorCSV evaluatorCSV = new EvaluatorCSV(ers);
evaluatorCSV.write();
Evaluation
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 19
Track track = TrackRepository.Multifarm.getSpecificMultifarmTrack("ar", "cn");
ExecutionResultSet ers = new ExecutionResultSet();
ers.addAll(Executor.run(track.getTestCases(), new Matcher(), "Matcher"));
EvaluatorCSV evaluatorCSV = new EvaluatorCSV(ers);
evaluatorCSV.write();
Full OAEI support: All tracks available (one-time automated download)
Evaluation
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 20
Track track = TrackRepository.Multifarm.getSpecificMultifarmTrack("ar", "cn");
ExecutionResultSet ers = new ExecutionResultSet();
ers.addAll(Executor.run(track.getTestCases(), new Matcher(), "Matcher"));
EvaluatorCSV evaluatorCSV = new EvaluatorCSV(ers);
evaluatorCSV.write();
Full OAEI support: All tracks available (one-time automated download)
Multiple evaluators available, extensible.
Exemplary Evaluation
“Show me the false positive class-class mappings for Multifarm on track en-de for matcher WiktionaryMatcher.”
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 21
Exemplary Evaluation
“Show me the false positive class-class mappings for Multifarm on track en-de for matcher WiktionaryMatcher.”
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 22
Exemplary Evaluation
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 23
“Show me the false positive class-class mappings for Multifarm on track en-de for matcher WiktionaryMatcher.” → Just filter the correspondences!
More Evaluation
Filter for Scores or Correspondences
• Micro Average Precision and Macro Average Precision over track Conference
• All residual true positives for track Anatomy
• Macro Average Class-F1 for all tracks
• …
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 24
What is MELT?
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 25
Matcher Development
Matcher Fine-Tuning
Matcher Evaluation
Matcher Submission
Matcher Submission
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 26
pom
xml
(fill out a template –no maven wizardry required)
> mvn install
> mvn deploy
• Creates SEALS zip file → can be submitted right away
• Creates HOBBIT docker container
• Uploads and deploys container directly in HOBBITautomatically
Usage Example
27
Usage Example
Motivation
“Which matchers are interesting candidates for combination?”
→ Matchers with a high F1 score and high “diversity”.
28MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE
Quantitative Analysis I
Analysis OAEI 2018 results for Conference and Anatomy: Jaccard overlap of Alignments rendered as heatmap in LaTex.
Executor.loadFromAnatomyResultsFolder(“myPath”);
// few lines of other code (available on GitHub as
// example)
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 29
Results for Anatomy
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 30
Results for Conference
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 31
Quantitative Analysis II
Mean Absolute Deviation (MAD) of Similarities plotted against F1.
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 32
Results for Anatomy
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 33
Results for Anatomy
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 34
Results for Conference
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 35
There is MUCH more to MELT
Ontology Caching Services
36
Matcher Pipelining
Baseline Matchers
One-Time Auto-Downloadof OAEI Tracks
Automatic Reading of OAEI Result Alignments
OAEI-Track Organizer Tools
TRY IT!
Execution of SEALS packages from within MELT
Alignment Extensions
Multi-Threaded Matcher Execution
ExecutionResultIndexing
Alignment Refiners
Thank you!
Sven HertlingData and Web Science Group, University of [email protected]
Jan PortischData and Web Science Group, University of [email protected]
Heiko PaulheimData and Web Science Group, University of [email protected]
37