benchmarking the interoperability of odts. april 7th 2005 1 © raúl garcía-castro, asunción...
TRANSCRIPT
Benchmarking the interoperability of ODTs. April 7th 2005 1 © Raúl García-Castro, Asunción Gómez-Pérez
Benchmarking the interoperability of ontology development tools
Raúl García-Castro, Asunción Gómez-Pérez
<rgarcia,[email protected]>
April 7th 2005
Benchmarking the interoperability of ODTs. April 7th 2005 2 © Raúl García-Castro, Asunción Gómez-Pérez
Table of Contents
• The interoperability problem
• Benchmarking framework
• Experiment to perform
• Participating in the benchmarking
Benchmarking the interoperability of ODTs. April 7th 2005 3 © Raúl García-Castro, Asunción Gómez-Pérez
Ontology development tools interoperability problem
• It appears due to ontology reuse.
Tool 1
Tool 2
Tool 3
Tool 4
Tool 5
Ontology development tools
Potential functionalities
Real functionalities
Benchmarking the interoperability of ODTs. April 7th 2005 4 © Raúl García-Castro, Asunción Gómez-Pérez
Ontology development tools interoperability problem
• Why is it difficult?– Different KR formalisms
frames description logic conceptual graphs
first order logic semantic networks
– Different modelling components inside the same KR formalism
• Some results:– It is difficult to preserve the semantics and the intended
meaning of the ontology– Interoperability decisions…
• At many different levels• Usually hidden in the programming code of ontology
exporters/importersO. Corcho. A Layered Declarative Approach to Ontology Translation with Knowledge PreservationFrontiers in Artificial Intelligence and Applications, Volume 116, January 2005
Benchmarking the interoperability of ODTs. April 7th 2005 5 © Raúl García-Castro, Asunción Gómez-Pérez
Knowledge Model comparison
Classes
Template Slots/properties/instance attributes
InstancesData types
RDF(S)
Protégé-2000
Subclass-of
Subproperty-of
Literals
Containers
Collections
Statements
Own slots/Class attributes
PAL constraints/WAB axioms
Concept groups
Disjoint decompositions
Exhaustive decompositions
Partitions
Constants
Relation properties
Synonyms
Abbreviations
Bibliographic references
Metaclasses
Benchmarking the interoperability of ODTs. April 7th 2005 6 © Raúl García-Castro, Asunción Gómez-Pérez
Elements outside RDF(S) K.M.
Thesis
MSc Thesis PhD Thesis
Disjoint-subclass
Thesis
MSc Thesis PhD Thesis
subclass
Thesis
MSc Thesis PhD Thesis
subclass
Partial loss
Total lossDon’t export
Insert ad-hoc RDF(S)
<rdfsClass rdf:about=“#Thesis”> <a:disjoint rdf:Resource=“#MsC Thesis”> <a:disjoint rdf:Resource=“#PhD Thesis”></rdfs:Class>
Doesn’t import
Thesis
RDFSEXPORT IMPORT
Benchmarking the interoperability of ODTs. April 7th 2005 7 © Raúl García-Castro, Asunción Gómez-Pérez
Example: WebODE and Protégé
RDFS <rdf:Description rdf:about='#distanceToSkiResort'> <rdf:type rdf:resource=‘&rdf;Property'/> <rdfs:comment>The distance from the hotel to a ski resort</rdfs:comment> <rdfs:domain rdf:resource='#accommodation'/> <rdfs:range rdf:resource=‘&rdfs;Literal'/> </rdf:Description>
WebODE
RDF(S)
Protégé-2000
Protégé-2000
Benchmarking the interoperability of ODTs. April 7th 2005 8 © Raúl García-Castro, Asunción Gómez-Pérez
Example: WebODE and Protégé
WebODE
RDF(S)
Protégé-2000
Protégé-2000
RDFS<rdf:Property rdf:about=“#distanceToSkiResort“
a:maxCardinality="1“ a:minValue="0.0“ a:range="float“ rdfs:comment="The distance from the hotel to a ski resort“ rdfs:label="distanceToSkiResort">
<rdfs:domain rdf:resource=“#accommodation"/> <rdfs:range rdf:resource="&rdfs;Literal"/></rdf:Property>
Protégé-2000 generatesad hoc RDF(S) code
Benchmarking the interoperability of ODTs. April 7th 2005 9 © Raúl García-Castro, Asunción Gómez-Pérez
Table of Contents
• The interoperability problem
• Benchmarking framework
• Experiment to perform
• Participating in the benchmarking
Benchmarking the interoperability of ODTs. April 7th 2005 10 © Raúl García-Castro, Asunción Gómez-Pérez
Benchmark and benchmarking
Benchmarking• Systematic evaluation• Comparison with the best tools• Extraction of best practices
Benchmark Benchmarking
IS A Test Continuous process
PURPOSE
• Measure• Evaluate
• Search for best practices– Measure– Evaluate
• Improve
TARGET
• Method• System
• Process• Product• Service
Benchmarking the interoperability of ODTs. April 7th 2005 11 © Raúl García-Castro, Asunción Gómez-Pérez
General framework for benchmarkingBENCHMARKING ITERATION
Recalibration task
PLAN PHASE
1. B. goals identification 2. B. subject identification 3. Participant identification 4. B. proposal writing 5. Management involvement 6. B. partner selection 7. B. planning and resource allocation
IMPROVE PHASE
11. B. report writing 12. B. findings communication 13. Improvement planning 14. Improvement 15. Monitor
EXPERIMENT PHASE
8. Experiment definition 9. Experiment execution 10. Experiment results analysis
General evaluation criteria:• Interoperability• Scalability• Robustness
Benchmark suites for:• Interoperability• Scalability• Robustness
Benchmarking supporting tools:• Testing frameworks• Workload generators• Monitoring tools• Statistical packages
García-Castro, Maynard, Wache, Foxvog and González-Cabero. Knowledge Web Deliverable 2.1.4 Specification of a methodology, general criteria, and benchmark suites for benchmarking ontology tools. December 2004.
Benchmarking the interoperability of ODTs. April 7th 2005 12 © Raúl García-Castro, Asunción Gómez-Pérez
Plan phase
Benchmarkinggoals
identification
Benchmarkingsubject
identification
Participantidentification
Need for benchmarking
Organisation goals and strategies Benchmarking
proposal writing
Benchmarking goals, benefits, costs
Benchmarking subject, tool functionalities, evaluation criteria
List of involved members, benchmarking team
Management involvement
Benchmarking partner selection
Benchmarking planning and
resource allocation
Benchmarking proposal
Management support
Benchmarking partners, updated benchmarking proposal
Benchmarking planning
Organisation's tools
Tools from outside the organisation
Organisation planning
Improve the interoperability of ontol. development tools
RDF(S) import and export capabilities
Identify ontology components
exported/imported
B.P.
Benchmarking the interoperability of ODTs. April 7th 2005 13 © Raúl García-Castro, Asunción Gómez-Pérez
Experimentdefinition
Experimentexecution
Experimentanalysis
Benchmarking planning
Benchmarking proposal Experiment
definition, experimentation planning
Experiment results
Experiment report
Experiment phase
RDF(S) Import benchmark suitesRDF(S) Export benchmark suites
• test 1• test 2• test 3• ...
• test 1• test 2• test 3• ...
NOOKOK
• test 1• test 2• test 3• ...
OKNOOK
OKOKNO ...
• test 1• test 2• test 3• ...
• test 1• test 2• test 3• ...
NOOKOK
• test 1• test 2• test 3• ...
OKNOOK
OKOKNO
E.R.
Benchmarking the interoperability of ODTs. April 7th 2005 14 © Raúl García-Castro, Asunción Gómez-Pérez
Benchmarkingreport writing
Benchmarking findings
communication
Improvementplanning
Updated benchmarking proposal
Experiment report
Improvement
Benchmarking report
Updated benchmarking report
Monitor
Necessary changes, improvement planning, improvement forecast
Organisation support
Improved tool
Monitorisation report
Improve phase
• Comparative analysis• Compliance with standards• Weaknesses• Recommendations on tools• Recommendations on practices
Benchmarking the interoperability of ODTs. April 7th 2005 15 © Raúl García-Castro, Asunción Gómez-Pérez
Table of Contents
• The interoperability problem
• Benchmarking framework
• Experiment to perform
• Participating in the benchmarking
Benchmarking the interoperability of ODTs. April 7th 2005 16 © Raúl García-Castro, Asunción Gómez-Pérez
Benchmarking goalsGoal 1:• To assess and improve the interoperability of ontology development tools using
RDF(S) for ontology exchange.
Goal 2:• To identify the subset of RDF(S) elements that ontology development
tools can use to correctly interoperate.
Goal 3:• Next step: OWL.
Benchmarking the interoperability of ODTs. April 7th 2005 17 © Raúl García-Castro, Asunción Gómez-Pérez
Experiment to perform1. Export to RDF(S)• Check if ontology development
tools can export the core elements of RDF(S).
• Check if ontology development tools can export other elements of their knowledge models.
2. Import from RDF(S)• Check if ontology development
tools can import the core elements of RDF(S).
• Check if ontology development tools can import the non-core elements of RDF(S).
• Check if ontology development tools can import the other elements of the knowledge model the tools exported to RDF(S).
RDF(S)
RDF(S)
Benchmarking the interoperability of ODTs. April 7th 2005 18 © Raúl García-Castro, Asunción Gómez-Pérez
Extended Benchmarks
Sample export benchmarks
Core Benchmarks Common to every tool
Particular to each tool
Benchmark 3 Export concept
NL description Export an ontology containing 3 concepts and no further property or relation between them.
Graphical representation
Expected result <rdf:RDF
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#">
<rdfs:Class rdf:about="#concept1“/>
<rdfs:Class rdf:about="#concept2“/>
<rdfs:Class rdf:about="#concept3“/>
</rdf:RDF>
Concept 1 Concept 2 Concept 3
Benchmarking the interoperability of ODTs. April 7th 2005 19 © Raúl García-Castro, Asunción Gómez-Pérez
Sample export benchmarks
Benchmark 6 Export linear concept taxonomy
NL description Export an ontology containing 3 concepts with a subclass-of relation between each pair of them forming a linear taxonomy.
Graphical representation
Expected result <rdf:RDF
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#">
<rdfs:Class rdf:about="#Concept 1“>
</rdfs:Class>
<rdfs:Class rdf:about="#Concept 2“>
<rdfs:subClassof rdf:resource=“Concept 1”>
</rdfs:Class>
<rdfs:Class rdf:about="#Concept 3”>
<rdfs:subClassof rdf:resource=“Concept 2”>
</rdfs:Class>
</rdf:RDF>
Concept 1
Concept 2
Concept 3
Benchmarking the interoperability of ODTs. April 7th 2005 20 © Raúl García-Castro, Asunción Gómez-Pérez
Sample export benchmarks
Concept 1
Concept 2 Concept 3 Concept 4
Concept 1
Concept 2 Concept 3 Concept 4
Concept 1
Concept 2
Benchmark 7
Export a concept with multiple children
Export an ontology containing four concepts, being three of them subclass of the fourth one.
Benchmark 8
Export a concept with multiple parents
Export an ontology containing four concepts, one of them subclass of the other three.
Benchmark 9
Export two concepts subclass of each other
Export an ontology containing two concepts, each one subclass of the other.
Benchmarking the interoperability of ODTs. April 7th 2005 21 © Raúl García-Castro, Asunción Gómez-Pérez
Export process
Load ontology
Concept 1
Concept 2
Export ontology
<rdf:RDF
xmlns:rdf="http://www.w3.org/2-rdf-syntax-ns#"
<rdfs:Class rdf:about="#Concept 1“>
</rdfs:Class>
<rdfs:Class rdf:about="#Concept 2“>
<rdfs:subClassOf rdf:resource=“Concept 1”>
</rdfs:Class>
</rdf:RDF>
Compare result with expected
<rdf:RDF
xmlns:rdfs="http://www.w3.org/f-schema#">
<rdfs:Class rdf:about="#Concept 1“>
</rdfs:Class>
<rdfs:Class rdf:about="#Concept 2“>
<rdfs:subClassOf rdf:resource=“Concept 1”>
</rdfs:Class>
</rdf:RDF>
<rdf:RDF
xmlns:rdf="http://www.w3.org/2-rdf-syntax-ns#"
<rdfs:Class rdf:about="#Concept 1“>
</rdfs:Class>
<rdfs:Class rdf:about="#Concept 2“>
<rdfs:subClassOf rdf:resource=“Concept 1”>
</rdfs:Class>
</rdf:RDF>
= ?YES
NO
Steps can be manual or automatic
Export strategy: minimal
knowledge loss in exports
Benchmarking the interoperability of ODTs. April 7th 2005 22 © Raúl García-Castro, Asunción Gómez-Pérez
Export resultsWebODE Protégé Tool A Tool B Tool C …
Benchmark 1 YES YES YES YES YES
Benchmark 2 YES YES YES YES YES
Benchmark 3 YES YES YES YES YES
Benchmark 4 YES YES YES YES YES
Benchmark 5 YES YES YES YES YES
Benchmark 6 YES YES YES NO YES
Benchmark 7 YES YES YES YES NO
Benchmark 8 NO YES NO YES YES
Benchmark 9 NO NO YES YES YES
Benchmark 10 YES YES NO NO YES
Benchmark 11 YES YES YES NO NO
Benchmark 12 NO NO YES NO NO
…
Benchmarking the interoperability of ODTs. April 7th 2005 23 © Raúl García-Castro, Asunción Gómez-Pérez
Extended Benchmarks
Sample import benchmarks
RDF(S) Core Benchmarks Common to every tool
Common to every tool. From export extended benchmarks
Benchmark 2 Import class
NL description Import a graph containing 3 classes and no further property between them.
Graphical representation
RDF(S) source <rdf:RDF
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#">
<rdfs:Class rdf:about="#concept1“/>
<rdfs:Class rdf:about="#concept2“/>
<rdfs:Class rdf:about="#concept3“/>
</rdf:RDF>
Expected result
Class 1 Class 2 Class 3
RDF(S) Extended Benchmarks
Class 1 Class 2 Class 3
Benchmarking the interoperability of ODTs. April 7th 2005 24 © Raúl García-Castro, Asunción Gómez-Pérez
Sample import benchmarks
Benchmark 5 Import classes with a property
NL description Import a graph containing 2 classes and a property between them.
Graphical representation
RDF(S) source <rdf:RDF
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#">
<rdfs:Class rdf:about="#Person“>
<ns:Is_author rdf:Resource=“#Book”>
</rdfs:Class>
<rdfs:Class rdf:about="#Book“/>
</rdf:RDF>
Expected result
Person BookIs_author
Person BookIs_author
Benchmarking the interoperability of ODTs. April 7th 2005 25 © Raúl García-Castro, Asunción Gómez-Pérez
Import process
Load graph
<rdf:RDF
xmlns:rdf="http://www.w3.org/2-rdf-syntax-ns#"
<rdfs:Class rdf:about="#Concept 1“>
</rdfs:Class>
<rdfs:Class rdf:about="#Concept 2“>
<rdfs:subClassof rdf:resource=“Concept 1”>
</rdfs:Class>
</rdf:RDF>
Import graph
Class 1
Class 2
Steps can be manual or automatic
Compare result with expected = ?
YES
NO
Class 1
Class 2
Class 1
Class 2
Benchmarking the interoperability of ODTs. April 7th 2005 26 © Raúl García-Castro, Asunción Gómez-Pérez
Import resultsWebODE Protégé Tool A Tool B Tool C …
Benchmark 1 YES YES YES YES YES
Benchmark 2 YES YES YES YES YES
Benchmark 3 YES YES YES YES YES
Benchmark 4 YES YES YES YES YES
Benchmark 5 YES YES YES YES YES
Benchmark 6 YES YES YES NO YES
Benchmark 7 YES YES YES YES NO
Benchmark 8 NO YES NO YES YES
Benchmark 9 NO NO NO YES YES
Benchmark 10 YES YES NO NO YES
Benchmark 11 YES YES YES NO NO
Benchmark 12 NO NO YES NO NO
…
Benchmarking the interoperability of ODTs. April 7th 2005 27 © Raúl García-Castro, Asunción Gómez-Pérez
Table of Contents
• The interoperability problem
• Benchmarking framework
• Experiment to perform
• Participating in the benchmarking
Benchmarking the interoperability of ODTs. April 7th 2005 28 © Raúl García-Castro, Asunción Gómez-Pérez
Benchmarking benefits
• For the participants:– To know in detail the interoperability of their ODTs.– To know the set of terms in which interoperability between
their ODTs can be achieved.– To show the rest of the world that their ODTs are able to
interoperate and are among the best ODTs.• For the Semantic Web community:
– To obtain a significant improvement in the interoperability of ODTs.
– To know the best practices that are performed when developing the interoperability of ontology development tools.
– To obtain instruments to assess the interoperability of ODTs.– To know the best-in-class ODTs regarding interoperability.
Benchmarking the interoperability of ODTs. April 7th 2005 29 © Raúl García-Castro, Asunción Gómez-Pérez
Participating in the benchmarking
• Every organisation is invited to participate:– If you are developers, with your own tool.– If you are users, with your preferred tool.
• Supported by the Knowledge Web NoE.
• The results will be presented in the EON 2005 workshop.
Benchmarking the interoperability of ODTs. April 7th 2005 30 © Raúl García-Castro, Asunción Gómez-Pérez
Timeline
April 22nd 2005 First definition of export and import benchmarks.
May 6th 2005 Agreement on the benchmark definitions and the experimentation.
July 1st 2005 Results from Export experiments.
September 16th 2005
Results from Import experiments.
If you want to participate in the benchmarking or have some further question/comment about it, please contact:
Raúl García-Castro <[email protected]>
Benchmarking the interoperability of ODTs. April 7th 2005 31 © Raúl García-Castro, Asunción Gómez-Pérez
KW Deliverable 1.2.2Deliverable 1.2.2: “Semantic Web Framework Requirements Analysis” • Analyse applications and tools for identifying the set of requirements for interoperation and
exchange of ontologies. • Identify the main components that an unified semantic web framework should have.
Should contain:• The main systems developed on the field.• For each application or system, include its architecture, design criteria, main components,
how they interoperate with other systems or exchange their ontologies.
These studies should allow you to identify: • The main and additional requirements for each type of tool/system.• The main and additional functionalities.• Results on evaluation of the requirements of interoperation and exchange. • Results on evaluation of other criteria like scalability, etc.• Summarize in a table such criteria in order to have a clear picture of the field.
Needs contributions from tool developers in order to obtain accurate descriptions of their tools.
Benchmarking the interoperability of ODTs. April 7th 2005 32 © Raúl García-Castro, Asunción Gómez-Pérez
Benchmarking the interoperability of ontology development tools
Raúl García-Castro, Asunción Gómez-Pérez <rgarcia,[email protected]>
April 7th 2005