The Third International Conference on Information Systems and Technologies  

ICIST 2013March 22 – March 24, 2013 - Tangier, Morocco

Position Paper: A New Approach for

Human Assessment of Ontologies

Authors:Leila ZEMMOUCHI-GHOMARI, l_ghomari@umbb.dz

UMBB, M’hamed Bouguerra University Boumèrdes, www.umbb.dzBoumèrdes, ALGERIA

&Abdessamed Réda GHOMARI, a_ghomari@esi.dz

LMCS LaboratoryESI, national Superior School of Computer Science, www.esi.dz

Algiers, ALGERIA

2

OUTLINE

ONTOLOGY EVALUATION

RELATED WORK

PROPOSED APPROACH

CASE STUDY

3

Why ontology evaluation ?

Involved in Selection of an ontology with regard to Objectives of use or reuse Several ontologies: suitable ontology Single ontology: Quality of ontology (good or

bad quality ontology) Involved in an ontology engineering process

Ontology evaluation is a crucial step in this process (at the end or through the whole process)

4

ONTOLOGY EVALUATION TYPES[Gomez-Perez, 2004]

ONTOLOGY VERIFICATIONDeals with building the ontology correctly

ONTOLOGY VALIDATIONDeals with the correspondence between the semantics of the model and the real world for which the ontology was designed

5

ONTOLOGY VALIDATION APPROACHES

Comparison with a gold standard or a reference ontology

Comparison with a source of data Application based-ontology assessment Human assessment

ontology developer

end-user

domain expert

6

RELATED WORK

Human assessment of ontologies fits into

ontology verification area. It is intended to detect mistakes and inconsistencies that occur with human modeling.

For example: in [Ceusters and Smith, 04, 05]:

NCI (National Cancer Institute thesaurus)

SNOMED (Systematized Nomenclature of Medicine)

7

RELATED WORK

Human assessment of ontologies fits into

ontology verification area. It is intended to detect mistakes and inconsistencies that occur with human modeling.

For example: in [Ceusters and Smith, 04, 05]:

NCI (National Cancer Institute thesaurus)

SNOMED (Systematized Nomenclature of Medicine)

missing or inappropriately allocated informal and formal

definitions shifts in terms meaning and redundancy in concepts.

8

Scope of this presentation

ontology validation area which relates the degree of correspondence of the ontology to that part of reality which it is designed to represent from the point of view of domain experts.

9

Why is ontology human assessment difficult?

Some quality attributes judged by domain experts, such as clarity, relevance and accuracy can be difficult to evaluate as they may not be easily quantifiable

10

PROPOSED APPROACH

11

STEP 1: DERIVATION OF QUESTIONNAIRE FROM ONTOLOGY

Ontology expressed in a web language (RDF, OWL)

Questionnaire expressed in natural language composed of four parts:

Hierarchical ontology levels (ontology depth) Axioms Relations between concepts Descriptive attributes of concepts

12

We propose possible answers to questions, we rely on principles mandatory for good quality ontologies:

Clarity: ontology is easily understood by the users so that it can be consistently applied and interpreted across the domain of interest

Lawfulness : knowledge described by the ontology is encoded

with meaningful terms. This is achieved by checking out that the words used by the ontology are appropriate

Accuracy: claims an ontology makes are right or wrong

Relevance: ontology satisfies ontology requirements or not

13

Criterion Possible Answers Answers’ Location

Clarity Not Clear Classes validation Axioms validation Relations validation Attributes validation

Lawfulness

Right but another term would be more appropriate Relations validation

Relevant but used terms are not appropriate

Attributes validation

Accuracy

Right Wrong

Classes validation Relations validation

Always Sometimes Never

Axioms validation

Relevance Relevant Not really relevant Not relevant at all

Attributes validation

14

THE OTHER STEPS

STEP 2 (Aggregation of questionnaire results) is performed automatically by web form module (like drupal webform)

STEP 3 & STEP 4 (Analysis and Synthesis of obtained results & questionnaire update):

Delphi method [Dalker & Helmer, 1963]: its purpose is to achieve convergence of opinions of experts concerning a specific topic using questionnaire.

Generally, 3 iterations of updated questionnaire are sufficient to reach a consensus

15

CASE STUDY we built an ontology called HERO ontology which

stands for “Higher Education Reference Ontology” we derived a questionnaire (100 questions) from

ontology elements and proposed MCQ as possible answers according to ontology quality criteria

16

17

18

19

QUESTIONAIRE AGGREGATED RESULTS

20

ONTOLOGY UPDATE

21

CONCLUSION

The purpose of this proposal is to define a methodological baseline for human assessment of ontologies and to carry out a practical case study for its applicability

Limitations Much more experiences are needed about the

practical usage of proposed guidelines Semi-automatic support of the approach is

required

22

REFERENCES

A. Gomez-Pérez, Ontology Evaluation, Handbook on Ontologies, pp 251-274, 2004.

J. Brank, M. Grobelnik, and D. Mladenic, “A survey of ontology evaluation techniques”, Proceedings of Data Mining and Data Warehouses (SiKDD), 2005.

N.C Dalkey, and O. Helmer, “An experimental application of the Delphi method to the use of experts”, Management Science, 9 (3), pp 458-467, 1963.

More references are included in the paper