SpotTheLink: Playful Alignment of Ontologies

Stefan ThalerSTI Innsbruck

University of InnsbruckInnsbruck, Austria

stefan.thaler@sti2.at

Elena SimperlInstitute AIFB

Karlsruhe Institute ofTechnology

Karlsruhe, Germanyelena.simperl@kit.edu

Katharina SiorpaesSTI Innsbruck

University of InnsbruckInnsbruck, Austria

katharina.siorpaes@sti2.at

Keywordsontology alignment, casual games, incentives, motivators, OntoGame

1. INTRODUCTIONA large share of tasks in semantic-content authoring crucially rely

on human intelligence [4]. This holds for many aspects of ontologyengineering, but also for ontology-based annotation, be that for data-oriented resources, such as images, audio and video content, or forfunctionality, such as Web services and APIs. In previous work ofours we have extensively discussed the importance of motivatorsand incentive mechanisms to encourage a critical mass of Internetusers - in particular, users beyond the boundaries of the semantic-technologies community - to contribute to such inherently human-driven tasks. Through OntoGame1 we have provided a frameworkfor casual games which capitalizes on fun and competition as twokey motivators for people to willingly invest their valuable timeand effort in semantic-technologies-related tasks, whose technicaldetails hide behind an entertaining collaboratively game experience[2]. This paper presents the newest release of the OntoGame series,called SpotTheLink, which addresses this challenge in the area ofontology alignment.

Ontology alignment is undoubtedly one of the most active andmature area of research and development in semantic technologies[1]. A multitude of approaches that aim at automatizing the match-ing, merging and integration of ontologies, both at the schema andthe instance levels, have been proposed and successfully applied toresolve heterogeneity issues; more recently, the topic has received at-tention in the context of interlinking RDF-encoded data sets exposedover the Web as part of the Linked Open Data Cloud. Human inputremains a key ingredient of ontology alignment, as a valuable sourceof domain knowledge used either to train matching algorithms and todevelop the underlying knowledge base, or to validate and augmentautomatically computed results. SpotTheLink provides a means tosystematically harvest such human input as a side-product of an en-tertaining collaborative online game. In a nutshell, in SpotTheLinkarbitrarily chosen pairs of players interact with each other to identify

1http://ontogame.org

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.SAC’11 March 21-25, 2011, TaiChung, TaiwanCopyright 2011 ACM 978-1-4503-0113-8/11/03 ...$10.00.

mappings between concepts and instances of two (Semantic-Web)ontologies. Mappings have to be defined consensually. This offersan effective mechanism to validate the data collected during theoperation of the system, and to discourage unwanted behavior - asplayers are randomly selected and cannot communicate to each otherthrough other channels than the ones pre-defined in the game, in thelong run their winning strategy converges towards providing truthfuland correct solutions to the challenges raised in each game turn. Inaddition, collaboration among players proves to be a strong socialmotivator - just as in other casual games, the appeal of the game liesless in resolving the particular challenge, but in the ability to ’guess’what is on the other player’s mind and to reach a consensus withoutresorting to typical inter-human communication means.

Our game-based approach to ontology alignment is complemen-tary to (semi-)automatic tools and techniques as those previouslymentioned. In particular, in the case of SpotTheLink, the game isnot meant to compete against the wide range of ontology alignmenttools and techniques that have been presented in the literature overthe past ten years or longer, but to provide these with an alternativeway to gather input from a potentially very large base of non-expertusers.

2. SPOTTHELINK: A GAME FOR ONTOL-OGY ALIGNMENT

SpotTheLink is designed according to the generic principles pro-posed in [3]. It is developed on top of the OntoGame platform andis available online for free use by semantic-technology enthusiastsor casual-game addicts.2 Its goal is to relate concepts of two on-tologies to each other. In the particular instance discussed in thispaper, we match DBpedia concepts to concepts from the PROTONupper-level ontology, but the game can be easily configured forother scenarios, as well as for different ontology alignment prob-lems, such as the generation of training data sets in order to enableautomatic algorithms to autonomously run matching tasks, and thevalidation of matching results which have been previously computedautomatically.

When the player starts a game a random partner is negotiatedand assigned. This team-of-two has to collaboratively solve a seriesof challenges, in other words, each player will only get points ifthe team gives consensual replies to each challenge. Every gameround consists of two challenges: (i) first, the players are presentedwith a random concept from DBPedia, along with a description andan image (if available), and have to choose and agree on a relatedconcept from the PROTON ontology; (ii) then the players have toagree on the type of correspondence between these concepts.

In the example, the players have to choose whether the Film Festi-

2http://ontogame.sti2.at:8080/OntoGameServer

1711

val concept (on the left-hand side) is a Happening, an Abstract or anObject. If their answers are identical, they earn points. In the nextstep, they further specify the nature of this match, that is, whetherthe concept they just selected in PROTON is more specific or thesame as the given DBPedia concept (cf. Figure 1). For the relation-ships we have chosen SKOS mapping relations (skos:exactMatch,skos:narrowMatch and skos:broadMatch), however, we renamedthem for a clearer understanding for a broad audience of players.If both players choose to skip or if they do not agree on any of thetwo challenges, they can continue playing on a new concept. Other-wise, if a broader match has been identified on the PROTON side,the game is resumed for sub-concepts of this particular PROTONconcept.

Figure 1: Step 2 - Agree on a correspondence

Each game has a time limit of 3 minutes and can include anarbitrary number of challenges. A game round ends in the followingcases: players disagree on one of the challenges, players agreeto skip, or the deepest level of the inheritance hierarchy has beenreached. Pairs of players are computed randomly and anonymouslyfor each new game round.

The data collected across multiple game rounds is analyzed of-fline and the results are encoded is SKOS. The analysis targets onlyalignments which have been confirmed throughout multiple gamerounds by different pairs of players. More concretely, for thoseconcepts that have been played at least six times by different players,we calculate the percentage for each unique answer combination -a combination includes the pair of concepts and the type of corre-spondence between them. Combinations that include concepts froma lower level in the hierarchy are assigned a higher weight, becausethey implicitly state that the higher-level concept has been agreedupon as well. If the percentage of an answer combination exceeds50 percent, the concept match will be marked for export.

3. EVALUATIONTo evaluate SpotTheLink we analyzed the results of the individual

challenges and the associated alignments, and conducted structuredinterviews with some of the players to assess their game experience.The trial involved a group of 16 players with some expertise insemantic technologies. They played the game in the same time - in

order to simulate the gaming experience where several users can bepaired randomly - for around 20 minutes, without having receivedany additional information on the purpose or goal of the game. Thisaccounts for a total of 5 hours of labor, across 190 game roundsinvolving 246 DBPedia concepts. After completion of the gameplaying session, we analyzed the collected data with the help of twoontology-engineering experts from outside the SpotTheLink teamand interviewed the players.

The evaluation was conducted with a focused user group in orderto catch design flaws at a relatively early stage in the developmentprocess.

The 16 participants matched 32 of 246 DBpedia concepts to thePROTON ontology. The total number of inputs recorded throughoutthe evaluation session was 882. On average, each player produced23.5 matches during the playing session. 146 of the total 190 gamerounds were logged as consensual, which means that in more threeof four cases players were able to find consensus about matchingconcepts and associated correspondence. This confirms findingsfrom previous OntoGame evaluations [2].

After playing the game the participants filled out an online ques-tionnaire. From the results of the user experience analysis, we canconclude that that the two types of challenges the game is builtupon - concept selection and relationship selection - must be evenfurther simplified and better explained through examples and a self-contained interface. Fast-paced casual games require repetitive andextremely simple tasks, and the latter is still to be achieved. Fur-thermore, the way in which concepts and inheritance hierarchies arerepresented should be improved to foster immediate understandingof the purpose of the first challenge. The results on the correctnessof ontology matches confirms previous findings: the output of thegames for general knowledge problems is of high-quality. A greaterchallenge is to make the game interesting and fun to play.

4. CONCLUSIONIn this paper we presented the latest release of the OntoGame

framework, called SpotTheLink, which allows for the definition ofmappings between Semantic Web ontologies as part of an collabora-tive game experience. We describe the SpotTheLink game as wellas the results of an early evaluation. The insights serve as a basisfor further development of SpotTheLink as well as other games ofthe OntoGame series.

AcknowledgmentsThe work presented has been funded by the FP7 project INSEM-TIVES under EU Objective 4.3 (grant number FP7-231181).

5. REFERENCES[1] J. Euzenat and P. Shvaiko. Ontology Matching. Springer, 2007.[2] K. Siorpaes and M. Hepp. Games with a Purpose for the

Semantic Web. IEEE Intelligent Systems, 23(3):50–60, 2008.[3] K. Siorpaes and M. Hepp. OntoGame: Weaving the Semantic

Web by Online Games. In Proceedings of the EuropeanSemantic Web Conference ESWC2008, pages 751–766, 2008.

[4] K. Siorpaes and E. Simperl. Human intelligence in the processof semantic content creation. World Wide Web Journal, 13(1),2010.

1712