A Tool Supporting Concept MapEvaluation and Scoring

Siddharth Ravichandran

November 29, 2010Master’s Thesis in Computing Science,30 ects credits

Supervisor at Cs-Umu : Jürgen BörstlerExaminer: Per Lindström

Umeå University,Department of Computing Science

SE -901 87 , Umeå SWEDEN

Abstract

Concept maps are a form of graphical knowledge representation [1]. Concept maps are drawn as directed or undirected graphs. Concept maps may carry a large and complex structure where concepts are described as a form of relationships between the nodes and the edges. Concept maps are widely used in teaching and learning where educational research and assessment are necessary functions. The common problem when dealing with concept maps is evaluation. This usually involves comparing concept maps for similarity. When developing a concept map the developer uses different ways and terminologies to describe the same concepts and relationships. Furthermore there is no commonly accepted way of scoring a concept map since not all scoring rules are suitable for all types of concept maps.

This thesis proposes a tool that reviews scoring techniques used for assessing the concept maps. A tool was developed to unify the terminology used in describing the concept and relationship in a concept map. The tool maps these terminologies into a common language and further scores it against a map master map. The tool also assesses concept maps based on different approaches of relational and structural scoring techniques as structure based scoring has shown better comprehension than scores based on concepts alone [9]. The tool produces the final score by summing up the scores of all these approaches. The tool is also user supported and is automated by the input prepositions given from the user.

The results suggest that the evaluation techniques used by the tool helps in assessing the concept map but the tool does not support all forms of concept maps. The tool shows that it can accommodate various aspects of scoring and has shown capacity to integrate with other tools like the word net and implement algorithms such as path finder. However the tool needs to be further extended to support assessment of hand written concept maps and other scoring schemes.

Contents

1 Introduction 11.1Introduction........................................................................................ 11.2 Problem Description..........................................................................3 1.3 Disposition.........................................................................................4 1.4 Goals………………………………………………………………..4

2 Literature Review and Background 62.1 Concept Map- Uses and Application..……………………………….6 2.2 Treatment by preposition…………………………………………..8 2.3 Terminology variations…………………………………………..82.4 Matching Problem- Master map vs student map…………………..8 2.5 Role of structure in Concept Maps ................................................10 2.6 Role of shapes in a concept map………………………………….12 2.7 Concept map scoring (in detail)…………………………………..14 2.8 Scoring by the tool (actual methods)..………………………….18

3 Basic Design and Implementation 21 3.1 Input Format……………………………………………………...213.2 Handling Prepositions………………............................................22 3.3 Design Overview and Architecture………………………………26 3.4 Application of scoring Algorithm………………………………..283.5 Package Diagrams………………………………………………..32

4 Results 38 4.1 Input constraints & Front end of the tool…………………………38 4.2 Repeatable Prepositions…………………………………………..404.3 Upload Window…………………………………………………..414.4 Holistic Overall Impression………………………………………424.5 Matching prepositions....................................................................434.6 Single Prepositions……………………………………………….474.7 Manually Resolve Window………………………………………494.8 Path Finder Algorithm and Shapes of Concept maps…………….504.9 Final Summary……………………………………...…………… 534.10 Evaluation of the Tool..................................................................53

5 Conclusion 57

6 Acknowledgment 58

References 59

List of Tables

2.1 Relational scoring ………………………………………………….. 16

2.2 Structural scoring …………………………………………………. 17

4.10 Metric Results…………………………………………………………. 54

1

Chapter 1

1.1 Introduction

Concept maps are graphical tools which are used for organizing and representing knowledge [1], mainly used in the areas of teaching. A concept within a concept map is usually represented in boxes or labels and is connected with labeled arrows. The label for the concept map can be a word or a + or % and sometimes more than the word itself. The relationship between concepts is represented as linking phrases between two concepts. In the figure 1.1 we note that “Represent”, “begins with “ and “is” are examples of relationships used to link concepts. The technique used in comprehending these relationships among different concept is called as “Concept Mapping”. A preposition contains two or more concepts connected using linking words or phrases to form a meaningful statement [3]. In the figure below” concept -> represent -> organized knowledge” is a meaningful statement which is a preposition from the concept map.

Figure 1.1: A classic concept map explaining notation and elements of a concept map, see [1].

2

A common characteristic of a concept map is concepts within a concept map are structured hierarchically with the most inclusive and general ones at the top of the structure and less general concepts hierarchically below [3].In concept mapping the learner identifies concepts, hierarchically organizes these concepts, differentiates between them and expresses more complex cross hierarchical relations. Cross links are relationships or links between concepts in different parts of the same map. However these characteristics may not be true in reality for all concept maps in general. These characteristics are true when we follow Nowak and his tools. In reality not always a concept map may follow a complex structure and may not always be hierarchical or possess cross links.

Concept maps are getting increasingly used in the areas of teaching and learning. They are very similar to mind maps. Concept maps are also used in areas of educational research, for capturing ideas and note making. Specifically in areas of education a concept map is usually used to explain a concept or a phenomenon for students to easily understand it [1, 3].Educational fields use concept maps for assessing the ability of student in understanding a topic .There are many resources to develop a concept map. The main ones are free mind, XMind, Compendium, IHMC and VUE [5, 6]. These tools are basically used to represent knowledge in the form of concept maps or mind maps. They have different features that help in drawing maps such as graph based links , exporting images to the maps ,icons on nodes , web and file hyperlinks to nodes , export maps into various file formats such as XML, JPEG. These concept mapping tools do not incorporate scoring of concept maps. They are usually restricted to drawing concept maps.

When we consider a collection of concept maps the key operations is to retrieve the maps, compare it for similarity and closeness with a master map [9]. For example instructors review student drawn maps by comparing them with expert knowledge [7]. This basically means comparing each concept map with other for similarity (when same concepts are involved). This becomes a nontrivial issue since different people use different ways to represent different structures when describing the same concepts and their relationships [2]. There is no rule in the use of terminologies used in the map and there is a clear distinction between one map builder with another in expressing knowledge while developing a concept map [3, 27]. This allows the map builder to build maps which are quite distinct in nature. So a common method to evaluate these maps is important since it helps in realizing one’s ability to understand a concept.

.In this thesis we develop a tool that evaluates concept maps by matching knowledge elements between concept maps, unifies terminology used in the map into a common language and then score the concept map based on certain scoring strategies [3][17]. This is interesting because different people exhibit different mental models in order to construct a map [3].

3

The tool is written completely in java. The main advantage of using java is ease of integration with other systems, prepares the tool to be extended in the future changes also is platform independent.

1.2 Problem Description

The problems that will be addressed in this thesis are divided as below:

Matching problems: While matching elements between two concept maps there are obvious comparisons between concepts and relationship nodes directly. The relationship names are usually problematic as it is often repeated more than one time in a concept map. A direct string matching algorithm between the 2 maps is not enough as we are not going to compare the elements string wise but we are going to consider the structure of the concept map as a whole. This means that direct string matching between concepts and relationship of two different maps may not yield fruitful results.

Variations in Terminology used by the map builder: Map builders often use different terminology to describe the same concepts or relationships. The terminology variations could pose as a serious problem since the map builder could express the same concept using a different term but finally mean the same as mentioned in the master map. The tool must look to map these similarities in terminology when matching the knowledge elements otherwise the comparison between the student map and the master map may not lead to accurate results.

Structural Differences in a concept map: .When the student is asked to draw a concept map on a particular phenomenon there are some complexities. Each student draws a map explaining the same concept but with a difference in representing the structure. The structure of the map varies from each student. A teacher could find it hard in this case to find out whether the student has understood the concept.

A concept map’s elements are represented in form of shapes that are graph based. They usually reflect the map builders mental structure in building the concept .The tool must find some ways to find the substructures used within a map. Graph theory algorithms that address the shape (list, tree, spoke, circular, network) of the concept map [9]. Since concept maps are graphs which represent knowledge in a unique way, a problem arises when there is a question about treating the structure of this map. To be precise there is a question of finding the path or the structure of the map.

Hand written concept maps: Concept maps that are drawn in a piece of paper should also be accommodated for evaluation by the tool. An approach to handle hand

4

drawn concept maps is needed since this tool is built for the purpose of handling concept map of any form.

Scoring Strategies: Scoring a concept map is assigning scores in the form of numbers based on certain scoring strategies. However there is no commonly accepted way of scoring concept maps. The literature however agrees on some approaches that are more or less subjective [9]. These scoring strategies are considering the closeness of a concept map with a master map, number of concepts, number of relationships, number of valid prepositions shape of concept maps and graph based measures. The weighted sum of all these approaches could be used as the final score.

These approaches are practical and can be coded for scoring these maps [2].However not all scoring rules is suitable for all types of concept maps. Nowak and Gowin’s scheme may be useful only when hierarchical levels are identified unambiguously. But in general the score assigned based on the structure of the concept map has given accurate results .So the problem is to find a suitable scoring method which is reliable and easy to use.

1.3 DispositionIn the following sections of the chapter the problem of the thesis are discussed, goals of the thesis are formulated. In chapter 2, supporting literature, similar tools that perform similar functions are reviewed and background of this thesis is furnished. It covers few but relevant strategies regarding the concept map scoring. It also helps in assimilating the reason behind scoring. In chapter 3 design and implementation of the tool is provided. The core data structure and scoring algorithms that are applied in building the tool is discussed. In chapter 4, we discuss the various test cases that were applied on the application divided by parts fulfilling all the possible criteria for testing. It also includes how each goal of the thesis was achieved reasonably. The results and evaluation of the tool are also discussed in the same chapter .It also covers the limitations of the tool and also future work. Chapter 5 consists of final conclusion of the thesis.

1.4 GoalsThe following list describes all the goals that were aimed when developing the tool:

To address differences in terminologies, match them clearly between two maps. Implement scoring strategies that help in realizing the amount of how well One understands the concept. Tool must be flexible and extensible and must accommodate more scoring strategies in future. Intuitive and easy to understand user interface.

5

The goal is to make the tool comprehensive as far as all scoring strategies are concerned and automate it. The tool must also be extendable for changes and it should be able to integrate with other tools. The user could bring varieties of prepositions which constitute the map builders knowledge. These varieties pose a huge challenge since there is no standard way of organizing, computing and scoring a concept map. The tool must also be able to organize each preposition in a standardized manner such that this tool processes them with less time. The next chapter reviews the literature and background of this thesis.

Note: All the concept maps drawn in this thesis are drawn using the concept mapping tool.

6

Chapter 2

Literature Review and Background

This chapter discusses the application of concept map, reviews the literature, addresses the issues of matching with supporting literature and analyzes the various scoring schemes that are suitable for the tool. The next section discusses the application of concept maps in common and specifically in the field of education.

2.1 Concept map – Uses and application. .Concept mapping techniques and structural organization of the concept maps shows meaningful learning, progressive difference in the core topic and integrative reconciliation [23][1][26]. Building concept maps try in identifying learners understanding by spotting key concepts and relationships. One of the most important uses of concept map in the field of education is to compare student map with the teachers map graphically [7].

In the field of education, students draw maps to express a particular concept or phenomenon [3]. The teacher further uses these maps to assess the student’s ability to understand the concept. Assessing these maps have shown considerable amount of success in understanding the ability of the student in expressing knowledge. However a student differs from another student in drawing maps by structure of maps he or she represents , by use of terminology in expressing concepts and also in shapes[27][3]. There must be a common method to assess all these maps taking into account the structural and terminological variations. They should also act as a platform for scoring these maps.

Figure 2.1 : a cricket concept map built for Student A.

7

For this reason, we assume two hand drawn concept maps described below in figure 2.1 and figure 2,2 to be two maps drawn by student A and student B. They focus on explaining the question “what are the important facts about cricket?” They differ in structure, there is variation in terminology in expressing the same concept and also differ in the shapes used in expressing the relationship.

Figure 2.2: a cricket concept map built for Student B.

Considering the distinct nature of concept maps a common form of evaluation is required to assess the concept maps drawn by students [2]. Studies have shown that evaluation of concept maps are very important since they reflect the map builders understanding [27] .

8

2.2 Treatment by prepositionPrepositions contain two or more concepts connected using linking words or phrases to form a meaningful statement [3]. In the figure 2.2 one of the preposition is cricket - >is played->by 22 players. This represents one meaningful statement which goes to explain that cricket is played by 22 players.

The studies previously made indicate the necessity of a tool that maps knowledge elements in concept maps. There is no written rule in naming the concepts and nodes when developing a concept map and they allow the map builder to express his or her own ideas on a wide variety of topics [27].Therefore tool must actually look to match elements in the form of prepositions.

2.3 Terminology variations

However previous studies have aimed at focusing on the overall map similarity without matching the elements. . A concept builder usually differs from other concept builders when he uses different terminologies in expressing the elements in a map [3][16]. For example in figure 2.1 we have the preposition cricket->includes->a batsmen and in figure 2.2 we have a preposition cricket->involves->a batsmen. They both actually mean the same but they are expressed in a different way.If the focus is completely based on overall map similarity then the evaluation tends to avoid the variations in terminology used by the map builder. Therefore this tool takes into account the idea of element matching rather than the overall similarity of the map. This feature however is observed to support assessment of concept map scoring [27][1].

This lead to necessity of an online dictionary or a tool which sorts out the problem of matching terminologies is very much required. An online dictionary like the word net was included to the tool and it was plugged in.

2.4 Matching problem -Master Map vs Student map

Previous studies have shown matching prepositions of a student map and a master or expert map created by experts or teachers can achieve better results in evaluating the maps. Chen , Lin and Chang evaluated student maps based on fuzzy algorithms and fuzzy based matching techniques which enlightened the importance of ranking concept nodes and relationship links between the concepts nodes. They constructed a master map from maps constructed by 3 experts and matched it with student maps using the fuzzy algorithm [27][23].

They observed some correlation between the performance of the student in a handwritten test and similarity between the student map and the master map. The correlation was more significant in the case of students who perform well and also for subject matters that were difficult. The accuracy went down when the organizational structure of these maps were distinct. They later concluded that matching techniques

9

must address the actual contextual representations rather than the whole map as an instance and also called for the need of preposition based matching mechanism .

For this reason we create a expert map for the concept map drawn in the figures 2.1 and figure 2.2. This map explains the same question “ what are the important facts about cricket?”. This map contains nearly perfect information about the question. We also take student map A drawn in figure 1 for comparison. A master map is assumed to have built with complete or nearly perfect understanding of the concept. A student’s map during the phase of analysis needs to require a map such as master map to find how similar his map has been with respect to perfection..

Figure 2.3: an expert map created for comparing with student map 1 and student map 2

When comparing the expert map and the student map concept wise there is a issue where one concept in the master map maybe similar to another concept in the student map but when it is represented as preposition it may vary[9]. Therefore it makes sense to compare each preposition from the master map with each preposition of the student map and find out how many prepositions are similar between these 2 maps.

Based on this assumption the prepositions for both the maps where automated from 2 two notepads and they were made to compare and checked for terminology variations. Studies seem to show that when concept map prepositions are taken and compared with maps created by experts they seem to show better results. Further studies show the importance of matching these techniques since concept maps are context based and they don’t have a restriction on the vocabulary used in building the tool [27][17].

10

.Figure 2.4: preposition of expert map.

Figure 2.5: prepositions of concept map developed by student A.

The prepositions of expert map and student map A were compared for similarity . There are 6 prepositions that seem to match between the student map and the master map. However, the similarity has to be further evaluated based on scores.

2.5 Role of structure in concept maps

Thomas Reichherzer and David leake (2006) [15] [17] analyzed the role played by structures in concept map. They developed models for analyzing concept maps. These models were divided in the form of four candidate models representing concepts as nodes in the concept map. The first 3 models consider map’s topology while the fourth model disregards the topology and considers each concept to be equally

11

important. The models are parameterized, to enable the actual contributions of structures and hierarchies and connectivity to be determined empirically.A final conclusion based on this experiment notes that subjects rank concepts higher if they are closer to the map’s root concept and if they have more outgoing connections or incoming connections relative to other concepts. They also observed that topology alone is a sufficient indicator to extract topic-relevant information from concept maps.

Linkage analysis

Linkage analysis is a concept devised by Liu, Don and Tsai (2005)[13] which helps to identify misconceptions and false assumptions by students . The whole idea of this concept is to take individual concept and link it with the concept map of a student and then to that of a teacher [9]. This helps in finding out the flaws in developing a concept map and also establishes ways to improve the map.

An example of how linkage analysis works:[9][13]We have a set of concepts C. We have teachers set of concepts C2 which has most of concepts connected to C. We also have student map C1 which seems to be connected with the concepts C. In this case he student may end up in confusing C2 with C1. This case shows C1 to be a concept that is possibly confused.So if a student wrongly connects C1 with concepts in C while most of the concepts in C seem to agree with the C2 i.e. teacher’s concepts then it points out that the student is wrong and he can substitute C2 with C1 now.

12

2.6 Role of shapes in a concept map

Figure 2.6: From top to bottom -Tree, Spoke and net differentiation [9].

There are 3 types of substructures from concept maps namely spoke, chain and nets (Kinchen and Hay 2000)[14] .

A chain represents single level hierarchy where concepts are arranged one below the other from top to bottom.

A chain corresponds to a sequence of concepts.

A net represent a substructure where a pair of concepts can be related to one another of different set of concept links.

The diagram above represent spoke, chain and a net substructure [14].The above substructures represent how well concepts are integrated into the mental models of the learner[9]. They also indicate how a concept map collapses when new information contradicts the leaner while developing it.

13

A spoke substructure fails to unite a former concepts but helps in identifying certain concepts that are related to the given core (or key organizing) concept. In the figure we could find that number 5 concepts in the spoke diagram indicates that a learner will be unable to specify the attributes of a given object without referring to the object’s class [9]. A chain substructure shows the order in which concepts were introduced initially as it corresponds to a sequential pattern [12]. They have chance of getting broken when they are presented with new information. They simply cannot change so quickly and accommodate new information in their flow of representing concepts. In net substructure concepts are integrated with one another strongly. The net structure seems to handle new information more smoothly and represents a regulated form of representing information. Evidence suggests that net structures indicate meaningful learning (Kinchin et al, 2005) [13].

Evaluation of concept maps depends on shapes of the concept map to an extent. The scorer or a teacher could actually judge what a student understood by looking at the different shapes which the student has used in representing the relationship between different concepts [27]. However, depending on shapes used within a map for evaluating the map has its own flaws. As, developing concept maps are context based a particular substructure used in representing a relationship need not be a good indicator in all occasions. For example , a student may represent a particular concept in a tree structure when the relationship can actually be represented only in a tree structure. A master map preposition could represent a spoke substructure when representing a relationship while the actual student map may represent a net substructure . This case indicates that a shape may not be the right indicator in certain cases.

For this reason a set of graph theory algorithm was applied which traverses the prepositions and indicates the user about the type of shapes used in the map. The graph theory algorithm can find out if it is a tree, spoke, list or net which ever the map builder has used in exhibiting relationships between the concepts.

Figure 2.7: shapes of concept map developed for student A.

14

The graph theory algorithm indicates the shape from the preposition file leaving the user with the convenience of knowing what shapes the map builder has used without looking at the actual map. The shapes are also one of the approaches in scoring however it is logical to leave it as an option for the user. This is because if the user feels the shapes could be an important factor in scoring the concept maps based on the context of the preposition that he has then the user could click continue to further process the shapes for scores.

2.7 Concept map scoring (in detail)Evaluating these concept maps manually is not a very easy task. It is tedious and a difficult job. Scoring is method of assigning scores based on scoring strategies [4]. Scoring a concept map is important as concept mapping usually is troublesome for many students as it tests the personal understanding of the student rather than knowledge that was merely accumulated [9].

The strategies basically assign points for the map based on various approaches and the final score is then computed. We need to find suitable scoring strategies to evaluate the map. A scoring strategy gets more logical when we have a master map since this map acts like a guide [17].

When building a tool that can evaluate maps one needs to know actual techniques that could be useful and feasible. This section studies these techniques which lay as major foundations in building a tool.

Evaluation Techniques

Shavelson et al. has proposed some techniques which is useful in evaluating these maps [27]. They possibly provide different ways to generate and score these maps. Some of the techniques are:

Quantitative measurement of number of map characteristics such as counting the number of prepositions.

The levels if hierarchy used in expressing relations within the map. Assigning scores to reward the validity of prepositions used in the map. Comparing the map with an expert map and observing the closeness that exist

between these maps.

Quantitative assessment method

The quantitative assessment method provides a means to calculate a numerical score for a given concept as a measurement of a students understanding of a particular domain [9]. The various schemes in this assessment method are listed below:

15

Holistic overall scoring method : This concept of scoring was developed by Mclure, sonak and suen (1999)[10] which allows the scorer to usually score the map based on what the scorer has actually understand about the domain on a scale of 0-10. The scorer will have a look at the map and based on his or her observation scoring is made from a scale between 0 and 10. This form f scoring seems to be easy and many previous studies which were conducted on relational and structural also seems to accept this scoring strategy to produce fruitful results [17].

Figure 2.8: holistic overall impression

The figure 2.8 is an example demonstrated in the tool based on the holistic overall impression. The student concept map for the question “ what are the important facts about cricket?” and the expert map for the same question appears in the figure 2.6. The user is allowed to rank the map based on a scale of 0-10. The rank value is then multiplied by 1.Weighted component scoring method : This idea of scoring the concept map is to assign partial points to each component of the map and / or links between them. The total sum of all these points would make the final score of the map [9]. The points are given usually based on the type of structure they add to the concept map.

The closeness index: This closeness index developed by Goldsmith, Johnson and action (1991)[12] compares the student map with the teachers map or master map to find out how close both are. This scoring approach helps in finding the value of similarity between actual concept and links in the student and teachers map that are common [9]. This allows a student map which is assumed as imperfect to be

16

compared with a expert or master map. The value showing how close it is helps us in finding how much the student has understood the concept. The closeness value is computed based on the similarity between the expert map and student map.

The comparison for the maps in figure 2.1 and figure 2.3 was done earlier in section 2.4. There were 6 prepositions hat were fund to be similar between the maps. A score of 3 is assigned to every preposition that seems to be similar. The final score is 18.

Qualitative assessment scoring

The Qualitative assessment methods are used to produce descriptive assessment of the concept map. They usually make a synthesis of various features and provide a descriptive diagnosis of the understanding [9]. The following tables highlight the pattern of structural and relational scoring the structural scoring method is based on the hierarchies represented in the concept map. They are developed by Novak and Gowin(1984)[11]. The relational scoring method usually gives points to each link between concepts in isolation. The scores r high when they are correctly labeled and they represent foundational relationship of the domain such as taxonomical and casual relationship [9]. The relational scoring method was adapter from a technique developed by Mclure and bell(1990) [28].

Feature of concept map ScoreValid , but incorrectly labeled link between 1 point each

conceptsValid and correctly labeled link between concepts 2 points each

that does not represent a hierarchical , casual orsequential relationship between concept

Valid and correctly labeled link between concepts 3 points eachthat does represent a hierarchical, casual Or

sequential relationship between conceptsLink between concepts where no relationship exists 0 points each

Table 2.1: Relational Scoring[9]

The relational scoring scheme has shown high reliability if master map which measures plain relations between the maps for the area experts. Relational scoring seems to be a useful method of scoring as shown by the results. They seem to be useful and easy to use and also easy to implement. Relational scoring also seems to be valid and reliable in other studies too [17].

The table below shows a different pattern of scoring. The main focus of scoring is based on the structure and hierarchies. This leads to the idea of choosing each concept one by one and analyzing each concept in relationship with the other from top to bottom or vice versa. The idea of choosing structural method is because the concept

17

map is in itself made up of diverse structure. Each component and relationship from and to affect the other component directly or indirectly in a concept map [9][3].

Features of concept maps Score

Valid hierarchical link between concepts 5 points eachValid link between concepts on different branches of 10 points each

hierarchical structureOther valid links between concepts 2 points each

Examples of concepts 1 point each

Invalid concept or link0 point each

Table 2.1: Structural scoring [9].

Usually these structures start from the main concept that appears on the top of the map and grow into a big tree like structure representing each concept. Therefore choosing the structural method actually solves the problem logically. When the scorer actually has a control over the structure of the concept the scorer is now able to judge how well the map builder has understood the concept [18] [11].

These methods however practically can be implemented as a software application and automated easily than rest of the methods. Literature seems to agree on most occasions that choosing the hierarchy of the concept can reach far more fruitful results.

The structural scoring method provides instructions to score which is easy to implement and has also shown great results. The instruction for structural scoring scheme was applied on a concept map containing 9 concepts and 9 different links.

The hierarchies depend on the map builder’s interpretation of the concept. The map builder’s interpretation can be judged by the number of valid prepositions he has employed in building it. In this case it is 8 valid prepositions out of 9 concepts. The number of hierarchies from the root is 5 and cross links if in case is valid will be considered as 10 points. The examples mentioned in the map are allotted with 2 points if it is valid [18].The final score based on assumption that they are valid comes to 30 which is not a bad number. When we carefully analyze this method we derive the fact that this scoring method makes sure that deeper the link is deeper the understanding of the concept is. It shows how the map builder has managed to break the topic in to levels of hierarchies and elucidate these levels with corresponding cross links and labels.

18

2.8 Scoring by the tool (actual methods)

Figure 2.9: Final summary of results for student concept map A.

The above figure shows the final summary window of the tool after scoring the student map in the figure 2.1 with the expert map in the figure 2.2 .The holistic overall impression is given to be 1 out of 10. The number of concepts said to unique and valid are 10 . So a score of 1 is assigned to it which makes it 10 in total.

The number of valid relationship is said to be 10. So the score is (10 * score 1) which is 10 in total. The closeness value after comparing the prepositions of a expert map with the preposition of student concept map A is 18.Closeness show the prepositions which are directly similar between the master map and the student map . 6 prepositions seem to be similar between each other. So the score for closeness is 6* 3 which is 18.

The number of valid preposition is the value of closeness plus the preposition which are valid but don’t match with the prepositions of the expert map. The prepositions which are student map which don’t match directly based on exact similarity and terminology will be discarded. But however there are some prepositions which could be unique and may not match directly with the prepositions in the expert map. The value for these prepositions will be added to the closeness value. The score for 4 valid

19

prepositions is 4* score of 2 i.e. 8. the closeness value is 18.The overall score for number of valid preposition in this case would be 26 (18 + 8).

A score of 2 for tree , 3 for spoke , 4 for chain and 5 for net was assigned by the tool. The map has 3 tree structures which gets a score of 6 and a score each of spoke and chain which gets 3 and 4 respectively. The weighted sum of all the scores adds up to 80. As discussed earlier the shape of the concept map may not be always a good indicator of map builders understanding. This example demonstrates the scoring based on shapes just in case the user wants to score for shapes.

The literature and analysis based on successful experiments goes to show that the approaches derived from these methods could possibly lead to better results in building a scoring application.

Studies also indicate the fact that by discarding a master map the scoring strategy becomes vague and unfruitful. Also relational scoring method fills most of the gaps in scoring by considering the importance of cross links and graphical representations of the map[18]. Most of the other scoring methods are either incomplete or just basic. The structural scoring method also considers important aspects such as hierarchy which shows the map builders understanding from the root of the map.

Graph based measures

Concept maps are used as conceptual graphs in the field of textual mining [28]. An algorithm which produces the structure of the concept map diagrammatically to the user could aid in further restructuring of the map for its betterment. This however has an effect of scoring the maps as well. A scorer can evaluate the map on looking at the structure of the map.A path finder algorithm that traverses a concept graph and finds the relationship between nodes would actually resemble the maps structure diagrammatically. A scorer from his perspective can judge the map and also allow it to be restructured.

20

21

Chapter 3

Basic Design and ImplementationThis chapter discusses the basic semantics and syntactic of the tool. This chapter also deals with basic design, core data structures, architecture and implementation of the tool.3.1Input format

The tool largely deals with prepositions which are in the form of texts. The form of inputs given to the tool will be in form of image files of the map and also a .txt file or a excel file which contains set of prepositions as input. The image file is useful in a way the user tends to score the map for overall impression and closeness. The other input should be an image file of both the concept maps. I.e. master and student map. The image file could be a jpeg or a bitmap image. This image file is supposed to allow the user to have a look at the student map and give an overall rating in a scale of 0-10 for it. To add to the purpose the user can compare the map with a master map and also affect the rating. However the requirement to treat concepts and nodes within the concept map is not the original goal of this tool

Figure 3.1: Input sets for master map.

The above prepositions shows simple master map designed for testing. This contains prepositions which have direct similarity with the student map. Then similar texts which could map prepositions with that in the student map. A general input form

22

would have the text ‘staff’ followed by a comma then the name of the staff or expert or a teacher. Then in the next line the focus question must be specified by mentioning the word ‘Question’ followed by a comma then the topic which the map covers. Then the prepositions are arranged in the order Concept -> Relationship->Concept.

Figure 3.2: A student map preposition

The figure 3.2 is a classic example of student map preposition which is automated from a notepad. These prepositions are compared with the master map for similarity and scored based on the relational and structural scoring method discussed in the previous chapter. In the next section the data structure and handling of prepositions are discussed followed the handling terminology variations between the prepositions of master and student map.

3.2Handling prepositions

The prepositions form the main crux of the tool. For the tool to unify all concept maps it requires a standardization of representing all prepositions. So while building

23

the tool it was made necessary to take the actual pattern of preposition as formulated in a concept map drawing tool such as the IHMC[6]. A preposition is made up of a concept A , concept B and a relationship $ .Concept map 1 drawn by the student could have prepositions such as (A , $, B) , (A 1 , $ 1, B1) , A 2 , $ 2, B 2)… and so on.Concept map 2 could have prepositions in the similar fashion. Namely, (C1, $, C2) and (C 5, $ 5, C7).The prepositions for exam in one map could be A 1- objects, B1 – class and $ - has. “A class has objects “is the preposition in this case. Further in the next section of this chapter we will see how the tool matches these prepositions for similarities.

Concept Relationship ConceptA $ BB $1 CC $2 D

Table 3.1: basic data structure of a preposition

The table shows the arrangement of prepositions. A ->$->B denotes one preposition. The following standard is what is used throughout the tool for processing. The scoring strategies are applied later on for processing. Manually developed concept maps ‘s preposition can be entered into a notepad or a excel file and fed to this tool for getting processed.

Concept Relationship Conceptsmoking causes cancercancer Is divided into many typesMany types include Blood cancer

Table 3.2: Basic data structure of a preposition (real examples).

Smoking->causes->Cancer is a live example of preposition that will be used by the tool for processing.

Hash maps for mapping similarities

Hash maps are basically a data structure which is used for comparing multiple structures by assigning indices and keys [20]. The tool was coded using this concept. The data structure created by the hash map is useful in realizing concept maps in the same form. A master map’s prepositions and a student concept map’s prepositions are taken in the form of a hash map for easy comparison of similarities. The hash table concept if required can be represented in the form of arrays also.

The hash function used in the hash map is used to transform the key to the index of the array element where the corresponding value is to be sought. The hash maps are

24

well dimensioned so the number of instructions for each look up is independent of the number of elements stored. This function helps in mapping similarities irrespective of the number of prepositions. The tool is now capable of handling vast number of prepositions.

Hash map that stores master map (every rows denote array lists).

Figure 3.3: Hash Maps

The above diagram shows the list of prepositions of the student map being stored in a array list further being compared to the list of preposition of the master map. The value of the comparison is further computed based on the scoring strategies and the final score is listed as shown in the diagram. The prepositions are not compared hierarchically in this tool or rather matched instantly. The every preposition in each side is compared for similarity. This would show how close a student map is with regards to a master map. The maps perform direct string matching. ‘Strcmp’ function is used by the map to match strings that are same from both sides. However a concept map as discussed earlier does not levy any norms and rules on the vocabulary for building the tool.

The main advantage of using hash maps is for its speed at which it can locate similarities [20]. There are cases where the hash table allows constant look ups. This function makes the tool to allow the users pick those prepositions which are valid yet not found matching with the master map. Another classic usage of this hash map is every location can be used as well [20]. This enables the preposition in master map to be compared with all prepositions in student map forming a hierarchical checking order. The keys used are ahead of time and helps in avoiding collision. It also establishes neat and smooth comparison between the prepositions [20].However the prepositions are texts and hash maps don’t support text match matching

3A $ BC $1 D 2E $2 F

4

Hash map that storesstudent map

Final scorePrepositions (rows denotearray lists).

A $ BB $1 CC $2 D

so well. This was a major disadvantage which was later solved by concurrently implementing word net.

25

Handling terminology variation

Word net is a large database of English. Synsets , a part of Word net is a cognitive synonym which are formed by grouping adverbs , nouns , adjectives and explain a distinct concept[21]. The role of word net played in this tool is to map synonyms between master map and student map. A case which occurs while scoring is the use of text by student. A student could actually be explaining the same concept as mentioned in the concept map but the student might be explaining it using different words r texts. The tool can handle different structure but when it comes to different texts it appeals to Word net. Word net is structured easily to suit computational linguistic and natural language processing. In the case of similarity matching it plays a very important role[21].

smokingcauses

cancer

causes _______________ S synonymmatching using

Source word net (score ofof 3)smoking

Sourceofcancer

Figure 3.4: Example for terminology matching.

The prepositions Smoking -> causes-> cancer and Smoking -> source of->cancer mean the same. The concepts smoking and cancer match directly in this case. But causes and source of don’t match directly. Word net helps in matching them successfully.

The tool takes each and every word of the prepositions separately and later on performs the comparison. This helps in indicating matching synonyms between two prepositions which are expressed in a sentence. The synonym matching in this case does occur between 2 words but they are picked anywhere within the sentence for comparison. This restricts the fact that similar words with same meaning are treated not only individually but also similar words in sentences are matched and later indicated to the user.

26

http://wordnetcode.princeton.edu/2.1/WordNet-2.1.exe (this is for windows OS alone). Download and install files from the above link. A jar file called jaws.exe is added to the project folder to enable dictionary. Dictionary path to database files is defined in the path file in the configuration folder in the project folder. Current path is C:/Program Files/WordNet/2.1/dict. Change the root folder if the word net is installed in different folder accordingly. The word net automatically checks for similarities in the prepositions and alerts the user. The user now can have all the similarities at a glance and now realize how well the map builder has been able to elucidate the concepts though with a different text.

The actual words found in jeopardy in the concept map is also checked . This is an interesting case because a word which is found in the form of concept with no connection i.e. no incoming and out going links could be found and may match with prepositions in the master map. For such cases hash maps aid in locating the destination an word net helps in locating the meaning and thereby its similarity with the master map. The hash maps do manual checking of text and maps similarities. However, it is restricted since it needs a lot of help from the user [19]. The user cannot be checking textual similarities as it may take time and make it tedious. Word net is also user friendly. It is a more professional tool to check synonyms and synsets , which basically what the tool looks up to. The implementation part of word net was successful since we can expect a lot of similarities within these maps. The similarities usually appear in most cases as a single word. The implementation of all the above sections could be manually scene in the coming section of class diagrams.

3.3 Design overview and Architecture

Figure 3.5 : Design overview of the tool.

27

The BO package contains the domain specific representation upon which the application operates. BO package contains all classes which has the data structure needed by the tool. The Util package contains all the necessary classes that define specific functions carried out by the dictionary ,graph algorithms etc. The controller package contains the main class which makes all the interactions between these package possible. The GUI package contains the implementation of the user interface and functions to render it.This application follows the MVC (Model View Controller) architecture closely. The GUI is the front end, with which the user interacts. These files are packaged as GUI together. Controller is the one that intercepts the requests from the user and delegates it to the appropriate BO class and finally renders the user interface back to the user. This follows a front controller design pattern and is present in the controller package.

BO classes are the business rules that actually process the request passed on by the controller and returns a result to the controller. These are the core classes and are packaged as BO.

The user interacts with the user interface by clicking the mouse button. The controller handles the input event from the user interface, by a registered handler and converts the event into appropriate user action, understandable for the BO package. The controller invokes the BO during a user action. This results to a state change in BO package (For example, the controller updates the user's rating which it gets from the user rating panel).

The GUI package queries the BO package in order to generate an appropriate user interface. The GUI package gets its own data from the BO package.The controller may issue a general instruction to the GUI to render itself. The GUI is automatically notified by the BO package of changes in state (Observer) which require a screen update. When a BO package changes its state, it notifies its associated package so they can refresh. The controller receives input and initiates a response by making calls on BO objects. The GUI renders the model into a form suitable for interaction. Multiple GUI can exist when there is a real purpose.

28

3.4 Application of scoring algorithm

The following algorithm represents the basic relational scoring methodology used in creating the tool. [2]. Each preposition P in a concept map is given a score between o and 3 according to the following protocol:

Figure 3.6: Structural scoring algorithm [9].

The scoring algorithm forms as the basic for the tool to choose before scoring prepositions. The algorithm checks if there is a relationships of any sought (direct or indirect) between 2 prepositions. In case there is a relationship it gives a score of 1. If not it gives a score of 0. The algorithm goes to check if the relationship is found in the order of hierarchy or is it just casual or maybe sequential. A score of 2 is given in case there is no such relationship structure else a score of 3 is awarded in case the relationship is any such order.

The tool uses a different algorithm which was derived from the algorithm mentioned above. Before going into the actual scoring algorithm the tool checks if the question in both the preposition is same and proceeds only if they are same. Otherwise it throws an ‘error’ message. This is to make sure that only two prepositions of the same map and same question is being compared for similarity and late scored. After this step the input file date are mapped to the array list in the order concept->relationship->concept.

29

Closeness to a master map: This approach is the heart of relational scoring technique. The tool checks each preposition in the master map with the preposition in the student map and computes all or no similarities between the 2 maps. Every single preposition in a master map is checked with all prepositions in a student map one by one. Scores are assigned simultaneously the final score from these similarities usually shows the closeness index [18].

The algorithm for scoring for closeness is as follows:Step 1: Compare student map preposition with each master map preposition.

Step 2: If there is a matching pattern found between the 2 files then increment the closeness value by 2.

Step 3: Else prompt the prepositions for manual resolving.

Step 4: If there is a matching pattern found after resolving manually then increment Closeness value by 1; else stop scoring.

The scoring for other parameters or approaches is done simultaneously while scoring the closeness value. The closeness value is done only after comparison. Scoring for other aspects like holistic over all impression, shapes, number of valid concepts, relationships and valid prepositions are calculated and displayed in the final summary window once the prepositions are mapped from the input file.

Scoring Schemes

The design is based on the ideas derived from structural scoring and relational scoring discussed in the previous chapter [18][2]. The theme of this tool is to score a concept map based on a number of approaches which were drafted from the structural and relation scoring method. The approaches which we are going to consider for scoring are discussed below:

Holistic overall impression: The overall impression of the concept map is given on a scale of 0-10[10] . A score of 10 shows that the user has been able to understand the concept well and structures of the concept has been strong and well correlated. A score of 0 shows poor or no understanding of the concept and the structure of the map seems to show bad coherence with respect to the concept. A score of 4 or 5 shows average understanding and moderate approach to representing the structure[2].

Number of valid concepts: This approach applied in this tool counts the number of concepts which seem to be valid. However not all concepts that were used in drawing the map would count. Those concepts which are supposedly similar to those concepts

30

in the master map will be taken into consideration. There is a void that a concept appearing in the student map maybe unique by itself and need not necessarily match with the one in the master map , in that case the tool asks the user to validate the concept. Such concepts are given more marks since it represents the map builders ability to illustrate a concept on his own[18].

Number of valid prepositions: A preposition would be to a valid relationship between one concept and another [3]. These could be in spoke, chain and a net form[14]. A preposition has to be valid with respect to its counter part in the master map. For this reason the tool checks each preposition appearing in the master map with that of preposition in the student map starting from the top level of hierarchy. Any preposition that founds to be matching with the preposition in the master map will be assigned a point of 2. This matching has to be direct i.e using similar texts or same texts or seems to be explaining the same phenomenon. However there are cases where there are prepositions that the student could illustrate that may not be comparable with the ones in the master map; in that case separate prepositions will have to be validated by the user. If found unique and non-matching then a score of 2 is assigned otherwise a score of 1 is assigned.

Number of Relationships: This approach of scoring is basically to find how much relationships have been exhibited by the student in his map [2]. This helps in finding the mental ability of the student to expresses the concept in the form of different structures in a well correlated fashion. The tool adds all the relationships that the student has drawn in his map by assigning 1 point for each relationship.

Shape of the concept map: The idea behind this approach is to know what type of structure the student has exhibited in the map to represent the map. The shapes could be tree, spoke, chain and net differentiation [14]. A map score of 3, 4 and 5 are assigned to the preposition depending on the shape of the map. A graph theory algorithm for finding the structure of the concept map through the preposition was developed in java and added to this tool [19]. This algorithm traverses through the preposition and realizes the shape of concept map through the structure in which it is assigned.

In case of (a, b) (b,c) and (c,a) this preposition when drawn a structure represents a chain structure whereas a (a,b) and (b,c) represents a simple tree. So in this order of preposition the tool uses this algorithm to find out the structure which the map represents. A score of 2 is assigned for a tree, spoke and chain whereas a score of 4 is assigned for a net differentiation [14]. The reason behind the changes in scoring pattern for different structure is explained in chapter 3. However this structure is solely based on the context at which the user develops the map. So this scoring approach is optional for the user. It can be left to the user whether or not to choose this approach in case the user thinks there is not much connection for this approach in that map. It can also be left to the user jus to know what type of the structure was used to represent the map and it need not be scored further.

31

Graph based measures: These measures could be useful in scoring the concept map structure wise [2]. A. This graph theory based algorithm is used to check the relationship between the concepts of the map and finally derived by the proximities of the pair of the entities [19]. They express the pattern of relationships in the map. However this tool tries to implement an algorithm which produces the relationship pattern and giving them a suitable score.

A path finder algorithm as a part of graphical representation was developed in this tool. The algorithm traverses from the root node to the top. It traverses hierarchically from the root to the top [22]. In our case the inputs will be in the form of prepositions that consists of data in the form of variety of texts. The texts are in the format (Concept A, relationship $, Concept B). The prepositions have to be in this order to make the algorithm conduct a search. The search starts from Concept A and takes its path through the relationship $ and finally meets Concept B. Then it moves onto the next line that contains Concept A1.

The implementation took place in a separate class that calls this algorithm on a set of prepositions and this class was integrated with the rest of classes. The Breadth First search traverses prepositions in the order of A, B, C, D, E, F. For example, let us assume that we have these prepositions:

(OOP, class, object, attribute, method, data type) then the root node here is OOP and it starts its traversal from OOP and goes until data type which is the final preposition. The BFS written in java for this tool computes the prepositions in the same order i.e (OOP , class , object , attribute , method , data type) and the DFS search traverses in the order of ( OOP , class , object , method , attribute , inheritance , data type).

The idea behind traversing the prepositions was to provide support for the tool to know path of the graph. By knowing the graph it could help in restructuring the map.Path finder algorithm is integrated in this tool and finally appears in the summary frame that shows the overall output of the tool. The algorithm was written in java and it provides the path as the output in the final frame just under the summary. The scoring scheme depends purely on the context of the user. So the path finder algorithm is kept optional in the tool for the user as concept maps are build on contextual information. The necessity of a path finder may or may not be suitable in some situations.The user can activate it by choosing it and then rate it. The score from this will be added to the summary. If the user finds the algorithm not relevant to that context keeping in mind the context in which the map builder has done the project then the user need not choose it.

Weights of all the scores: The weighted sum of all these approaches will be the final score of the map [2][9].

32

3.5 Package Diagrams

The tool has 4 packages namely package BO , package controller , package util and package gui. In this section we will see the description of the class diagrams and the implementation of these packages in detail. The descriptions carry details about the relationship between various classes within a package and also the significant methods and functions that constitute the classes.

Package BO

Figure 3.7: class diagram of package BO;

The package.BO; contains the basic data structures of the tool. The methods, structures and all functionalities are provided by this class. The classes and its main functions and purposes in this package is listed below.

Question.java: This class gets the question for the comparison for the input files (student map and master map prepositions). The Question.java class is named as a question since this tool follows a standardized format in giving the input prepositions. It is indexed with the name question after which the array of prepositions is taken in.

33

Concept.java: The class has the structure of the maps with created question given first importance followed by the concepts and relationships involved in the prepositions of the input files. This class is used for adding and setting relationship between both the maps. It contains all the basic data structure for the preposition file to be used in this tool. It is basically done in hash maps. The mapping between the two classes is done through the hash maps called in this class.

Student.java : This class has methods like get() , add() and set() data for the student map. This class is used for setting up the student map file.This also creates an array list containing the student map prepositions which is to be compared with the master map prepositions.

Staff.java: This class is used to set up the master map file, creating an array list with which the comparison with the student map preposition will be made. The master map prepositions are taken as strings or texts in a huge array list

Package util

Figure 3.8: package.util

The package.util consists of classes that are considered for auxiliary purposes in this tool.

Dictionary.java: This class is used to look up synonyms for given word that appears between the student map prepositions and master map prepositions.

Graph.java: This class constructs the traversal of nodes . For this the input file prepositions are considered and they are traversed from bottom to top. This helps in finding the path of the concept graph.

Node.java: This class defines the structure of the node.

Path.java: This class reads the configuration file for the path of dictionary files.

34

ScaleImage.java: This is sued to adjust the size of the picture in order to fit into the window properly.

Score.java: This class reads the configuration file for the score defined by the user.

Package GUI

Figure 3.9: package.gui.

MainGui.java : This class creates the data structures for the gui and its frames. It calls all other Gui frames and panels as per the flow of processing of the class. MainGui is the main class for this package. This acts as the Frame (or Parent Window) of this Application. Acts as another controller within the GUI classes.

ImagePanel.java - This is a Jcomponent which is a base class for all Swing components except top-level containers JFrame, JDialog, and JApplet. This component displays pictures to the user.

35

MessageDialog.java: This class is a model dialog box which throws error messages, information to the user.

ResolvePanel.java: This panel is one of the panel that is used when manually resolving the conflicts.

UploadGui.java: This panel is used to select the files that are given as input for marking.

UserRating.java: This displays rating option in order for the user to select. This aids the holistic overall impression window for choosing the overall impression.

Package Controller

Figure 3.10: package. Controller.

Controller.java is the main class which handles the all the events accordingly. Controller is the one that intercepts the requests from the user and delegates it to the appropriate model class and finally renders the view back to the user. This class imports data from GUI package to setup the graphical user interface, display messages, upload messages,

36

imports data from BO package to get the array list prepared and sets the rules for comparison . While comparing this class imports data from the Util package mainly such as the auxiliary data like dictionary, loading and scoring functions.

The controller.java class lists out the sequences or the order by which processing of the classes based on various scenarios. The java.io.file is assigned by the controller class for the input output stream through which the input file can be realized and an necessary output can be spawn. Util.LoadData.java is used by the controller for uploading the data. The master map prepositions and student map prepositions are handled directly and made to run by the controller.java class. LoadStaff() method assigns the controller with particular string that needs to be uploaded.

The scoring involves the usage of the following methods.

initial_scoring () - when this method is called the controller sets the initial values which stored which the input is loaded and mapped.

user_mark() - when this method is called the controller sets the value set by the thru the user interface as the value or the score given by the user.

set_score() - this will set the value which is the result of comparison of the two files after the automatic and manual resolving functions are carried out.

final _score () - this method when called will result in the summing up the scores which have been collected.

final_scoring() - this method will be responsible for the displaying of the summary of the scores in the summary page on the user interface.

The scoring is done through comparison of the array list which have been created from the files input by the user. Once the user uploads the file through the user interface the files are uploaded and the array lists are prepared and then when the score option is selected thru the user interface the controller handles this event by passing the array lists for comparison and getting a value, based on the result of the comparison, the obtained value is again passed by the controller by a response event to the user interface which results in the display of the score in the user interface.

37

38

Chapter 4

ResultsThis chapter furnishes the results achieved by the tool after it was tested on various scenarios. The results also discuss the problems that were faced and the required solutions given to the problem. The problems are analyzed and it reveals on what cases the tool works and what cases the tool cannot work. The tool was later checked for quality using SD metrics. The last section discusses the case of manually drawn concept maps in detail and further enhancement which could be made to this tool to solve the problems that were discussed earlier in this section.

4.1 Input constraints and front end of the tool

The input sets as discussed earlier in chapter 3 have been designed in such a way that it should have a question. The topic of the concept map is supposed to be specified with question followed by a comma and then the topic name. The tool cannot match the preposition of both the maps if the question is different in character. The diagram below shows the front end of the tool. The friend end has the list of scoring strategies as described in the previous chapter just to show the user on what scoring approaches

39

does the tool score. The front end has a next button on clicking that it moves to the upload window which uploads the student map , master map and prepositions of these maps.

Figure 4.1 : input files for wrong questions (master map)

Figure 4.2: input files for wrong questions (student map).

The tool cannot accept a preposition which has questions that are different in text and character used. This is because the tool used hash maps to represent a student concept

40

map and a master map for comparison. The hash maps as discussed earlier in chapter 3 takes the concept maps in two different hash maps and compares the prepositions for similarity. The matching process starts first by checking for the question. The tool cannot check further if there is a change in the format of question. So this demands the user to write the question in the same format for both the maps. The above example shows the difference in expressing the question by varying a word ‘rules’ and ‘rule’ will be considered as a mistake and tool cannot take it. This makes sure that concept maps which belong to the same topic question is only compared. On the flip side this restricts users who want to directly enter prepositions and score it. This issue can be later addressed in the future and required technical changes can be made to counter this.

4.2 Repeatable prepositions

This case involves prepositions which repeat more than once. This could be an issue since this preposition could be counted more than once for scoring. The tool basically treats repeatable prepositions as ‘duplicate’ and discards them.

These type of prepositions could affect the accuracy in giving correct scores if they are treated. For this reason the tool takes the string just once and compares it with master map. However the tool does not restrict the use of same preposition again and again. But when it is being scored these repeatable prepositions is matched using a string algorithm which usually doesn’t take the same string again.

41

4.3 Upload window

Figure 4.3 : upload window

The upload window takes the image and the text files containing the prepositions. The staff input files stream must have the file path of the staff image file in .jpeg and the input file for prepositions .txt. This holds good for the student files also. Then on giving upload these files are uploaded. The student text file and staff text files are then sent for scoring and the image file appears in the next window for allowing the user to score based on holistic overall impression.

42

4.4 Holistic overall impression

Figure 4.4 : Holistic overall impression window.

After the files are uploaded the user is asked to rate the map based on 0-10. The window shows 2 maps in comparison. The map created by the experts or the master map comes on top while the student map comes in the bottom. The student map is the map that is going to be scored. But the user is given an option to see both the maps so that the opinion on the student map is affected. On giving a score of 0-10 the score gets recorded and then sent to the final summary window for publishing. It is also added to other scores for computing the weighted score of the whole map. After giving next the tool starts to match the prepositions between the master and student maps through the text file which was uploaded before. The next section discusses the matching in detail and also discusses the problems that were faced and the solutions that were offered henceforth.

43

4.5 Matching Prepositions

This section deals with preposition matching and various problems faced when matching these prepositions for similarity and terminology differences.

Self calling prepositions

The preposition highlighted below is a self calling preposition. The preposition enzyme->cuts->enzyme calls itself when drawn in a map. In terms of concept mapping it means one of its outgoing link actually points to itself This preposition could pose a problem when scoring since they are unique in nature. They may or may not match with the master map.

Figure 4.5 : Self calling preposition

The preposition enzyme->cuts->enzyme involves only a single concept.The concept ‘enzyme’ calls itself or rather mentions that enzymes cut enzymes. This is also a meaningful preposition.The solution to this issue is to treat the preposition like any other preposition. Firstly, it is checked for similarity with the master map. If the preposition is found matching with any preposition of the master map then it will be assigned with a score of 3.Otherwise, it is treated as an unique preposition.

44

Unique prepositions fall under the category of those which don’t match the master map but come with a possibility of being valid on its own. This further invokes a manually resolve window. The manually resolve window usually has a list of prepositions which are assumed to be valid or invalid in any case but cannot be scored until the map builder verifies it. So based on this the map builder can choose if this preposition is valid or not. If it is valid then it is passed to the scoring section and scored under the points given for number of valid prepositions.

Treating terminology differences

Figure 4.6 : input files for treating terminology differences (student map).

45

Figure 4.7 : input file for treating terminology differences (master map).The above prepositions are arranged in different order and also those which don’t even match with each other when we compare both the files. The word opposite and antonym are similar and they are highlighted in the input file. The Word net that is a part of the tool finds this similarity and then it maps the similarity. In this case antonym and opposite are the only 2 words which are similar short and tall don’t make sense with light and heavy. But in this case the tool provides a partial point. If the prepositions are exactly similar the tool is going to give full marks, To strike a balance between these two cases there is a option of manually resolving the preposition in case of doubt. Word net does not work for words like ‘is a’ , ‘was a’ etc. Word net has the capacity only to take one word at a time to check in this case there is a window called manually resolve. In this window we can find the prepositions of both master and student map. The tools ask the user to confirm if these prepositions are similar in nature though their texts don’t match according to word net.

Figure 4.8: Manually resolve window for treating terminology differences.

46

The tool will not score if there is only a relationship specified in the text file. For example it can have a, relationship, which means there are no concepts but a relationship. This actually does not work since it does not follow the preposition standard which was formulated for this tool.

Terminology differences between long prepositions

Figure 4.9: input file for treating terminology differences in long sentences (student map).

Figure 4.10: input file for treating terminology differences in long sentences (master map).

47

The previous case discussed about the synonym matching between 2 words that are similar. But this case is unique because there are 2 words which are found to be similar but are arranged within a sentence.

These sentences however mean the same. Most of the prepositions which use similar words between the 2 maps and also arranged in different sentences seem to show great similarity. In the above case Track, ->on the right hand side-> is off side and track -> towards the right hand side ->is off side. In this case tracks and off side are 2 concepts which are matched directly by the tool. Direct text matching or direct synonym matching does not work for comparing relationship node for this preposition. The tool reads every word alone and also finds similarity between these words. So it presents a combination of these prepositions to the user in the manually score window. The user now can see the prepositions and later score it. The score is then added for closeness.

4.6 Single prepositions

Figure 4.11: single preposition found in student map.

The above window shows another new form of preposition. The concept ‘method’ that is highlighted is a concept used in this map which does not have connection with any of the other concepts. It has no incoming or outgoing links to it.This could pose a problem. This concept may match or may not match directly with a preposition. The prepositions in the master map could have expressed the same thing in a different form while the student has expressed it as a single concept and still find some logic in it. In this case the manually score option is again invoked. This is just to validate if the usage of single concept in the student preposition text was really necessary or not.

48

Figure 4.12: Manually resolve window for solving single prepositions.

The manually score window now shows the concept ‘method’ to the user after computing other prepositions. The user should choose to say if it is correct one or not. Then after calling yes or no the user must press the score button to make this case affect the final score.

49

4.7 Manually resolve window

Figure 4.13: Manually resolve window

50

The manually resolve window is invoked when there is another unique case of single prepositions. A preposition matching depends on direct similarity matching and also from the word net. The word net since it is an online dictionary has some restrictions. There are cases where word net may fail to plot synonyms as the dictionary is restricted. There are cases where prepositions which are found to be valid may not have any form of similarity with the master map in this case. This case is usually considered unique because a master map which is supposed to be perfect map may forget that a student could also draw prepositions which are expressed in different text and also don’t match with the prepositions in master map directly.

This could pose a serious problem on the tools ability to match accurately. For this reason the manually score window is invoked just to ask the user to validate if the preposition was right or wrong. The manually resolve window will appear only on these unique cases. The window will not appear when matching is done based on terminology and direct similarity. These unique cases are considered by the tool to make scoring more accurate. The manually resolve window will not show prepositions unless the tool locates an unique case.

4.8 Pathfinder algorithm and shapes of concept maps

Figure 4.14: Output of pathfinder algorithm.

The path finder algorithm as explained in the previous chapter uses DFS and BFS search to traverse through the graph. This finds the path of the map which could be easy for the user to know the structure of the map. The tool very much adheres to the principles of relational and structural scoring so the reason for including this

51

algorithm bears its own fruit. However the path finder may not be so accurate in telling the score of the user. It is an optional approach and the user need not choose it if the user feels that path finder may not be accurate in analyzing the map in his case. For every successful search 2 points is awarded. The final path of the algorithm is displayed in the final output window. This is just to allow the user to restructure the map.

Shape of the concept map

The prepositions when taken into input are traversed using a graph algorithm to find the substructure it represents. The substructures as discussed earlier may be a tree, spoke , list and net differentiation. The tool actually traverses and intimates the scorer about the substructures used by the map builder from the student map preposition.

These substructures have separate patters of scoring. However there is a case where the user could have used a spoke sub structure to represent a relationship whereas the master map might have a normal tree structure in this case the user could get more score for no reason. Concept maps are drawn based on context. The information in the concept maps are also context based. So to treat this issue the tool allows the user to have this scoring strategy as an option. After having a look at the map the user can decide whether or not to include this as scoring strategy.

Figure 4.15: Output on shapes of tree and chain

52

The user after rating the map based on the overall impression is presented with a window which represents the substructure that were used in the map. These substructures are directly automated from the input preposition file. This allows the user to know what shapes have been used in building the map without having to look at the map.Later by clicking on ‘yes’ or ‘no’ button the score for that particular sub structure will be calculated and added to the final score. This could be a tedious process when the map is large. But in this case the user has the option to leave this scoring strategy.The window will ask the user to pick shapes for scoring or not. This happens after the user rates the map based on overall impression. In this case the map just shows the shapes that were used but not score each of the shapes one by one. All the shapes in this case will appear in one single window. So the user can have a look at it. This carefully reduces the time involved in the process and makes it less tedious for the user. Otherwise the user will have to click ‘yes’ button to score for every substructure.The graph algorithms are used to determine shapes works as given in the diagram. John is a friend of, peter and peter, son of, smith. They are direct tree like relationships. The map just pops up the user and makes him know that it is a tree. In the output window you could see that ‘yes’ or ‘no’ button to seek continuation. This on choosing computes the score for a tree. The user may or may not use it. This applies to all different shapes exhibited in the map . The tool intimates the user about the shape of the concept map in a different frame. This approach is also optional. It doesn’t score if the user doesn’t want to score but it generates the shape of the map to keep the user interested. The shapes of concept maps plays a tricky role so it is left to the user to chose, But as illustrated above the user has the opportunity to see the structures used and also get the final number of different structures that were used appearing in the output window.

53

4.9 Final summary

Figure 4.16: Output window of final summary.

This result is the output for the 2 input files given previously. The impression of the user in this case is 10 and then the number of concepts is 22. The number of relationships as manually counted by the tool is 19. The closeness to the master map is 19 which means there has been partial and direct similarities between the preposition of the student and the master map. The validity of prepositions is 26. The different between the number of relationships and validity of preposition is a bit distinct as far this tool is concerned. The tool adds the preposition that doesn’t match with the master map yet they are unique on its own and developed correctly by the user. There were 4 concepts that did not match with the master map but they were unique on their own. These maps were added to the closeness value to provide the final valid prepositions. These are those prepositions which have been left by the tool to manually resolve and chosen by the user.

The final summary displays the number of trees, spoke , chains and net used by the user to be scored as per the users choice. The weighted sum of this will be the final score of the user. However, the path finder algorithm is just for a graphical representation and they don’t play a major role in numerical scoring.

4.10 Evaluation of the tool

This section discusses the evaluation of the tool based on SD metrics , analysis and discussion of the tools performance which further discusses the problems and future

54

enhancement proposed for this tool to counter the problems and make it more extensible.

Measurement of the tool using SD metrics

The tool was checked for consistency and quality using the SD metric. To solve this purpose a metric plug in was added to the tool and the final results were satisfactory. Since this tool has to be extensible we will choose measures such as instability, abstractness, afferent and efferent coupling . However the other important aspects such as mcabe cyclomatic complexity showed 17 which is not a bad number. This goes to show that the tool is moderately complex. A number of 17 for cyclomatic complexity indicate the tool is only moderately complex. However since the tool is new and it has classes like controller.java in the package controller which takes most of the burden there is a reason for this number.But values of coupling (Afferent coupling and Efferent coupling) show encouraging signs. It shows that number of packages that depend upon classes within packages is less. A number of 6 for afferent coupling show that if there is a problem in one package it will not affect the rest of the package so much.

Abbreviation Measures Tool’s valueCa Afferent coupling 6Ce Efferent coupling 5I Instability 0.714A Abstractness 0

Table 4.5: Results representing metrics

Similarly, the number of other packages that this particular class depends on is also less. For the controller package it is a meager 1 which shows encouraging signs. The overall value is 5. The value of instability value is moderate for the entire tool it is around .714 while abstractness is 0. The value of 0 usually indicates completely concrete package.

This evidence goes to show that the tool has the capacity to integrate. with other application. It has the capacity to integrate and it is easy to reuse and maintain this tool. The tool however is new and it is in early stages.

Discussion and future enhancement

This section discusses the limitations, future enhancements on which the tool could be extended.The tool now works on all the approaches of scoring and it has shown some capacity to integrate itself with other tools. A path finder algorithm, Word Net and graph

55

algorithms are external applications which have been added to the tool successfully and used to get maximum performance out of it. This goes to show that the tool can be extended to even wider variety of applications.

Manually developed maps: The map cannot score manually developed concept maps because the form of input. How ever the tool can be extended to take in manual maps as inputs by some form of signature recognition technology by adding intelligence to the tool. The only solution we have in our hand is to write all the prepositions of the map in a set of map. Compare it successfully with master maps prepositions.

Abbreviations and synonyms (not found in word net): There are abbreviations and synonyms which may not be found in word net. These abbreviations can be added to the dictionary and also some words which is found frequently being substituted in place of another word in the dictionary. Once it is added to the dictionary the tool would use it to score it.

Future workThe tool can be extended to provide more functionality with respect to scoring some of them are listed belowApart from Adding various graph theory algorithms.

The tool should be integrated with algorithms that read into the image directly so that this preposition handling works in the background and direct image file could be fed instead of uploading prepositions in text files.

The tool can be integrated with concept map drawing tools so that the user has a whole package of drawing and also finding out the score.

Many new schemes of scoring can be added. Like the HARD and CRD model which analyzes the global structure of the tool [10].

Applying algorithms like nearest neighbor algorithm makes sense in this tool since we are looking for relationship between once concept to another. Also an addition could be to work on making Dijkistra and ford –Fulkerson algorithm [19].

The scoring schemes can be written in J2EE and later they can be published as web services on the web .To do this we need a glass fish server deployed in axis 2 web. These scoring schemes can be converted into BPEL services and can be published in the web. This allows user to share the schemes as services and can take these schemes to the web world.

Applying similarity flooding algorithm to counter terminology matching in a more efficient manner [27].

Should be integrated with technologies that can recognize paper drawn maps that recognize the hand drawn maps through signature recognition.

56

57

Chapter 5

Conclusion

The goal of the project was to make the tool work in different scoring schemes and the goal is achieved successfully. I realize that concept map scoring is an intriguing task, but it is also very challenging one. It is challenging because of the vastness of knowledge represented in the map and also the dependency on the user. To write a code for unifying the concept map preposition is one major step considering the freedom the user gets when developing a concept map. This tool however offers a wholesome package of different varieties of approaches that help the user to pick what he or she wants in terms of scoring. The tool is a basic one and it needs to be tested further. However, the primary goals of the project were achieved.

58

Chapter 6

AcknowledgmentI would like to express my gratitude to Jürgen Börstler (Thesis supervisor) for his special guidance and for providing valuable inputs for developing this tool. I owe a lot to the department of computing science which has given me the power of knowledge to write this thesis and has shown me a path for a bright future henceforth.

The work would not have been possible without the support of my family and friends who motivated and encouraged me in every situation. Last but not lease , my special thanks to Per Lindström( course coordinator) for his support.

59

References

[1] Josesph D. Nowak , Alberto j. Canas . The Theory Underlying Concept Maps and How to Construct and Use them. Technical Report IHMC Cmap Tools 2006-01 Rev 01-2008.

[2] Jurgen Bostler. Relational Scoring of Concept Maps. Unpublished.

[3]Nowak J.D. & Gowin, D.B.(1984). Learning how to learn.New York,NY:Cambridge University Press.

http://cmap.ihmc.us/docs/ConceptMap.htm l , Alberto J. Cañas & Joseph D. Novak Institute for Human and Machine Cognition September 2009, Accessed 2009 -12-12.

[4] Farrand, Paul; Hussain, Fearzana and Hennessy, Enid (May 2002). "The efficacy of the 'mind map' study technique". Medical Education 36 (5): 426–431.

http://cmap.ihmc.us/conceptmap.html.Accessed 2009-13-12.

[7]Canas , A.J.,Cariff,R.Hill,G.,Carlvalho , M.Arguedas, M.,Eskridge ,T.et al.(2005).Concept maps :integrating knowledge and information visualization. Tn S.-0 Tergan & T.Keller (Eds.), Knowledge and information Visualization: Searching for synergies (pp. 205-219).

[8]Derbensteva,N.,Safayeni,F.& Canas ,A.j.(2004 ).Experiments on the effect of map structure and concept quantification during concept map construction. In A.J. Canas ,J.D Novak & F.M.Gonzalez (Eds.), Concept maps:Theory , methodology ,technology, proceedinga of the first international conference on concept mapping.

[9]Jeroen Keppens and David hay. Concept map assessment for teaching computer programming. Vol. 18 , No. 1 , March 2008 , 31-42.

[10]Mclure , J., Sonak, B., & Suen , H. (1999). Concept map assessment of classroom learning: Reliability , validity, and logistical practicality. Journal of Research in Science Teaching , 36(4), 475 – 492.

[11]Novak, J., & Gowin, D.(1984). Learning how to learn. New york : Cambridge University Press.

[12]Goldsmith, T., Johnson, P., & Action, W. (1991). Assessing structural knowledge. Journal of educational psychology, 83, 88-96.

[13]Liu,C-C.,Don,P.-H.,& Tsai, C.-M.(2005). Assessment based on linkage patterns

60

in concept maps.Journal of information science and engineering.21, 873-890[14]Kinchin.I., DeLeij, F ., & Hay, D.(2000).How a qualitative approach to concept map analysis can be used to aid learning by illustrating patterns of conceptual development. Educational research, 42(1), 43-57.

[15]Thomas Reicherzer , David leake. Understanding the Role of Structure in Concept Maps, Computer Science Department , Indiana university Lindey Hall 215, Bloomington , IN 470405 ,USA , proc cog sci 2006 , uncovered.

[16]Kleinberg, J.(1999). Authoritative sources in hyperlinked environment, journal of the ACM 46(5), 604-632.

[17] Canas, A. , Leake , D., & maguitman, A. (2001). Combining concept mapping with CBR: Experience-based support for knowledge modeling. In proceedings of the fourteenth international Florida Artificial intelligence research society conference(p286-290). Menlo Park, CA: AAAI Press.

[18]John R. Mclure, Brian Sonak, Hoi K.Suen. Concept Map Assessment of Classroom Learining: Reliability , Validity and Logistical Practicality. Journal of research in science teaching . Vol .36.NO. 4 PP. 475-492(1999).

[19] Gibbons, Alan (1985), Algorithmic Graph Theory, Cambridge University Press.

"9: Maps and Dictionaries". Data Structures and Algorithms in Java (4th ed.). Wiley. pp. 369–418. ISBN 0-471-73884- 0 .

http://wordnetcode.princeton.edu/2.1/WordNet-2.1.exe. Accessed on 2009-10 - 0 9 .

Dijkstra, E. W. (1959). " A note on two problems in connexion with graph s ". Numerische Mathematik . J. Nowak, Learning, Creating and Using Knowledge: Concept maps and facilitative tools in schools and corporations. ( Mahwah New Jersey: Lawrence Erlbaum Associates,1998)

[20] B.R Gaines and M.L.G Shaw, Web Map: concept mapping on the web, presented at proceedings of WWW4: Fourth international world web wide Conference, Boston, 1995. [21] M. Weidman and W. Kritzinger, Concept mapping vs. webpage hyperlinks as an information retrieval interface: preferences of postgraduate culturally diverse learners, presented at the Proceeding of 2003 annual conference of South African institute of computer scientists and information technologists on enablement through technology, 2003.

61

[22] R.R. Hoffman, J.W.Coffey, K. Ford, M.J. Carrot, Storm-1k: A human centered knowledge model for weather forecasting, presented at the Proceedings of the 45th Annual Meeting of the Human Factors and Ergonomics Society, Minneapolis, M N, 2001.

Byron Marshall( byron.marshall@bus.oregonstate.ed u ), Hsinchun Chen , Therani Madhusduan : Matching knowledge elements in concept mapping using a Similarity Flooding Algorithm, Accepted by Decision Support System , October 2005 Pg 1-13.

62