Semantic Map Assessment Project Overview for McREL

Dr. Roy B. ClarianaPenn State, Great Valley

RClariana@psu.eduhttp://www.personal.psu.edu/rbc4

12/18/02

Project intent Semantic maps (SMs) are considered to be

valid and reliable measures of science content knowledge (Ruiz-Primo, Schultz, Li, Shavelson).

SMs are described as “authentic”, teachers and students use semantic maps in the science classroom as a tool to represent their understanding of that content (Novak).

The intent of this project is to establish an automatic computer system that scores semantic maps by comparing students’ maps to an “expert” map.

EARLYDAYS

Student-developed SMs

The student recalls important terms from a science lesson and drags related terms closer together and unrelated terms further apart to form clusters or categories, and then draws lines between directly related terms.

lungs

oxygenateblood

removeCO2

pulmonaryartery

pulmonaryvein

leftatrium

rightventricle

lungs

oxygenateblood

removeCO2

pulmonaryartery

pulmonaryvein

leftatrium

rightventricle

Scoring Semantic Maps To date, SMs are scored by teachers or

trained raters using scoring rubrics (Lomask) Although this marking approach is time

consuming and fairly subjective, map scores usually correlate well with more traditional measures of science content knowledge (multiple choice, fill-in-the blank, and essays)

No one has tried automatic assessment yet To automatically mark SMs, the graph is

converted into an array (matrix)

SMs can be represented by two kinds of arrays Link array – if a line (link) is used to

connect two terms, a “1” is placed in the corresponding array cell, and a “0” is used in array cells to show that there is not a link between terms.

Association array – represents the strength of relationship between pairs of terms as distances scaled from 0 (highly related) to 1 (unrelated). This array can be converted into a link array of implicit clusters equivalent to a PathFinder neighborhood.

Semantic map w/ link array

lungs

oxygenateblood

removeCO2

pulmonaryartery

pulmonaryvein

leftatrium

rightventricle

lungs

oxygenateblood

removeCO2

pulmonaryartery

pulmonaryvein

leftatrium

rightventricle

LA L OB PA PV RCO RVleft atrium (LA) 1Lungs (L) 0 1oxygenated blood (OB) 0 1 1pulmonary artery (PA) 0 1 0 1pulmonary vein (PV) 1 1 0 0 1remove CO2 (RCO) 0 1 0 0 0 1right ventricle (RV) 0 0 0 1 0 0 1

Most studies use link information,usually called “propositions”.

Semantic map w/ distance array

lungs

oxygenateblood

removeCO2

pulmonaryartery

pulmonaryvein

leftatrium

rightventricle

lungs

oxygenateblood

removeCO2

pulmonaryartery

pulmonaryvein

leftatrium

rightventricle

LA L OB PA PV RCO RV left atrium (LA) 0 0.20 0.25 0.18 0.12 0.26 0.11 Lungs (L) 0.20 0 0.06 0.14 0.17 0.07 0.17 oxygenated blood (OB) 0.25 0.06 0 0.20 0.19 0.09 0.23 pulmonary artery (PA) 0.18 0.14 0.20 0 0.23 0.14 0.07 pulmonary vein (PV) 0.12 0.17 0.19 0.23 0 0.24 0.19 remove CO2 (RCO) 0.26 0.07 0.09 0.14 0.24 0 0.20 right ventricle (RV) 0.11 0.17 0.23 0.07 0.19 0.20 0

0.17

Some use association information,usually called “neighborhoods”.

SM automatic scoring Pilot A group of 12 practicing teachers enrolled in

CI 400 at PSU completed SMs to describe the structure and function of the heart and then wrote essays on this topic from their maps.

SM “distance” data were obtained using Concept Mapper software (available at: http:/www.personal.psu.edu/rbc4/cm1.htm) that I developed last March for this purpose.

SM “link” array data were colleted by manually entering 1’s for linked terms and 0’s for unlinked terms into an Excel spreadsheet.

. . . Pilot Computer-derived LSA Essay scores were

obtained by pasting the participant’s essays into the Web-based form available at: http://www.personal.psu.edu/rbc4/frame.htm

Manually determined SM and essay scores were determined by 5 pairs of judges

Variables and correlation results are shown on the next slides

Internal variables Links (L) – arithmetic sum of all links in the map Link agreement with an expert (L/Exp) – the

arithmetic sum of the links that exactly match the expert map

Associations (A) – first convert the proposition closeness to a link array (.13 as cut-off), then the arithmetic sum of all links in the map

Association agreement with an expert (A/Exp) – convert proposition closeness to links (.13 as cut-off), then the arithmetic sum of the links that exactly match the expert map

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.00 0.05 0.10 0.15 0.20 0.25

Cut-off distance

Lin

k vs

Ass

ocia

tion

corr

elat

ion

(r)

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.00 0.05 0.10 0.15 0.20 0.25

Cut-off distance

Lin

k vs

Ass

ocia

tion

corr

elat

ion

(r)

Criterion Variables LSA Essay – Essay score established by

Latent Semantic Analysis software, using Landau and Kintsch's Web site

Semantic Map (Map) – Map scores established by averaging the scores from 6 pairs of judges using Lomask et al., 1992 rubric for assessing semantic maps

Essay – Overall essay scores established by averaging the scores from 5 pairs of judges (using a rubric)

HUMAN

Correlation matrix

Judge Metric L L/Exp A A/Exp LSA Map Essay

computer L 1

L/Exp 0.74 1

A 0.89 0.65 1

A/Exp 0.79 0.81 0.91 1

LSA 0.69 0.83 0.78 0.82 1

human Map 0.68 0.87 0.76 0.87 0.74 1

Essay 0.61 0.80 0.76 0.82 0.75 0.98 1

Computer

HumanHuman

Significant correlations shown in bold.

Maps and Essays The strongest correlation, r = 0.98, was

shown for “Map” and “Essay”, both human derived metrics.

Since the semantic maps were used as an aid in writing the essays, it seems reasonable that the two should be highly related.

This high correlation (based on human raters) provides criterion-related evidence of semantic map validity. If confirmed in later studies, science teachers may reasonably use semantic map scores for student assessment.

A/Exp The automatically derived variable

association agreement with an expert (A/Exp) was significantly correlated with the human derived “Map” (.87) and “Essay” (.82) scores, as well as with the computer derived “LSA essay” score (.82).

Thus the first pilot suggests that association agreement with an expert is a promising automatic measure of science content knowledge (like PathFinder C scores).

The automatically derived variable link agreement with an expert (L/Exp) was significantly correlated with the human derived “Map” score (.87), as well as with the computer derived “LSA essay” score (.83).

Thus, link agreement with an expert is also a promising automatic measure of science content knowledge (most extant studies involving SMs use some variant of L/Exp).

L/Exp

Links VS. associations In a multiple regression analysis of these

two automatic variables to human derived essay scores, association agreement with an expert accounted for 67% of the variance, link agreement with an expert accounted for an additional 6%, so the two together accounted for 73% of the variance in human rater essay scores (multiple r = 0.85).

So association (neighborhood) and link (propositions) information each account for some unique components of the essays.

Next steps

Field trials – Confirm pilot results; examine criterion-related validity for SMs to MC, CR, essay, and other test forms; determine cut-score approach for association arrays; find the best algorithm for score generation and automate it; improve the software, especially automating link capture; much more…

Present follow-up investigation 60 undergraduate students in intro

EdPsyc completed an instructional text on the heart, developed concept maps of the content on paper, then completed a verbatim- and an application-level multiple-choice posttest on the lesson content.

I am using the Concept Mapper software to establish the distance arrays, and the same manual procedure for link arrays.

Example concept map

Student TN-10 Concept Map

..follow-up investigation So far, data is collected, I’m establishing the arrays now, Then I will determine the cut-score for

transforming the distances to link arrays, Then, I’ll calculate correlations with the

MC tests, and finally Submit a manuscript early 2003, and use

these two as a basis for obtaining a grant for a larger field-trial and for software development

What is the potential of automatic SM assessment? Can automatic semantic map marking

support higher-level learning? Higher-level assessment?

How could teachers/districts use an automatic semantic map marking system?

What value would automatic semantic map marking have for “test” companies?

Next steps . . .

Is there a fit here at McREL?NCLB:-State standards must be developed for science by the 2005-06 school year.-Beginning in the 2005-06 school year, tests must be administered every year in grades 3 through 8 in math and reading.-Beginning in the 2007-08 school year, science achievement must also be tested.

A Roadmap to Professional Practice, Norm 5:- Choose teaching and assessment strategies that help students develop understandings of math and science.- Choose teaching and assessment strategies that are compatible to one another.- Use multiple methods and tools and systematically gathering data about students’ scientific and mathematical reasoning skills and their understandings about math and science concepts.- Incorporate ongoing, embedded, diagnostic, prescriptive, and summative assessment into instruction.- Provide opportunities for students to demonstrate their knowledge, understandings, and skills in a variety of ways.- Use the results of assessments at different levels and in a variety of ways to improve teaching and learning.- Communicate student progress to the student and his/her parent(s) or guardian.- Review assessment tasks for the use of stereotypes, offensive or irrelevant language, or assumptions that reflect the perspectives or experiences of a particular group.- Recognize that the purpose of an assessment may be different in different situations.

Next steps . . .

Is there a fit here at McREL?

If so, what are our next steps?