CS1 Assessment Using Memory Diagrams

Mark A. HollidayDavid R. LuginbuhlDept of Mathematics and Computer ScienceWestern Carolina University

SIGCSE Technical Symposium on Computer Science EducationNorfolk, VAMarch 5, 2004

10/21/2003 2

Overview

Background Introduction to Memory Diagrams

– A simple example

Qualitative Assessment– Noting and correcting common errors

Quantitative Assessment– Results so far

Conclusion

10/21/2003 3

Motivation

Problem:– How to introduce and reinforce object-oriented

concepts in an introductory CS course – How to help CS1 students visualize the state of the

computer as the Java program executes

10/21/2003 4

Motivation

Solution:– Develop a visual representation of each change in

the state of memory (abstractly) due to the object-oriented effects of executing a sequence of Java statements

– Memory diagrams are more detailed than the code since there is at least one change in the state of memory per statement.

– Often there are several changes in the state of memory for a single statement.

10/21/2003 5

Motivation

Solution continued:– Write the fragment in both Java and the visual

language– The visual language must be detailed and precise– The visual representation accesses an alternative

and supplemental mode of thinking

10/21/2003 6

Relationship to Psychology

In other application domains psychologists have studied the effect on text comprehension by supplementing with visual representations– P. Goolkasian, Picture-Word Differences in a

Sentence Verification Task, Memory and Cognition, 1996.

– B. Henderson, personal communication. 2003.

10/21/2003 7

An “Elementary” Analogy

In elementary school, many of us were taught the rules of good grammar through sentence diagramming

– The teacher would show students the parts of a sentence with diagrams

– Students would also be required to diagram sentences to better understand the concepts of grammar

The white rabbit eats green lettuce.

rabbit eats lettuce.

Thewhite

green

10/21/2003 8

Method Invocation

otherColor = “red”;

aColor = “blue”; // a)

aColor = aColor.concat(otherColor);

// rhs call is b)

// rhs return is c)

// lhs is d)

String

“blue”

String aColor String otherColor

String

“red”

concatconcat

( )

String

“bluered”concat

X

10/21/2003 9

Language Design Principles

Use shape to reinforce when concepts are the same– e.g., all kinds of variables (local variables, parameters, and

fields) are rectangles

Use shape to reinforce when concepts differ– e.g., variables, objects, classes all have different shapes

(rectangles, circles, and diamonds)

Use position to reinforce concepts– e.g., the reference contained in a reference variable starts

inside the variable, not on the border– e.g., the rectangle representing an instance variable is

placed inside the circle representing the object

10/21/2003 10

Student Exposure to Diagrams

We introduce these diagrams to students on the first day of CS1 class

We ask students to produce diagrams of their own– In in-class groupwork – In weekly closed labs– On quizzes and tests

10/21/2003 11

Student Exposure to Diagrams

Example: in-class groupwork– all groups at blackboards so that everyone can

see the diagrams and multiple group members can be drawing simultaneously

– everyone must be able to explain during the demonstration

– no chalk for the strongest group member– at least a third of “lecture” time is spent doing this

(with an upperclass student helper assisting)

10/21/2003 12

Student Exposure to Diagrams

How do we have the time for all the groupwork and diagramming?

Ours is a minimalist CS1:– no exception handling– no inheritance, abstract classes, or interfaces– no recursion– no inner classes or event-driven code– no graphics– but we do real console I/O (not hidden)

Bottom line: reinforcement of concepts in a number of contexts

– But not just for learning…

10/21/2003 13

Student Assessment

By having students use diagrams themselves, we have them demonstrate their comprehension of object-oriented concepts

We believe these diagrams have potential for measuring programming comprehension

10/21/2003 14

Qualitative Student Assessment

Test problem from last year:

– Diagram the program fragment at the right

Note last line particularly

– Static public variable (System.out)

– Method composition– PrintStream object– Dog object– Creation of a String

object

Dog spot; // fig a)spot = new Dog("spot");

// right hand side is fig b)// left hand side and equal sign is fig c)

System.out.println(spot.toString());// fig d)

Student attempts to produce figure a

10/21/2003 15

Common Mistakes

Figure a– creating the object as well as the variable– putting variable name inside variable rectangle

Dog spot; // fig a)spot = new Dog("spot");

// right hand side is fig b) // left hand side and equal sign is fig c)

System.out.println(spot.toString()); // fig d)

Dog spot

Solution:

Student attempts to produce figure b

10/21/2003 16

Common Mistakes

Figure b– not creating an object: putting “spot” inside spot

rectangle– name field problems:

• not showing field’s rectangle• putting field’s rectangle on object boundary• not showing String object for “spot” and that the name field

holds a reference to that String object

– reference problems:• arrow in wrong direction• arrow starts on the boundary of the rectangle instead of

inside the rectangle

Dog spot; // fig a)spot = new Dog("spot");

// right hand side is fig b) // left hand side and equal sign is fig c)

System.out.println(spot.toString()); // fig d)

10/21/2003 17

Common Mistakes

Figure b– already placing the reference in the spot variable

Dog spot; // fig a)spot = new Dog("spot");

// right hand side is fig b) // left hand side and equal sign is fig c)

System.out.println(spot.toString()); // fig d)

Dog spot

Dog

String

“spot”

name

Solution:

Student attempts to produce figure c

10/21/2003 18

Common Mistakes

Figure c– placing the reference on the boundary of the spot

rectangle instead of inside

Dog spot; // fig a)spot = new Dog("spot");

// right hand side is fig b) // left hand side and equal sign is fig c)

System.out.println(spot.toString()); // fig d)

Dog spot

Dog

String

“spot”

nameSolution:

Student attempts to produce figure d

10/21/2003 19

Common Mistakes

Figure d

Dog spot; // fig a)spot = new Dog("spot");

// right hand side is fig b) // left hand side and equal sign is fig c)

System.out.println(spot.toString()); // fig d)

PrintStream out

name

()

Dog

Dog spotSystem

PrintStream

String

“spot”

printlntoString

Solution:

()

String

“name=spot”

10/21/2003 20

Common Mistakes

Figure d– not show toString method invocation correctly– not show the String object created correctly– not show the reference to the new String object

being passed as the argument to the println method call

– not show that System.out is a static public field of the System class

– now show println method invocation correctly

Dog spot; // fig a)spot = new Dog("spot");

// right hand side is fig b) // left hand side and equal sign is fig c)

System.out.println(spot.toString()); // fig d)

10/21/2003 21

Quantitative Student Assessment

Previous example illustrates how the diagrams are used qualitatively to assess student comprehension– to pinpoint problems and provide proper

reinforcement of specific concepts

Can we also use them for quantitative assessment?

Can we also evaluate how effective they are in assessment?

10/21/2003 22

Quantitative Assessment (cont.)

Completed Study: Correlation of score on memory diagram question with rest of test and with course score

Three test questions from two CS1 classes Two studies (correlation with rest of test and

correlation with course score) for each of the three questions (three experiments)

10/21/2003 23

Quantitative Assessment (cont.)

An example quantitative comparison using the third test question (experiment three)

Histogram is sorted by memory diagram question score

The three series of scores appear to track each other

Mean score of the memory diagram question is significantly lower

– not surprising since a correct memory diagram requires a deeper understanding of the effect of a program fragment

Experiment Three for Both Studies

0

0.2

0.4

0.6

0.8

1

1.2

1 2 3 4 5 6 7 8 9 10 11 12 13

Question

Rest of Test

Course

10/21/2003 24

Conclusion – Memory Diagrams

A relatively low-tech approach for teaching OO concepts

– Well-suited for classroom, labs, exams– “Write” a program fragment in both Java and in the

visual language– Importance of shape and placement for reinforcing

concepts– Having students make their own diagrams adds to

this reinforcement

10/21/2003 25

Conclusion – Memory Diagrams

Promise of diagrams for measuring comprehension

– If students can diagram what is happening in memory, they are probably understanding the deeper meaning of the program

– Measured assessment effectiveness relative to rest of test and course score

– Currently measuring assessment effectiveness compared to corresponding Java program fragment

10/21/2003 26

Conclusion – Memory Diagrams

To learn more Mark A. Holliday and David Luginbuhl, “Using Memory

Diagrams When Teaching a Java-Based CS1”, Proc. of the 41st ACM Southeast Conference, Savannah, GA, March 2003.

Mark A. Holliday and David Luginbuhl, “CS1 Assessment Using Memory Diagrams”, Proc. Of SIGCSE 2004, Norfolk, VA, March, 2004

Mark A. Holliday, “Introducing Java Using Memory Diagrams”, (include short guide to the notation); http://cs.wcu.edu/~holliday/LectureNotes/150/lectures.html

Presentation slides: http://cs.wcu.edu/~holliday/sigcse04.ppt