choosing your tools

Created by Michael Gos

Choosing Your ToolsHow Do We Measure the Learning

Outcomes?

For the Instructional Learning Outcomes Council (I-LOC)


The Next Step

• We have now set learning outcomes based on skills and knowledge we want students to retain in five years.

• With those specific learning outcomes in hand, the next step is to select an appropriate measurement tool.

• There are two broad types of measures that can be used for any learning outcome: • Direct• Indirect

• Direct measures seek evidence that the outcomes are being met. This evidence is found in learning products created by students.

• Indirect measures seek opinions on whether they are being met.



Direct and Indirect Measures

Direct Measures Indirect Measures

Standardized Tests Student Surveys

Locally-created tests Student Focus Groups

Classroom Assignments Alumni Surveys

Portfolios Employer Surveys

A sampling of each type of measure:



Choosing your measuring Tool

• The single most important factor in choosing a measuring instrument is Content Validity

Content Validity: the concepts examined by the measuring tool are representative of the concepts laid out in our learning outcome statement(s)

• If the tool doesn’t assess the concept laid out in the learning outcome, choose different learning outcome or tool



Assessment Tools

• Chances are there exist, or you can create, multiple measures with good content validity.

• So how do we choose from the various options?

• In general, it is a matter of money, workload and faculty preferences.

• Let’s take a look at the various types of direct measures.



Standardized Tests

• These are tests sold commercially (although they can be created locally given enough time and work). These are evaluated for validity, reliability, pre- and post-test effects and many other matters of statistical importance.

• They measure what they are supposed to measure• They require a small time commitment• They can be given as part of a class exam,

guaranteeing students will take them

Positive Points:

Negative Points:• They can be expensive to buy

• Faculty often are resistant to their use



Locally-Created Tests

• These are tests created by professors for use in one or more classrooms. Every quiz and exam you write falls into this category.

Positive Points:•Faculty trust them

•Moderate workload (no additional work if used as a regular course exam such as the final, but see below)•Low Cost

Negative Points:•Must be especially careful to guarantee content validity

•If graded by a committee rather than the individual classroom instructor, these can be time consuming



What kind of test?

• Different test types serve different purposes.

• If you are testing for content mastery, then true/false and multiple choice tests will work well.

• True/False and multiple choice questions can work for testing mastery of a skill, but this requires a great deal of thought and creativity on the part of the designer.

• Essay tests can test for any issue, content or skill, but are more time consuming to grade.



Classroom Assignments

• Any paper, quiz, project, exam or performance indicator that would normally be a part of the course requirements.

Positive Points:•Non-intrusive. The class routine is not altered in any

way•If evaluated by the individual instructor, low workload (but see below)•Less faculty resistance

Negative Points:•Difficult to determine content validity, though possible with good training.•Can be time consuming if evaluated by a committee



Portfolios

• A collection of work done by a single student over some period of time (a single course to an entire college career)

Positive Points:

•Can give a broad view of student abilities and accomplishments•Can be used to test several learning outcomes at the same time

Negative Points:•Difficult to collect and assemble. Requires some student

cooperation•Evaluation is either expensive (if contracted out) or time consuming (if graded by faculty)



Some Factors to Consider

• Cost: What is your budget? Can you afford to purchase standardized tests? Can you afford to hire outside evaluators or adjuncts to evaluate measures?

• Faculty time commitment: Are faculty willing to put in the out-of-classroom hours needed to go through training, establish inter-rater reliability and evaluate artifacts?

• And, perhaps most important of all, what do we really want to know about the results of this learning outcome? Let’s look at this last point more closely! For the Instructional Learning Outcomes Council (I-LOC)


What do we really want to learn?

• There are three possible things a measure can tell us:

1. How well did our students perform on this learning outcome?

2. How much have they improved in this area this semester?

3. How much did they improve as a result of this class?

These are important distinctions. Let’s look at each possibility in a little more detail.



Performance on this Learning outcome

• A measure taken at any point in a student’s education will tell us where that student is at that moment.

• Unfortunately, it tells us nothing about how he got there, or when.

• With measures of this type, we can improve student performance on the measure simply by raising college admissions standards.

• In effect, they tell us nothing about what students have learned at Lee College.



Improvement this semester

• If we can determine where a student was before entering a course, and then measure again later, we can narrow our findings considerably

• Using a pre and post measure, we can determine the student’s growth during that time period

• Unfortunately, we still can’t say it came from the course in question, only that it occurred in that time period. However, this information advances our understanding more than a single measure at one point in the semester.For the Instructional Learning Outcomes Council (I-LOC)


Improvement as a result of this class

• The ideal situation, of course, would be to be able to say the improvement seen is a result of what we did in class (sometimes called “value added”).

• This is a difficult threshold to meet. It requires not only pre and post assessing, but requires an experimental design that includes randomization, treatment and control groups, and statistical analysis

• While powerful, this is not always practical.

• Remember, we are only looking for indicators of trend, not proof.

• Hypothesis testing must be reserved for constrained, controlled, concise research questions.



So which is right for You?

• If you are assessing only where your students are now, a post assessment will do. It won’t however, tell us how effective our courses have been.

• If you are assessing what students have learned since the start of class (or their college career), use a pre and post assessment. It won’t guarantee they learned this IN our classes, but it does assure the learning came DURING them.

• If you absolutely need to know what they learned in our classes, use experimental design.

• That may not be practical. We may, in some cases, have to settle for determining what they learned during our classes, regardless of where the learning is acquired.



A word about ethics

• It is unethical to report findings beyond the level that your assessment design measures.

• If you are doing a single assessment, be sure to only state that “our students are here, now.” Do not overstate by identifying this as a measure of anything done by us.

• If you are doing a pre and post assessment, report only that “our students achieved this, in this time period.” Do not attribute it to what we did in the classroom.



Things to watch out for…

• The choice you make here will have a tremendous effect on faculty workload. Make it carefully.

• Beware of unexpected resistance to certain forms of measurement. (Faculty often fear standardized tests, for example).

• Be sure you pick a method that you can actually accomplish. Cutting corners invalidates the best designs.

• Don’t be afraid to do it right. If well thought-out, a good design is no more expensive or work intensive than a poorly designed one.



Reducing Workload

• Obviously, the easiest way to reduce workload is to use standardized tests. Locally created Scantron tests are also low workload options

• If your choice of instrument is something that must be graded or evaluated by the faculty, there are still ways to reduce the workload.

• Create or use an assignment or test that will be a graded part of the class.

• Let the individual faculty members evaluate their own students’ work and forward only the results (not the artifact) to the division

• This is easily accomplished with the use of grading rubrics



A Sample Grading Rubric (Instruction Writing Assignment)

Objective/Criteria

Meets criteria

Meets criteria is most areas

Fails to meet criteria in most places

Instructions are numbered, not lettered or bulleted

(10 points) (8 points) (3 points)

Instructions in command form


No more than one operation per step


Visuals used when appropriate and are accurate representations


Cautions, Dangers, Warnings used as needed and in proper format

(5 points) (3 points) (1 point)

Grammar, spelling, wording and capitalization rules followed

(10 points) (6 points) (2 points)For the Instructional Learning Outcomes Council (I-LOC)


Using Rubrics

• You may have used the “grade forms” in WebCT. These are grading rubrics.

• They make for quick and easy assessment of student work, whether for a course grade or for program assessment.

• Professors can simply add the assessment criteria to the regular grading rubrics used for any assignment.

• If the faculty member decides the learning outcome being assessed by the division is not important to the course grade, points can be set at or near zero for that criterion

Note: Introduction to Rubrics is a resource available in each division or the Center for Teaching Excellence.



Using Rubrics

• Faculty members then send the rubric scores on to the department

• There is no need to forward artifacts

• Student work is graded only once, by the classroom teacher, saving the massive time commitment of multiple readings

• An added bonus: Once instructors try using rubrics, most faculty will like them, even for student grading



A Word about indirect measures

• These are used to seek someone’s opinion on whether a learning outcome has been met.

• Opinions that might be of value can come from students, alumni or employers.

• A common way to get student opinion is to place a question about the learning outcome in the end-of-semester student evaluation.

• Alumni and employer opinion can be had through phone or mailed surveys or through focus groups.



The big decision

• Our goal in all this has been to make our teaching better.

• The decision you make on measuring tool selection will have a huge impact on the quality and usefulness of your findings.

• It will also impact your budget and workload.

• Choose carefully!


choosing your tools

Documents