developing and applying tools

8/13/2019 Developing and Applying Tools

1/43

Developing and Applying Tools:

Rubrics and Exam Questions

n a c ens, .

Dawn Zimmaro, Ph.D.

John Kucsera, Ph.D.

The University of Texas at Austin


2/43

Agenda

1. Overview of SACS

2. Mapping Assessments to Program Outcomes

3. Writing Critical Thinking Exam Questions4. Creating Rubrics


3/43

Academic units

Administrative units


4/43

Agenda

1. Overview of SACS




5/43

Mapping Assessments to Program Outcomes

At what level should you collect data to measure your

outcomes?

Program


6/43

Issues to consider

What existing course assignments, exams, projects, etc.

can you use to measure your learning outcomes?

What program level assessments can you identify or

What is unique about your program that is measurable?


7/43

Classroom level

Exam

Assignment

Presentation Paper

Project


8/43

Program level

Exit exam

Thesis/dissertation

Capstone course paper or project Portfolio

Licensure/professional exam

Exit survey


9/43

English Department B.A.

Example learning outcome:

Students will be able to construct arguments and

communicate them effectively in writing.


10/43

Classroom level assessment

Read description of E 370W Major Authors (Emerson)


11/43

Assignment

Writing assignment that asks students to develop an

argument about an issue

Emersons efforts to examine the authority and effects ofreli ious and educational institutions hel frame

discussions about literature and education for subsequent

generations.

Given the above statement develop a central thesis with

supporting arguments about how Emersons works havehelped to frame discussions about literature and

education.


12/43

Program level assessment

A collection of individual students writing assignments

on constructing arguments taken from several English

courses

Over the course of the whole degree program


13/43

Second learning outcome

Example learning outcome:

Students will be able to demonstrate knowledge of the

history and the structure of the English language.


14/43

Classroom level assessment

Classroom exam

Topics:

History of English: Old to Early Modern English

Topics should all relate to the learning objective


15/43

Program level assessment

Exit exam

Questions taken from all the courses that relate to the

learning outcome


16/43

Sample assessment map

See handout


17/43

Discussion

Have you tried this approach? What worked? What

didnt?

If you havent tried this, why not?


18/43

Agenda

1. Overview of SACS




19/43

Writing Crit ical Thinking Exam Questions

Purpose of session

Learning Outcomes

Critical thinking exam questions Objective and Subjective exam questions

Application activity

Final thoughts


20/43

Purpose

Objective:

To compose critical thinking questions for program or

classroom level exams

Aligning outcomes with methods


21/43

Learning Outcomes

Program to classroom level (alignment)

General statements to specific outcomes

Person centered ear, spec c, an measura e

Classroom learning outcomes should contain

Procedure/method (assignments that will be provided)

Behavior (what the students will do; usually the action verb)

Criteria (how well a learner performs to be judged adequate and thequalitative description of the criteria)


22/43

PhD Program Example

Program Educational Objective: Sixty percent of graduates from the College of Education will attain positions as

college faculty.

(Program) Learning Outcome 1:

Will prepare students as future researchers Graduatin students will demonstrate fundamental knowled e of uantitative,

qualitative, and mixed research designs.

Procedure 1.1 (Classroom learning outcome): Course EDP 382 will focus on the fundamental knowledge necessary to

conduct quantitative research.

Given an exam administered in EDP 382, at least 80% of students willdemonstrate a fundamental knowledge of quantitative research designs by

achieving a score at or above an 80 out of 100. The exam will test the extent towhich students can compute common statistics, identify assumptions ofresearch designs, evaluate.


23/43

Writing Crit ical Thinking

Exam Questions

For critical thinking:

Focus on higher levels of Blooms taxonomy


24/43

Blooms taxonomy for the cognitive domain

Remember

recall or memorize information

Understand

translate from one form to another (e.g., restate in own words)

Apply

Analyze

examine a concept and break it down into parts

Evaluate

make judgments using standards of appraisal Create

put information together in an unique or novel way


25/43


Exam Questions



Behavior component of learning outcome ,

of students will demonstrate a fundamental knowledge

Action verbs (handout)


26/43


Exam Questions



Behavior component of learning outcome __ ,

students will demonstrate a fundamental knowledge

Action verbs (handout)

Outcome to exam questions (and instruction)

Can use the action verb

Ex: Compute the standard deviation of.

Test blueprint (handout)


27/43

Objective and Subjective

Exam Questions

Objective questions

Select a correct response from several alternatives

Subjective questions

Present an original answer reng s wea nesses o eac

Assumptions (True/ False/ or Undetermined)

1. Essay exams require more thorough student preparation andstudy time than objective exams.

2. Essay and objective exams can be used to measure the samecontent or ability.

Examples of objective, critical thinking exam questions (packet)


28/43

Application Activity

In groups or pairs and with using the Emerson content

from Dawns previous presentation,

1. Construct one objective and one subjective examuestion.

2. Explain why you believe your questions are assessing

critical thinking.


29/43

Application Activity

Objective examples:

(Analysis) Emerson might be characterized as any of thefollowing EXCEPT:

(A) a transcendentalist

(B) an American Renaissance writer

(C) a public speaker

(D) a political prophet

(E) a literary critic

(Evaluate) Determine whether the following statement is true orfalse:

Melville and Hawthorne were both disillusionedtranscendentalists.


30/43


31/43

Agenda

1. Overview of SACS




32/43

1.What is a rubric?

2.When should you use a rubric?

3.What are the elements of a rubric?

Creating Rubrics

4.How do you create a rubric?


33/43

What is a rubric?

-A scoring guideline that measures student

achievement systematically:

specifies instructor expectations

lists performance criteria

describes levels of quality


34/43

When should you use a rubric?

On assignments that require subjective grading

Places structure on a subjective process

Essa s

Presentations

Multi-media


35/43

What are the elements of a rubric?

What the

student needs

to do

Descriptors and criteriathat identify performance

expectation for each

point on the scale

I.

Example:

Include a compelling, descriptive, and persuasive

conclusion that summarizes arguments set forth inpaper


36/43

Standards that identify

the range of quality orperformance levels

How well the

student does itII.

Example:--no or poor conclusion or summary of argument

--some summary of points made, but nothing beyond

summary; no broad conclusions/lessons

--

a conclusion going beyond summary of what was written inthe body of the essay


37/43

A scale of points

awarded to specificelements of student

performance

III.

What score the

student willreceive

5 = outstanding

3 = acceptable

1 = poor


38/43

Critical Thinking Rubric

A. Clearly and precisely

analyzes key information,

questions, and problems

B. Uses inference to reason

carefully from clearly stated

premises to important

A. Is unable to analyze

information, questions, and

problems or does so

superficially

B. Is unable to or infrequently

uses inference to reason

1 [2 3 4] 5

implications and

consequences

C. Uses deductive and

inductive reasoning and

problem-solving skills

consistently and with ease

from clearly stated premises

or recognize implications and

consequences

C. Is unable to or infrequently

uses deductive and

inductive reasoning andproblem-solving skills


39/43

Analytic Rubrics

Holistic Rubrics


40/43

How do you create a rubric?

Develop assignment Identify learning objectives

Decide which criteria to evaluate

Decide what kind of rubric to use Define performance indicators

Identify standards of performance

Decide on grading or scoring scale Review rubric to ensure it measures what you intend


41/43

Determine key criteria

If the assignment

objectives are

then consider these criteria

Creative products Wow factor, novelty

Technical quality

Adherence to conventions of the field

Development of ideas

Clarity

Oral presentation Voice projection

Body language

Organization


42/43

Identify Indicators

Concrete sign or evidence of a criterion beingmetnot meant to be exhaustive

Example:

Learning objective: Students will be able to write in

Criterion: Student clearly and precisely analyzes

key information, questions, and problems

Indicator:

Identifies main tension in story and relates itto sub-plots


43/43

developing and applying tools

Documents