Assessment Assessment WorkshopWorkshopPresented by:Presented by:The Office of Institutional Research and The Office of Institutional Research and AssessmentAssessment

O ffice of Institu tional P lanning and Budget

VacantM anagem ent Assistant

Ann M cM illanAdm inistrative Assistant

Jane BaillargeonAssistant D irector, P rogram R ev iew and Assessm ent

R ay W allaceInstitu tiona l R esearch Analyst

Jason M ayerP rogram m er Analyst

A rch ie G eorgeD irector, Institu tional R esearch and Assessm ent

Beverly R hoadesAssociat D irector

C retia BunneyBudget Specia list

Janice T odishBudget Specia list

M ark B ra inardD irector, Budget O ff ice

W ayland W insteadExecutive D irector

Assessment WorkshopAssessment Workshop

• Why do assessment?Why do assessment?

• What is assessment?What is assessment?

• When to assess?When to assess?

• How to assess?How to assess?

Why do assessment?Why do assessment?

Improvement Improvement

AccountabilityAccountability

AccreditationAccreditation

ImprovementImprovement

• CurriculumCurriculum

• Instructional methodology and Instructional methodology and practicepractice

• Student servicesStudent services

““Nothing is so perfect Nothing is so perfect that it cannot be that it cannot be improved upon.”improved upon.”

Trudy Trudy BantaBanta

AccountabilityAccountability

• State Board of Education/ Board of State Board of Education/ Board of RegentsRegents

• Public accountabilityPublic accountability

• Competition for limited resourcesCompetition for limited resources

““Every publicly supported Every publicly supported social services agency social services agency now has an outcome-now has an outcome-based agenda.”based agenda.”

Trudy BantaTrudy Banta

AccreditationAccreditation

• Northwest Association of Schools and Northwest Association of Schools and Colleges (NASC) Colleges (NASC) – We are to demonstrate through regular We are to demonstrate through regular

and systematic assessment that and systematic assessment that students who complete their programs, students who complete their programs, no matter where or how they are offered, no matter where or how they are offered, have achieved a specified set of learner have achieved a specified set of learner goals established for each program. goals established for each program.

““There must be evidence There must be evidence that students are that students are mastering the course and mastering the course and curriculum objectives. curriculum objectives. Given the ferocious Given the ferocious competition developing competition developing among learning among learning organizations worldwide, organizations worldwide, these [assessments] are these [assessments] are necessary steps.”necessary steps.”

Trudy BantaTrudy Banta

What is assessment?What is assessment?

Assessment is a systematic process Assessment is a systematic process of looking at student achievement of looking at student achievement within and across courses by within and across courses by gathering, interpreting and using gathering, interpreting and using information about student learning information about student learning for educational improvement.for educational improvement.

American Association of Higher Education American Association of Higher Education (AAHE)(AAHE)

Characteristics of good Characteristics of good assessmentassessment• comprehensive, ongoing and evolutionary comprehensive, ongoing and evolutionary

• broad involvement from facultybroad involvement from faculty

• clear, assessable educational goals and objectivesclear, assessable educational goals and objectives

• uses a variety of assessment and evaluation uses a variety of assessment and evaluation methodologiesmethodologies

• collects meaningful and accurate datacollects meaningful and accurate data

• primary emphasis is on improvement of teaching and primary emphasis is on improvement of teaching and learninglearning

(adapted from Seybert, 1998)(adapted from Seybert, 1998)

Why aren’t grades Why aren’t grades enough? enough? • grading practices are not standardgrading practices are not standard

• need different ways of structuring need different ways of structuring program assessmentprogram assessment

• grades reflect many things other grades reflect many things other than course content and masterythan course content and mastery

• objectives differobjectives differ

• good assessment requires multiple good assessment requires multiple ways of measuring goal achievementways of measuring goal achievement

When to AssessWhen to AssessCan be Decided According to When Critical Can be Decided According to When Critical Decision-Making and Communication OccursDecision-Making and Communication Occurs

• Entry assessment helps determine who should be Entry assessment helps determine who should be admitted and who is prepared to benefit from which admitted and who is prepared to benefit from which programs and courses.programs and courses.

• Midpoint assessment occurs when students reach a Midpoint assessment occurs when students reach a crucial decision point or level of attainment in their crucial decision point or level of attainment in their program of studies.program of studies.

• Exit assessment helps determine which students have Exit assessment helps determine which students have attained the prerequisite knowledge, skills, and abilities attained the prerequisite knowledge, skills, and abilities associated with the program goals.associated with the program goals.

• Follow-up (Post Completion) assessment helps Follow-up (Post Completion) assessment helps determine the effectiveness of the educational determine the effectiveness of the educational programs in preparing students for further education, programs in preparing students for further education, transfer, entry or reentry into the workforce, or the transfer, entry or reentry into the workforce, or the students’ personal goals.students’ personal goals.

NCTLANCTLA

How to assess? How to assess? • Identify each degree and certificate program to be Identify each degree and certificate program to be

assessed.assessed.

• Identify student learning goals and the Identify student learning goals and the educational criteria and experiences for each goal.educational criteria and experiences for each goal.

• Identify appropriate assessment methods and Identify appropriate assessment methods and strategies.strategies.

• Collect, analyze and interpret data.Collect, analyze and interpret data.

• Specify program improvements.Specify program improvements.

Identify each degree and Identify each degree and certificate program to be certificate program to be assessed.assessed.

• All certificate programsAll certificate programs

• All undergraduate programsAll undergraduate programs

• All graduate programsAll graduate programs

• All off-campus programsAll off-campus programs

Identify student learning goalsIdentify student learning goals

• establish three to six goals for each establish three to six goals for each program, both graduate and program, both graduate and undergraduateundergraduate

• identify education criteria and identify education criteria and experiences for each goalexperiences for each goal– What is to be learned?What is to be learned?

– What is the level of learning?What is the level of learning?

– What is the learning applied to?What is the learning applied to?

Identify Key Components of Each GoalIdentify Key Components of Each GoalIdentify Key Skills for Each GoalIdentify Key Skills for Each Goal

Goal: Students will understand and Goal: Students will understand and apply logical and ethical principles to apply logical and ethical principles to personal and social situations.personal and social situations.

• What logical and ethical principles are learned regardless of specific What logical and ethical principles are learned regardless of specific coursework taken?coursework taken?

• How do students show their understanding and ability to apply these How do students show their understanding and ability to apply these principles?principles?

• How do we see students apply principles to their personal lives and How do we see students apply principles to their personal lives and development?development?

• How do we see students applying principles to social settings and How do we see students applying principles to social settings and circumstances?circumstances?

NCTLANCTLA

What if you do not have assessable What if you do not have assessable goals?goals?

1. Examine the set of required courses.1. Examine the set of required courses.

2. Ask, “What have we been trying to teach?” (Outcomes)2. Ask, “What have we been trying to teach?” (Outcomes)

– Content knowledge?Content knowledge?

– Cognitive skills?Cognitive skills?

– Values and attitudes?Values and attitudes?

3. Ask, “What should students know before they enter the 3. Ask, “What should students know before they enter the curriculum in order to succeed?’ (Entrance Criteria)curriculum in order to succeed?’ (Entrance Criteria)

4. Ask, “What should students know when they complete the 4. Ask, “What should students know when they complete the curriculum in order to graduate?” (Exit Criteria)curriculum in order to graduate?” (Exit Criteria)

5. Ask, “At what points in the curriculum are students doing well 5. Ask, “At what points in the curriculum are students doing well or having difficulty?” (Midpoint Criteria)or having difficulty?” (Midpoint Criteria)

6. Ask, “Are our alumni successful in the field?” (Post Completion 6. Ask, “Are our alumni successful in the field?” (Post Completion Criteria)Criteria)

Adapted from NCTLAAdapted from NCTLA

What if you have assessable goals, but no What if you have assessable goals, but no specific curriculum to support them?specific curriculum to support them?

1. Ask, “Do we really teach students (the goal)?”1. Ask, “Do we really teach students (the goal)?”

2. If “Yes,” then identify if the goal is:2. If “Yes,” then identify if the goal is:

– embedded throughout coursework, orembedded throughout coursework, or

– achieved through an identifiable pattern of courseworkachieved through an identifiable pattern of coursework

3. State the specific coursework pattern required to attain the 3. State the specific coursework pattern required to attain the general education goal.general education goal.

4. Identify entry ability required for students to succeed at the 4. Identify entry ability required for students to succeed at the collegiate level.collegiate level.

5. Identify key midpoints in the development of student abilities 5. Identify key midpoints in the development of student abilities along the general education goal.along the general education goal.

6. Identify levels of attainment or performance required for 6. Identify levels of attainment or performance required for graduation.graduation.

7. Identify levels of attainment or performance in an employment 7. Identify levels of attainment or performance in an employment setting.setting.

NCTLANCTLA

What to Assess:What to Assess:• Knowledge outcomes - core of concepts Knowledge outcomes - core of concepts

and material knowledgeand material knowledge

• Skills outcomes - what a student can doSkills outcomes - what a student can do

• Attitudes and values outcomes - those Attitudes and values outcomes - those faculty believe to be importantfaculty believe to be important

• Behavioral outcomes - behaviors crucial Behavioral outcomes - behaviors crucial to the curriculum’s impactto the curriculum’s impact

Bloom’s Classification of Cognitive Bloom’s Classification of Cognitive SkillsSkills• KnowledgeKnowledge

• ComprehensionComprehension

• ApplicationApplication

• AnalysisAnalysis

• SynthesisSynthesis

• EvaluationEvaluation

Knowledge OutcomesKnowledge Outcomes

• Describe the basic components of Describe the basic components of empirical research.empirical research.

• Give examples of major themes or Give examples of major themes or styles in Music, Art, or Theatre.styles in Music, Art, or Theatre.

• Recognize in complex text logical, Recognize in complex text logical, rhetorical, and metaphorical patterns.rhetorical, and metaphorical patterns.

NCTLANCTLA

Comprehension OutcomesComprehension Outcomes

• Correctly classify a variety of plant Correctly classify a variety of plant specimens.specimens.

• Explain the scientific method of inquiry.Explain the scientific method of inquiry.

• Summarize the important intellectual, Summarize the important intellectual, historical, and cultural traditions in historical, and cultural traditions in Music, Art, or Theatre from the Music, Art, or Theatre from the Renaissance to Modern times.Renaissance to Modern times.

NCTLANCTLA

Application OutcomesApplication Outcomes

• Demonstrate in the laboratory a working Demonstrate in the laboratory a working knowledge of lab safety procedures.knowledge of lab safety procedures.

• Apply oral communication principles in Apply oral communication principles in making a speech.making a speech.

• Compute the area of a room.Compute the area of a room.

• Use editing symbols and printers’ marks.Use editing symbols and printers’ marks.NCTLANCTLA

Analysis OutcomesAnalysis Outcomes

• Distinguish between primary and secondary Distinguish between primary and secondary literature.literature.

• Diagram a sentence.Diagram a sentence.

• Listen to others and analyze their Listen to others and analyze their presentations.presentations.

• Differentiate between historical facts and Differentiate between historical facts and trivia.trivia.

NCTLANCTLA

Synthesis OutcomesSynthesis Outcomes

• Revise faculty copy for a news story.Revise faculty copy for a news story.

• Formulate hypotheses to guide a Formulate hypotheses to guide a research study.research study.

• Create a poem, painting, design for Create a poem, painting, design for a building.a building.

NCTLANCTLA

Evaluation OutcomesEvaluation Outcomes

• Compare art forms of two diverse cultures.Compare art forms of two diverse cultures.

• Critically assess an oral presentation.Critically assess an oral presentation.

• State traditional and personal criteria for State traditional and personal criteria for evaluating works of art.evaluating works of art.

• Draw conclusions from experimental Draw conclusions from experimental results.results.

NCTLANCTLA

Identify the educational Identify the educational experiences for each goal.experiences for each goal.

List actions intended to enable the students to achieve these List actions intended to enable the students to achieve these goals.goals.

Goal # Course/Experience

Level ofLearning

Applicationof Learning

1 Philosophy 101 Knowledge None1 Philosophy 210 Comprehension Social issues1 Psychology 310 Analysis Personal

situations1 Sophomore Field

ExperienceApplication Group

situations1 History 200 Comprehension Historical

events1 Internship Application Business

situations

Identify appropriate Identify appropriate assessment methods and assessment methods and strategiesstrategies• Choose one or two goals from each Choose one or two goals from each

programprogram

• Identify appropriate measures of goal Identify appropriate measures of goal attainmentattainment

• Identify the appropriate point of Identify the appropriate point of measurementmeasurement

• Use multiple methods of assessmentUse multiple methods of assessment

Characteristics of Effective Performance Characteristics of Effective Performance MeasuresMeasures• Relate to goalsRelate to goals

• Focus on the vital few elements to measureFocus on the vital few elements to measure

• Foster improvementFoster improvement

• Are well communicatedAre well communicated

• Are reviewed as often as appropriateAre reviewed as often as appropriate

• Provide information on level, trend and comparative/ competitive Provide information on level, trend and comparative/ competitive datadata

• Focus on the long-term well-being of the student and the programFocus on the long-term well-being of the student and the program

Adapted from Engelkemeyer, 1998Adapted from Engelkemeyer, 1998

Measurement BlockersMeasurement Blockers

• Fuzzy goals or action strategiesFuzzy goals or action strategies

• Unjustified trust in informal feedback systemsUnjustified trust in informal feedback systems

• Entrenched measurement systemsEntrenched measurement systems

• Incorrect focusIncorrect focus

• No agreement on prioritiesNo agreement on priorities

Adapted from Engelkemeyer 1998Adapted from Engelkemeyer 1998

Collect, analyze and interpret Collect, analyze and interpret datadata• What did you find?What did you find?

• What were the program’s strengths and What were the program’s strengths and weaknesses?weaknesses?

• How well are the students learning the How well are the students learning the concepts?concepts?

• Is the learning achieved appropriate for Is the learning achieved appropriate for the level?the level?

• Is the learning being appropriately Is the learning being appropriately applied?applied?

Assessment Reveals the Gap

Curricular Goals

The Performance Gap

Student Outcomes

National Center on Postsecondary Teaching, Learning, & AssessmentNational Center on Postsecondary Teaching, Learning, & Assessment

Assessment Helps Close the Gap

Curricular Goals

Student Outcomes

National Center on Postsecondary Teaching, Learning, & AssessmentNational Center on Postsecondary Teaching, Learning, & Assessment

Identify Strategies for Identify Strategies for Change Change • What will you do to improve student learning? What will you do to improve student learning?

• Which program elements should be reinforced?Which program elements should be reinforced?

• Which program elements should be maintained?Which program elements should be maintained?

• Which program elements should be Which program elements should be strengthened?strengthened?

• Which program elements should be modified?Which program elements should be modified?

– at the undergraduate level? at the undergraduate level?

– at the graduate level? at the graduate level?

– off campus?off campus?

Be flexible, adaptive and prepared to change. Be flexible, adaptive and prepared to change.

• There will There will alwaysalways be problems. be problems.

• Things Things alwaysalways change (mandates, change (mandates, circumstance, personnel, priorities.)circumstance, personnel, priorities.)

• View assessment as an evolutionary process.View assessment as an evolutionary process.

(Seybert, 1989)(Seybert, 1989)

ReferencesReferencesThe Assessment Institute Resource Book.The Assessment Institute Resource Book. Assessment Institute. Assessment Institute.

National Center for Postsecondary Teaching, Learning and National Center for Postsecondary Teaching, Learning and Assessment, Burlington, VT. October 15-17, 1998.Assessment, Burlington, VT. October 15-17, 1998.

Ball State University. Ball State University. Assessment Workbook.Assessment Workbook. Munci: Ball State Munci: Ball State University Offices of Academic Assessment, 1992.University Offices of Academic Assessment, 1992.

Black, Karen E., Trudy W. Banta, and Jane L. Lambert. “Best Black, Karen E., Trudy W. Banta, and Jane L. Lambert. “Best Practices in Program Review.” AAHE Conference on Practices in Program Review.” AAHE Conference on Assessment. American Association of Higher Education, Assessment. American Association of Higher Education, Cincinnati, OH. June 14, 1998.Cincinnati, OH. June 14, 1998.

Bloom, Benjamin, et al. (ed) Bloom, Benjamin, et al. (ed) Taxonomy of Educational Objectives: Taxonomy of Educational Objectives: Handbook I. Cognitive DomainHandbook I. Cognitive Domain. NY: David McKay, 1956. NY: David McKay, 1956

Ewell, Peter T., ed. “Assessing Educational Outcomes.” Ewell, Peter T., ed. “Assessing Educational Outcomes.” New New Directions for Institutional ResearchDirections for Institutional Research 47 San Fransisco: Jossey- 47 San Fransisco: Jossey-Bass, Sept, 1985.Bass, Sept, 1985.

Halpern, Diane F., ed. “Student Outcomes Assessment: What Halpern, Diane F., ed. “Student Outcomes Assessment: What Institutions Stand to Gain.” Institutions Stand to Gain.” New Directions for Higher EducationNew Directions for Higher Education 59 San Fransisco: Jossey-Bass, Fall, 1987.59 San Fransisco: Jossey-Bass, Fall, 1987.

References Cont.References Cont.

Idaho State Board of Education. Idaho State Board of Education. Governing Policies and Governing Policies and Procedures.Procedures. Boise: Idaho State Board of Education, 1994. Boise: Idaho State Board of Education, 1994.

Engelkemeyer, Susan West. “Key Measures in Organizational Engelkemeyer, Susan West. “Key Measures in Organizational Performance.” AAHE Conference on Assessment. American Performance.” AAHE Conference on Assessment. American Association of Higher Education, Cincinnati, OH. June 13, Association of Higher Education, Cincinnati, OH. June 13, 1998.1998.

Wright, Barbara D. “Assessment for Beginners: Getting Wright, Barbara D. “Assessment for Beginners: Getting Started.” AAHE Conference on Assessment. American Started.” AAHE Conference on Assessment. American Association of Higher Education, Miami Beach, FL. June 11, Association of Higher Education, Miami Beach, FL. June 11, 1997.1997.

Seybert, Jeffrey A. “Community College Assessment Seybert, Jeffrey A. “Community College Assessment Strategies.” AAHE Conference on Assessment. American Strategies.” AAHE Conference on Assessment. American Association of Higher Education, Cincinnati, OH. June 14, Association of Higher Education, Cincinnati, OH. June 14, 1998.1998.

Sims, Serbrenia J. Sims, Serbrenia J. Student Outcomes Assessment: A Historical Student Outcomes Assessment: A Historical Review and Guide to Program Development.Review and Guide to Program Development. Connecticut; Connecticut; Greenwood Press, 1992.Greenwood Press, 1992.

NEED MORE NEED MORE INFORMATION?INFORMATION?

Archie GeorgeArchie George Jane BaillargeonJane Baillargeon

885-7995885-7995 885-5828885-5828

Archie@uidaho.eduArchie@uidaho.edujane@uidaho.edujane@uidaho.edu