assessment methods

Assessment MethodsAssessment MethodsOffice of Institutional Research, Planning, and AssessmentOffice of Institutional Research, Planning, and Assessment

Neal F. McBride, Ed.D., Ph.D.Neal F. McBride, Ed.D., Ph.D.Associate ProvostAssociate Provost

How would you assess these SLOs?

Graduates are able to critique a brief draft essay, pointing out the grammatical, spelling, and punctuation errors and offer appropriate suggestions to correct identified deficiencies

Senior undergraduate psychology majors perform above the national average on the GRE Psychology Subject Test

In a “capstone course” during the final semester prior to graduation; required to critique a supplied essay containing predetermined errors; evaluated by a 3-person faculty panel (criteria: appropriate suggestions to remediate 90% of the errors)

GRE Psychology Subject Test; completed during the senior year, required for graduation. Compare average GRE Psychology Subject Test scores with average scores of all examinees nationwide

Assessment MethodsAssessment Methods

Assessment methods are ways to Assessment methods are ways to ascertain (“measure”) student achievement ascertain (“measure”) student achievement

levels associated with stated student levels associated with stated student learning outcomes (SLOs)learning outcomes (SLOs)

“Outcome” is a generic term for goals, objectives, and/or aims

MissionMissionVision Vision

UniversityUniversityStudentStudent

Outcomes Outcomes

StudentStudentLearningLearning

Outcomes Outcomes

AssessmentAssessmentMethodsMethods

A specific assessment method(s) is selected for A specific assessment method(s) is selected for a specific outcome. . . “How do I ‘measure’ a specific outcome. . . “How do I ‘measure’ thisthis outcome?”outcome?”

Basis for Selecting Basis for Selecting Appropriate Assessment Appropriate Assessment MethodsMethods

Assessment MethodsAssessment Methods

Assessment methods include both Assessment methods include both directdirect and and indirectindirect approaches. . . We’ll define approaches. . . We’ll define these terms in a few minutes.these terms in a few minutes.

First, let’s explore a few criteria or First, let’s explore a few criteria or considerations to keep in mind as you considerations to keep in mind as you select appropriate assessment methods...select appropriate assessment methods...

Qualitative Versus Quantitative MethodsQualitative Versus Quantitative Methods

Qualitative assessment:Qualitative assessment: collects data that does not collects data that does not lend itself to quantitative methods but rather to lend itself to quantitative methods but rather to interpretive criteria; “data” or evidence are often interpretive criteria; “data” or evidence are often representative words, pictures, descriptions, representative words, pictures, descriptions, examples of artistic performance, etc.examples of artistic performance, etc.

Quantitative assessment:Quantitative assessment: collects representative collects representative data that are numerical and lend themselves to data that are numerical and lend themselves to numerical summary or statistical analysisnumerical summary or statistical analysis

Programs are free to select assessment methods Programs are free to select assessment methods appropriate to their discipline or service.... choices appropriate to their discipline or service.... choices must be must be valid and reliablevalid and reliable

Valid and Reliable MethodsValid and Reliable Methods

ValidValid:: The method is appropriate to the academic The method is appropriate to the academic discipline and measures what it is designed to discipline and measures what it is designed to measuremeasure

ReliableReliable:: The method yields consistent data each The method yields consistent data each time time it is used and persons using the method are it is used and persons using the method are consistent in implementing the method and consistent in implementing the method and interpreting the data interpreting the data

Basic AimBasic Aim: : “defensible methods”“defensible methods”

Embedded assessmentEmbedded assessment - - “measurement” strategies “measurement” strategies included as part of the requirements within existing included as part of the requirements within existing courses, internships, or other learning experiences– courses, internships, or other learning experiences– “double duty” assessment; e.g., “critical assignments”“double duty” assessment; e.g., “critical assignments”

Ancillary assessment - Ancillary assessment - “measurement” strategies “measurement” strategies added on or added on or in addition toin addition to requirements within existing requirements within existing courses, internships, or other learning experiences– courses, internships, or other learning experiences– “additional duty” assessment“additional duty” assessment

Locus of AssessmentLocus of Assessment

Sources for FindingSources for FindingAssessment MethodsAssessment Methods

Professional associations and organizations

Other programs/departments at CBU

Similar programs/departments at other universities

Published Resources

Dunn, D. S., Mehrotra, C. M. and Halonen J. S. (2004). Measuring Up: Educational Assessment Challenges and Practices for

Psychology. APA: Washington, DC.

Web... In general or for your specific areahttp://www.liberalarts.wabash.edu/assessment

Literature search by a professional librarian

Personal experience – yours or colleagues

Does it “fit” the SLO?

Did the faculty or student services staff select the method and are they willing to participate in its use?

Will all students in the program or provided service be included in the assessment (ideally, yes) or a sample of students (maybe)?

How much time is required to complete the assessment method? Determine how this affects faculty, staff, and students

When SELECTING ANY ASSESSMENT method, here are some questions to consider carefully:

When and where will the assessment be administered?

Are there financial costs? Are program and/or university resources available?

Is the method used at one point in time (cross-sectional method) or utilized with students over several points in time (longitudinal method)?

Does the program faculty/staff have the skills and/or knowledge necessary to use the method and analyze the results?

Most importantly... WHO is responsible to make certain the assessment is accomplished?

TIPTIP

Ideally…. as you write or rewrite SLOs keep in mind the question: “What method(s) can I use to assess this SLO?”

Why is this tip potentially useful?

Direct MethodsDirect Methods

Direct assessmentDirect assessment methodsmethods are “measurement” are “measurement” strategies that require students to strategies that require students to actively demonstrateactively demonstrate achievement levels related to institutional and program-achievement levels related to institutional and program-specific learning outcomes specific learning outcomes

Direct assessment methodsDirect assessment methods focus on collecting focus on collecting evidence on student learning or achievement directly evidence on student learning or achievement directly from students using work they submit (assignment, from students using work they submit (assignment, exam, term paper, etc.) or by observing them as they exam, term paper, etc.) or by observing them as they demonstrate learned behaviors, attitudes, skills, or demonstrate learned behaviors, attitudes, skills, or practicepractice

Capstone or Senior-Level projectsCapstone or Senior-Level projects, papers, presentations, , papers, presentations, performances, portfolios, or research evaluated by faculty or performances, portfolios, or research evaluated by faculty or external review teams... effective as assessment tools when external review teams... effective as assessment tools when the student work is evaluated in a standard manner, focusing the student work is evaluated in a standard manner, focusing on student achievement of program-level outcomeson student achievement of program-level outcomes

ExamsExams - locally developed comprehensive exams or entry-to- - locally developed comprehensive exams or entry-to-program exams, or national standardized exams, certification program exams, or national standardized exams, certification or licensure exams, or professional exams or licensure exams, or professional exams

Internship or PracticumInternship or Practicum - evaluations of student knowledge - evaluations of student knowledge and skills from internship supervisors, faculty overseers, or and skills from internship supervisors, faculty overseers, or from student participants themselves. This may include written from student participants themselves. This may include written evaluations from supervisors focused on specific knowledge or evaluations from supervisors focused on specific knowledge or skills or evaluation of student final reports or presentations from skills or evaluation of student final reports or presentations from internship experiences.internship experiences.

Direct Methods: ExamplesDirect Methods: Examples

PortfoliosPortfolios (hard-copy or web-based) - reviewed by faculty (hard-copy or web-based) - reviewed by faculty members from the program, faculty members from outside the members from the program, faculty members from outside the program, professionals, visiting scholars, or industrial boardsprogram, professionals, visiting scholars, or industrial boards

Professional Jurors or EvaluatorsProfessional Jurors or Evaluators to evaluate student to evaluate student projects, papers, portfolios, exhibits, performances, or recitalsprojects, papers, portfolios, exhibits, performances, or recitals

Intercollegiate CompetitionsIntercollegiate Competitions - useful for assessment when - useful for assessment when students are asked to demonstrate knowledge or skills related students are asked to demonstrate knowledge or skills related to the expected learning outcomes within appropriate programsto the expected learning outcomes within appropriate programs

Course assessmentsCourse assessments - these are projects, assignments, or - these are projects, assignments, or exam questions that directly link to program-level expected exam questions that directly link to program-level expected learning outcomes and are scored using established criteria; learning outcomes and are scored using established criteria; common assignments may be included in multiple sections common assignments may be included in multiple sections taught by various professors (assuming prior agreement)taught by various professors (assuming prior agreement)

Direct Methods, continued

Direct Methods: AdvantagesDirect Methods: Advantages

Require students to actively demonstrate Require students to actively demonstrate knowledge, attitudes, and/or skillsknowledge, attitudes, and/or skills

Provide data to directly measure expected Provide data to directly measure expected outcomesoutcomes

Demand less abstract interpretationDemand less abstract interpretation

Usually “easier” to administerUsually “easier” to administer

Direct Methods are always our first choice;Direct Methods are always our first choice;indirect methods support indirect methods support

but cannot replace direct methodsbut cannot replace direct methods

Achievement Levels or CriteriaAchievement Levels or Criteria

Rarely does every student achieve all SLOs Rarely does every student achieve all SLOs completely, 100%; nor can we expectcompletely, 100%; nor can we expect this this

What “level” of achievement is acceptable? What “level” of achievement is acceptable? Identified in the “OPlan”Identified in the “OPlan”

Rubrics recognize varying achievement levelsRubrics recognize varying achievement levels

Rubrics are a Rubrics are a scoringscoring method or technique method or technique appropriate to many assessment methodsappropriate to many assessment methods

Outcome Novice Developing Proficient Accomplished

Correctly analyzes research data

1 Limits analysis to correct basic descriptive analysis.

2 Selects and executes correct basic statistical analyses

3 Selects, articulates, and executes an inferential statistical analysis

4 Selects, articulates, and executes the statistical analysis suitable to the research question

A Rubric ExampleA Rubric Example

Excellent resource:

Stevens, D. D. & Levi, A. J. (2005). Introduction to Rubrics.

Sterling, VA: Stylus.

CBU utilizes 4-point rubrics, with the specific level CBU utilizes 4-point rubrics, with the specific level criteria appropriate to the outcome in questioncriteria appropriate to the outcome in question

Guidelines for ImplementingGuidelines for ImplementingImbedded, Direct AssessmentImbedded, Direct Assessment

Link class assignments to both SLOs and course Link class assignments to both SLOs and course objectivesobjectives

If multiple sections of the same course exist and the If multiple sections of the same course exist and the intent is to aggregate data across sections, ensure that intent is to aggregate data across sections, ensure that the assessment is the same in all sections (same the assessment is the same in all sections (same assignment and grading process)assignment and grading process)

Make certain faculty collaboration underpins Make certain faculty collaboration underpins assessment across multiple course sectionsassessment across multiple course sections

Tell students which assignment(s) is being used for Tell students which assignment(s) is being used for SLO assessment as well as course assessment…Why?SLO assessment as well as course assessment…Why?

Indirect MethodsIndirect MethodsMethods requiring the faculty and student life staff to Methods requiring the faculty and student life staff to inferinfer actual student abilities, knowledge, and values actual student abilities, knowledge, and values rather than observing direct evidence of learning or rather than observing direct evidence of learning or achievementachievement

Indirect assessment is gathering information through Indirect assessment is gathering information through means other than looking at actual samples of student means other than looking at actual samples of student work... e.g., surveys, exit interviews, and focus groupswork... e.g., surveys, exit interviews, and focus groups

Indirect methods provide Indirect methods provide perceptionsperceptions of students, of students, faculty, or other people (often alumni or employers) who faculty, or other people (often alumni or employers) who are interested the program, service, or institutionare interested the program, service, or institution

Indirect methods expand on or confirm what is Indirect methods expand on or confirm what is discovered after first using direct methodsdiscovered after first using direct methods

Indirect Methods, Indirect Methods, ContinuedContinued

Exit interviews and Student Surveys - to provide meaningful assessment information, exit interviews and/or student surveys should focus on students’ perceived learning (knowledge, skills, abilities) as well as students’ satisfaction with their learning experiences; including such things as internships, participation in research, independent projects, numbers of papers written or oral presentations given, and familiarity with discipline tools

Faculty Surveys aimed at getting feedback about faculty perceptions of student knowledge, skills, values, academic experiences, etc.

Alumni Surveys aimed at evaluating perceptions of knowledge, skills, and values gained while studying in a particular program. . . surveys frequently target alumni who are 1-and 5- years post-graduation and include program-specific questions

Indirect Methods, Continued

Surveys of Employers / Recruiters aimed at evaluating specific competencies, skills, or outcomes

Tracking Student Data related to enrollment, persistence, and performance... may include graduation rates, enrollment trends, transcript analysis (tracking what courses students take and when they take them), and tracking student academic performance overall and in particular courses


External ReviewersExternal Reviewers provide peer review of academic provide peer review of academic programs and the method is a widely accepted in programs and the method is a widely accepted in assessing curricular sequences, course development assessing curricular sequences, course development and delivery, as well as faculty effectiveness. . . and delivery, as well as faculty effectiveness. . . using external reviewers is a way to assess whether using external reviewers is a way to assess whether student achievement reflects the standards set forth student achievement reflects the standards set forth in student learning and capacity outcomes. . . skilled in student learning and capacity outcomes. . . skilled external reviewers can be instrumental in identifying external reviewers can be instrumental in identifying program strengths and weaknesses leading to program strengths and weaknesses leading to substantial curricular and structural changes and substantial curricular and structural changes and improvementsimprovements


Curriculum and syllabus analysisCurriculum and syllabus analysis – Examining – Examining whether the courses and other academic whether the courses and other academic experiences are related to the stated outcomes... experiences are related to the stated outcomes... often accomplished in a chart or “map.”often accomplished in a chart or “map.”


Syllabus analysis is an especially useful technique Syllabus analysis is an especially useful technique when multiple sections of a course are offered by a when multiple sections of a course are offered by a variety of instructors. . . provides assurance that variety of instructors. . . provides assurance that each section covers essential points without each section covers essential points without prescribing the specific teaching methods used in prescribing the specific teaching methods used in helping the students learn the outcomes helping the students learn the outcomes

Keeping records or observingKeeping records or observing students' use of students' use of facilities and services... data can be correlated with facilities and services... data can be correlated with test scores and/or course grades test scores and/or course grades

Example:Example: Logs maintained by students or staff Logs maintained by students or staff members documenting time spent on course work, members documenting time spent on course work, interactions with faculty and other students, interactions with faculty and other students, internships, nature and frequency of library use, internships, nature and frequency of library use, computer labs, etc.computer labs, etc.


Advantages of Indirect MethodsAdvantages of Indirect Methods

Relatively easy to administerRelatively easy to administer

Provide clues about what could/should be Provide clues about what could/should be assessed directlyassessed directly

Able to flesh out subjective areas direct Able to flesh out subjective areas direct assessments cannot captureassessments cannot capture

Particularly useful for ascertaining values and Particularly useful for ascertaining values and beliefsbeliefs

Surveys can be given to many respondents at Surveys can be given to many respondents at the same timethe same time

Indirect Methods Advantages, Indirect Methods Advantages, ContinuedContinued

Surveys are useful for gathering information from alumni, Surveys are useful for gathering information from alumni, employers, and graduate program representativesemployers, and graduate program representatives

Exit interviews and focus groups allow questioning Exit interviews and focus groups allow questioning students face-to-face; exploring and clarifying answers is students face-to-face; exploring and clarifying answers is done more easily done more easily

External reviewers can bring objectivity to assessment External reviewers can bring objectivity to assessment and answer questions the program or department wants and answer questions the program or department wants answered or questions based on discipline-specific answered or questions based on discipline-specific national standardsnational standards

Disadvantages of Indirect MethodsDisadvantages of Indirect Methods

Indirect methods provide only impressions and Indirect methods provide only impressions and opinions, not “hard” evidence on learningopinions, not “hard” evidence on learning

Impressions and opinions may change over time and Impressions and opinions may change over time and with additional experiencewith additional experience

Respondents may tell you what they think you want to Respondents may tell you what they think you want to hearhear

Survey return rates are often low and, consequently, Survey return rates are often low and, consequently, not representativenot representative

Indirect Methods Disadvantages, Indirect Methods Disadvantages, ContinuedContinued

You cannot assume those who did not respond would You cannot assume those who did not respond would responded in the same way as those who did respondresponded in the same way as those who did respond

Exit interviews take considerable time to completeExit interviews take considerable time to complete

Focus groups usually involve a limited number of Focus groups usually involve a limited number of respondents who are not representativerespondents who are not representative

Unless the faculty agree upon the questions asked Unless the faculty agree upon the questions asked during exit interviews and focus groups, there may not during exit interviews and focus groups, there may not be consistency in responsesbe consistency in responses

Suggestions for ImplementingSuggestions for ImplementingIndirect, Ancillary AssessmentIndirect, Ancillary Assessment

Use “purposeful samples” when it is not possible to Use “purposeful samples” when it is not possible to include all students (which is include all students (which is alwaysalways the first choice) the first choice)

Offer incentives to participantsOffer incentives to participants

Anticipate low turn-out and therefore over-recruitAnticipate low turn-out and therefore over-recruit

Plan carefully logistics and question design (i.e., Plan carefully logistics and question design (i.e., surveys, interviews, focus groups)surveys, interviews, focus groups)

Train group moderators and survey interviewersTrain group moderators and survey interviewers

Implementation SuggestionsImplementation Suggestions, , ContinuedContinued

Consider using web-based or telephone as well Consider using web-based or telephone as well as face-to-face interviews or focus groupsas face-to-face interviews or focus groups

Set time limits for focus groups and interviewsSet time limits for focus groups and interviews

Develop and provide very careful, explicit Develop and provide very careful, explicit directionsdirections

Be wary of FERPA regulations when using Be wary of FERPA regulations when using archival recordsarchival records

Only use archival records that are relevant to Only use archival records that are relevant to specific outcomesspecific outcomes

Capitalize on what you are already doingCapitalize on what you are already doing

Integrate Integrate imbedded assessment as much as possibleimbedded assessment as much as possible

Schedule ancillary assessment during regular class Schedule ancillary assessment during regular class times or times when students are presenttimes or times when students are present

Make assessment a graduation requirementMake assessment a graduation requirement

Plan an “assessment day”Plan an “assessment day”

Seek to make assessment a routine activity within Seek to make assessment a routine activity within your curriculum or student services programsyour curriculum or student services programs

Implementing Assessment in GeneralImplementing Assessment in General

REVIEW:REVIEW:Assessment Assessment

Strategy CombinationsStrategy Combinations

Imbedded, Imbedded, directdirect assessment assessment

Imbedded, Imbedded, indirectindirect assessment assessment

Ancillary, Ancillary, directdirect assessment assessment

Ancillary, Ancillary, indirectindirect assessment assessment

Depending on the specific SLO, there are four Depending on the specific SLO, there are four assessment strategies or frames:assessment strategies or frames:

REMEMBER:REMEMBER: There is more than one way to assess any given SLO! There is more than one way to assess any given SLO! It’s your choice as long as it is valid and reliable.It’s your choice as long as it is valid and reliable.

assessment methods

Documents