compiling and reporting your assessment plan data office of institutional assessment &...
TRANSCRIPT
Compiling and Reporting Your Assessment Plan Data
Office of Institutional Assessment & EffectivenessSUNY OneontaSpring 2012
Providing a Context: Oneonta’s Process for Assessing Student
Learning Fall 2009: Approval by President’s Cabinet of APAC Guidelines for Academic Program Assessment
Spring 2009: Submission by programs of Step 1 (Establishing Objectives) of guidelines
December 2010: Submission by programs of Step 2 (Activities & Strategies) of guidelines
June 1, 2011: Submission of Steps 3 (Assessment) and 4 (Closing the Loop) [plans only] All plans approved by academic deans by the end of Fall 2011
2011-12 academic year: First round of data collection, with reports due June 1, 2012 as part of annual reports
Emphases Throughout the Process
Establishing congruence among institutional goals, programmatic and course objectives, learning opportunities, and assessments
Linkages to disciplinary (and, as appropriate, accreditation/certification) standards
Using a variety of assessment measures, both quantitative and qualitative, in search of convergence
Value of course-embedded assessment Course- vs. program-level assessment
Course- Vs. Program-Level Assessment
Focus of SUNY Oneonta assessment planning is programmatic student learning objectives, not individual students or faculty
For the most part, data collection described in approved assessment plans will take place in the context of the classroom (i.e., course-embedded assessment)
Focus at this point, however, is how to compile and aggregate data across courses and course sections, as appropriate, and to report these results in a meaningful way
What You’ve Done So Far
1. Development of programmatic student learning objectives
2. Curriculum mapping3. Development of plan for collecting data across a
three-year period APAC guidelines recommend that programs assess 1/3 of SLOs
every year
4. Development of plan for reviewing data and using that information to revise courses and programs, as appropriate (i.e., “closing the loop”)
Follow Your Curriculum Map – It’s Your Assessment Database!
COURSE
SLOs 1 2 3 4 5 6 7
Introductory Course E E History/Theories E P P Methods E, L E, L L Required Course 1 E E E, P P Required Course 2 P P P
Required Course 3 E E P Required Course 4 I, PO I, PO Capstone PO PO PO PO PO Assessment Key:
P-Paper E-Exam PO-Portfolio O=Oral Presentation L-Lab Assignment I-Internship
Designed correctly, your curriculum map tells you what to assess, when to
assess, and how to assess it!
Collecting Data: Important Points from Spring 2011
Workshop Collaborative work among faculty to assure: SLOs are being assessed using direct measures Measures are of sufficient quality, either by review process or
through the use of a checklist that includes good practice criteria A priori success indicators are in place for each measure as well
as common categories for results (e.g., exceeding, meeting, approaching, not meeting standards)
Decisions on how issue of multiple course sections will be addressed
Process in place for assuring: Reliable scoring of qualitative measures Courses (and students) to be included in assessment are
representative
Value of Assessment Data Will Be Maximized IF:
A variety of assessment measures is used Quantitative and qualitative Course-embedded and “stand alone” measures Benchmarking as available
For each SLO, data are collected from multiple courses/course sections
Criteria for scoring and determining what meets standards are consistent across courses Do measures have to be the SAME? No, but…….
And Perhaps Most Important
There’s no point in collecting useless
data!
Compiling and Reporting Assessment Data Across Courses
Aggregating and Documenting Your Assessment Results
Approaching the Task Systematically
1. Start with your curriculum map, which depicts: The courses in which you’re covering – and
assessing – SLOs And, depending on the detail of your map, the levels
at which SLOs are being addressed and the measures being used to assess them
2. Select courses – and faculty – to be included in the assessment and communicate that expectation clearly and establish a time frame for administering assessment and submitting results
Approaching the Task Systematically (cont.)
3. Collect and compile data (categorized as agreed upon by department) on spreadsheet for review
4. Review results collectively, reach conclusions regarding strengths and weaknesses, and make recommendations for change as appropriate
Be sure to document all this (i.e., write it down)
A Step-by-Step Example
1. Following Your Curriculum MapThe Simplest Case: Which Courses Cover Which SLOs?
To What Extent Do Courses Cover SLOs?
COURSE
SLOs 1 2 3 4 5 6 7
Introductory Course 4 2 History/Theories 4 4 3 Methods 4 4 4 Required Course 1 3 3 2 4 Required Course 2 4 3 4
Required Course 3 4 4 4 Required Course 4 4 4 Capstone 4 3 3 2 4
COURSE
SLOs 1 2 3 4 5 6 7
Introductory Course X X History/Theories X X X Methods X X X Required Course 1 X X X X Required Course 2 X X X
Required Course 3 X X X Required Course 4 X X Capstone X X X X X
1. Following Your Curriculum MapAt What Level Do Courses Cover SLOs?
How Do Courses Assess SLOs?
COURSE
SLOs 1 2 3 4 5 6 7
Introductory Course I I History/Theories I R R Methods I R R Required Course 1 R R R R Required Course 2 R M M
Required Course 3 R R M Required Course 4 M M Capstone M M M M M
Assessment Key:
I-Introduced R-Reinforced M-Mastery
COURSE
SLOs 1 2 3 4 5 6 7
Introductory Course E E History/Theories E P P Methods E, L E, L L Required Course 1 E E E, P P Required Course 2 P P P
Required Course 3 E E P Required Course 4 I, PO I, PO Capstone PO PO PO PO PO Assessment Key:
P-Paper E-Exam PO-Portfolio O=Oral Presentation L-Lab Assignment I-Internship
2. Selecting Courses to be Included in Assessment
Do all courses covering a particular SLO need to be included?
Ideally, yes – remember, each SLO is only assessed every 3 years
Especially important if SLOs are being assessed at different “levels” (in terms of proficiency and/or course levels)
Do all course sections covering a particular SLO need to be included?
No, but why not? If not, important that:
Adequate number of students are included in the assessment Sections are selected randomly
2. Selecting Courses to be Included in Assessment
(cont.) Once courses are selected, communicate this fact quickly and clearly to faculty, being sure to emphasize:
SLOs to be assessed and categories to be used in reporting results
Mapping of assignments – and eventually assessment results – to specific SLOs and the need for a priori criteria in categorizing student performance
Time line for submitting results (late enough in semester for students to demonstrate competence but early enough for data to be compiled)
Must assignments and a priori criteria be the same? No, but the more comparable, the more meaningful the results Ultimately, it’s a programmatic decision
A Few More Hints About Assignments
Final grades are NEVER an adequate measure for a specific SLO
Overall assignment grades are rarely – if ever – an adequate measure for a specific SLO
Often only minor adjustments are needed in existing assignments to gather information on specific SLOs
Limit assignments to one SLO Break assignments into subparts, each of which addresses a
specific SLO Identify test questions that map to specific SLOs
3. Compiling Data for Review and Discussion
Important for all results be submitted in agreed-upon format, organized, and maintained in a single place (an Excel spreadsheet is perfect)
Different ways to organize results for review and deliberation
Best to start with all data for each SLO, showing each course and its overall results for student performance organized by performance categories
The more you can aggregate, the better (i.e., by course, course level, level of proficiency), being attentive to variations within aggregating categories
Importance of removing identifiers
Sample Spreadsheet (for Internal Review by Program
Faculty)Course # # Students Exceeding Meeting Approaching Not Meeting
GER 100 25 24 56 12 8
GER 100 22 95 0 5 0
Sub-Total - GER 100 47 59.5 28 8.5 4
GER 150 24 76 14 10 0
GER 175 12 24 42 27 7
Sub-Total 100-level 83 54.75 28 13.5 3.75
GER 225 47 53.2 29.8 12.8 4.3GER 275 19 15.7 73.7 5.3 5.3
GER 280 23 61 22 13 4
Sub-Total 200-level 89 43.3 41.83 10.37 4.53
GER 330 28 21 50 25 4
GER 335 28 39 29 27 5
GER 350 17 24 46 29 0
Sub-Total 300-level 73 28 41.67 27 3
GER 481 9 33 66 0 0
GER 481 5 0 100 0 0
GER 481 21 24 43 33 0
Sub-Total 400-level 35 19 69.67 11 0
Total 280 37.68 43.96 15.32 2.89
4. Reviewing and Discussing Results and
Closing the Loop Important points of discussion How are data consistent?
Do students at different course levels perform similarly? Eventually, it will be possible to look at this issue over time
How are they distinctive? Do students perform better on some objectives than others?
Are assessment results “acceptable” to program’s faculty? What strengths (and weaknesses) are revealed and what
explains them? What should (and will) the program do to improve areas of
weakness?
Incorporating Assessment Data into Annual Report
At present, directions for Annual Report include the following:
“Descriptions of student learning outcome assessments conducted for your students during the year. These descriptions should include a summary of the assessment methods that were used and the results, including benchmarking as appropriate. Also, whenever possible, detail course-specific student learning outcomes and their relationship(s) to departmental or programmatic expectations.”
Incorporating Assessment Data into Annual Report
(cont.) APAC’s recommendation to the Provost and Deans is that Annual Reports include two distinct components with respect to the assessment of student learning, effective June 2012 Chart that summarizes the assessments that were
done during the academic year and results Narrative that summarizes the program’s discussion of
the assessment results, overall conclusions, and planned changes based on the assessment data
Programs may also report other relevant data (e.g., results from senior/alumni surveys)
Sample Outcomes Assessment
Chart – Aggregate DataSimplest Case (and All That’s Required)
CoursesAssessment Measure(s)
Performance Criteria
# of Students
% Exceeding Standards
% Meeting Standards
% Approaching
Standards
% Not Meeting
StandardsAll Portfolio 4-5=Exceeding
3=Meeting2=Approaching1=Not Meeting
338 18% 59% 13% 10%
Sample Outcomes Assessment
Charts – More DetailedBy Course Level
CoursesAssessment Measure(s)
Performance Criteria
# of Students
% Exceeding Standards
% Meeting Standards
% Approaching
Standards
% Not Meeting
Standards100-Level Tests 90%+=Exceeding
80-89%=Meeting70-79%=Approaching69% and below=Not Meeting
203 18% 59% 13% 10%
200-Level Papers 4=Exceeding3=Meeting 2=Approaching1=Not Meeting
78 18% 59% 13% 10%
Tests 90%+=Exceeding80-89%=Meeting70-79%=Approaching69% and below=Not Meeting
112 18% 59% 13% 10%
300-Level Portfolios 4-5=Exceeding3=Meeting2=Approaching1=Not Meeting
96 18% 59% 13% 10%
Sample Outcomes Assessment
Charts – More DetailedBy Competency Level
CoursesAssessment Measure(s)
Performance Criteria
# of Students
% Exceeding Standards
% Meeting Standards
% Approaching
Standards
% Not Meeting
StandardsBeginning Tests 90%+=Exceeding
80-89%=Meeting70-79%=Approaching69% and below=Not Meeting
203 18% 59% 13% 10%
Inter-mediate
Papers 4=Exceeding3=Meeting 2=Approaching1=Not Meeting
78 9% 61% 20% 10%
Tests 90%+=Exceeding80-89%=Meeting70-79%=Approaching69% and below=Not Meeting
112 11% 54% 25% 10%
Advanced Portfolios 4-5=Exceeding3=Meeting2=Approaching1=Not Meeting
96 15% 54% 18% 13%
Issues to Be Addressed in Narrative Overall conclusions of student performance on
SLO as revealed by assessment Description of strengths and weaknesses
revealed by data Planned revisions for improvements as
appropriate Planned revisions to the assessment process
itself as appropriate
APAC Members
Paul French Josh Hammonds Michael Koch Julie Licata Richard Lee
Patrice Macaluso Anuradhaa Shastri Bill Wilkerson (Chair) Patty Francis (ex
officio)