Enrich Program Evaluation With Student Learning Outcomes Assessment

CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation

2013 CRLA ConferenceThe Boston Park Plaza HotelNovember 9, 2013Karen S. Agee, Ph.D.Reading and Learning Coordinator EmeritaUniversity of Northern Iowakaren.agee@uni.edu1CAS Assessment: Using Student Learning Outcomes to Transform Program EvaluationOn a scale of 1 (clueless) to 10 (guru), how knowledgeable are you of student learning outcomes (SLO) assessment strategies?

1 2 3 4 5 6 7 8 9 10

2CAS Assessment: Using Student Learning Outcomes to Transform Program EvaluationDeveloping SLO Expertise:Do our programs and services help students to succeed?

3CAS Assessment: Using Student Learning Outcomes to Transform Program EvaluationSelf-assessment of student learning outcomes is now required by accrediting agencies for all programs.

4CAS Assessment: Using Student Learning Outcomes to Transform Program EvaluationHigher Learning Commission Requirement

4.B. The institution demonstrates acommitment to educational achievement and improvement through ongoing assessment ofstudent learning.(continued. . . )5CAS Assessment: Using Student Learning Outcomes to Transform Program EvaluationHigher Learning Commission Requirement, continued:

1. The institution has clearly stated goalsfor student learning and effective processes for assessmentof student learning and achievement of learning goals.

6CAS Assessment: Using Student Learning Outcomes to Transform Program EvaluationHigher Learning Commission Requirement, continued:

2. The institution assesses achievement of the learning outcomes that it claims for its curricular and co-curricular programs.

7CAS Assessment: Using Student Learning Outcomes to Transform Program EvaluationHigher Learning Commission Requirement, continued:

3. The institution uses the information gained from assessment to improve student learning.

8CAS Assessment: Using Student Learning Outcomes to Transform Program EvaluationHigher Learning Commission Requirement, continued:

4. The institutions processes and methodologies to assess student learning reflect good practice, including the substantial participation of faculty and other instructional staff members.

9CAS Assessment: Using Student Learning Outcomes to Transform Program EvaluationPostsecondary institutions seek reaccreditation every 10 years but are expected to show long-term data and evidence. 10CAS Assessment: Using Student Learning Outcomes to Transform Program EvaluationThe Higher Learning Commissionwants every institution to develop a culture of assessment.11Are we concerned about/with student learning outcomes assessment?

12Stages of Concern Related to Change

Stage 0, Awareness:I am not concerned about SLO assessment.Stage 1, InformationalI would like to know more.Stage 2, PersonalHow will SLO assessment affect me?Stage 3, ManagementI seem to be spending all my time assessing student learning.13Stages of Concern Related to Change (cont.)

Stage 4, ConsequenceHow will my assessment strategies improve student learning?Stage 5, CollaborationHow can I relate what I am doing to what others are doing?Stage 6, RefocusingI have ideas about some even better assessment strategies! Adapted from Hord, Rutherford, Huling-Austin, and Hall, 1987, Taking Charge of Change14CAS Assessment: Using Student Learning Outcomes to Transform Program EvaluationAssessing student learning outcomes is a necessary and fascinating aspect of evaluation,

15CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation. . . and the CAS standards for learning assistance programs provide an outline of learning outcomes to consider.

16CAS Assessment: Using Student Learning Outcomes to Transform Program EvaluationIs it even possible to assess student learning in a learning center ?

How can tutoring center staff document gains in student learning ?

17CAS Assessment: Using Student Learning Outcomes to Transform Program EvaluationOur thesis:Nobody on campus can better assess student learning outcomes (SLO) than we can.18CAS Assessment: Using Student Learning Outcomes to Transform Program EvaluationDo we share these premises? 1. Program evaluation is necessary.2. Student learning assessment is an important part of program evaluation.19Program EvaluationNew to higher education in the 1960sDifficult for learning centers in the 1970sRare for developmental education programs before 1980sAfter publication of A Nation at Risk, considered essential for the improvement of teaching and learning[T]he educational foundations of our society are presently being eroded by a rising tide of mediocrity that threatens our very future as a Nation and a people.If an unfriendly foreign power had attempted to impose on America the mediocre educational performance that exists today, we might well have viewed it as an act of war.20CAS Assessment: Using Student Learning Outcomes to Transform Program EvaluationAssessments in program evaluation?

Assessment of the need for the programAssessment of program designAssessment of how the program is being implementedAssessment of the program's outcome or impactAssessment of the program's cost and efficiency

21Assessment Exercise #1

~ volunteers needed ~22CAS Assessment: Using Student Learning Outcomes to Transform Program EvaluationWhat is assessment?

Assessment is measurement against a standard.23CAS Assessment: Using Student Learning Outcomes to Transform Program EvaluationA learning assistance center is any place where learners, learner data, and learning facilitators are interwoven into a sequential, cybernetic, individualized, people-oriented system to service all students (learners) and faculty (learning facilitators) of any institution for whom learning by its students is important (Christ, 1971, p. 39).24CAS Assessment: Using Student Learning Outcomes to Transform Program EvaluationA learning center must then by definition be a goal-focused program that steers by feedback.

25Program Evaluation Measuresper Hunter Boylan, 1981Student grade point averages

Gain scores from pretest to posttest

Ratings of student satisfaction with program services

Student retention26Limitations of Typical Evaluation Measures: GPAs

Many variables may affect course grades and grade point averages

Difficult to measure impact of any specific activity

Stepwise regression incorporating all factors of GPA requires sophisticated statistical skills27Limitations of Typical Evaluation Measures: Gain ScoresInteractive effect of pretesting

Invalidity of commercial, norm-referenced test for assessing student gain

Potential difference between gain score and actual behavior

28Limitations of Typical Evaluation Measures: Student SatisfactionQuestions constructed to elicit favorable responsesBias from survey setting or conditionsBias from student attitude and unrelated experiencesNot always differentiated by student usageSatisfaction not synonymous with skill

29Limitations of Typical Evaluation Measures: Student RetentionEven more generalized than GPAs

Confounded by transient issues causing dropping out temporarily

Lower retention sometimes associated with program success30Program EvaluationMeanwhile, back in the 1970s, student affairs professionals decided to write standards for student services programs. 31CAS StandardsCouncil for the Advancement of Standards in Higher Education (CAS)www.cas.eduFounded in 1979Promotes standards in student affairsCreates book of professional standards and guidelines Develops self-assessment guides

32CAS Assessment: Using Student Learning Outcomes to Transform Program EvaluationOverview of CAS StandardsGeneral Standards (2011), www.cas.edu43 functional area standards available for purchase as digital documents or printed booksLearning Assistance Program Standards and Guidelines and LAP Self-Assessment Guide available for $35 at www.cas.edu Standards store

33CAS StandardsWhich standards appropriate for us?

Learning Assistance Programs (LAP)

TRiO and Other Educational Opportunity Programs (TOEOP)34CAS StandardsLearning Assistance Programs (LAP)

Approved by CAS in 1986 with expert input from CRLA and NADEand revised in 1996 and 200735CAS StandardsUltimate purpose of CAS?

To foster and enhance student learning, development, achievement, and citizenship. 36CAS Standards: 12 PartsMissionProgramOrganization and LeadershipHuman ResourcesEthicsLaw, Policy, and Governance37CAS Standards: 12 SectionsDiversity, Equity, and AccessInstitutional and External RelationsFinancial ResourcesTechnologyFacilities and EquipmentAssessment and Evaluation38CAS Standards: ProgramCAS Learning and Developmental Outcomes (2008)Six domains, 28 dimensions

Programs and services must assess relevant and desirable student learning and development outcomes and provide evidence of their impact on student learning and development. Programs must articulate how they contribute to or support students learning and development in the domains not specifically assessed.

39CAS LAP Standards Lets look now at the LAP standards (pp. 5-10 of handout) and the Learning and Developmental Outcomes (separate handout)

40Orthodox Methods of Assessing SLOsDirect methodsStudents demonstrate learning via some form of standardized test focusing on aspects of student learningExamples: CAAP, CLA, ETS Proficiency Profile (formerly MAPP), GRE subject tests, PRAXIS exams

Maki, P. L. (2004). Assessing for learning: Building a sustainable commitment across the institution. Sterling, VA: Stylus.

41Orthodox Methods of Assessing SLOsDirect methodsStudents demonstrate learning via some form of standardized test focusing on aspects of student learningExamples: CAAP, CLA, ETS Proficiency Profile (formerly MAPP), GRE subject tests, PRAXIS exams . . . but couldnt we assess even more directly?

42Orthodox Methods of Assessing SLOsIndirect methodsStudents report perceptions of their learning and the educational environment that supports that learningExamples: CSEQ, NSSE, NSLLP

Maki, P. L. (2004). Assessing for learning: Building a sustainable commitment across the institution. Sterling, VA: Stylus.

43Orthodox Methods of Assessing SLOsIndirect methodsStudents report perceptions of their learning and the educational environment that supports that learningExamples: CSEQ, NSSE, NSLLP

Couldnt students report their perceptions even more powerfully?

44Orthodox Methods of Assessing SLOsPerformance-based methodsStudents represent learning in response to assignments or projects that are embedded in their educational experiences

Examples: Successful solution of a problem not previously understoodDemonstrated writing skill in a senior honors paper

Maki, P. L. (2004). Assessing for learning: Building a sustainable commitment across the institution. Sterling, VA: Stylus.

45Orthodox Methods of Assessing SLOsPerformance-based methodsStudents represent learning in response to assignments or projects that are embedded in their educational experiences

Examples: Successful solution of a problem not previously understoodDemonstrated writing skill in a senior honors paper

Couldnt we assess students performance?

46CAS Self-Assessment OutlineReview/revise missionIdentify program goals and learning goalsAlign major programs and services with goalsSpecify desired outcomes Program and operation outcomes Learning and development outcomesMap learning and development goals to CAS learning domains and dimensionsDetermine assessment approach47CAS Self-Assessment OutlineReview/revise missionIdentify program goals and learning goalsAlign major programs and services with goalsSpecify desired outcomes Program and operation outcomes Learning and development outcomesMap learning and development goals to CAS learning domains and dimensionsDetermine assessment approach48Specify Desired OutcomesProgram and Operation Learning and Development49Write SLO Statements:Specify who, under what conditions, what, to what extent, how, why

Consider immediate learning and longer-term outcomes

50CAS Self-Assessment OutlineReview/revise missionIdentify program goals and learning goalsAlign major programs and services with goalsSpecify desired outcomes Program and operation outcomes Learning and development outcomesMap learning and development goals to CAS learning domains and dimensionsDetermine assessment approach51CAS Self-Assessment Outline Map learning and development goals to CAS learning domains and dimensions

52CAS Self-Assessment OutlineReview/revise missionIdentify program goals and learning goalsAlign major programs and services with goalsSpecify desired outcomes Program and operation goals Learning and development goalsMap learning and development goals to CAS learning domains and dimensionsDetermine assessment approach53Enrich Program Evaluation With SLO AssessmentWhat student learning outcomes do you want to assess?

Any volunteers?

54Enrich Program Evaluation With SLO AssessmentKeeling & Associates Format for Writing Student Learning OutcomesSomeone who does ___ will be able to do ___ to a degree, and it is demonstrated by their doing ____, which can be shown to ___ who requires it for ____.

55Enrich Program Evaluation With SLO AssessmentIn learning assistance programs, we seek productive behavior change as evidence of learning.

Students who complete Speed Reading will report an increase in assignment reading of at least 50%, as demonstrated by pre- and post-course surveys.Students who complete Speed Reading will be able to read twice as efficiently, as demonstrated by doubling reading efficiency score on pre/post DRTs.

56Enrich Program Evaluation With SLO AssessmentIn learning assistance programs, we seek productive behavior change as evidence of learning.

Participants in an In One Ear and Out the Pen workshop will be able to list and describe the four basic steps of Quiz Notes in their own words, as demonstrated by reporting these steps orally to the presenter at the end of the workshop.Participants in an In One Ear and Out the Pen workshop will be able to use Quiz Notes to prepare for an exam in a lecture course as demonstrated by bringing completed notes to an Ask-a-Tutor session and answering at least 75% of Quiz Notes questions correctly without peeking when quizzed by a Peer Instructor.

57Enrich Program Evaluation With SLO AssessmentIts time to look at the assessment samples handout.

58Three Assessment Approaches for LAPsDorothy Williams Measures of Metacognitive Development

UNI Reading & Learning Centers Knowledge-Skills-Attitudes

May and Harris Scale: Where Are You Now?

59Dorothy Williams Measures of Metacognitive Developmentwww.lsche.net/resources/lrnr_asses/assess_slo_metacgntv_rubric.htmSLOs and instruments developed for metacognitive development, levels of critical thinking, locus of control, and use of learning stylesRubrics developed for metacognitive motivation, acquisition, retention, and performanceSLOs and instruments used in general tutoring, writing center, math center, reading center, academic skills centerSLOs and instruments used for both appointment-based and drop-in services of these offices60UNI Reading & Learning Centers Knowledge-Skills-Attitudes

Developed from UNIs Qualities of an Educated Person project by Karen Agee and Peer InstructorsK-S-A used for Ask-a-Tutor appointments and Coordinator consultationsK-S-A assessment made collaboratively with student whenever possible End-of-session assessment techniqueGrowth and change in Knowledge-Skills-Attitudes

61UNI Reading & Learning Centers Knowledge-Skills-AttitudesKnowledgeUnderstand specific subject knowledge (1)Synthesize multiple readings, text and notes, or diverse sources (2)Synthesize course content with lived experience (3)See implications of new knowledge (4)Seek new information to solve problems (4)Think critically (5)Rethink previous assumptions (6)Reason effectively (7)Formulate creative approaches (8)

62UNI Reading & Learning Centers Knowledge-Skills-AttitudesSkillsAccurately self-appraise skills, abilities, and needs (9)Develop relationships with faculty, staff, and students (13)Seek and offer help appropriately (14) Work collaboratively with others (15)Set and pursues goals (21)Communicate effectively (22)Demonstrate technological competence (23)Manage time effectively (24)63UNI Reading & Learning Centers Knowledge-Skills-AttitudesAttitudesDemonstrate ethics and integrity (11)Demonstrate the joy of learning (12)Explore issues of purpose, meaning, courage, and hope (12)Treat others with respect (13)Appreciate diversity and difference (17)Demonstrate social responsibility (18)

64May and Harris Scale: Where Are You Now? Developed to assess problems and progress in tutoring sessions

Used by tutors and student clients to make progress visible

65May and Harris Scale: Where Are You Now? Setting Goals for a Tutoring SessionCRLA Handbook for Training Peer Tutors and Mentors Mark S. May, Clayton State University, & Jacqueline Harris, Ball State University As soon as [session] goals have been set, the tutor can ask, On a scale of 1-10, how far along are you to reaching this goal? This simple exercise sets the stage for the end of the session, when the tutor can ask the same question again. It gives both the tutor and student the opportunity to provide each other with immediate feedback and positive reinforcement. It also helps define short- and longer-term goals for the client and establishes a possible framework for the next session.

1 2 3 4 5 6 7 8 9 10

66Mission StatementsDelta College (Michigan):The mission of the Teaching/Learning Center is to support students to become successful independent learners. This support from professional staff, peer tutors, other students, or by way of T/LC services to faculty includes determination of needs; individual, group, and distance facilitation; and services needed to promote lifelong learning and the mission of Delta College.67Mission StatementsLiberty University (Lynchburg, VA):The primary purpose of the Bruckner Learning Center is to provide University-wide academic support services for all students and faculty in general and special needs students in particular.

68Mission StatementsOwensboro Community and Technical College (Kentucky):The Teaching and Learning Center's Mission is: to promote student success by sharing our knowledge of the learning process with students, faculty, administration, and staff in a collaborative partnership for the enhancement of learning; andto promote student success by providing academic support services to faculty. 69Mission StatementsParadise Valley Community College (Phoenix, AZ):

The mission of the Learning Support Center is to support learning and help everyone achieve their potential. We provide hope and encouragement in a friendly, safe environment.

70Mission StatementsUniversity of Texas (Austin):UT Learning Center: Our mission is to empower students in their pursuit of lifelong learning within a safe and inclusive learning environment.71Mission StatementsVirginia Wesleyan College (Norfolk):The Learning Center coordinates tutorial, placement, and academic support services at Virginia Wesleyan College. Academic support services include transitional advising during Summer Orientation and Winter Session, academic counseling for provisional students and those on academic probation, and disability services. Our mission is to give students the resources they need to become successful independent learners while maintaining an open channel of communication with faculty. We supplement faculty instruction by providing peer and professional tutoring, appropriate resource materials to supplement course work, study skills workshops, and alternate testing facilities.

72Mission StatementsSouthwestern Michigan College:The Teaching & Learning Centers of SMC are dedicated to creating an environment where students, staff, faculty and community members may seek support to help achieve success in both their academic and personal lives.

73What is a rubric?A rubric is a scoring scale utilized to measure a students or staff members performance against a predetermined set of criteria.

74What does a rubric do?A rubric

divides a desired outcome into its component parts

identifies what are acceptable and unacceptable degrees of performance for a specific outcome

75These component parts serve as criterion points,

and

the rubric provides explanations of appropriate degrees of performance for each criterion

76Enrich Program Evaluation With SLO AssessmentBack to our first question:

On a scale of 1 (clueless) to 10 (guru), how knowledgeable are you of student learning outcomes (SLO) assessment strategies?

1 2 3 4 5 6 7 8 9 10

77Enrich Program Evaluation With Student Learning Outcomes AssessmentReferencesBoylan, H. R. (1981). Program evaluation: Issues, needs, and realities. In C. C. Walvekar (Ed.), Assessment of learning assistance services (pp. 3-16). San Francisco, CA: Jossey-Bass.

Christ, F. (1971). Systems for learning assistance: Learners, learning facilitators, and learning centers. In F. L. Christ (Ed.), Interdisciplinary aspects of reading instruction: Proceedings of the Fourth Annual Conference of the Western College Reading Association (pp. 32-41). Los Angeles, CA: WCRA.

Hord, S. M., Rutherford, W. L., Huling-Austin, L., & Hall, G. E. (1987). Taking charge of change. Alexandria, VA: ASCD.

Maki, P. L. (2004). Assessing for learning: Building a sustainable commitment across the institution. Sterling, VA: Stylus.

78 Karen S. Agee, Ph.D., served the College Reading and Learning Association (CRLA) as secretary, president-elect (conference chair), president, past president (coordinator of state, region, and chapter leaders and special interest group leaders) and executive assistant to the Board. Karen currently represents CRLA on the Council of Learning Assistance and Developmental Education Associations (CLADEA) and on the Board of Directors of the Council for the Advancement of Standards in Higher Education (CAS). She serves on the editorial board of the Journal of Developmental Education, The Learning Assistance Review, and the Journal of College Reading and Learning. She has received CRLAs Robert Griffin Award for Long and Outstanding Service and the CRLA Boards Special Recognition Award, UNIs Exemplary Service Award, and the Iowa Board of Regents Award for Staff Excellence. Karen coordinated the Reading and Learning Center of the learning center at the University of Northern Iowa 1984-2009. She had originally intended to teach Latin and Greek but was sidetracked by interest in the reading comprehension difficulties of her elementary and college students in Indiana and New Mexico. In 2012 Karen co-edited with Russ Hodges the Handbook for Training Peer Tutors and Mentors (Cengage Learning). She previously collaborated with Dr. Hodges on the program management chapter in Handbook of College Reading and Study Strategy Research (2nd ed., edited by Rona F. Flippo and David C. Caverly, 2008, Routledge).