Learning Outcomes Assessment: An Introduction

Laura J. Rosenthal, English Department, University of Maryland http://www.english.umd.edu/profiles/lrosenthal [Angelica Kauffman, Lady Scarlett as the Muse of Literature]

What is Outcomes Assessment?

• Figuring out what and much students have learned

• Figuring out what students haven’t learned

• Assessment is based on empirical data (information)

• Tends to work best in an aggregate (a program or a college as opposed to an individual class

• Distinct from grading

Why assess?

Needed for accreditation Essentially a compromise

with those who wanted more direct governmental control over higher education

In this sense, very different from No Child Left Behind

Not standardized; process differs by regional accrediting agencies

Student Learning Parallel history in

education, psychology and other fields as a way to improve learning

Increasing need to explain value of arts and humanities degree and education

Sparks conversations about student learning within programs

Improvement of programs

Who controls assessment?

• Accreditors are member-funded organization and NOT governmental organizations

• Accreditation is similar to the academic peer review process

• Current accreditation system keeps academic standards in the hands of the academy

• Accreditation is a system for quality control; regulates standard for institutions to grant degrees

• In general, assessment processes are designed LOCALLY at the institution

• Institution vary greatly on how much they standardize assessment throughout programs

Measuring: Direct Measures

Pre-test and post-test Eportfolios Observations of student performance Strategic questions on exams Evaluation of student work by someone

other than the instructor Evaluation of student work against a rubric Interviews with students

Direct measures are generally more desirable; should have some of them in any assessment

Measuring: Indirect Measures

Survey of students Signs of success, such as publishing a paper

or getting a job Evidence of student engagement or interest Student self-reporting about learning Student achievement in subsequent classes Continued interest in topic (such as novel

reading or going to museums)

Assessment across disciplines

• Some disciplines (Education, Engineering, Business—ie, professional schools) have long required assessment to be accredited by their profession organizations

• Liberal Arts disciplines less accustomed to assessment

• Liberal Arts disciplines include science, math, and the humanities

• Tends to be greater resistance to assessment in the Liberal Arts than in professional schools

• Learning goals may be harder to measure

WHY does assessment pose particular challenges in the Arts and Humanities?

Ineffability

Learning goals feel mysterious and hard to define

Assessment feels like an intrusion

Goals of courses can be unclear

Disagreement within regarding about the discipline

Faculty unaccustomed to collaborating

Myths about assessment (some taken from the forthcoming report from the ADE/MLA committee)

• Assessment is the same as grading (see later slide)

• Assessment requires specialized expertise

• No one will care about the results from the process

Genuine Concerns about Assessment

• Assessment is time-consuming Toward an answer: find ways to integrate assessment into existing work flow and educational process

• Assessment work will meet with insufficient institutional reward

Toward an answer: consider having assessment work “count” under teaching rather than service; raise issue of its value with administrator

Arts & Humanities Views of Institutional Assessment

(information courtesy of NSSE)

• 75% indicate institution is involved “Quite a bit” or “Very much”

• 62% indicate results are disseminated effectively

• 56% indicate that institutional assessment efforts are useful to them

What do we know about assessment in humanities departments?

• Survey conducted for the 2010 Association of Departments of English (ADE) East summer meeting

• ADE is part of the Modern Languages Association (MLA) • Conducted by David Lawrence, in collaboration with Laura

Rosenthal • Only covers participating English departments • Survey taken by chairs • About 50 responded Full survey results: • https://www.surveymonkey.com/sr.aspx?sm=QClKErAfvTgrLM

npZuomZ5bCMiF8wCQgnEK2BFIw_2fGE_3d

Survey results

1. Has your department implemented an assessment process or project? (Check all that apply.)

Yes, for assessment of student learning in specific courses

68.6% 35

Yes, for assessment of bachelor's degree program(s)

86.3% 44

Yes, for assessment of MA degree program(s)

37.3% 19

Yes, for assessment of MFA degree program(s)

7.8% 4

Yes, for assessment of doctoral program(s)

13.7% 7

No, but we are now or will soon be engaged in developing a process

2.0% 1

Assessment Strategies Used

Collegiate Learning Assessment (CLA) 7.8% 4

A faculty committee reviews assignments and papers for a sample of students

60.8% 31

Individual faculty members specify student learning outcomes for particular assignments they give or courses they teach and develop rubrics to measure for those outcomes

43.1% 22

We don't have an assessment plan or program, so this question doesn't apply to us

2.0% 1

Other (please explain) 25.5% 13

Do you think that outcomes assessment has the potential to improve learning in your programs? Who

has led assessment efforts?

Yes 89.8% 44

No

10.2% 5

Chair 80.4% 41

Associate chair

9.8% 5

Undergraduate director

21.6% 11

Graduate director

27.5% 14

Writing program director

35.3% 18

Tenured faculty member

51.0% 26

Tenure-track faculty member

27.5% 14

Non-tenure-track/adjunct faculty member

2.0% 1

Staff person without faculty status

2.0% 1

Assessment professional

7.8% 4

Other (please explain)

15.7% 8

What resources have you found most helpful?

Not

usefu

l

Somewh

at useful

Very

useful

Indispensibl

e N/A

Rating

Averag

e

Respons

e

Count

Workshop on my campus

13.6% (6)

47.7% (21)

20.5% (9) 0.0% (0) 18.2

% (8) 2.08 44

Meeting with campus administrators

12.5% (6)

45.8% (22)

18.8% (9) 8.3% (4) 14.6

% (7) 2.27 48

Attending an assessment conference

4.3% (2)

17.4% (8)

17.4% (8) 6.5% (3)

54.3%

(25) 2.57 46

Sessions at other conferences not devoted to assessment

6.8% (3)

36.4% (16)

9.1% (4) 4.5% (2)

43.2%

(19) 2.20 44

Talking to colleagues

8.3% (4)

25.0% (12)

33.3%

(16) 27.1% (13) 6.3%

(3) 2.84 48

Published (print or internet) materials and resources on assessment

4.3% (2)

40.4% (19)

25.5%

(12) 12.8% (6) 17.0

% (8) 2.56 47

Grading vs Assessment

Grading • Grade for course is a

composite • Grades are not

necessarily measures of learning

• Grades are inconsistent across faculty members

Assessment • Assessment tries to

capture how much students have actually learned

• Assessment generally done at the level of the program or the institution rather than the individual class

• Assessment information can be embedded into a class

Student Learning Assessment vs Teaching Evaluations

Evaluation • Individual • Survey of student

satisfaction • Not necessarily correlated

with learning • Often correlated with

grading • Provides information for

students ‘shopping’ for courses and to programs about student satisfaction

• Critique [Steven Brint]: part of the “consumer” model of education

Assessment • Collaborative and aggregate • Student engagement can be

an indirect measure of learning

• Goal is to improve learning • Assessment should provide

information to instructors and programs

• Assumes that there are ways to improve learning, and that this is a worthy goal

• Critique: Slippery slope to standardization; violates classroom autonomy

Assessment can help renew teaching

Traditional • Lecture [Attention

span=15 minutes] • Emphasis on product

rather than process • Transfer of

information • Student responsible

for success or failure • Handed down from

previous teachers

Student-Centered • Collaborative • Independent research • Writing in drafts • Oral presentations • Strategies for engagement • Discussion orientation • Use of digital tools • Based on research about

effectiveness; cognitive science

• Shared responsibility for student learning

Online and Blended Learning Outcomes assessment emerging as the most important issue: how do we know whether or not students are learning?

Faculty need to be aware of assessment for training graduate students for new interview questions

Before • What is your teaching

philosophy? • What are your classes

like? • What kinds of

questions do you raise about the text?

After • How do you know that

your students are learning?

• What do you do to engage students?

• What kinds of opportunities do you set up for student inquiry?

• How do you manage group work?

Tips for implementation

• Be patient and listen • Find allies within departments • Hold a workshop • Avoid excessive jargon • Explain the bigger picture • Tap into faculty priorities • Find some funding

Benefits to Faculty

• Will learn more about students • Can clarify goals of program • Makes grading easier (rubrics, great clarity) • Can promote greater collaboration • Can inspire new teaching strategies • Helps faculty think more clearly about

engagement • Can streamline class preparation • Turn more responsibility over to students

Advice for you to offer departments

• Make assessment a collaborative activity • Develop rubrics • No need to reinvent wheel • Expect to modify constantly (just like anything else) • Put some resources into it (course release? Research fund?

Assistant? Release from other committees?) • Integrate assessment into work you already do • Only ask things you really want to know • Involve students in assessment • Keep your eye on the prize: Improvement of learning!

Rewrite Narratives about Assessment

If you think your subject is valuable, then you want students to learn it Critical thinkers help solve problems Imagine how much better a place the world would be if more people were

truly educated Some of your students will become educators themselves The arts and humanities are crucial. We want as many people as possible to understand them. Today’s students are tomorrow’s readers, arts donors, audiences, museum goers, creators, voters, administrators, leaders, writers, citizens.