high value evaluation strategies in foreign …nflrc.hawaii.edu › evaluation › files ›...

John Norris – University of Hawaii – July 29, 2011

High‐value evaluation strategies in foreign language educationPlease site as:Norris, J. M. (2011). High‐value evaluation strategies in foreign language education. Plenary address presented at the annual meeting of the Western Consortium of Middle Eastern Language Programs, University of Texas, Austin (July 29, 2011).

Bologna Process:

A Multinational Evaluation Bureaucracy…?

A somewhat different take on evaluation…

What effects do these projects have on participants, artists, the community, etc.?

How can we evaluate things which are by nature difficult to discern and measure?

How can evaluation maintain a balance between results and the process of getting them?

Is evaluation a candid, honest exercise or a means of seduction?

2 ways of seeing evaluation & change

Educative inquiry

capacity for dealing with change

internal, owned by us

taking responsibility

understanding, improving

defending, promoting

Regulatory requirement

agent of change

external, mandated

maintaining control

monitoring, managing

comparing, determining

How do we choose to see evaluation in MELPs?

Evaluation as change agent

Western Consortium 2009: Approaching useful assessment and evaluation

Facing change in foreign language education

“New and increasingly complex challenges —

political, cultural, technological, and financial —

are profoundly altering conditions for the humanities

in the United States.”

American Academy of Arts and Sciences

“At the state level, the humanities per se have a difficult task achieving any sort of prominence on the public policy agenda. Like the federal funding agenda, where billions go to science and a comparative pittance goes to the humanities, there is little hope that poetry, literature, and language can realistically compete with roads, prisons, and health care for direct support.”

Summers (2004), p. 68

Pressure to evaluate

US Department

of Education

“DOE”

Council forHigher

EducationAccreditation

“CHEA”

Recognize

Regional

Accreditation

Agencies

Middle States Association -MSA

New England Association -NEASC

North Central Association -NCA

Northwest Association -NWCCU

Southern Association -SACS

Western Association -WASC

ACCREDIT

Colleges

& Universities

Representation of evaluation

Spellings Commission (2007), on higher education accountability:

“…higher education institutions should measure student learning…”

Technocratic measurement problem

Traditions of evaluation

Managerial evaluation

Jet-in-jet-out Expert

(JI JOE)

Accountability testing

Under these familiar approaches, evaluation gets done efficiently, but it generally meets only program-external bureaucratic or political needs; evaluation is done to programs (and teachers, and learners), not with or for programs.

Misguided practices

RateMyProfessors

Quality Rating Categories

☺How easy?

☺How fair?

☺How good?

HOW HOT???

Misguided practices

One size fits all…?

Common European Framework of Reference for Languages

C2

C1

B2

B1

A2

A1

Misinterpretations, negative reactions

Berger (2008), on assessment in the humanities:“This emphasis compels us to justify our values and methods by translating them into the quantitative, quasi-scientific methods… We are not asked to identify what we want our students to know or understand or be prepared intellectually to grapple with. Rather, we are asked for the behaviors that our students will exhibit that will demonstrate their learning—and we are told that we must develop a quantitative instrument that will measure these behaviors.”

Misperception that outcomes assessment = standardized measurement

MLA website blogger:“What I would much rather see is a definitive statement from the MLA rejecting the assessment madness altogether. Let’s admit that, when all is said and done, what we do is not something that we can ‘know’, or that can be measured…”

Useless evaluation

College foreign language educators:

“Sometimes we think they are just collecting dust on some administrator’s shelf in the Dean’s office, cause we never hear anything from them…So, I’m not sure what those program evaluations are really accomplishing in our college or in our university.”

“Although required by our university and accrediting association, faculty see it as a burden that is essentially a waste of time. Some faculty refuse to participate. Conclusions drawn from evaluations have little, if any, impact on decision making.”

“frankly, a waste of time; it just causes us to jump through meaningless hoops. Good FL instructors already assess their students constantly both inside and outside of class and in a variety of ways. It's what we do. Much of the assessment craze seems to be a waste of time for us.”

My main concern is… “That it would not be a waste of everyone's time with no concrete results. That the people running it would be so afraid of stepping on toes that they just babble about quality without looking at the details.”

College foreign language educators:

Re-envisioning evaluation as a useful process

“I feel a personal responsibility to be accountable …”

“We have a social and moral responsibility towards our students and towards society at large to state as clearly as we can what it is that we do for them and why what we do is valuable.”

“I think you got to realize that it’s not punitive. It’s to improve yourself.”

“Maybe it’s just a part of the feeling of professional responsibility that has been to some extent sort of energizing.”

Distinguishing terminology

EVALUATIONof programs

ASSESSMENTof students

MEASUREMENTof quantifiables

Learners

Assessment

Curriculum

Instruction

Teachers

Materials

Needs

L2 Education Programs

SLA theory

Educational theory

Teacher preparation

Resources

Planning and policy

Contexts

L2 Educational Affordances

Implementation?Outcomes?

Value?

What gets evaluated? The nature of programs

Multiple uses for evaluation

PURPOSES

AccountabilityRevising

curriculum

DevelopingPrograms

EncouragingParticipation

DeterminingNeed

Improving teaching

Raising Awareness

GeneratingKnowledge

DemonstratingOutcomes

Justifying$ requests

Improvinglearning

IlluminatingValue

Multiple methods for evaluation

METHODS

Tests

Externalreview

Document review

Self/peerassessment

Observations

SWOTAnalysis

Interviews

Performanceassessment

Standardizedmeasures

Journals

Meetings

Portfolios

Surveys

Focus groups

Corrected vision of useful evaluation

Traditional view: Begin by asking…

What are the outcomes targeted by the program?

How can they be measured?

Are they being met?

Evaluative vision: Begin by asking…

Who is in a position to utilize information for the betterment of learners, the program, the discipline?

What questions do they have about learners, teachers, courses, curriculum, etc.? What challenges do they face?

What needs to happen on the basis of evaluation?

Who is asking forthat information?

Who is doing the measuring and interpreting?

Who is held responsible?

What is the starting point for developing useful evaluations?

Proceduralizing useful evaluation

1. Participation – stakeholders, representatives, primary intended users

2. Prioritization – challenges, questions in immediate need of answers

3. Instrumentation – what information will answer the questions?

4. Collection – how can we get info with available time/resources?

5. Interpretation – what do findings mean in context?

6. Utilization – what decisions & actions are taken?

Foreign language educators are ultimately responsible for what happens in FL education.

Participation by FL educators is essential throughout all phases of evaluation if contextual

relevance is sought.

A focus on specific intended uses for evaluation is essential from the outset, if the process is to

make any difference.

University of Hawaii, National Foreign Language Resource Center

Summer Institute 2007

“enables the field to articulate and demonstrate—internally and externally—the unique contributions of language studies in a pluralist and globalized world.”

What is the value of evaluation in language education?

Provides a framework for

discussion

Encourages heightened

commitment

Increases awareness,

communication

Makes student learning more

efficient

Democratizes, unifies,

engages…

Facilitates solving of problems

Sheds light on how programs

function

Why bother? From the perspective of MELPs…

empirical basis for systematic change

proactive approach to targeted, focused program improvement

medium for enhanced communication, within and beyond programs

methodology for illuminating strengths and weaknesses, demonstrating value

Western Consortium 2011:New threats, new opportunities, and a mandate to evaluate

MELP evaluation 2011: A SWOT analysis

Strengths: Weaknesses:?

Opportunities: Threats:!

What is the status quo for evaluation in

MELPs?

MELP evaluation 2011: Strengths

Evaluation Tools for Quality Language ProgramsNMELRC Roundtable

MESASan Diego

20 November 2010

Awareness


http://www.nflrc.hawaii.edu/evaluation

Resources


Examples of FL evaluation


Notre Dame Arabic

Engagement


Strengths:Increased awareness in MELPsAvailability of eval resourcesExamples of useful FL evaluationLegitimate interest and engagement by major centers of ME language ed.

Weaknesses:?



Strengths:Increased awareness in MELPsAvailability of eval resourcesExamples of useful FL evaluationLegitimate interest and investment by major centers of ME language ed.

Weaknesses:? Limited capacity for evaluation? Lack of published MELP examples? Uncertain purposes, methods?Widely differing MELP needs


MELP evaluation 2011: Threats

•LSU: German, Russian programs (and faculty)•George Washington: FL requirement•SUNY Albany: French, Italian, Russian programs (and faculty)•University of Iowa: MA and PHD in German and Linguistics•Brandeis University: BA in Hebrew and Yiddish•Indiana University of Pennsylvania: BA in French and German•Etc.

“…no clear value to the institution…”


Holquist (2011), on dangers of outcomes assessment:

“…to sacrifice all other goals in the service of standardized outcomes across the board, no matter what the effect might be on different areas of scholarship, university systems, or individual students and professors—the humans who are the subject of the humanities.” (p. 79)

“But in the end, it is only in the scale of a whole lifetime that the worth of literary education may be measured, and that is a scale that cannot be ‘tuned.’” (p. 86)Teagle Foundation

(2011)


•April 13, 2011

Language and International-Studies Programs Face 'Devastating' Cuts Under Budget Deal

By Ian WilhelmWashingtonThe federal budget plan expected to be approved by Congress this week would make sharp cuts in foreign-language and international academic programs, with some university officials saying they could result in staff layoffs.




Opportunities: Threats:! FL program closures! Heightened resistance by colleagues! De‐funding of international education




Opportunities:Commitment by LRCs (as long as…)Other (nongovt.) sources of supportPossibilities for collaborationCoalescing interest in useful evalEmerging scholarship of FL evaluation

Threats:! FL program closures! Heightened resistance by colleagues! De‐funding of international education

MELP evaluation 2011: Opportunities

Build upon…NFLRC + NMELRC investment

(while it lasts)

Contribute to…Scholarly exchange on evaluation

in higher education Benefit from…Coalescing focus on useful evaluation in the humanities

Participate in…ME and FL collaborations on

useful evaluation

Pursue…Nongovernmental (e.g., foundation) support

Getting toUseful evaluation in

MELPs???

SWOT ‐ R

(SomeWhere Over The Rainbow…?)

Strengths, Weaknesses, Opportunities, Threats Response

Response: A mandate to evaluate

Can we put evaluation to work in support of FL education (MELPs in particular), not simply in reaction to a variety of external demands, but as a core practice focused on better understanding and improving what we do, and as a means of demonstrating our value to society?

Can evaluation in turn help us respond to forces that threaten the sustainability of what we are trying to do in FL education?

What works?Useful strategies for making evaluation valuable in language programs

What are ‘high-value’ evaluation strategies?

Ways of approaching evaluation that are feasible, efficient, and produce meaningful data, but also enable program improvement, participant learning, and organizational culture change, and ultimately communicate program worth.

There are a lot of ways to do evaluation

What do recent experiences and research indicate about ‘high‐value’ strategies for FL program evaluation?

Strategy 1: If you build it, they will come

Basic idea:Making available, and providing assistance with, easy‐to‐use tools and procedures can both initiate and sustain systematic data collection for evaluation.

Example:

Student exit survey project

Associate Dean Faculty AdvisorsEval PHD GAs

•Help to programs in item dev.•Focus on graduating students•Online administration by CLLL

College‐level questions

Program‐level questions

GA support•Data analysis•Reporting•Facilitation

Value:Persistent data

stream that did not exist priorFocus on crucial

endpoint student perspectivesAll programs

doing somethingSome programs

build strategically from feedbackCost = 1 GAshipPrograms = 49

Strategy 2: Follow the leader

Basic idea:When program leadership takes an active role in promoting and facilitating evaluation, things tend to happen, through resource allocation, recognition, influence, and ‘big‐picture’ vision.

Value:Sustained dept.

commitment to evaluation cultureRaised awareness

about curricular thinking and actionInfluenced other

programs, institutionLed to national

collaborationCost = Leader’s

time, dept resources

Example:

Department of German

Curriculum evaluation 1999‐2011

Chair

Evaluation foci:Assessment systemsOral proficiency outcomesWriting ability outcomesStudent perceptions of

curricular implementationAlumni perceptions of

degree valueHumanistic outcomes

Leadership

Resource allocation (time, $)

Direct participation

(learning, doing)

Scholarship (e.g., evalpublishing)

Raising local, professional awareness

Strategy 3: Just do it!

Basic idea:Getting an initial, small‐scale and highly focused, evaluation project done can set the stage for (a) improved understandings of the possible contributions of evaluation and (b) subsequent program development and evaluation activities.

Value:Empirical basis for

resource requestsIdentified unique

stakeholder groupsGave clear

direction to program devlpt.Demonstrated

value of evaluation, within program & to deanCost = free…?

Example: University of New Mexico

Portuguese learners needs analysis

LCTL Course design? Learner populations? Program growth? How to change?

Program chair Ed. PHD Student

Why not start with a Needs Analysis?

Document reviewFocus groupsStudent surveys

Distinct students: Spanish L1, Non‐Spanish

Differing learning targets, levels,

purposes

Need for 2 tracks, new courses, new

faculty

Strategy 4: All hands on deck

Basic idea:Participation in all phases of evaluation by multiple stakeholders, often in the form of a committee, enhances the likelihood of learning, buy‐in, and consensus‐building through evaluation

Value:Increased

communication, vertical/horizontalIncreased

collaboration across FL programsConsensus on

common goals, good practicesClarity on data

feasibility & utilityCost = Time, $Programs = 4 FLs

Example:

Duke University: Trinity College FLsEvaluating the Foreign Language Requirement

Eval Committee Assessment Office

DeanChineseFrenchGermanSpanish

+

1. FL Proficiency: SOPI + Portfolios + Questionnaire

2. Cultural learning: IDI + GPI + Questionnaire

3. Add Factors: Registrar data + Questionnaire

4. Students’ views: Focus groups + Questionnaire

Pilot data collection

simultaneously

Interpreting & Analyzing together

How to use so much data???

0. Stage setting: Generate ideas, plans, intended uses

Strategy 5: Where’s the beef?

Basic idea:An initial evaluation focus on stating and assessing program outcomes (learning or otherwise) can have long‐term and far‐reaching ripple effects, if taken seriously by program stakeholders.

Value:Consensus on

value of FL studiesArticulation of

curriculum, coursesEnhanced

communicationAssessment leads

to changesProgram survival,

growthCost = Faculty

time over 1 yearPrograms = 5 FLs

Example:

Eval Committee Senior StudentsFull Faculty

Draft, vet, revise

SLOs for all majors, all FLs

Stating and assessing learning outcomes

Mapping

Curriculum designExisting courses

Senior Assessment

Strategy 6: Best face forward

Basic idea:Dissemination of products of evaluation, as well as the fact that evaluation is being done, can lead to public awareness about the program and the extent to which its efforts are being monitored and improved.

Example:

Value:Prospective students

and parents respond positively (“others don’t do that…”)Increase enrollmentsInstitutional

recognitionMaintains

accountability to students by making outcomes & evaluation public

Putting evaluation to work in FL programs

Support

Leadership

Participation

Prioritization

Feasibility

Action

Dissemination

? ?

How do we accomplish a sustainable evidence‐based practice of educational decision‐making and action within MELPs?

What might work?

Taking advantage of evaluation in Middle Eastern language education

Scaling up, scaling out…

Support

Leadership

Participation

Prioritization

Feasibility

Action

Dissemination

? ?

Scaling up, scaling out: Scholarly exchange

1. Why not establish a persistent venue (or venues) for the scholarly exchange of MELP evaluation projects, methods, findings, and so on? Why not participate in opportunities for scholarly exchange on evaluation with other FL communities?

American Academy of Religion (2009)

“The American Academy of Religion should inaugurate a consultation on “The Assessment of the Religious Studies Major” with the goal of integrating the section into the permanent structure of the Annual Meeting.”

“As we learn more about our students, their strengths and their weaknesses, we need simultaneously to establish structures that will promote a sustained dialogue on effective means of maintaining and refining what we do well and identifying and improving what we do less well.”

Scaling up, scaling out: Scholarly exchange

1. Why not establish a persistent venue (or venues) for the scholarly exchange of MELP evaluation projects, methods, findings, and so on? Why not participate in opportunities for scholarly exchange on evaluation with other FL communities?

ADFL

Scaling up, scaling out: Collaboration

2. Why not develop a network of practice focused on program evaluation (and related topics) in ME language education?

Web‐based resources

Annual meetings

Program‐to‐program collaborations

Instrument exchange

Scaling up, scaling out: Peer review

3. Why not initiate a standard practice for voluntary MELP peer review, with procedures and criteria developed by the discipline, with recommended self‐study guidelines, and with teams of trained external evaluator peers from all participating programs?

Homegrown initiative Promoting excellence through evaluation, according to standards set by the discipline of English Language Teaching (i.e., through a TESOL commission)

Scaling up, scaling out: Impact evaluation

4. Why not seek foundation funding to support a large‐scale evaluation of individual and social impact related to MELPs in the United States?

Purposes and Values of Education

The challenges of finding ways to close "test score gaps" among groups of students and to better prepare people for work are both urgent and very real. These challenges should not, however, be permitted to push from our consciousness abiding questions about the larger purposes and social values that animate education. Indeed, a good case can be made that too single-minded an obsession with the most "practical" aspects of education may in the long run be counterproductive even for its own limited purposes.

We value education for its contributions to civic, political and community life, for its role in advancing social justice, for its capacity to open to people worlds of cultural and artistic excellence, and in the largest sense for its contributions to "human flourishing." Questions at this less immediate but ultimately deeply practical level are often posed by philosophers and social critics, the best of whom show a lively interest in and skilled use of findings from the social

Areas of InquiryWe value education for its contributions to civic, political and community life, for its role in advancing social justice, for its capacity to open to people worlds of cultural and artistic excellence, and in the largest sense for its contributions to "human flourishing."

Scaling up, scaling out: Impact evaluation

4. Why not seek foundation funding to support a large‐scale evaluation of individual and social impact related to MELPs in the United States?

RQ: What is the value of Middle Eastern language and culture

education programs in the U.S.?

Most Significant Change

Impact narratives: degree programs, area studies centers, study abroad, institutions, communities, etc.

Stakeholder Committees•Review narratives•Determine range & type of impact

•Identify gaps•Establish

baseline value statements

Disseminate Strategically:

Within the disciplinesThrough the pressAt ME education policy summitIn congressional letter‐writing, testimony

Lots of pressures to do evaluation,

but why?

Uncertain roles for evaluation in

MELPs

Challenging to get started and sustain, feasibly, usefully

But… Considerable resources and

support available

And… Strategies for making it

usefulPossibilities for

putting evaluation to use within programs

Scholarly exchange might reap diverse

benefits

And maybe large‐scale evaluation could help us

demonstrate and focus on MELP value

So, what should we do about it?

Value of evaluation: It’s up to you

A few priority issues in evaluation:The rest of the sessions

Making the most of MELP evaluation

Given where we are and where we might go with evaluation in MELPs, the rest of the sessions are intended as a way of encouraging next steps…

Focus on common methods, issues, strategiesBased on experiences and examples in FL evaluationLeading towards discussion and …Recommendations for practice

Making the most of MELP evaluation

Friday:•John Davis on sorting out useful surveys•Roundtable discussion on a common, if ambiguous evaluation practice (program reviews)

Saturday:•Yukiko Watanabe and Bonnie Sylwester on value and impact of outcomes‐based evaluations•Martha Schulte‐Nafeh on the utility of holding ourselves accountable through external eval•Nahal Akbari on the use of logic models as a way of conceptualizing programs•Esther Raizen and Joanna Caravita on evaluating the impact of mentor‐student relationships•Exchanging useful ideas: Breakout sessions on priority topics in MELP evaluation•Getting beyond measurement: Roundtable discussion on assessing challenging outcomes

Sunday:Thinking big in difficult times: Strategic planning session of all participants focusing on how to put evaluation to work in the service of Middle East language education

Thank you!

high value evaluation strategies in foreign …nflrc.hawaii.edu › evaluation › files ›...

Documents