john m. norris university of hawai´i at mānoa using assessment for understanding and improving...
TRANSCRIPT
John M. Norris
University of Hawai´i at Mānoa
Using assessment
For understanding and improving
Language education
“Ich lerne sehen”
~R. M. Rilke
Dealing with change
In language education
Rethinking the value of language education
“English language learners (ELLs) are the fastest growing student population in America. Today, one out of every nine students is learning English as a second language…By 2025, English language learners will make up one out of every four students in our classrooms.”
Margaret Spellings (2005) – U.S. Secretary of Education
Rethinking the value of language education
“Americans need to be open to the world; we need to be able to see the world through the eyes of others if we are going to understand how to resolve the complex problems we face.”
Daniel Akaka, U.S. Senator from HawaiiLanguage Crisis in
the U.S.
Rethinking the value of language education
“The stakes are very high. The challenges college graduates face over the next 50, 60, and 70 years will intensify. They involve the understanding of different cultures, the balance of global power, the depletion of environmental resources, and the ability to continue to grow in a rapidly changing world. It is because the stakes are so high that I believe a focus on student learning is so critical and why FL programs have such an important role to play.” (Chase, 2006, p. 585).
Geoffrey Chase – Dean of Undergraduate Studies
San Diego State University
Rethinking the value of language education
AAAL, ACTFL, MLA, Northeast Conference
Modeling Representation of Foreign Language Education at the Federal Level in the United
States
MLA White Paper
Foreign Languages and Higher Education: New Structures for a Changed World
Standards
Standards for foreign language learning in the 21st century
High-quality teachers
NCATE – TESOL/ACTFL Teacher Development Program Standards
Rethinking the value of language education
LanguageEducation
How do we bridge the content vs. language divide?
Are we seeking communicative
competence or quality of mind?
How do we ensure advanced language
learning?
What’s the relationship
between L1, L2, and academic
achievement?
Should we adopt an
instrumental, aesthetic, ethical, or
moral justification for
FL learning?
Who trains teachers how?
Determining the value of language education
“What we assess is what we value”
--Lauren Resnick“How we choose to assess will determine what gets valued”
--Norris
Why worry about assessment?
Top 3 sources of pressure for assessment in college FL
programs:
1.University administration
2.The dean
3.Accreditation process
“As part of its re-accreditation, the
university has required all undergraduate
programs to create and implement outcomes-oriented assessment
plans.”
(survey respondent)
“Time-consuming. Takes away from the business of teaching. Many aspects of learning can’t be measured.”
(survey respondent)
Why worry about assessment?
NOCHILDLEFTBEHIND
COLLEGE STUDENTCollege Leaving Exam
“Does the Spellings Commission think about language education
at all?”
Michael Holquist (ADFL 2007 Summer Seminar West)
Why worry about assessment?
“thoughtless mastery”
ASSESSMENT
“bureaucratic nonsense”
“fails to improve teaching”
“duplicates existing efforts”
“reductionist”
“discourages teaching skills that are difficult to measure”
“Orwellian, punitive process”
“utilitarian technocracy”
Frequent teacher reactions to assessment:
Why worry about assessment?
capacity for change
faculties of problem-solving
FLs need to develo
p
Domna Stanton (2007, ADFL Summer Seminar West)
Perception of
assessment
managerial model of efficiency
not appropriate to academe
BUT…
Why worry about assessment?
Despite such problems, college FL survey respondents desired increased use of assessments and related processes for:
1. Understanding & improving program outcomes
2. Understanding & improving program functions
3. Improving FL education on the whole
4. Understanding & Improving the worth of the program
5. Raising awareness about FL programs
Why worry about assessment?
And some expressed a professional ethic to engage in outcomes assessment:
“We have a social and moral responsibility towards our students and towards society at large to state as clearly as we can what it is that we do for them and
why what we do is valuable.”
(Anonymous survey respondent)
Why worry about assessment?
TESOL•Language•Culture•Planning, managing•Assessment•Professionalism
ACTFL•Language, linguistics•Cultures, literatures•Acquisition, instruction•Integration of standards•Assessment•Professionalism
NCATE Teacher Preparation Program Standards
Assessment as change agent
Value of
LanguageEducation
KeyChallenge:
DefineDelimitDetractDismiss
Key Opportunity:
EnableEnhanceEngenderEmpower
Within this milieu, what is the role to be played by assessment?
Traditions, trends, and the status quo
In language education assessment
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
ADFL FLA LT MLJ UP
Ass
essm
ent
arti
cles
%Assess
%FL
%USCFL
%Proficiency
Percentage and type of FL assessment/evaluation articles in five journals, 1984-2002
0%
5%
10%
15%
20%
25%
30%
35%
ADFL 84-93 ADFL 94-02 FLA 84-93 FLA 94-02 LT 84-93 LT 94-02 MLJ 84-93 MLJ 94-02 UP 84-93 UP 94-02
US co
llege
FL
asse
ssm
ent
High, low, and average yearly percentage of articles on U.S. college FL assessment/evaluation, 1984-1993 and 1994-2002
Swender (2002), on FL teacher professional development in assessment:
“After all, if teachers do not know how to measure what students can do with language, how will they be able to determine whether their students are measuring up to the expectations of the 21st century”.
Received traditions of language assessment
Technocratic measurement problem
WebCAPE Foreign Language Placement Exam
Received traditions of language assessment
Commercial assessment problem:
One size fits most purposes & settings
TSE – SPEAK – SLEP - TOEIC
Received traditions of language assessment
Commercial assessment problem:
One size fits most purposes & settings
Received traditions of language assessment
When we do address
assessment, in…
• FL teaching texts
• State of the art collections
• Professional organizations
• NCATE/ACTFL teaching standards
• Accountability requirements
We focus on…
• How to measure the four skills
• How to rate oral proficiency
• How to test cultural knowledge
• How to assign grades
• How to give feedback
Without much concern for…
• WHY are we assessing?
• What assessment methods fulfill what purposes?
• How do we contextualize it?
• What good is it doing?
Advanced vocabulary learning: individual vocabulary notebooks
Learners identify meaningful
words to studyBased on
individual needsor interests
Record 30/week in vocabulary
notebookRevisit, review, and acquire 30over the week
Instructional Intent
From: Moir & Nation (2002)
Advanced vocabulary learning: individual vocabulary notebooks
Learners identify meaningful
words to studyBased on
individual needsor interests
Record 30/week in vocabulary
notebookRevisit, review, and acquire 30over the week
Instructional Intent
Learners identifyunsuitable words
to studyCram the
night beforethe weekly test
Pass the recalltest and promptlyforget the words
Assessment Reality
The ‘assessment mindset’ in teacher practice:
“Although this student still has problems with grammar, the ideas are there. He is working through the choices the community has about their need for a better water supply system. Hmm, this is difficult. I just wish his grammar errors weren’t so bad then I could give him an ‘A’.”
Mohan example, TBLT 2007 Conference
Received traditions of language assessment
What is the appropriate proficiency level to adopt as a student learning outcome for the 2-year language requirement?
Received traditions of language assessment
INTERMEDIATE -
LOW ? ? ?
Outcomes embody the essential purpose of an educational program: developments in knowledge, skills, dispositions of learners
Requires rethinking of educational programs as something more than the delivery of experiences or the exposure of learners to information
Calls for articulation of curriculum and instruction in support of targeted outcomes, demands integrated thinking
Provides a clear statement of educational program value; answers the question “How do you know?” with evidence of educational effectiveness
Current trends in assessment
Student Learning Outcomes
Assessment
Current trends in assessment
Process
•State outcomes
•Identify indicators
•Measure them
•Analyze the results
•Then what?
•(Let the chair/dean do it)
Student Learning Outcomes
Assessment
Large public institution Accreditation
pressures to assess learning
How about an electronic portfolio?
Huge expenditure, $$$, time, effort Thousands of student
portfolios created
BUT…
We have to assess our “liberal studies” core,
ASAP!
Current trends in assessment
1. *!%#$!@*
…faculty didn’t understand it…students thought:
2. waste of time
Electronic Portfolio
…administrators wanted to do something with it but weren’t sure exactly what
NEVER GOT USED
Current trends in assessment
Barrington (2003), on assessment in the liberal arts:
“To design and administer (intellectually honest) assessment plans that will measure such capabilities with a dozen or more standardized ‘learning objectives’ is next to impossible” leading to “pestilent repercussions” for the truly valued learning objectives that constitute the liberal arts, in that it “discourages teaching such skills because they are difficult to measure”.
Current trends in assessment
Perception problem
Traditions, trends, and the status quo
1. Focus on doing…
2. Based on commercial testing, proficiency mvt.
3. Reactive v. proactive praxis
4. Driven by external impetuses
5. Technocratic measurement emphasis
6. Little scholarly investment
Traditions and trends1. Not useful—not used!
2. Potential negative washback, reductionism, waste
3. Not relevant to curriculum & instruction, program values
4. Not ours—mandated or purchased off the shelf
5. Not perceived as worth the effort by faculty
6. Minimal professional development
Status quo
in language education
Re-envisioning assessment
Resolving terminological confusion
Measurement is the consistent elicitation of quantifiable indicators of well-defined constructs via tests or related observation procedures; it emphasizes efficiency, objectivity, and technical aspects of construct validity.
Norris (2006) MLJ Perspectives
Resolving terminological confusion
Assessment is the systematic gathering of information about student learning in support of teaching and learning…It may be direct or indirect, objective or subjective, formal or informal, standardized or idiosyncratic…It provides locally useful information on learners and learning to those individuals responsible for doing something about it.
Norris (2006) MLJ Perspectives
Resolving terminological confusion
Evaluation is the gathering of information about any of the variety of elements that constitute educational programs, for a variety of purposes that include primarily understanding, demonstrating, improving, and judging program value; evaluation brings evidence to bear on the problems of programs, but the nature of that evidence is not restricted to one particular methodology.
Norris (2006) MLJ Perspectives
The nature of useful evaluations
Evaluation gets used when…
(a) intended users of evaluation participate;
(b) evaluation is pursued as a process, not an end-game;
(c) sufficient time and resources are allocated;
(d) evaluation produces interesting, credible, relevant findings;
(e) findings are reported in a timely fashion;
(f) Interpretations and recommendations are contextualized.
Evaluative assessment
Light (2001) on outcomes assessment:
“…a process of evaluating and improving current programs, encouraging innovations, and then evaluating each innovation’s effectiveness. The key step is systematic gathering of information for sustained improvement. And always with an eye toward helping faculty or students work more effectively.”
Richard Light (2001, p. 224)
PURPOSES
AccountabilityRevising
curriculum
ProgramDevelopment
Articulation
Diagnosis
Improving teaching
Raising Awareness
Motivating Learners
Certification
Justifying$ requests
Improvinglearning
Illumination
An evaluative approach to assessment
METHODS
TestsLanguageProfiles
Quizzes
Selfassessment
Observations
PeerAssessment
Interviews
Performanceassessment
Standardizedassessment
Journals
Meetings
Portfolios
An evaluative approach to assessment
Surveys
Focus groups
Approaching assessment: •The most important consideration is supposed to be the use for which it is intended (cf. Bachman & Palmer, 1996; Brown, 2005)
•BUT, in test development, what is the first question that gets asked? What are you going to measure?
“Decide on purpose of assessment:•What abilities are you assessing?
--What is your construct or model of these abilities?”
Coombe, Folse, Hubley (2007, p. 4))
An evaluative approach to assessment
An evaluative approach to assessment
An evaluative approach to assessment
PrimaryIntended Users
AssessmentAdvisors
Stakeholders &Audiences
Negotiate & specify:
• priority uses
• methods
• analyses
•reporting
• timelinesENABLING USE
EMPOWERING USERS
An evaluative approach to assessment
Products:
(a)public documents on the exact roles to be played by assessments in the FL program and the different forms that
those assessments take
(b) program policies on assessment practice at the individual, classroom, and program levels
(c) assessment methods that lead to local actions
(d) evaluative/evidentiary justification for assessments and their uses
(e) Identification of other factors in need of evaluation
Why bother?
Align assessment with curriculum, instruction, program values: foster ownership Raise awareness & buy-in
among students, faculty, others about roles for assessment and evaluation Increase the likelihood
that data will be used
Decrease frequency and number of (useless) assessments and evaluations
Support improvements in student learning
Why bother?
Received view of assessment:
What do we test? How do we test? What tests are available? Evaluative Vision:
What questions do we have about our learners, courses, and programs? How do we gather data appropriate to answering those questions? In what ways can we utilize those data to resolve the challenges we face?
What does it look like?
Using assessment to understand and improve language programs
Example: Needs assessment for curricular revision
Problem
•Perception of curricular ‘antiquation’
•Relevance to learners’ wants and needs?
•Shifting standards for college FL learning
•New expectation to pursue performance outcomes
•How to evolve? What values to prioritize?
Context
•Japanese at University of Hawaii
•2-yr. requirement (mostly non-majors)
•@1000 students per year
•@50 instructors
•Diversity of students, diverse uses of Japanese
Role of assessment
•Identify learners’ perceived L2 needs
•Identify teachers’ perceptions of same
•Identify gaps between the two
•Prioritize outcomes expectations for 2-yr. program
•Provide basis for revision of materials
Example: Needs assessment for curricular revision
Instruments and ProceduresQuestionnaire
s
L2 Use Domains
Social
Occupational Tourism
Likert scale ratings of priorities
Open-ended questions+
Teachers (N = 46)
Students (N = 688)
Academic
Example: Needs assessment for curricular revision
Identified Learner Needs
Teachers + Learners Prioritize
Learners Prioritize,
Teachers do notIn Hawaii…
•Academic routines
•Socializing
•Hosting guests
•Working retail
•Interacting w/tourists
•Using computers
In Japan…
•Academic routines
•Socializing
•Surviving as a tourist
•Participating in complex discussions
In Japan…
•Medical interactions
•Residing long-term (housing, etc.)
•Working abroad
•Formal ceremonies
Example: Needs assessment for curricular revision
Other Curricular Implications
•Instruction moves too rapidly
•Overemphasis on Kanji memorization
•Not enough attention to culture
•Not enough focus on situation-specific speaking tasks, performance practice
Example: Needs assessment for curricular revision
Using Assessment for Curricular Change
•Basis for review of texts, materials & pedagogy development
•Revised 2-yr expectations emphasize speaking, culture, performance
•Attention to Japanese use in Hawaii & long-term residence abroad (both authentic to learners)
•Development of performance-based outcomes assessments
•Revised balance/pace of instruction
Related Changes
•Dean commits to Department-level Assessment Specialist
•Evaluation utilized as campus-wide example of ‘closing the loop’ back to curriculum
•Increased enrollments, improved course evals
•Principal Investigator tenured…
See Iwai et al (1999)
Example: Realizing the curriculum via assessment
Problem
•New curriculum, new instruction, old assessment
•Accuracy of curricular learning trajectories?
•Students meeting expected outcomes?
•Feasible expectations for intensive and non?
•Relevance of external assessments?
Context
•German at Georgetown Univ.
•New curriculum: Developing Multiple Literacies
•Fully integrated Language & Content instruction
•Task- & genre-based
•Advanced L2 literacy target
Role of assessment
•Operationalize curricular expectations
•Provide evidence regarding learner development & outcomes
•Enable between-level, and intensive/non comparisons
•Fit seamlessly into teaching and learning processes!!!
Example: Realizing the curriculum via assessment
Level performance profiles
Task assignment sheets
Sample student performances
TEACHERS
Development
Analysis
LEVEL
GROUPS
Deliberation
Revision
Prototypical performancewriting tasks
Example: Realizing the curriculum via assessment
Task
Content
Language
Assignment 1Assignment 2
Semester…
Assignment 3Assignment 4
PrototypicalPerformance
WritingTask
Explicit performance
criteria
Curricular level expectations
Consistent assignment framework
Example: Realizing the curriculum via assessment
“Assessment in this kind of a context is, I would almost say probably an indispensable
aspect in order to clarify any number of things. Because it is in the discourse about assessment and how we would do that that our knowledge became articulated or the
holes in that knowledge became clearer to ourselves. Or the cover-ups that we had engaged in were no longer possible if we wanted to be honest with ourselves about
it.”
Example: Realizing the curriculum via assessment
Using Assessment for Curricular Change
•Forced the curriculum to become real
•Close specification of L2 progress within/across curricular levels
•Disambiguation of learning outcomes in terms of task, content, language
•Curricular ‘map’ for use by teachers and learners (what happens when?)
•Forged agreement between levels on what can and cannot be expected
See Byrnes (2002)
Example: Assessing and improving effectiveness
Problem
•Does it help?
•Does it get used?
•Do students and teachers understand how it is supposed to be used?
•Are conditions appropriate for benefiting from it?
•Should it be maintained?
Context
•LAUSD
•Kindergarten/First, ELLs with varying proficiencies (41%)
•Wide-spread reading ‘failure’
•Supplemental software introduced to enhance reading instruction
Role of assessment
•Evidence of reading development
•Enable comparisons between ages, proficiencies, treatment/control
•Capture the context of implementation
•Provide thorough interpretation of processes + outcomes
Example: Assessing and improving effectiveness
Instruments and Procedures
Standardized, mandated reading assessments
Extensive classroom observations
Teacher interviews
200 classrooms
(100 treatment, 100 control)
Enable estimation of added effect of new
software
Provide for meaningful interpretation of findings
in terms of classroom realities
Indicate English reading abilities among children
Example: Assessing and improving effectiveness
Findings
•No differences in reading ability between treatment and control classes or proficiency groupings
•BUT, Low implementation in classes and by individuals
•Lots of reasons: Lack of time, competition with standard curriculum, computer malfunction, lack of understanding by learners and teachers, lack of ‘on-task’ engagement
Uses
•Provide for professional development in software
•Articulate/align with standard curriculum
•Individualize for students with greatest need
•Shift school schedule to allow for implementation to occur
•Retain the software but continue assessing effectiveness with studentsSee Llosa & Slayton
(forthcoming)
Facing change:Learning to see assessment as a useful process
•There is more to useful assessment than just good measurement; measurement informs construct interpretations…assessment enables informed change
•Contextualization in classes, curricula, institutions is essential for assessment to be interpretable and meaningful
•Intentionality for specific purposes, by specific users, with specific consequences is essential for assessment to be used
•These processes should be built in from the beginning of assessment design; specification of intended use is one heuristic for doing so
•Educational assessment is only useful insofar as it does good; hold assessments accountable to that principle
A few resources…
University of Hawai‘i at Mānoa, Second Language Studies
http://www.nflrc.hawaii.edu/evaluation
Thank you!