Measuring Student Learning at the Classroom Level: A Longitudinal Study

Aleidine MoellerLeyla MasmaliyevaDallas Malhiwsky

University of Nebraska-Lincoln

Research Focus

A longitudinal study at the classroom level that attempts to more accurately depict students’ acquisition of language proficiency that ultimately could contribute to a more accurate and smooth articulation tool for documenting language progress.

The Need: Clearly Articulated Descriptors

• What students know and are able to do at the end of each year of classroom language study

• Accurate and smooth articulation requires empirically tested

common yardstick at the classroom level that defines language progression from one level to the next in a cohesive manner

Smooth Articulation

• Accurate and smooth articulation requires an empirically tested common yardstick at the classroom level that defines language progression from one level to the next in a cohesive manner.

ACTFLconstructed a chart approximating what students should know and be able to do given specific years of language study, however, this chart does not reflect classroom based research, but rather is based on proficiency interviews with individual language students and “on information gathered from foreign language professionals representing a variety of program models and articulation sequences” (ACTFL Performance Guidelines for K-12 Learners, 1998).

Classroom Based Evidence• There is a dearth of classroom

tested studies that have investigated the level and rate at which language learners achieve proficiency after a sequenced course of language study.

• Classroom-based evidence has been limited to a few studies restricted to one or two classrooms that do not allow for generalizable results.

Subjects

7-12 Teachers of Spanish2 week Spanish Immersion funded by Teacher Quality Grants– Language, literature, culture

pedagogy, technology, LinguaFolio

– Teacher participants implemented LF (120 teachers)

– Selected 17 – Followed teachers & students for

5 years

Context for the Study: LinguaFolio• Modeled after European

Language Portfolio (ELP) of the Council of Europe (COE)

• Adapted by National Council of State Supervisors of Foreign Languages (NCSSFL)

• Promote reflective, autonomous language learning and cultural interactions through a self-assessment tool

• http://linguafolio.unl.edu

LinguaFolio Nebraska• seeks to inform students about

themselves and their own learning, showing them exactly where they are in relation to learning goals

• with the help of their teachers, knowing which steps to take to achieve these goals.

• through systematic self-assessment, goal setting, feedback and reflective learning tools, the LinguaFolio guides students to monitor and map their own learning.

Students must make what they learn part of themselves (Chickering & Gamson, 1987, p. 3)

• Talk about what they are learning, write about it, relate to past experiences, apply in their daily lives

• Too often we see students as subjects of assessment, not users of assessments. The student must be considered the primary user of assessment information.

Promote Self-regulation

• How do we position students to think about their own learning and make connections inside and outside the classroom to monitor their learning development?

Pilot Test: Lessons Learned

10 teachers of Spanish 7-12Tested & revised instruments

teacher observation formteacher beliefs & attitudes

Goals: needed trainingDevelopment of Teachers HandbookOnline LinguaFolio

Why goal setting?

“Goal setting influences learning and motivation by providing a target and information about how well one is doing.”

(Alderman, 2005, pg. 105)

Goal setting → increased self-efficacy→ increased intrinsic motivation

Goal setting:Monitor progressIncreased self-efficacyprovides expectation for

success and increases the likelihood of engaging in future tasks without external incentives

Increased intrinsic motivation

Characteristics of goals

Performance vs Learning based Goals

Performance- focus on comparing oneself to othersLearning- focus on gaining newknowledge and skills

Characteristics of goalsShort vs long term

• Near future versus distant future

• Both needed

Characteristics of goalsEasy vs. difficult

Analysis of 110 studies published from 1969 to 1980 demonstrated that 99 studies showed that difficult learning goals produced better performance than medium, easy, do-best, or no goals at all (Griffee and Templi,1997).

Characteristics of goals

Specific vs. Broad Goals• Students setting too specific goals may

focus their attention to such a degree that they will ignore “information not specifically related to their goal”

• (Marzano et al, 2001, p. 94).

• Broad goals are vague and have a nonspecific outcome

Characteristics of goals

Assigned versus Self-chosen Goals• Assigned goals foster motivation because

they imply that the individual is capable of achieving the goal or task

• Self-chosen- power of ownership• Crucial factor in assigned goals Is acceptance- once involved in a goal, the goal itself becomes more important than how it was set or whether it was imposed

Simply goal setting is enough?

Many educators overestimate their students’ ability to set high quality goals (Boekaerts, 2002).

“A perceived discrepancy between performance and the goal may create dissatisfaction” (Schunk, 2003, p. 163).

Examples of students’ poor goals:

• “speak fluently”

• “understand everything”

• “get an A (grade)”

For goals to enhance student achievement goals should meet specific criteria (Schunk, 2003).

SMART Goals

• SSpecificpecific

• MMeasurableeasurable

• AAgreed upongreed upon

• RRealistic (challenging yet realistic)ealistic (challenging yet realistic)

• TTime-basedime-based

SMART+ER =SMARTER

Return to Your Goals• What evidence do you have that indicates

whether you met your goals?• What could you have done to better meet your

goals?• How could you have restated your goals in

order for them to be more appropriate?• Are you satisfied with your progress today? Did

you put forth your “best effort” to achieve your goals?

Formula for Success:

Setting goals +documenting progress +self-assessing learning=student achievement

Results of Goal Setting on Student Achievement

Correlation between goal setting and student achievement at .01 level of significance in reading, speaking and writing

Correlations

LinguaFolio Goal Sc ore

LinguaFolio Action Plan

Score

LinguaFolio Reflection

Score

STAMP Readi ng

score STAMP

Writing score

STAMP Speaking

Score Pearson Correlation 1 .363(**) .585(**) .329(**) .317(**) .331(**) Sig. (2 -tailed) .000 .000 .000 .000 .000

LinguaFolio Goal Score

N 201 175 201 194 190 193 Pearson Correlation .363(**) 1 .158(*) .218(**) .222(**) .260(**) Sig. (2-tailed) .000 .036 .004 .004 .001

LinguaFolio Action Plan Score

N 175 175 175 169 165 168 Pearson Correlation .585(**) .158(*) 1 .215(**) .176(*) .311(**) Sig. (2 -tailed) .000 .036 .003 .015 .000

LinguaFolio Reflection Score

N 201 175 201 194 190 193

Pearson Correlation .329(**) .218(**) .215(**) 1 .756(**) .661(**) Sig. (2 -tailed) .000 .004 .003 .000 .000

STAMP Reading score

N 194 169 194 529 525 526 Pearson Correlation .317(**) .222(**) .176(*) .756(**) 1 .652(**) Sig. (2 -tailed) .000 .004 .015 .000 .000

STAMP Writing score

N 190 165 190 525 525 522 Pearson Correlation .331(**) .260(**) .311(**) .661(**) .652(**) 1 Sig. (2 -tailed) .000 .001 .000 .000 .000

STAMP Speak ing Score

N 193 168 193 526 522 527

** Correla tion is si gnificant at the 0.01 level (2-tailed). * Correla tion is s ignificant at the 0.05 level (2-tailed).

LinguaFolio Goal Score

LinguaFolio Action Plan Score

LinguaFolio Reflection Score

STAMP Reading score

Pearson Correlation

.315** .292** .253**

N 673 644 672

STAMP Writing score

Pearson Correlation

.436** .507** .256**

N 664 635 663

STAMP Speaking Score

Pearson Correlation

.404** .454** .236**

N 641 617 640

Assessment for Learning

• Students engage in assessment for learning when they use assessment information to learn how to judge the quality of their own work and set goals for their own improvement.

This interactive LF NE website • students log in and establish

base line knowledge and skills from which to build long term goals

• Self-assessment of competency chart: identify what they are able to do in their learning process while also reflecting upon the level of language competency they would like to attain

I can ask for and give directions with the help of a map or other visual aid. I can ask for directions to a place. I can tell someone how to get from one place to another (e.g. go straight, turn left, turn right). I can tell someone where something is located (e.g. next to, across from, in the middle of). I can exchange information with other people about what to do, where to go, and when to meet. I can make simple plans to do something with someone. I can exchange information about where or when to go or meet (e.g. store, movie, concert, restaurant). I can invite someone to do something or go somewhere. I can accept or refuse an invitation to do something or go somewhere.

6. Write

reflection based on work

(evidence) and goals

5. End of

chapter: review goals. Choose

work to represent goal

attainment.

4. Save goals, tasks and all

completed class work in a folder

3. Students determine tasks for achieving chapter goals.

2. Visit Chapter Content:

(Students) write personal goals

based on chapter content.

1. Review

Self-Assessment of Competency Level Rubric

(LF NE!)

Dossier of Evidence Student Driven Cycle:

Once Per Chapter

Measuring Student Learning

• STAMP– Archived student samples– Proficiency based– Accessible – Provided a control group– Measures speaking, writing

and reading

STAMP

• Teacher independent proficiency assessment that archives language skills over time

• Access to all reading, writing, speaking scores and individual student samples over time

STAMP• A summative assessment

tool, measuring student proficiency levels in foreign languages in four skills

• Test items are based on benchmarks linked to ACFTL guidelines

• Securely and efficiently delivered through the internet

• Computer adaptive online assessment

STAMP

Report can be displayed in terms of a class, group or individual

Is criterion-referenced (item evaluation is based on how they perform as students approach and then cross a mastery level)

STAMP• Assessment is made on basis of what

students can actually perform in real-world context

• Provides detailed and user-friendly report that aims to demonstrate that the best measure of educational success is progress in student performance.

• The ultimate purpose is to improve language teaching and learning

• Gives the teachers concrete feedback on students overall proficiency and breakdowns of performance according to topic and task

Individual Report for É ÉÉÉ. 5/1/2007 10:19:27 AM

Interpretive Presentational Reading 3

Topics Needing Further Study:

Community, Holidays/Celebrations, Leisure/Activities, Occupations/Professions, Shopping/Stores, Travel/Vacations, Weather/Seasons

Writing 3

Writing Sample Breakdown:

Sample 1 Does Not Meet Standards

Sample 2 Meets Standards

Sample 3 Meets Standards

Speaking 3

Speaking Sample Breakdown:

Sample 1

Does Not Meet Standards

Sample 2

Meets Standards

Sample 3

Meets Standards

When reading about introductory topics such as:

community daily routines shopping/stores food/beverage

When writing about introductory topics such as:

community daily routines shopping/stores food/beverage

When speaking about introductory topics such as:

community daily routines shopping/stores food/beverage

STAMP Reading

Minimum 1.00

Maximum 4.00

Yr 1 Mean N 274 1.3885

Yr 2 Mean N 683 1.6676

Yr 3 Mean N 204 2.2745

Yr 4 Mean N 17 2.6471

STAMP Writing

Minimum 1.00

Maximum 4.00

N 562 1.4217

N 673 1.9173

N 204 3.0882

N 17 3.2529

STAMP Speaking

Minimum 1.00

Maximum 5.00

N 525 1.3619

N 626 1.7923

N 197 2.8477

N 17 3.2353

Mean Year One Year 2

STAMP Reading score 1.3875 1.7488 2.2400 STAMP Writing score 1.3010 1.6408 2.4600 STAMP Speaking Score 1.1708 1.5670 2.4898 Pronunciation score 1 1.7670 2.0111 2.3061 Pronunciation score 2 1.7903 1.9865 2.3469 Pronunciation score 3 1.7908 1.9859 2.3404 Grammar score 1 1.6878 1.9224 2.3673 Grammar score 2 1.6874 1.9216 2.5102 Grammar score 3 1.6964 1.9181 2.4681 Vocab score 1 1.6222 1.8726 2.3673 Vocab score 2 1.6244 1.8703 2.5102 Vocab score 3 1.6752 1.9153 2.4681 Fluency score 1 1.6433 1.8753 2.3061 Fluency score 2 1.6521 1.8804 2.3673 Fluency score 3 1.6675 1.8955 2.3830

Context of the project

• Doc seminar• Sat. mornings• 10 students (6 Spanish

language experts; 4 researchers)

• 6 Spanish language (3 native speakers/3 non-native speakers)

The Process with Speaking• Started looking at student

speaking samples from 1st year, 2nd year and 3rd year

• From this initial analysis—the team of 6 lang. experts determined what could students do at the end of each year.

• The team of researchers did a lit. review on rubrics and how to create an effective one

• A rubric emerged based on student production

The Process (Cont.)

• Next came the transcription of the speaking samples

• Almost 1000 speaking samples transcribed (954)

• Average student response time was roughly 2 min.

Creating a Rubric

1. Gather samples of student performance

2. Describe samples3. Sort by levels of quality4. Cluster into groups

Creating a Rubric5. Identify levels—determine what

major categories, dimensions or traits seem to be emerging from the sorting process, eliminating redundancies and including descriptors from the list that best describe the traits at each achievement level

6. Samples that identify each ability level (exemplars)—will assist in maintaining consistency with each as you score across time, students and assignments. What each achievement level looks like—several samples at each level

Speaking Rubric

1st year N=105 students/N=31

5 samplesExceeds

StandardsMeets

StandardsDoes Not Meet

Standards Not Ratable

Pronunciation rating 3 180 66 62

Grammar rating 17 105 127 62

Vocabulary rating 22 87 140 62

Fluency rating 20 93 136 62

2nd year N=104

students/N=312 samples

Exceeds Standards

Meets Standards

Does Not Meet Standards Not Ratable

Pronunciation rating 9 232 19 43

Grammar rating 49 146 65 43

Vocabulary rating 64 117 79 43

Fluency rating 62 123 121 43

3rd year N=109

students/N=327 samples

Exceeds Standards

Meets Standards

Does Not Meet Standards Not Ratable

Pronunciation rating 47 169 47 53

Grammar rating 45 167 62 53

Vocabulary rating 52 154 68 53

Fluency rating 41 174 59 53

The Process with Writing

• The methodology was the same.

• The rubric was based on student production and created before analysis of samples.

• No transcription was necessary since they were writing samples

Writing Rubric

The Experience

• Interactive activity• Directions: You all have a

copy of the writing rubric. Please now look at the following three samples and use the writing rubric to assess these actual student samples.

Sample #1

Prompt Transcribed Sample

You are with your new host mother, and she would like to know more about your family. Describe at least 2 members of your family. Be sure to include for each one: • Name• Relationship to you• Age• His or her favorite food or activity

Esta es mi hermano Kane. Tiene diez y siete años. Le gusta mucho la lucha. Este es mi madre Marsha. Tiene cuarenta años. Le gusta preparar la cena. Este es mi padre Brad. Tiene cuarenta y dos años. Le gusta jugar deportes.

Sample #2Prompt Transcribed Sample

Your friend Alejandro is visiting from Honduras and is going to attend school with you for a few weeks. Tomorrow will be his first day and he wants to know what it will be like. Give as detailed a description as possible about a typical day, including: • what time you get up• what you do before school • how you get to school • how many classes you have • which classes you like or don’t like and why • what you do after school.

Mevanto a las seis y media por la manana.

Sample #3Prompt Transcribed Sample

Select one of the rooms pictured below (bedroom/kitchen), in Spanish: • name the room• describe at least 2 of the items that you see in the room including: • the name• color• location in relation to other items in the room

El tazón rojo está al lado de la cuchara. La cuchara está al lado del cuchillo. La ventana está cerca del tazón rojo.

Results

• Sample #1---Exceeds Expectations

• Sample #2---In Progress• Sample #3---Meets

Expectations

Interrater Reliability

• The 6 language experts rated samples to check for interrater reliability

• At first the percentage was low but after negotiation of sample ratings the interrater reliability was 83%

Influential Factors• The 4 researchers reviewed the

factors which influenced the study and did a lit. review on these:– Effect of time limits on testing– Teacher variance– STAMP prompts (listing activities)– Teacher attitudes– Degree of commitment by teachers– Time spent on goal setting– Degree of feedback students

received– Students at different levels– Reluctant learners– Challenges of 1st year language

learning

Related Questions that Emerged during the study• Teacher Beliefs &

instructional practices• Teacher

competency/content knowledge & student achievement

• Effect on student achievement across content areas

Teacher beliefs/change

• Multiple case study (4 teacher participants)

• Interviews, observations, field notes, document analysis

Themes

• Instruction more interactive

• More discovery learning• Students own their

learning• More facilitator of learning• Less textbook driven• Learning more of a

partnership• More accountability

• My instruction is “more interactive, more them doing the work, and them discovering”

• “more accountable to what I am going to be teaching them”

• ‘I can show their parents what they’ve learned, their best work”

• “not so much oriented by the textbook”

• “more purpose in mind of what I am trying to achieve”

• “more as a facilitator of knowledge”

• “much more interactive, I have the students create things themselves, not just looking at the texts”

• “more of a partnership in learning than what we had before”

• “they own what they learn”

• “a lot more self-discovery”

Questions

Ali Moelleramoeller2@unl.edu