measuring of student subject competencies by sam: regional experience
DESCRIPTION
Measuring of student subject competencies by SAM: regional experience. Elena Kardanova National Research University Higher School of Economics. Outline of presentation. SAM based model for assessment of student subject competencies - PowerPoint PPT PresentationTRANSCRIPT
Measuring of student subject competencies by SAM: regional experience
Elena Kardanova
National Research University Higher School of Economics
Outline of presentation› SAM based model for assessment of student subject
competencies
› Regional diagnostic study: sampling and procedures of test administration
› SAM regional norms and presentation of results
› Interpretation and uses of SAM information: primary analysis of factors that influence educational results in primary school
› SAM uses for improving teaching and learning: relation between teacher’s pedagogical approaches and educational results in primary school
› SAM experience in other countries
SAM based model for assessment of student subject competencies
Approaches to results interpretation
Norm-Referenced
The result of individual student is interpreted depending on the achievement of the whole population
Each student gets test score
Norms are set
Approaches to results interpretation
Criterion-referenced
The gradual option of achievement scale is developed. It’s based on students integrated test scores and benchmarks that divide all participants of testing into groups that relevant to different proficiency levels
Each participant is assigned a proficiensy level
Students estimation
› Rasch model is used as a test model
› Test scores are reported on a 1000-point scale with a mean at about 500 and standard deviation of 50
› Test scores of all participants are on the same metric scale regardless of the time of test administration and specific set of test items completed
Mathematical competence scale
Proficiency level 3
We expect student A to successfully complete at least 50% of level 3
items
Student A
Proficiency level 2
Proficiency level 1
Below level 1
We expect student B to successfully complete at least 50% of level 2
items
Student B
We expect student C to successfully complete at least 50% of level 1
items
Student C
We expect student D to be unable to
successfully complete even 50% of level 1
items
Student D
500
570
430
Items of
the 3rd
level
Items of
the 1st
level
Items of
the 2nd
level
Estimation of examinees
Interpretation of benchmarks
Benchmarks:
570 (border btw. 2 and 3 proficiency
levels)
500 (border btw. 1 and 2 proficiency
levels)
430 (border btw. 0 and 1 proficiency
levels)
Regional diagnostic study: sampling and procedures of
test administration
Velikiy Novgorod and its area
Regional diagnostic studyMay 2012 Sample size: 4406 students of 4-th grade (the region’s whole population of fourth grade students )
No selection at the school or classroom level
Description of research sampling
47% boys,
53% girls
72% urban,
28% rural
Number of students 4406
Number of schools 189
Number of classes 297
Number of settlements 134
Regional diagnostic study: procedure
› Paper&pencil form
› Administration to the whole class by the teacher
› Two 45-minute testing sessions with a 15-minute break
› Whole region in 1 week
SAM regional norms and presentation of results
SAM results
Integrated test score( relation of result to the metric scale)
Proficiency level (relation of result to grade scale)
3D profile (relation btw. results of 3 subtests)
Normative-referenced interpretation: Statistical norms (Mathematics)
Average group norms Mean Standard deviation
517 34
Socio-cultural norms Mean
561
Average individual norms Mean Standard deviation 522 49
Percentile individual norms 10th percentile 90th percentile 459 581
Math profile for a sample of students
Distribution of test participants on proficiency levels (Mathematics)
Distribution of students of different schools of the region at proficiency
levels (mathematics)
› Schools put in order by increasing of the mean test score
› For every school the nean test score is indicated in brackets.
Distribution of students of different classes within the same school by achievement levels (mathematics)
Interpretation and uses of SAM information: primary analysis
of factors that influence educational results in
primary school
Key questions
1. What is the efficiency of different educational programs?
2. How different factors influence on students learning?
3. What characteristics of learning environment influence on educational quality in primary school?
4. How teachers and school work can be improved?
Educational environment and its characteristics that can be
examined
Level Responsibl
e entity
Domains
Regional /
Federal
Federal or local Government
Regional educational policy, Federal educational standard, unified exams, curriculum
School School principal
School policy, type of school, curriculum, condition of building and classes, sports sections, school activities, etc. Recruiting of teachers and administrative personnel
ClassTeacher Quality of teaching, methods of teaching,
pedagogical approaches. Quality of students feedback. Educational tasks and goals
Outside of
school,
family
Parents and student
Out-of-school activities, additional education, social-economic status, parents education, books, computer, Internet access, personal motivation
Information sources for analysis of factors
Contest questionnaries (for teachers, for administrative personnel)
› Set of contest characteristics can vary depending on regional research tasks
› Focus at characteristics of school environment that can be corrected to improve the quality of education
Technical informationThe detailed information about school learning and teaching features can be collected purposefully (e.g. educational programs).
Relation btw. SAM math results and type of educational institution
› There are 15% of student study in gymnasium.
› Differences btw. schools are statistically significant: gymnasiums get better tests results.
› Number of children at 2nd level is the same. Difference is btw. children at 1st and 3rd levels.
Relation btw SAM math results and type of settlement
Percentage of children at 3rd level is higher in the city and decreases in towns and villages. Its vice versa for percentage of children at 1st level – it’s higher in villages.
Relation btw SAM math results and type of settlement
Most of students are on the 2nd proficincy level In Velikiy Novgorod the results are slightly better – bigger percentage of children is at 2nd and 3rd levels.
Relation btw SAM math results and size of the class
› We can single out 2 types of classes – big and small
› Small classes are those that have less than 11 students, big classes have 11 and more students (maximum number of students in one class is 33)
› All together we analyzed 76 small and 152 big classes
Relation btw SAM math results and size of the class
Results of children in small and big classes are not statistically different.
SAM uses for improving teaching and learning: relation between teacher’s pedagogical approaches and educational results in primary school
Pedagogical approaches› Currently it is widely assumed that teachers’ beliefs
about the nature of teaching and learning include both “direct transmission beliefs about learning and instruction” or, so called, “traditional beliefs” and “constructivist beliefs about learning and instruction” (OECD, 2009).
› 2 educational approaches: traditional and constructivist
– The traditional approach implies that teacher communicates
knowledge in a clear and structured way, explains correct solutions, gives learners clear and resolvable problems and ensures peace and concentration in the classroom
– The constructivist approach implies that students are active participants in acquisition of knowledge, students’ own inquiry is stressed developing problem solutions
Teachers survey
› Special teachers questionnaire›228 teachers in total›56 teachers work in Velikiy Novgorod, and 172 in Novgorod region›17 teachers work in gymnasium and 211 in comprehensive school›186 teachers graduated from university and 42 graduated from college›Work experience varies from 2 to 48 years. Average is 25 years
Correlation -,204** is significant at the 0.05 (2-tailed)
Traditional
Constructivist
Teachers’ general pedagogical approach
Clusterization of classes
Pedagogical approaches
› Constructive approach positively relates to results of learning in math and Russian language: the higher level of constructivism of a teacher the higher test scores students have.
› Constructive approach positively relates to the number of students in class at 3rd level and negatively with number of students at 1st level and below 1st level.
› Traditional approach doesn’t have significant relation with learning results – neither with test scores or distribution of children at levels.
SAM experience in other countries
› Kazakhstan
› Kyrgyzstan
› Tajikistan
Thank you for your attention
Elena Kardanova
Center for monitoring and quality of education
Institute of education
Higher School of Economics
http://ioe.hse.ru/monitoring/