POSITIONING EVALUATION AS RESPONSIVE TO INFLUENCES AND AS BEING INFLUENTIAL
Cheryl Ballantyne
NSW Department of Education and Communities
Western Sydney Region
Australasian Evaluation Society International Conference
29 August – 2 September 2011
Hilton, Sydney, Australia
AIMS OF PRESENTATION
To explore the notion of evaluation as an intervention by:
Describing a current evaluation in progress
Identifying influences to date
Outlining future directions for the evaluation
THEORETICAL CONTEXT Evaluation as an intervention has an end
goal of social betterment (Henry, 2005; Henry and Mark, 2003)
Evaluation as an intervention has potential to influence at individual, interpersonal and collective levels (Henry and Mark, 2003)
Integrated theory of evaluation influence incorporates three dimensions, source of influence, intention and time frame (Kirkhart, 2000)
KEY CLAIM
Evaluation, when incorporated as a regular intervention within program implementation can increase the program’s effectiveness and its responsiveness to contextual influences.
STRUCTURE
1. Description of program
2. What have been the contextual features that have influenced the program development and implementation?
3. What influences has the evaluation had to date?
STRUCTURE
4. Why has this evaluation had these influences?
5. How was the evaluation data generated?
6. What are the implications for developing future evaluation instruments?
1. WESTERN SYDNEY NINGBO CHINESE VOLUNTEER TEACHER-RESEARCHER PROGRAM
Partnership involving DEC Western Sydney Region, Ningbo Municipal Education Bureau, PRC, and University of Western Sydney
PROGRAM AIM
To promote the teaching and learning of Chinese language and culture in WSR schools by generating interest in schools and embedding that interest through a sustained program of support
Long term – development of students’ intercultural competence in Chinese
PROGRAM FEATURES2008-2012 18 participating schools, 10 primary and 8 high
schools selected by expression of interest, majority of students are non-background speakers of Chinese
Up to 10 graduates from Ningbo, China arrive at the end of June each year
Volunteer in schools two days per week to support the teaching and learning of Chinese language, culture and society
Additional human resource to participating schools equal to approximately $25,0000 per year
PROGRAM FEATURES2008-2012
Volunteers complete Master of Education (Honours) at University of Western Sydney
Research informs and is informed by school experience
Return to China in December the following year
2. CONTEXTUAL FEATURES THAT HAVE INFLUENCED PROGRAM DEVELOPMENT AND EVALUATION
Globalisation
Expansion of China in the worldApprox 1.2 billion English speakers in China
Approx 700 billion Chinese speakers in other countries
Implications for students’ future communities and employment
CONTEXTUAL FEATURES THAT HAVE INFLUENCED PROGRAM DEVELOPMENT AND EVALUATION
Commonwealth Government commitment to increasing Asia literacy of Australian students
2020 target
National Curriculum Languages learning area
Mandarin Chinese to be one of two target languages to be taught in NSW
Across curriculum dimension – Asia and Australia’s engagement with Asia
CONTEXTUAL FEATURES THAT HAVE INFLUENCED PROGRAM DEVELOPMENT AND EVALUATION
NSW DEC memorandum of understanding with China
NSW Confucius Institute in partnership with China
Confucius classroom in Western Sydney Region
CONTEXTUAL FEATURES THAT HAVE INFLUENCED PROGRAM DEVELOPMENT AND EVALUATION
Western Sydney Region memorandum of understanding with Ningbo Municipal Education Bureau
annual cultural exchange visits sister schools
EVALUATION RESPONSE 2010 – 2012 evaluation of Volunteer
Teacher-researcher Program to inform ongoing program development and future directions
Focus question: What can a Chinese native-speaker volunteer program contribute to the teaching and learning of Chinese language and culture in schools?
ANALYSIS OF EVALUATION INFLUENCES:
Individual
Interpersonal
Collective (Henry and Mark, 2003)
3. WHAT INFLUENCES HAS THE EVALUATION HAD TO DATE?
Individual
Increased the importance individual principals and regional staff attribute to the Program
Renewed individual principals’ commitment to the Program
Shaped Regional leaders’ attitudes towards importance of Chinese language learning and the partnership with Ningbo
WHAT INFLUENCES HAS THE EVALUATION HAD TO DATE?
Interpersonal
Principals share ideas about their Chinese programs with one another
Principals emphasise importance of Chinese programs to staff and community
WHAT INFLUENCES HAS THE EVALUATION HAD TO DATE?
Interpersonal
Additional schools have expressed interest in participating in the Program
Inquiries about/interest in the Program from interstate and national levels
Meeting of partners to be held in September 2011 to discuss extension of partnership
WHAT INFLUENCES HAS THE EVALUATION HAD TO DATE?
Collective Signing of new Memorandum of Understanding
planned for late September 2011
Increased allocation of funding to program
Regional planning to target outreach programs delivered through Confucius Classroom to high school learners
Planning of professional learning to focus on areas for further program development
EVALUATION INTERVENTION AND SOCIAL BETTERMENT In 21st century Australia, Asia literacy and
development of intercultural competence in Chinese has potential to lead to social betterment (increase students’ world view, improve employment prospects).
Evaluation has identified student, parent and teacher interest in Asia literacy through Chinese
Evaluation has revealed that students, parents and teachers regard Asia literacy through Chinese as important
Evidence has influenced further evaluation strategies, to enable a deeper analysis of influence
4. WHY HAS THIS EVALUATION HAD THESE INFLUENCES? (EVIDENCE)
0
1
2
3
4
5
6
7
Schools with Mandarin teacher
Schools with volunteer only
Schools with other support
Number of WSR primary schools providing Chinese language/culture programs by type
of support
WHY HAS THIS EVALUATION HAD THESE INFLUENCES? (EVIDENCE)
480
500
520
540
560
580
600
620
640
Year K Year 1 Year 2 Year 3 Year 4 Year 5 Year 6
Primary students in WSR learning Chinese language/culture
WHY HAS THIS EVALUATION HAD THESE INFLUENCES? PRIMARY SCHOOL SURVEYS
702 student responses from eight schools
67% - learning Chinese language important 80% - learning Chinese language interesting 63% - learning about China important 78% - learning about China interesting 65% would like to continue learning Chinese
in 2011
WHY HAS THIS EVALUATION HAD THESE INFLUENCES? PRIMARY SCHOOL PARENT SURVEYS
117 parent responses from seven schools
More than 80% - learning Chinese language and culture important and valuable for students
WHY HAS THIS EVALUATION HAD THESE INFLUENCES? HIGH SCHOOL SURVEYS
245 student responses from five schools
178 student responses from one school only
10 parent responses all from one school
WHY HAS THIS EVALUATION HAD THESE INFLUENCES? (EVIDENCE)
0
100
200
300
400
500
600
700
800
Year 7 Year 8 Year 9 Year 10 Year 11 Year 12
Secondary students in WSR studying Chinese language courses
WHY HAS THIS EVALUATION HAD THESE INFLUENCES? STAFF SURVEYS
50 staff responses representing 12 schools38 from 6 primary schools12 from 6 high schools
95% - learning Chinese language important
100% - learning Chinese culture important
93% - Chinese Program has a positive influence on schools
WHY HAS THIS EVALUATION HAD THESE INFLUENCES? STAFF SURVEYS
0
5
10
15
20
25
30
30 minutes to one hour one to two hours more than two hours
Time spent on Chinese language/culture per week
WHY HAS THIS EVALUATION HAD THESE INFLUENCES? PRINCIPAL FOCUS GROUPS
Volunteers’ rapport with students has stimulated interest in Chinese language
Chinese language is accepted as part of the curriculum (no longer exotic – High School Principal)
Planning for Chinese language and culture programs occurring in primary schools and high schools
WHY HAS THIS EVALUATION HAD THESE INFLUENCES? PRINCIPAL FOCUS GROUPS
Chinese language has replaced other language programs - two high schools
Links between Chinese language and culture and other learning areas - three high schools
Chinese classroom allocated - one high school
WHY HAS THIS EVALUATION HAD THESE INFLUENCES? PRINCIPAL FOCUS GROUPS
Collaboration across learning communities to support transition Year 6 to 7.
System support important to the long term effectiveness of school programs
5. HOW WAS THE EVALUATION DATA GENERATED
DEC annual census information on languages programs
Focus group of Principals term 1 2010
Surveys of students, parents and staff distributed to 18 participating schools term 3 2010
EVALUATION INFLUENCE: SOURCE, INTENTION, TIME FRAME
Source – process (focus groups) and reporting of evidence (results)
Intention – intended (focus groups, reporting of evidence)
Time frame – immediate, evaluation embedded in program implementation
(Kirkhart, 2000)
6. WHAT ARE THE IMPLICATIONS FOR FUTURE EVALUATION INSTRUMENTS? FUTURE DIRECTIONS
Student self-assessment of intercultural competence to enable deeper analysis of Program’s impact on student learning
Parent and teacher assessment of student intercultural competence to enable deeper analysis of impact on student learning
6. WHAT ARE THE IMPLICATIONS FOR FUTURE EVALUATION INSTRUMENTS? FUTURE DIRECTIONS
School self-evaluation (based on program standards) to enable deeper analysis of school programs and influence school planning
Document analysis of school plans, policies and annual school reports to analyse extent to which Chinese programs are valued, planned for and embedded in whole school operation
CONCLUSION This evaluation (in early stages) has had
influence at school, regional, state, interstate and national levels
Further planning of evaluation instruments is intended to deepen influence
This evaluation, as an intervention embedded in program planning and implementation, has potential to make a difference for Western Sydney students
THANK YOU FROM CHERYL BALLANTYNE
FOR CONSIDERING - POSITIONING EVALUATION AS RESPONSIVE TO INFLUENCES AND AS BEING INFLUENTIAL
Australasian Evaluation Society International Conference
29 August – 2 September 2011
Hilton, Sydney, Australia