Evaluation – some ideas and suggestions

Educational psychologist, focus on motivation, self-

regulated learning and designing powerful problem-oriented learning environments

member of the national counseling group in PISA 2004-> assessment of students‘ attitudes and perceptions of family and school environment

I‘m in charge of the university, responsible for the organizational development

University of Bielefeld

Elke Wild

Evaluation - Definition Evaluation is often seen as unnecessary or threatening

because it‘s used to justify allocations of ressouces, responsibilities etc.

The actual purpose of evaluation, however, is to assess the effects of your interventions (observation) and to value them in terms of standards (by referring to objective

criterias, temporal and/or social comparisons; most common are social comparisons by referring to matched control groups)

Thus, evaluation is a means to obtain or preserve (attributed or self-defined) standards of

quality („Qualitätssicherung“) to get feedback how to improve quality / desired goals or

standards.

„traditional“ vs. „innovative“ kinds of evaluation researcher-designed field studies aiming at the

documentation of success or failure (empirical support of theoretical assumptions)

„collaborative innovation“ as an alternative (and probably superior) form of evaluation; it means that practicians contribute to the definition of goals (theoretical constructs) and criterias

(empirical indicators) the development of conceptual features (including scripts,

materials, assessment instruments) the implementation of measures (i.e., exchanges in the

course of teacher education) the kind of evaluation (decisions concerning the relative

importance of goals, the dedication of time and money)

proposal held out the promise of two kinds of evaluation

formative evaluation means an ongoing assessment; emphasis is on the continous and qualitative feedback concerning the process

-> evaluation as a kind of coaching, counseling designed to optimize interventions/projects

final evaluation („summative Evaluation“) means to assess goals attaiment at the end of an intervention in order to come to definite conclusions concerning the benefit of the project („product oriented evaluation“)

issues to clarify frequency / times of measurements: before projects starts

(expectations)? after the first run (problems, challenges)? after each run (progress, new goals?) and/ or at the end (lasting effects in terms of implemantation, transfer, dissemination)?

relevance of content areas/dimensions: knowledge (repertory of instructional strategies in different domains, languages etc.), attitudes (awareness of the „european dimension“; sensitivity for social and political issues), changes in (professional) self-concepts and motivation to continue exchange, intra- and interdisciplinary cooperation, use information technologies (internet, email)

frequency of feedback: third and sixth meeting (see proposal)? interviews in two selected sites / countries: most „typical“

institutions/sites? most divergent institutional / organizational conditions (restrictions, resources, experiences, ....) to consider a variety of potential problems?

issues to clarify: target (groups of) persons members of the organizing committee /coordinators, mentors, fellow worker, guest families, students (Refrendare), students (Schüler), teachers / teacher training institutions who need informations

concerning the establishment of intercultural exchange-projects; strategies to distribute CD-ROMs (Nachhaltigkeit und Verbreitung – see expert comments!)

materials / strategies: a) to facilitate participants‘ problem solving b) to improve the communication within INCLUDEME-

members: utility of includeme-domain, homepages, INTERWISE conference system;

c) to ensure dissemination: utility of handouts / reports / „survival kids“; instruments for ongoing self-evaluation; marketing

d) to influence probability of acceptance and transfer: informations concerning „critical issues“ (requirements that are necessary or desirable; factors that may moderate success / effects),

formative evaluation – methodological suggestions questionnaires (questionnaires, online scales, diaries/portfolio as a means to

foster self-monitoring) in ordera) to assess informations concerning similarities and differences between locations/countries (issue of generalizability) b) to examine effects on participants (vs. comparative students, teachers etc.)c) to get some transfer information (changes in attitudes of staff members not participating directly)

interviews (semi-structured, with selected individuals that represent participation students, mentors and coordinators) in order to identifiy challenging issues and moderating factors

expert ratings (professionals for multimedia learning environments, teachers interested in european exchanges) to assess the utility of handouts / reports / procedures / instruments used for evaluation

additional methods: tests (competencies)? observations (variance in videotyped instructional strategies)? records / files (representativenes of participants)

some final remarks...

result of the first evaluation (on base of the application for grant) of two experts

critical feedback: only few ideas to improve long-lasting effects and dissemination within institutions and beyond ideas how to inform and motivate colleagues,

students, classmates... idea how to distribute CD-ROM