aes conference sydney 2011 ralph g straton murdoch university evaluation design for differential...
TRANSCRIPT
AES Conference Sydney 2011
Ralph G Straton Murdoch University
Evaluation Design for
Differential Effects Evaluation
Evaluation Design for Differential Effects Evaluation
Overview
Nature of Evaluation Design
Evaluation Questions
Inference Issues
Evaluation Design for Differential Effects
Qualitative Evaluation and Differential Effects
Conclusion
Evaluation Design for Differential Effects Evaluation
Nature of Evaluation Design
Blueprint for the Study
What information to be gathered
From which Sources
When it is to be gathered
How will it be Obtained
Limits Nature and Extent of Information Obtained
Evaluation Design for Differential Effects Evaluation
Evaluation Questions
• Does the program have an effect? and
What are the effects of the program?
• Does the program achieve its goals? and
To what extent does the program achieve its goals?
• Is the program better than the present program (or some other specific program)? and
Is the program the best program?
Evaluation Design for Differential Effects Evaluation
Evaluation Questions
• To what extent are program outcomes related to status on Characteristic X?
• Do NESB students perform as well as ESB students on the new ‘Culture Free’ maths program?
• Is performance among NESB students related to their culture of heritage?
Evaluation Design for Differential Effects Evaluation
Inference Issues
Idealised View of Evaluation Inference
Inference to the Population Sampled From utos > UTOS
Reality of Inference in Evaluation
Inference to ‘Other’ Populations utos > *UTOS
Maximising Utility of Evaluation Inferences
Choose Sample to Reflect ‘Other’ Populations
Observe Relevant Person, Program and Setting Characteristics
PROGRAM TARGETPOPULATION
OTHERPOPULATION
A
OTHERPOPULATION
BEVALUATIONSAMPLE
Evaluation Design for Differential Effects Evaluation
Evaluation Design for Differential Effects
Identify and Observe Relevant Characteristics Variables
Build Program Variations into the Design
Observe ad hoc Program Variations
Determine and Report Diversity on Relevant Variables
Determine and Report Program Interactions
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Evaluation Design for Differential Effects Evaluation
Qualitative Evaluation and Differential Effects
Include Qualitative Descriptions of Program Variations
Include Qualitative Descriptions of Program Settings