group4 present3 3-15
TRANSCRIPT
![Page 1: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/1.jpg)
The 2010 User Friendly
Handbook of Project
Evaluation
Chapter 4: The Evaluation
Process: Carrying Out the
Study and Reporting
![Page 2: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/2.jpg)
Presented by
M. Bess-Frazier
T. Flemming
V.Lee
A. Smith
N. Wooten
![Page 3: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/3.jpg)
Introduction
The Process of Evaluation and Reporting
consists of 4 general steps:
• 1. Data Collection
• 2. Data Analysis
• 3. Report the Findings
• 4. Publishing the information
![Page 4: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/4.jpg)
Step 1: Data Collection
![Page 5: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/5.jpg)
Step 2: Analyze the Data
![Page 6: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/6.jpg)
Step 3: Report the Findings
![Page 7: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/7.jpg)
Step 4: Publish (Disseminate) the Information
![Page 8: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/8.jpg)
Step 1: Data Collection
![Page 9: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/9.jpg)
Conducting the Data Collection
Things to consider:
• Get permission
• Consider needs and sensitivities of
participants
• Make sure data collectors are properly
trained, unbiased, and objective
• Get data from as large of a sample as
possible
• Disrupt as little as possible
( Frechtling, 2010, p. 39)
![Page 10: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/10.jpg)
The Necessary Clearances and
Permissions
Think about the setting of your data
collection!
• Talk to teachers, administrators, bosses,
or parents where applicable
• Consider permission forms
• Consider who needs a copy of the final
data collection and conclusion
It may help with cooperation if you offer to
share the results after you’re done!
![Page 11: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/11.jpg)
Consider the Needs and Sensitivities of Participants
• Be honest with participants
• Assure participants that no personal
repercussions will result from any
information provided
As an anecdote, consider a survey on what
percentage of the population uses their seatbelt
when in a care. Do you think the results would
differ if a police officer gave the survey?
![Page 12: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/12.jpg)
Make Sure Data Collectors are Properly Trained
• Provide data collectors information about the community or culture where they will be collecting
• Data collectors need to ask the same questions and use the same prompts
• They should be supervised to avoid “coaching” and a distortion of data
• Data collectors should be unbiased
![Page 13: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/13.jpg)
Collect Data from as Many
Members as Possible
• More members will decrease the margin
of error (Moore, 1993)
• A response rate of 70% or higher within
the sample is considered to be high
quality
• Follow up on non-participants
![Page 14: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/14.jpg)
Avoid Disruption
Consider the setting of your data collection!
Consider the schedules of participants and
the project as well.
![Page 15: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/15.jpg)
Step 2: Analyze
the Data
![Page 16: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/16.jpg)
Data analysis checklist
✓Check the raw data and prepare them for
analysis
✓Conduct initial analysis based on the
evaluation plan.
✓Conduct additional analyses based on
the initial results.
✓ Integrate and synthesize findings.
![Page 17: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/17.jpg)
Check data for unusual responses.
– Example: selecting
more than one answer
instead of the one
required
– Choosing the same
response for all
question (i.e. “c”)
– Inconsistent answers
These questions may need to
be eliminated from the data that
will be analyzed. The data is
then prepared for coding and
entering for computer analysis.
Verification and quality control
procedures should already be in
place.
✓Check the raw
data and prepare
them for analysis.
![Page 18: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/18.jpg)
– The analysis may raise
more questions as results
are calculated. Be careful
to stay with the set of
analysis that were
originally of interest.
– Updated software is better.
These offer support to
analysts and a way to
manage large sets of
narrative data.
– quantitative data collection
should be consistent to
avoid unnecessary
invalidation or affect the
evaluation integrity.
✓Conduct initial
analysis based on
the evaluation plan
![Page 19: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/19.jpg)
“It is very likely that the initial analyses
will raise as many questions as they
answer.”
– The second analyses will address
these.
» Example: “The first analysis
looked at teacher performance, a
second analysis might subdivide
the total group into subunits of
particular interest—e.g. more
experience versus less
experienced…..examine if any
significant differences were found
between them. “
» These cycles can continue as
resulting data suggest other
interesting avenue to explore;
even if they were not planned
(Frechtling, 42).”
– Available time and money determines
which tasks can be completed.
✓Conduct
additional
analysis based
on the initial
results.
![Page 20: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/20.jpg)
Integration of the separate analyses,
the development of the findings’
conclusions are the last steps of
data analysis. The results may not
produce completely consistent
findings or fit into a report that
explains ambiguities. Some report
findings can remain unanswered...
✓ Integrate
and
synthesize
findings
![Page 21: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/21.jpg)
Step 3: Report the Findings
● Requires:
o pulling together the collected data
o condensing the findings to correlate the
original questions of the evaluation
o publicize the findings
![Page 22: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/22.jpg)
Main Sections of Formal Reports
“1. Background
2. Evaluation study questions
3. Evaluation procedures
4. Data analyses
5. Findings
6. Conclusions (and recommendations) “ (Frechtling,44)
![Page 23: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/23.jpg)
BackgroundThis section can include:
• a need or focus of the project and objectives• possibly a journal or literature review
• the stakeholders and their background
• the study participants
• the activities and components
• project expected timeframe and venue
• the project resources used
• the expected measurable outcomes
![Page 24: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/24.jpg)
Evaluation Study Questions
• Lists the questions that the study
addressed.
• Also includes variables that may affect the
project outcome, for instance, issues with
data collection, time constraints.
![Page 25: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/25.jpg)
Evaluation
Procedures ● This section also describes the
types of data collected and the
instruments used for the data
collection activities.
● It is helpful at the end of this
section to include a matrix or
table that summarizes the
evaluation questions, the
variables, the data collection
approaches, the respondents,
and the data collection schedule.
● This section
describes the
project
participants.
● The participants’
selection method
and whether
sampling was
used.
![Page 26: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/26.jpg)
Data AnalysesSection Explains:
• Data analyses methods
• Data analyses stages and how they were applied
• Data integrity safeguards to exclude study confounds
• Description of measures to ensure the study
participants represented the subject population
• A table summarizing the data analyses, if feasible
![Page 27: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/27.jpg)
Findings
• Includes the analyses results
• Lists the study/ project questions asked even if no
answer was produced from the study
• Visual displays such as graphs if applicable
• A final project summary that explains the major
conclusions
![Page 28: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/28.jpg)
Conclusions and Recommendations
• Summarizes more generally
• Addresses the project’s findings,
evaluation questions and program goals
![Page 29: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/29.jpg)
Other Sections
• If applicable , there can also be formal
reports that have:
– A study abstract with findings summary
– An executive summary which contains
more detail as to the study synopsis,
conclusions, including implications for
relevant practices
![Page 30: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/30.jpg)
Developing an Evaluation Report
• Ask colleagues or stakeholders for a critical
review of the report and any suggestions for
improvement prior to the final publication
• The report should be presented in a way
that is suitable for intended audience
![Page 31: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/31.jpg)
Step 4: Disseminating the Information
![Page 32: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/32.jpg)
● Dissemination is the last part of project
evaluation. (Spreading your information)
● It should include the project funding source or prospective funding sources
![Page 33: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/33.jpg)
The Dissemination should have:
• What the group needs to know and the best
way to communicate the information to them
• In what way did the project accomplish the
goals according to the National Science
Foundation?
![Page 34: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/34.jpg)
Sharing your Findings
• Make a list of the various audiences to
share findings.
• It is important to share projects that
worked and projects not so fruitful.
![Page 35: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/35.jpg)
Conclusion
An effective study evaluation should include
collecting data, analyzing the data, and reporting and
publishing the study information. There are various
elements to consider to maintain integrity and avoid
biased in collecting study data. Objectivity in reporting
the study evaluation findings to stakeholders, school
district administrators and study funding sources will
enhance the ongoing development of educational
research for all professionals.
![Page 36: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/36.jpg)
References
Frechtling, J., Mark, M., Rog, D., Frierson, H., Thomas,
V., Hood, S., Hughes, G. (2010). The 2010 User Friendly
Handbook of Project Evaluation. Arlington, VA: National
Science Foundation, Directorate for Education & Human
Resources, Division of Research, Evaluation and
Communication.
Moore, D., & McCabe, G. (1993). Introduction to the
practice of statistics (2nd ed.). New York: Freeman.
![Page 37: Group4 present3 3-15](https://reader034.vdocuments.mx/reader034/viewer/2022052413/55aa17431a28ab4c248b45fe/html5/thumbnails/37.jpg)
Open Ended Questions for the
Class
● In data collection, we must consider the needs and
sensitivities of the participants. For example, it would
be biased to have a police officer ask people if they
broke the law. What other examples can you think of
where the data collector might affect the information
received? How might this affect us in the classroom
as teachers?
● For Reporting the Findings and Study Dissemination,
what are the most important factors to consider in how
and to whom the findings are reported?