Purpose and Focus of Evaluation

The purpose and focus of the evaluation is to better understand the results of student

learning in a multiple discipline areas with the use of technology. The evaluation will attempt to

evaluate how project based learning improves student learning of technology tools and

materials. This evaluation is a formative evaluation which aims to prepare needs assessment as

well. Since the program is new it will need to make necessary adjustments as the program

grows. This program has been developed to allow students to make independent decisions on

how they can learn and express themselves creatively with technology.

Description of Program

The program to be evaluated is a high school technology requirement. This program is a

required course in which students complete the course using their choice of subject area (science,

English, social studies) develop a project using technology. This course is proposed to start in the

2011 school year. The course is taught by technology department teachers and utilizes teacher

collaboration from multiple subject areas. Students need to develop their own project concepts

and what technology they will use to complete the project.

What stakeholders will be informed?

The stakeholders who should be informed are the Director of Technology, Principal of the

High school and prominent and effected department heads. These stakeholders should be

informed of the results of this evaluation in order to understand whether the program they are

using promotes certain outcomes.

Evaluation of Capabilities and Risks of Evaluation

The capabilities and risks of doing this evaluation are relatively small. The evaluation will

require as few as three individuals to work as individuals to administer interviews, collect video

data and administer testing procedures. The major investment in the completion of this

evaluation is time for data collection and organization. The cost of having individual

administering the testing and surveying materials is anticipated to be minimal.

What are the political ramifications?

The political ramifications for this evaluation are that it could empower the technology

department to have a larger role in curriculum development or it could hurt the need for

technology in the school curriculum. The results of the questions could provide solid data that

technology learning improves project based learning techniques. This program could also show

the weaknesses of student’s technology learning.

Description of Approaches and Models Used

“The distinguishing feature of an objectives orientated evaluation approach is that the

purposes of some activity are specified, and then the evaluation focuses on the extent in which

these purposes are achieved.”(Fitzpatrick,Worthen,Sanders,2004) It is the object-orientated

approach in which my evaluation will seek to determine the extent of student learn across

content areas, skill levels and technology projects. The key questions of the evaluation will seek

to define specific goals and learned skills.

Object Orientated Approaches to Key Questions

1. How does the WHS tech requirement increase the student use of technology across the

“content areas”?

Objectives-orientated approach

This question requires that specific data regarding students using different content areas in

order to complete their technology projects. This question gets data pertaining to the specific

content areas students will use. This question will require on qualitative and quantitative data

collection. The described content areas like English, math or Social Studies will be asked in both

quantitative questions and in qualitative interviewing scenarios.

2. Based on the standards of the course requirement will students engage more in the project

based learning?

Objectives –orientated approach

This question will measure student’s level of engagement and learning with project-orientated

school work by having to use technology. This question will be answered primarily in qualitative

interview questioning. This specific question is aimed primarily at teachers but there is an

anticipation that all participants will be asked to reflect on their experiences with subject matters

similar to project based learning. There will be quantitative data collected from likert surveys to

measure the tendencies of learning styles. One of the anticipated learning styles anticipated in the

questions will be “Project Based”.

3. Will the use of project based learning increase student use of technology?

Objectives-orientated approach

This question will measure the student’s use of technology before and after the beginning of

this program. To answer this question most definitively will be to create test score summaries

from the pre and post technology skills test. This data can be constructed to show the growth of

learning as well as possible areas of limited learning.

With Qualitative data students can express in interviews or focus groups their perceptions of

learning. In this forum students will hopefully respond to their understanding of their learning

and expectations. Students and teachers can reflect on specific uses of technology and scenarios

in which they felt as though they expanded their knowledge base.

Who is the Audience or Stakeholders?

The audience for this evaluation would be the High School Principal, Director of Technology,

department heads of different departments, Assistant Superintendant of Curriculum. The core

group of teachers and administration and committee to help develop and implement the program

would also hold interest. The data from the evaluation will help the stakeholders make changes

in the curriculum or implementation of the program.

The stake holders will also be part of the formative evaluation of the program as their input

into future evaluations and goals will be taken into account. It will be important to work

successfully with the stakeholders in order to ensure their participation and interest in continually

evaluating the program.

An important factor in this evaluation is the school community of parents, administrators and

teachers. In Winchester, Massachusetts the affluent community cares deeply about the education

of the students and the programs they are involved with. “Dialogue with multiple audiences also

clarifies the reasons behind an evaluation.”(Fitzpatrick,Worthen,Sanders,2004) involving these

audiences will empower the evaluation and show clear reasons for the evaluation on all sides.

What are the key questions?

The evaluation questions are based on the outlined focuses of the program. Under

agreement between major stakeholders (High School Principal, Technology Director, High

School Department Heads) the focused areas of the High School Technology Program will teach

technology courses which allow students to take advantage of multiple subject disciplines,

creating technology final projects and learning specific technology skills. Having the key

questions aligned with the stakeholders thinking should enhance buy in from the

administration.

The Big Questions

1. How does the WHS tech requirement increase the student use of technology across the

“content areas”?

2. Based on the standards of the course requirement will students engage more in the project

based learning?

3. Will the use of project based learning increase student use of technology?

Summarized Researched Findings from a Project Based Learning Article

The research article attempts to make sense of current data and definitions of project based

learning defined in the article as “PBL”. The article clarifies what project based learning is

predominantly defined as and also explains that part of the problem with assessing project based

learning is that there have been different definitions of what it entails and how it has been

instructed in various contexts.

One of the key concepts of instruction with “PBL” is the “project as the central teaching

strategy.”(Thomas,2000) This means that all work and instruction is aimed at completing the

project not teaching subject areas in a central way only as needed. The projects are student

driven and when they are completed are realistic and not school-like. (Thomas,2000)

Conclusively PBL has students constructively investigate their question and use problem solving

to follow the investigation to completion.

The article describes common characteristics of project based learning as a “model which

organizes learning around projects” (Thomas,2000). This model “culminates in realistic projects

or presentations which use an authentic driving question and the use of tools of technology.”

(Thomas,2000) These open descriptions and programs which have adjusted the variables of PBL

have made it difficult to define PBL to one type of instructional models.

The “Expeditionary Models” which are similar to the predominant one described in the article

as Project Based Learning (PBL) has been shown to increase learning in content areas as well as

skill based learning. One example is the researches conducted at the New American Schools

Development Corp. Using “Expeditionary Learning” (PBL) students have improved standardized

tests scores on the Iowa Basic Skills in Dubuque, Iowa from “Well Below Average” to “District

Average”. The magnitude of change in schools through the schools in Dubuque, Iowa was from

15%-90%. (Thomas,2000)

This research substantiates the evaluation of the learning of technology skills in the Required

Technology Course Program with the use of Project Based learning Initiatives. Students in the

program will be self directed learners who will problem solving skills in order to complete

projects. The evaluation of the learned technology skills will determine if the PBL approach is

appropriate and successful in the Required Technology Course context.

Key Question Analysis

The key questions will be addressed with a series of data collection techniques which will

collect quantitative and qualitative data. Each type of data is necessary to best assess the

program’s success at teaching the skills that need to be learned with technology but all the

experience and process of the students projects in the program.

The qualitative questions will be addressed primarily in interviews and focus groups. It is

important to collect this data because it will give insight to the learning process involved with

specific student’s projects as well as their happiness with the program.

Quantitative questions will be asked in the form of likert surveys and test summaries. These

questioning methods are necessary to understand in clear numbers the learned skills and

knowledge as well as percentage data which can reflect the student happiness and perceived

success with the operation of the program.

Research Methods & Feasibility Issues

How does the WHS tech requirement increase the student use of technology across the

“content areas”?

Research Method: Students will take a likert scale survey which will assess their inclusion of

other subject disciplines with technology learning before and after their participation in the

course. These questions will ask students to choose a content area in which they worked and try

to develop an understanding of their selection for specific content areas. The variable with this

question is if the students who are using technology across disciplines are learning new skills or

are they just applying it differently to a new subject area.

Based on the standards of the course requirement will students engage more in the project

based learning?

Research Method: Students will be given a likert survey which assesses the student’s likelihood

to complete a long range project using technology before and after their participation in the

project. Students will also be asked to define specifically their work ethic in projects. Teachers

will be asked to measure student reaction to multiple step project assignments. Administrators

may be asked to express their opinion on the showcasing of multi-step projects. A variable in this

questioning is individual’s completion of projects. Some may complete the project successfully

but not learn new tools or do it in a project based way. These variables allow for new questions

about the learning methods.

Will the use of project based learning increase student use of technology?

Research Method: Students will be surveyed on their use of technology habits before and after

their participation in the course. This question is looking for a growth of technology tools being

used as well as the student’s knowledge of technology tools. The pre and post test of technology

skills will give in a clear presentation of quantitative data the stronger and weaker areas of the

students technology knowledge gained over the course of the program. A variable for this

question is the normal difference in some students learning compared to others and how this

might affect the data.

Data Types

The quantitative data that I will receive will be in the form of questionnaires and likert

surveys from primary sources. These questionnaires should reflect in quantitative percentages the

habits, confidence, skills of students use of technology, project based learning and independent

work skills and experience.

There is qualitative data to be collected from interviews and focus groups. This data will also

be a primary source. This data will be from students, teachers, administration and community.

This data will allow individuals to reflect openly on their experiences with the program. The

questioning will be open ended and encourage personal expression with inclusions of scenarios

and portrayals of the sources experience.

Questionnaires-Open question questionnaires, for those who cannot be interviewed. Data will

provide open answers to questions regarding participation and final outcomes in the program.

Test Data-Test scores and summaries which will provide percentage data of learned skills in

technology.

Interview transcriptions-Transcribed data of interviews of participants providing experiences and

scenarios.

Interview videos-Interview data which can be used for continual viewing and for final

presentation.

Likert Surveys-Surveys to determine opinions and feeling pertaining to student involvement with

the program as well as their learning tendencies.

Measurement of Quantitative & Qualitative Data

This evaluation will be a formative evaluation which will provide data of the success of major

components of this plan. This evaluation will provide quantitative data which will show the

students tendencies with technology and learning on their own. It will be important that students

are able to provide a broad range of responses in regards to their use of technology. It will also

be important that a broad range of students are surveyed in order to gain quality data which

includes students who willingly take technology courses and those who may have a limited

experience with technology learning.

Feasibility Issues of Evaluation: Data Sources, Sampling Strategy & Validity Issues

The data sources for this evaluation will be prospective students for the high school

technology required course. They will be randomly sampled. Some of the validity issues may

develop with students who have taken no technology courses or those students who have taken

other high school courses or have been using technology regularly in their classes to complete

projects.

The sampling strategy for the data is to use 100 participating students as well as all

participating administration and teachers. The sampling size allows for focus groups as well as

individual interviews. The sampling size also allows for complete testing to be completed in a

relatively short time.

Potential validity issues exist with students being able to discuss their experiences before tests

and interviews. Potential bias and validity issues exist with the evaluator working within the

program being evaluated. Considering the relatively short testing times the data collection should

show little to no validity issues. The evaluator will hire individuals to administer tests and direct

focus groups and interviews.

The Content, Audience and Focus of the Final Report

The presentation of this report would be a presentation night specifically at the high school

in which everyone could receive a copy of the report and watch a multimedia presentation

which outlined some of the major parts of the report.

The multimedia presentation would use a series of videos of student experiences and stories

about their participation in the program. These “experience stories would give real life scenarios

and portrayals of their learning in technology and the skills they gained. The video can also show

some of the qualitative interview data given from major stakeholders and express some of the

key goals of evaluating the program.

A key piece of the presentation will be to show some of the pertinent quantitative data in a

PowerPoint presentation. This will reveal the results of the evaluation and then open up the

presentation for a question and answer period. An open and honest presentation of the experience

of all of the students involved will be very powerful with the audience. The expected audience

would be students, parents, teachers, administrators and community members.

Time and Task Management Plan Visual

Time and Task Plan Visual: High School

Technology Requirement

September 1, 2011

Pre-Evaluation Site Visit: Evaluate site context

of learning, resources, anticipated learning

environment.

Participants: Students, Teachers,

Administration, School

Time Period: One Month

Prior to evaluation start.

September 15,2011

Pretest Of Technology Skills: Technology skill

test based on National Technology Standards.

Participants: Participating Students

Time Period: 2-3 Days

September 19,2011

Likert Surveys Pre-test: Evaluate student,

teacher, administration opinions, attitudes

and dispositions with technology curriculum

learning and process.

Participants: Teachers, Administrators,

Time Period: 2-3 Days

Students

September 23-December 10, 2011

Continual Site Visits During Program

Time Period: Once a

Week for duration

Of program.

December 12-13,2011

Interviews with Teachers, Administrators

Time Period:2 Days

December 13-15,2001

Interviews with Students

Time Period: 3 Days

December 16-18, 2011

Focus Groups Meet 3 Sections

Subjects: Project based learning

Technology skills

Content Areas

Time Period: 3 Days

January 2-20

Data Analysis and Synthesis Time Period: 3 wks.

February 20th: Final Presentation of Findings Time Period: 2 Days

References Fitzpatrick, S.,Sanders,J.,Worthen,B. (2004). Program Evaluation:Alternative Approaches and Practical

Guidelines 3rd Edition (. Boston: Pearson Education.

Thomas,J. W. (2000). A Review Of Research On Project Based Learning. San Fafael,California: The

Autodesk Foundation.