year five evaluation report: netc techshare star · pdf fileyear five evaluation report: netc...

Year Five Evaluation Report:

NETC TECHShare Star Schools Project

June 2004

Prepared by

Rebecca H. Zittle, M.A.

Frank J. Zittle, Ph.D

Center for Educational Evaluation & Research (CEER)

Executive Summary ............................................................................................................ 5

Introduction................................................................................................................... 11 Background of the Problem .......................................................................................... 11 TECHShare Goal and Objectives ................................................................................. 15

Objective 1 - Lesson Development........................................................................... 16 Objective 2 - NETtrain.............................................................................................. 18 Objective 3 – Network Infrastructure ....................................................................... 19

Evaluation Activities and Results ..................................................................................... 20 Introduction................................................................................................................... 20 Key Evaluation Questions............................................................................................. 20 History of the Project .................................................................................................... 21 Purpose of the Evaluation ............................................................................................. 21

Model Classroom Study............................................................................................ 22 Data Analysis ............................................................................................................ 26 How Evaluation Activities Addressed Project Objectives........................................ 26 Preparation for the Model Classroom Study............................................................. 29

Results from the Model Classroom Study: Evaluating the Impact on Students (GPRA 8.1.1) ............................................................................................................................. 32

Results Of the Pre- and Post-Assessments ............................................................... 40 Summary ................................................................................................................... 55 ??Recommendations ................................................................................................. 57

Curriculum Preparation Studio Surveys ....................................................................... 58 Lesson Development:................................................................................................ 59 Communication and Collaboration:.......................................................................... 62 Project Leadership/Project Organization: ................................................................. 63 Summary ................................................................................................................... 64 Lessons Learned/Recommendations:........................................................................ 65

Development of TECHShare Lessons (Modules): ....................................................... 65 Online Lesson Evaluation Rubric (GPRA 8.1.2).......................................................... 69

Method ...................................................................................................................... 69 Results....................................................................................................................... 70 Summary ................................................................................................................... 80 Recommendations..................................................................................................... 81

Content Development Specialist Survey ...................................................................... 83 Curriculum Development.......................................................................................... 84 Work with NETC Teachers ...................................................................................... 85 Model Classroom Study Support .............................................................................. 86 Summary and Recommendations ............................................................................. 87

NETtrain Web Site Evaluations.................................................................................... 88 Findings..................................................................................................................... 89

Evaluation Summary and Discussion of the Findings ...................................................... 93 Major Findings.............................................................................................................. 94 Recommendations......................................................................................................... 96

Evaluation Strategy and Lessons Learned ........................................................................ 98 Products Recommended for Wide Dissemination .......................................................... 101

2

References....................................................................................................................... 102 Appendix A Model Classroom Study Instruments ........................................................ 103 Appendix B Curriculum Preparation Studio Surveys A & B ........................................ 137 Appendix C NETtrain Web Site Evaluation Checklist.................................................. 142 Appendix D Facsimile of Online Lesson Rubric and Summary Comments ............... 145 Appendix E Content Development Specialist Survey ................................................... 165 Appendix F Miscellaneous Statistical Analyses Results ............................................... 169 Appendix G GPRA Indicator Data ................................................................................ 173 Appendix H Year 5 Evaluation Budget ......................................................................... 179

3

List of Figures and Tables

Figure 2 Results of an ANOVA Showing Significant Difference in Pre- and Post-test Gains Between Experimental and Control Groups................................................... 41

Figure 3 Teacher Estimations of Student Mastery of Standards....................................... 42 Figure 4 Student Mean Score Gains on TECHShare Lessons by Subject........................ 43 Figure 5 Lessons Used Met State Standards and Benchmarks ......................................... 44 Figure 6 Comparison of Student Performance on High Culturally Relevant Lessons vs.

Low-Culturally Relevant Lessons............................................................................. 45 Figure 7 Changes in Student Behavior Perceived by Teachers ........................................ 46 Figure 8 Navajo Cultural Elements Increased Student Engagement ................................ 47 Figure 9 Teachers' Responses to Item on Adding Navajo Cultural Elements .................. 48 Figure 10 Multimedia Features Seen to Contribute to Student Engagement.................... 49 Figure 11 Comparison of Smartboard® Use vs. No- Smartboard® Use in Experimental

Classrooms with Primarily Navajo Populations ....................................................... 51 Figure 12 Number of Lessons Taught in a Lab vs Classroom.......................................... 52 Figure 13 Lessons Used Stand-alone vs Integrated with Curriculum............................... 53 Figure 14 Ease of Integration of Lesson with Curriculum ............................................... 54 Figure 15 TECHShare Lessons Could be Integrated by Most Educators......................... 55 Figure 16 Reported Speed of Internet Connection ........................................................... 71 Figure 17 Reported Ability to Receive Audio .................................................................. 72 Figure 18 Summary of Responses to Item on Lesson Integration .................................... 77 Figure 19 Summary of Responses to Item on Student Mastery of Objective(s) .............. 79

Table 1 Model Classroom Study Units & Lessons........................................................... 31 Table 2 NETC Schools that Participated in the Model Classroom Study ........................ 33 Table 3 Demographics of Model Classroom Teachers..................................................... 36 Table 4 Model Classroom Student Demographics ........................................................... 38 Table 5 Model Classroom Student Computer Use ........................................................... 39 Table 6 Year 5 Lessons Produced by Dine College ......................................................... 66 Table 7 Year 5 Lessons Produced by UNM ..................................................................... 67 Table 9 Year 5 Lessons Produced by NAU ...................................................................... 67 Table 10 Year 5 Lessons Produced by ASU..................................................................... 68 Table 11 Number of Reviews by Reviewer Type............................................................. 70 Table 12 Type of Internet Browser................................................................................... 71 Table 13 Summary of Responses to Lesson Mechanics Domain ..................................... 72 Table 14 Summary of Responses to Instructional Design Domain .................................. 73 Table 15 Summary of Responses to Culturally Responsive Curriculum Domain............ 76 Table 16 Content Analysis of Lesson Reviewers' Comments .......................................... 80 Table 17 Web Site Reviewers by Type............................................................................. 89 Table 18 Results from the NETtrain Site Reviews........................................................... 90 Table 19 Number of Modules in Core Content Areas Aligned with State Standards .... 178

4

Year Five Evaluation Report:

NETC TECHShare Star Schools Project

June 2004

Executive Summary This report describes the evaluation activities conducted by CEER (the Center for

Educational Evaluation and Research) in Year 5 of the Navajo Education Technology

Consortium’s (NETC) TECHShare project, funded under the Star Schools program. The

focus of the TECHShare project is to improve student learning through the distribution

and use of standards-based multimedia lessons. The primary target population for the

project is the member schools of the Navajo Education Technology Consortium that are

located within or near the Navajo Nation. The majority of students at these schools are

Navajo, and many are Limited English Proficient (LEP), low income, or both, and most

perform well below the national average in academic achievement. The TECHShare

project aims to ameliorate these conditions by developing and disseminating standards-

based multimedia lessons, with some lessons incorporating culturally relevant and

appropriate content specifically for Navajo students.

Many of the lessons are developed from materials collected from local teachers in New

Mexico and Arizona. Development teams at four universities (Arizona State

University/ASSET, Northern Arizona University/NAU, New Mexico State

University/NMSU, and the University of New Mexico/UNM), and Dine College fine

tune the materials and add multimedia to create standards-based lessons for electronic

delivery. A sixth partner, NITI (National Indian Telecommunications Institute), has

worked on the development of additional resources for teachers to use with their students,

such as videotaped interviews with local poets, writers and artists to serve as role models.

5

Completed lessons and resources are placed in an online database where they are

available for teachers and students to download from the NETC’s website, known as

NETtrain (http://nettrain.unm.edu).

One of the major activities of TECHShare in Year 5 was the reiteration of the Model

Classroom study that was first implemented in Year 4. The Model Classroom study was

devised as a means both to test the effectiveness of TECHShare lessons with teachers and

students and to demonstrate, or model, technology integration at NETC schools. In Year

5, eighteen teachers at fifteen schools from across the consortium participated by

implementing one of five units of TECHShare lessons during the 2003-04 school year.

CEER used an experimental design combined with convergent mixed methods to

evaluate the study, including pre- and post-tests, surveys, online teacher journals, and in-

person observations and interviews. Year 5 project activities also included continued

development of the NETtrain dissemination infrastructure and multimedia standards-

based lessons. Accordingly, CEER continued to monitor lesson development and

usability of the web site.

CEER takes a collaborative approach to evaluation because we believe that collaboration

with key project stakeholders increases the likelihood that the results of the evaluation

will meet their needs and provide stakeholders – both at the project and the program level

- with meaningful data to aid in decision-making. Evaluation methods in Year 5 included

paper and electronic surveys and journals, paper-based tests, an online lesson evaluation

rubric, and site visits. Key findings from Year 5 evaluation activities include:

Pre/post-test scores: The results of an analysis of variance (ANOVA, F(1,1334) =

37.42, p < .001, R2 = .027) of pre/post test change (Change) on two levels of group

assignment (Experimental and Control) suggest that there were significant differences

between the Experimental Group (M = 18.67, SD = 23.91) and the Control Group (M

= 11.44, SD = 19.14) with the Experimental Group outperforming the Control Group

by an average of seven percent (7%) across all lessons. Due to the random

assignment of participants to conditions, there is a high degree of confidence that the

6

observed differences were not attributable to chance fluctuations.

Other significant findings from the Model Classroom study:

o The results of a one-way ANOVA suggest that students in the

Experimental group who completed TECHShare lessons that were rated as

possessing High Navajo Cultural Relevance (as measured by the Online

Lesson Rubric instrument) achieved higher score increases from pre-test

to post-test (M = 16.06, SD = 21.44) than students who completed

TECHShare lessons that were rated as Low (M = 13.31, SD = 22.36) on

Navajo Cultural Relevance measures (ANOVA f(1,1334) = 5.14, p = .023,

ή2 = .004). This accords with findings from the Year 4 Model Classroom.

o Although students in the Model Classroom study whose teacher

participated in the NETC’s professional development project (ETIP)

showed gains marginally greater than those students whose teachers were

not in ETIP, the difference was not significant as it had been in Year 4.

o Students in the Experimental group whose teachers employed a

Smartboard® during instruction demonstrated significantly higher (M =

21.07, SD = 26.89) score percentage increases than students whose

teachers did not (M = 15.47, SD = 23.33) use a Smartboard® (ANOVA

f(1,404) = 4.32, p = .040, ή2 = .01).

o In most cases, TECHShare lessons could be integrated into the curriculum

in both New Mexico and Arizona with little or no adaptation, and were

considered by teachers to aid students in mastering the standards and

benchmarks.

TECHShare lesson development: Over the five years of the project, the curriculum

development studios have met the development targets of Objective One by

completing 1516 modules. These modules make up 246 standards-based lessons that

7

are available for use by teachers and students from the NETtrain web site.

TECHShare lesson quality: The evaluation of a large sample of TECHShare lessons

over the course of Years 4 and 5 confirms that the majority of lessons contain

challenging content aligned with state standards. A total of 1165 modules were

evaluated in Year 5: Of these, 1027 fall within the GPRA categories of Language

Arts, Math and Science, and all of these modules were found to be aligned with state

standards.

Web site usability: Reviews of the NETtrain web site against an evaluative checklist

were positive overall. One area identified as in need of improvement was navigation

and manageability, with some users reporting problems with slow downloading of

pages. Given the complexity of the site and the number of resources and features that

it contains, it seems that the developers have done a very good job of creating a web

site that is generally user-friendly and utilitarian.

The Content Development Specialists continued to play a critical role in the project,

although the focus of their work in Year 5 shifted away from new lesson development

towards providing essential support to teachers – especially Model Classroom

teachers – to integrate TECHShare lessons into their instruction.

Recommendations drawn from evaluation findings include:

Continue to encourage all TECHShare Curriculum Preparation Studios to include

cultural components in their lessons since data from both Year 4 and Year 5 have

indicated that students show greater gains on tests of lessons with high Navajo

cultural context. If the lesson itself cannot be set in a Navajo context, include an

extension activity that helps students to apply the new learning in a familiar context.

Ensure that all future lessons developed give strong support to English language

learners, including audible narration of the text, multiple language versions of the

same lesson, and visuals that serve to reinforce content. This type of support is

8

especially critical in lessons developed for the elementary grade levels especially in

light of the fact that a large proportion of the NETC student population is limited

English proficient (at least 37% of the Model Classroom sample population were

LEP).

Encourage NETC schools to purchase Smartboards and to provide training and

support for teachers to use them to integrate technology into the curriculum since

using Smartboards to facilitate instruction with TECHShare lessons resulted in higher

score gains for students in the Model Classroom studies.

Work on sustainability from Day One; identify it as one of the project goals and

work on it throughout the project.

Before developing lessons or modules, conduct a needs assessment with teachers

in the target population to identify areas where lessons are most needed and

wanted.

Allocate money for marketing from the inception of the grant and begin

marketing efforts early in the project.

Design lessons around a universal template to provide the end user with a

consistent, recognizable and easy to use environment.

Require studios to collaborate on selection of grade levels and content areas to be

developed in order to provide a more cohesive and coherent body of lessons.

Also, require studios to collaborate on one lesson or lesson unit per year to

facilitate and promote greater communication and sharing of resources between

partners.

Have curriculum and cultural specialists on site at each studio.

9

Supplement in-person quarterly meetings with regular telephone conferences in

between.

Provide downloadable assessment activities that can be used to support the online

instruction. Teachers find more instructional value in lessons that contain some form

of assessment that can be used to gauge and document student learning. Be sure that

assessments align with specific state standards and benchmarks to further support

student achievement and the goals of the No Child Left Behind legislation.

The results of the more rigorous Model Classroom study conducted in Year 5 were

generally positive, and served to confirm many of the findings from Year 4. Findings

from both iterations of the Model Classroom study indicate important connections

between culturally relevant curriculum, technology and student learning for the target

population of primarily Navajo students, which could have potentially significant

ramifications for improving academic achievement. These findings should be

disseminated nationally as they are likely to be of interest to the broader field of

educators working with indigenous populations.

10

Project Description

Introduction The Navajo Education Technology Consortium (NETC) was awarded funding for a five-

year Star Schools project, known as TECHShare, in June 1999. The NETC is a

partnership of thirteen school districts and ten individual schools (as of June 2002) that

joined forces in 1997 to leverage funding for technology in their schools. Located in

New Mexico, Arizona and Utah within or near the Navajo Nation, NETC member

districts and schools serve student populations that are predominantly Navajo (roughly

85%). The NETC anticipates that the TECHShare project will help to ameliorate the

negative effects of isolation, mobility and scarce resources on Navajo students’ academic

achievement through the development and distribution of standards-based multimedia

online lessons.

Background of the Problem The Navajo Nation covers an area of approximately 25,000 square miles and crosses

three states. The area is sparsely populated and many NETC schools are in small

communities, where teacher turnover is high and qualified substitutes are scarce. Many

Navajos utilize extended families in raising their children, with the result that it is not

uncommon for students to change schools – even districts – once a year or more. These

conditions result in students being exposed to multiple standards and curricula within a

given year so that articulation of instruction is lacking and academic progress is slow.

Reports from the National Assessment of Educational Progress (NAEP) indicate that as

early as the fourth grade, Native American students score below the national level in

basic reading, math and history (Hale, 2002).

Native Americans also have one of the highest dropout rates in the country, with about

36% not finishing high school – almost twice the national average (Swisher and

Tippeconic III, 2000). And contrary to expectations, research indicates that in the case of

Navajo students, those who dropout perform no worse academically than those who stay

in school: 45% of Navajo dropouts were ‘B’ or better students (Platero et al. 1986 as

cited in Hale, 2002). Hale (2002) lists several school-related deficits that have been

11

identified in research as critical factors in dropout rates across groups, including Native

Americans; the list includes:

• Large schools

• Uncaring teachers

• Irrelevant curriculum

• Inappropriate testing

• Passive teaching methods

• Tracked classes

• Lack of parent involvement

While not all of these indicators necessarily apply to NETC schools (few would be

considered ‘large’ relative to many urban schools) two indicators are particularly worth

noting, and these are ‘irrelevant curriculum’ and ‘passive teaching methods.’ Relative to

‘irrelevant curriculum,’ the vast majority of textbooks and curricula are not written with

native students in mind. For most publishers, the costs of producing culturally relevant

materials for the relatively small native student market are perceived as being too

prohibitive. In spite of research that indicates that Native American students are more

responsive and engaged when the curriculum is culturally relevant and incorporates

opportunities for interaction (e.g., McREL, 2002), schools too often make only

superficial efforts to provide culturally relevant curricula through one-shot events such as

a Thanksgiving unit or Native American Day (Indian Nations At Risk report, 1991, cited

in Hale, 2002). Furthermore, most teachers are not provided with adequate professional

development on the home cultures of native students or their learning styles, outside of

one general course on multiculturalism or an occasional inservice. Of the 557 certified

teachers who teach in Navajo Nation area schools, more than half are non-Navajo

(Navajo Nation School Survey, 1999).

While teacher preparation programs have improved in their provision of training in active

student-centered instructional strategies, many teachers tend to fall back on the passive

teaching methods and strategies they experienced earlier in their own schooling,

especially when standardized testing is the focus. In addition, few teacher preparation

programs adequately prepare teachers for teaching in classrooms that include large

12

numbers of minority and English language learners (ELL), in spite of the fact that

enrollments of both minority and ELL students continue to rise rapidly nationwide.

Traditional transmission methods of teaching based on teachers lecturing and students

sitting passively and memorizing information are in marked contrast to the experiential

learning that Native American students are accustomed to at home and in their

communities (Nelson-Barber & Estrin, 1995), and are not supportive of ELL students’

learning.

As Fouts (2000) states in his report on computers and education for the Gates

Foundation, one of the central components of school reform – as evidenced in the No

Child Left Behind legislation – is the goal of higher academic standards and a stronger

focus on higher order thinking, problem solving skills and real world applications. These

ends can only be accomplished in a learning environment that is substantially different

from the traditional classroom, and technology proponents are confident that new

technologies provide the means to this end.

Trends toward the use of educational technology and telecommunications in schools are

on the increase. Where technology was once viewed as an add-on to education, it is now

being more fully integrated into teaching and learning. Levels of integration tend to vary

from state to state, however, usually depending upon the level of priority given to

technology within state administrations. Also, a number of states are struggling with

attempts to merge different types of educational technology and telecommunications,

such as educational television, satellite technology, computer aided instruction, Internet

or web-based classes, teleconferencing, and interactive television or videoconferencing,

into one cohesive system. As Hezel and Associates (1999) found, “‘distance learning’ is

becoming more difficult to distinguish from ‘technology based learning,’” and

technology is becoming the “sine qua non” of education:

“More and more, technology is being characterized as a tool to assist educators in

three important ways: as a means to redress inequities, as a tool to support statewide

subject area standards, and as a way to facilitate administration and disseminate

public information” (Hezel & Associates, 1999).

13

In Sivin-Kachala’s (1998) review of 219 research studies conducted from 1990 to 1997 to

assess the impact of technology on learning and achievement, his analysis revealed some

consistent positive findings:

Students in technology-rich environments experienced positive effects on

achievement in all major subject areas.

Students in technology-rich environments showed increased achievement in

preschool through higher education for both regular and special needs children.

Students’ attitudes toward learning and their own self-concept improved consistently

when computers were used for instruction.

His analysis also revealed one important inconclusive finding as well:

The level of effectiveness of educational technology is influenced by the specific

student population, the software design, the educator’s role, and the level of student

access to technology (Schacter, Milken Exchange on Education Technology, 1999).

Many Native Americans tend to be global or holistic learners who think reflectively and

respond to visual and tactile stimuli (Reyhner, 2002). They often prefer to observe a task

from start to finish before attempting it themselves, and learn more effectively through

cooperation than competition. The TECHShare project is using technology to more

appropriately meet the learning needs of Navajo students, and to ease the problems of

isolation, inequitable resources, staff shortages and irrelevant curriculum in NETC

schools by using telecommunications to reach even the smallest, most remote school.

With its combination of culturally relevant, standards-based multimedia lessons and

telecommunications technologies, TECHShare has the potential to become just the

resource NETC teachers need to create technology-rich learning environments for the

benefit of their students anytime, anywhere.

The TECHShare project builds directly on a Technology Innovation Challenge grant

(TICG) that the NETC received in 1998. Because of the inter-dependence of the two

projects, it is important to know something of the Challenge grant. NETC’s TICG

14

project, known as ETIP (for Education Technology Improvement Plan), provides

professional development on the integration of technology into the curriculum to teachers

within the consortium. To support this professional development, the ETIP project

created the web-based infrastructure, known as NETtrain. One anticipated outcome of

the ETIP summer training is that teachers would develop ideas for how to teach core

curricula in ways that integrate technology while addressing their state’s standards.

During the first four years of ETIP, teachers developed their ideas into ‘modules,’ and

these later formed the basis for some of the multimedia lessons developed by the

TECHShare project and disseminated via NETtrain back to NETC teachers and students.

TECHShare Goal and Objectives The TECHShare project is intended to be student-focused. As stated in the project

proposal, the goal of the project is “to measurably improve achievement for up to 80,000

students through the modification, distribution and teacher use of digitized, standards-

based curriculum modules (in the core areas of English, science and math).” In order to

achieve this goal, the project has three stated objectives:

1. Project Objective One: Digitizing and fine-tuning of electronic, standards-based

curriculum modules originally generated by NETC teachers under the auspices of

the ETIP project. Modules will incorporate TESOL strategies and will be

constructed around templates that will incorporate embedded assessments and be

customizable for use beyond the NETC consortium.

2. Project Objective Two: Primary dissemination of the curriculum modules to

NETC teachers, administrators and students via web/streaming technologies, high

quality video and/or via face-to-face connectivity (e.g., NetMeeting, Collabra,

etc.) as appropriate for the technological capability of each site. Secondary

dissemination will be to teachers and classrooms in other districts throughout the

United States.

15

3. Project Objective Three: Create a network (for curriculum accumulation and

dissemination) of terrestrial, microwave and satellite connections (as applicable)

to permit asynchronous and synchronous communication throughout the network.

Six partners – four universities, one college, and one non-profit organization – work on

the development of TECHShare lessons and resources, and one partner is also

responsible for the development of the dissemination infrastructure. Activities being

conducted to meet the objectives are described in more detail below.

Objective 1 - Lesson Development In creating lessons, the Curriculum Preparation Studios established in Year 1 leverage

existing resources wherever possible. Although Objective 1 only cites the ETIP project,

the TECHShare project partners draw materials for lessons from other sources as well.

The Curriculum Preparation Studios (CPS) at the University of New Mexico and

Northern Arizona University draw upon materials from the ETIP project (described

previously in this report), in which they are both partners. The CPS at New Mexico State

University draws upon materials generated by another Technology Innovation Challenge

grant, known as RETA (Regional Education Technology Assistance project) for their

lessons, and the CPS at Arizona State University draws upon materials from the National

Teacher Training Institute (NTTI) and from local teachers. At Dine College, TECHShare

developers draw upon the work of the college’s students, many of whom are preservice

teachers learning to develop culturally relevant curriculum and learning materials such as

storybooks for Navajo children. Additional content and support is provided by the

Content Development Specialists hired by TECHShare in Year 2. The sixth partner,

NITI (National Indian Telecommunications Institute), developed some lessons in the past

but is currently responsible for developing various support resources, such as videotaped

interviews featuring Dine (Navajo) role models, a digital database of images and sound

files pertaining to the Navajo culture and local area, and resources specifically for

parents. NITI also advises other partners on cultural components.

16

The three projects that TECHShare draws from provide professional development to

teachers to help them become more proficient at integrating technology into the

curriculum. As part of their training, these teachers develop learning activities that

require the integration of computers and the Internet. Some teachers only get to the stage

of describing their activity in text, but others are able to develop their ideas further and

actually create a draft of a computer-based module themselves. Whatever the stage of

development, these modules become the “raw materials” for some of the TECHShare

lessons.

Once the raw material for lessons has been turned over to the Curriculum Preparation

Studios, a team works on developing the lessons for electronic delivery. The teams are

generally made up of some combination of curriculum specialists, web developers,

graphic artists, editors, and instructional designers. Teachers and parents who originally

created the modules are encouraged to provide specific suggestions for how the material

might be developed, including ideas for graphics, animations, extension activities and

assessments. Teachers are also asked to specify the state standards that their modules

address. The intention is to develop standards-based, student centered, interactive

learning activities in the core content areas of math, science, social studies and English.

Some lessons are developed specifically with Dine (Navajo) students in mind, and

incorporate elements of Dine culture and/or are produced in the Dine language.

Culturally Relevant Curriculum

The goal of developing culturally relevant and appropriate curriculum for Dine students

is an important facet of the TECHShare project. It is estimated that eighty-six percent

(86%) of the consortium’s student population is Navajo / Dine. As the research suggests,

the lack of culturally relevant curriculum is believed to be a factor in these students’ low

academic achievement and disinterest in school, which in turn often lead to high drop out

rates. The TECHShare project anticipates that the provision of culturally relevant

curriculum will help to alleviate these problems and make learning more meaningful for

Dine students.

17

Several steps are being taken to promote the development of such curriculum for

consortium schools. The ETIP project trains teachers from NETC member schools in

teams that include parents as well as school administrators. These teams are encouraged

to develop lesson modules that incorporate aspects of Dine culture, and may rely on the

parent members for their cultural knowledge. It is hoped that as parents gain technology

skills, they will become more involved in schools and continue to contribute their cultural

knowledge and technology skills in support of the development of curricula. NITI draws

upon ideas they get from parents to develop some of their TECHShare resources.

Completed lessons are uploaded to a database on the NETtrain1 website, which is the

dissemination infrastructure developed by the University of New Mexico. From the

website, teachers and students can search for lessons by standard or by content area and

download the lessons they need via the Internet, when they need them. The lessons may

also prove to be a valuable resource for substitute teachers and help to prevent students

from missing out on a day of instruction due to staff shortages or illness. Alternative

means of access to the lessons, such as via CD-ROM, is being provided to schools that do

not have adequate Internet access or technological infrastructure.

Objective 2 - NETtrain In addition to providing the primary means of disseminating lessons to the NETC

population, the NETtrain website provides a means of communication and collaboration

for project partners and NETC members. The website has been enhanced with the

following features:

1. Technology-focused tutorials (such as how to use a Smartboard®);

2. A message board for threaded discussions;

3. Links to training schedules and resources for ETIP participants;

4. A calendar of events;

5. Minutes from NETC and TECHShare meetings, and

6. An area for posting and sharing documents.

1 The NETtrain website was initially developed under the ETIP project but is being further enhanced with funding from TECHShare.

18

Since Year 2, the website has also contained links to evaluation instruments such as the

online lesson evaluation rubric and the NETtrain site evaluation checklist, so that project

participants can participate in evaluation activities at their convenience.

Objective 3 – Network Infrastructure

In the first two years of the project, satellite dishes were purchased and installed at a few

member schools to provide them with wireless Internet access. Other connectivity issues

have been addressed through E-rate applications and all NETC member schools are now

connected to the Internet.

Together, TECHShare’s components of standards-based multimedia lesson development

and web-based lesson dissemination are intended to support NETC teachers and students

in their development and learning, for the ultimate benefit of the local communities in

which they live.

19

Evaluation Activities and Results

Introduction This section of the report presents the Year 5 evaluation plan, a description of the

evaluation activities conducted to address Year 5 project activities and the findings

derived from the data collected. The presentation follows the general guidelines provided

by the funding agency, the US Department of Education. First, the key evaluation

questions for Year 5 are presented, followed by a description of the evaluation plan

designed to address those questions. Next, a table is used to illustrate specifically how

individual evaluation activities addressed project objectives. Next, evaluation findings

are presented along with descriptions of the methods and instruments used. Incorporated

within the results are data that address the two GPRA indicators central to the Star

Schools program [the GPRA tables can be found in Appendix G]. The last part of the

evaluation results section provides a summary and reiteration of the major findings, a

discussion of the findings, and recommendations and suggestions for the future drawn

from the findings. The report closes with a brief discussion of the evaluation lessons

learned and a section on products recommended for wide dissemination.

Key Evaluation Questions Based on the goal and objectives of the project and in accordance with the final year of

the project, CEER designed the evaluation to answer the following key questions:

• Who participated in the Model Classroom Study?

• Did students who participated in the Model Classroom study

experience learning gains?

o Did students who used TECHShare lessons (i.e. the

experimental group) experience significantly different

learning gains than students who used text-based versions

(i.e. the control group) of the same lessons?

• What variables might have contributed to/detracted from effective use

of TECHShare lessons?

• How many lessons has the project produced?

20

• What is the quality of these lessons?

• What are the lessons learned from the project?

History of the Project Year 1 of the TECHShare project was devoted to start-up activities, and the enhancement

and creation of the first online lessons. During Year 2, a pilot test was conducted in

schools to check the usability and grade appropriateness of a sample of these lessons, and

to test the dissemination system. Year 3 project activities primarily involved the

dissemination of information about the NETtrain website and TECHShare lessons to

NETC schools, and the continued development and refinement of lessons. The focus of

Year 4 project activities was the Model Classroom study, which was used as a means to

field-test units of TECHShare lessons at the same time as modeling the integration of

technology in NETC classrooms. Pre- and post-tests designed specifically for the lessons

provided an initial measure of the lessons’ impact on students, but did not compare the

impact of TECHShare lessons with traditional instruction. In Year 5, CEER added

comparison groups and random assignment to the evaluation design for the Model

Classroom study in order to be able to provide stronger evidence of causal links between

TECHShare lessons and improved student achievement in the final year of the project.

Below is a description of the general purpose of the evaluation, followed by a detailed

discussion of the experimental design that was employed in Year 5.

Purpose of the Evaluation The purpose of the evaluation in Year 5 was to collect summative measures of

TECHShare project activities to determine the extent to which the project attained its

goal and objectives and impacted student learning. Year 5 evaluation findings should

assist project management and partners in determining 1) how effective TECHShare

lessons are for the target population, 2) whether some lessons are more effective than

others, and 3) what solid evidence of the project’s success can be documented for future

marketing and funding efforts. CEER continued to use the CIPP (context, input, process,

and product) evaluation model initiated in Year 1 as a framework for the evaluation,

while utilizing methods most appropriate to the implementation of evaluation research.

21

Below is a description of evaluation activities carried out in Year 5; the budget for the

evaluation may be found in Appendix H.

Model Classroom Study CEER anticipated that approximately the same number of teacher volunteers (16-20)

would participate in the Model Classroom study in Year 5 as in Year 4. Four to five

teachers from each of the ETIP/TECHShare training centers (located in Gallup, Shiprock,

Window Rock, and Kayenta) were invited to participate in order to obtain a small but

representative sample of NETC teachers.

Teachers who agreed to participate in the Model Classroom study were asked to commit

to the following:

1. To integrate one unit of selected TECHShare lessons into their curriculum,

following the experimental design for implementation (described below). The

five units will be in the areas of elementary math, elementary science, mid-school

math, mid-school language arts, and Navajo language;

2. To complete all evaluation instruments and participate in evaluation activities;

3. To allow an external evaluator to observe their classroom while a TECHShare

lesson is being used.

As the Year 4 Model Classroom Study demonstrated, participation as a Model Classroom

teacher requires teachers to alter familiar practices and to conduct activities that fall

outside the scope of their regular duties and responsibilities. For this reason, CEER

recommended that the project continue to provide additional support for Model

Classroom study teachers in Year 5 through the Content Development Specialists. The

following strategies and methods were used to evaluate the Model Classroom study in

Year 5:

(a) Student Assessments – Experimental Design The pre/post tests used in Year 4 provided useful information as to the effectiveness of

TECHShare lessons, but without control groups it was not possible to say whether the

22

effects were due to the lessons alone. For Year 5, an experimental design that included

random assignment to condition and between-groups comparison was used to help

control for confounding variables such as teacher effects, individual differences,

environmental factors and timing, and to provide stronger evidence of the causal links

between use of TECHShare multimedia lessons and student achievement.

For the Year 5 Model Classroom study, teachers who volunteered to participate were

matched in pairs according to grade level and content area. Both teachers committed to

complete the same unit of lessons with their students. Before starting, the teachers were

randomly assigned to either the control or the experimental group for the first lesson (see

Figure 1). The teacher in the experimental group would use the computer-based

multimedia version of the TECHShare lesson; the teacher in the control group would use

a text-based “regular” version of the lesson created for the experiment. Students in both

groups would take the same pre-test. The content of the two lessons would basically be

the same, but the text-based version would more closely approximate “traditional”

instruction with no multimedia and no cultural enhancements (some TECHShare lessons

are designed specifically for the primarily Dine student population).

After completing the lesson, students in both groups would take the same post-test; the

assessments used were developed specifically to measure the performance objectives of

the lessons. When it came time to do the second lesson in the unit, the paired teachers

and students would “switch” conditions, so that the experimental group for lesson 1 was

now the control group for lesson 2 and vice versa. This procedure was repeated for the

remaining lessons in the unit. After the pre- and post-assessments were completed for

each lesson, the teacher in the control condition had the option of using the multimedia

version of the lesson with his/her students in order to ensure that no potential benefits

were withheld from participating students.

23

TECHShar

TECHShar

competenci

(objective

curriculum

established

assessment

developed

Specialists.

as NAEP

(Arizona’s

assessment

achievemen

8.1.1 requir

Figure 1 Experimental Design for Year 5 Model Classroom Study
e lessons are d
e lessons are designed to be standards-based and to help students attain the

es contained within the standards. One of the objectives of the project

1.C in the proposal) states, “At least 50% of those students accessing the

modules will perform at grade-level in the core subjects, as measured by the

benchmarks of the state standards.” For the Year 5 Model Classroom study,

s based on the performance objectives of lessons selected for the study were

as a collaborative effort between CEER and the four Content Development

Assessment items were selected from published and validated sources, such

(National Assessment of Educational Progress), Spectrum and AIMS

Instrument to Measure Standards) standardized tests. The results of the

s are being used to evaluate the impact of TECHShare lessons on student

t compared to “traditional” instruction and to address performance indicator

ed by GPRA.

24

(b) Site Visits During site visits, the evaluators would observe teachers and students working with the

lessons in their unit to document and describe how the lessons, TECHShare or text-based,

were taught. While it is not possible to observe learning physically taking place, years of

research and experience have resulted in the development of specific indicators that are

accepted as evidence of engaged learning. These indicators include: time on task, taking

responsibility for own learning, motivation/excitement, working collaboratively,

performing authentic challenging tasks, completing assignments, teacher acting as coach

or facilitator, and others. CEER developed an observation protocol using these objective

criteria to be used when observing Model Classroom teachers and students. Using a

protocol also serves to standardize data collection across sites. CEER also planned to

conduct structured interviews with teachers to collect more in-depth feedback on how the

study was progressing, and with their principals to gather background information on the

school. All protocols used in the evaluation may be found in Appendix A.

(c) Online Journal All Model Classroom teachers were asked to maintain an electronic journal. In the

journal, teachers documented the conditions under which the TECHShare versions of

lessons were used (i.e., number of computers used, amount of time spent, supplemental

activities used, etc.) and other pertinent details. Teachers also reported on any changes

they experienced in their own instructional practice as a result of using the TECHShare

lessons, as well as how their students responded to the lessons. This information

provides important supplemental data, and is being used to triangulate the data collected

by means of the assessments and site visits. Data obtained by means of the journal

include:

1. Titles of TECHShare lessons used with students.

2. A description of the context (e.g., in a classroom or computer lab, number of

computers used, amount of time spent on the lessons, technical problems

encountered, etc.).

3. Observed changes in student engagement.

4. Estimated level of student mastery of the objectives.

25

5. Contribution of multimedia and cultural features of lessons to student

engagement.

(d) Additional Instruments A number of other instruments were used to collect data for the Model Classroom study,

including a student demographic survey and a teacher background survey, both of which

were completed by teachers. The student demographic survey gathered information

necessary for CEER to address Star Schools GPRA reporting requirements, and provided

additional information on student characteristics that might affect the implementation and

results of the study. On this survey, teachers reported demographic characteristics of

their school and students, such as Title 1 status, geographic location, race and ethnicity.

In addition to general information on demographics, certification and teaching

experience, the teacher survey included a one-item measure known as “Stages of

Adoption2,” which is a self-assessment instrument of a teacher’s level of adoption of

technology. This measure was repeated at the end of the study to see if teachers

perceived that they had increased their use of technology as a result of their participation

in the study.

Data Analysis The evaluation activities outlined above resulted in the collection of both

quantitative and qualitative data. Each type of data has its strengths and weaknesses;

collecting both provides a measure of balance between the precision of numbers and

the nuances and richness of description. Quantitative data are analyzed using SPSS

software and qualitative data are analyzed using content analysis to identify patterns

and/or discrepancies.

How Evaluation Activities Addressed Project Objectives The table below lists the three objectives of the TECHShare project, the sub-components

or activities designed to achieve those objectives, the degree to which the objective has

been met over the five years of the project, and the corresponding evaluation methods

and strategies that were implemented in Year 5 to assess progress and outcomes.

2 Christensen, R. (1997)

26

Project Objective 1: Digitizing and fine-tuning of electronic, standards-based curriculum

modules originally generated by NETC teachers under the auspices of the ETIP project.

Modules will incorporate TESOL strategies and will be constructed around templates

that will incorporate embedded assessments and be customizable for use beyond the

NETC consortium.

-Met. Project Objective 1 is broken down into four sub-components, 1.A-1.D, described below:

Objective Sub-component: Evaluation Activity:

Objective 1.A: To establish and maintain

four Curriculum Preparation Studios.

Curriculum Preparation Studio Surveys

(x2/year)

Objective 1.B: To produce and disseminate

at least 1000 curriculum modules for the

five years of the project.

Curriculum Preparation Studio Surveys

(x2), Lesson Development tracking, Online

Lesson Rubric

Objective 1.C: At least 50% of those

students accessing the curriculum modules

will perform at grade level in the core

subjects, as measured by the established

benchmarks of the state standards.

Model Classroom study (includes student

pre- and post-tests, online teacher

journals, online lesson evaluations,

observations, and interviews).

Objective 1.D: Provide Master Teacher

mentorships for ETIP teachers who are

producing the preliminary curriculum

modules, in order to ingrain procedures

and styles found by the Curriculum

Preparation Studios to be most needed in

the modules when they arrive at the

studios.

Content Development Specialist Survey

27

Project Objective 2: Primary dissemination of the curriculum modules to NETC

teachers, administrators and students via web/streaming technologies, high quality video

and/or via face-to-face connectivity (e.g., NetMeeting, Collabra, etc.) as appropriate for

the technological capability of each site. Secondary dissemination will be to teachers

and classrooms in other districts throughout the United States.

-Met. Project Objective 2 is also broken down into four sub-components, 2.A-2.D, described

below:

Objective Sub-component: Evaluation Activity:

Objective 2.A: Anticipate bandwidth

availability by developing a

communications infrastructure using the

Internet.

Objective 2.B: Establish a central place for

management, configuration, webcasting,

archival and retrieval of learning modules

and related learning support material.

Objective 2.C: Exploit video streaming over

the Internet as an alternative to

conventional video conferencing and

satellite communication in order to provide

on-demand learning support and activities.

Objective 2.D: Create a knowledge

database/knowledge management system

to store and archive curriculum standards-

based learning modules and other

knowledge created through the work of the

three Technology Innovation Challenge

Grant projects.

Objectives 2.A – 2.D describe the tasks

involved in creating the infrastructure that

is being used to disseminate TECHShare

lessons, known as NETtrain. CEER is

evaluating NETtrain from the perspective

of the target population, that is, NETC

teachers and students who use NETtrain

to access the lessons and other resources.

The following Year 5 evaluation activities

provide general feedback on the utility of

the NETtrain website: NETtrain Web Site

Evaluation Checklist, Online Lesson

Rubric, Online Teacher Journal, Model

Classroom teacher interviews.

28

Project Objective Three: Create a network (for curriculum accumulation and

dissemination) of terrestrial, microwave and satellite connections (as applicable) to

permit asynchronous and synchronous communication throughout the network.

-Met – all NETC member schools are connected to the Internet. No evaluation conducted for Objective 3 in Year 5.

As the table shows, one evaluation activity or instrument was often used to address more

than one project objective or subcomponent. For example, the Curriculum Preparation

Studio surveys addressed subcomponents 1.A and 1.B, and 2.B and 2.D. Therefore, for

the sake of simplicity and clarity, evaluation findings will be organized by evaluation

activity rather than by objective.

Preparation for the Model Classroom Study Identification of Model Classroom Teachers: The procedure for identifying teachers to

participate in Year 5 followed that used in Year 4: since the Content Development

Specialists (CDSs) are most familiar with the teachers and schools in their service area,

they were asked to recommend teachers for the study. Recommendations were also

sought from school principals. Both ETIP and non-ETIP participating teachers were

eligible to participate, as were teachers who had participated in the Model Classroom

study in Year 4, as this factor could be controlled for when analyzing the data. If more

than five teachers were nominated and willing to participate from each area, all names

would be put in a “hat” and five teachers drawn at random. The selection strategy

involved having one teacher from each of the four training center service areas do one of

the five units of lessons to get a representative sample of NETC teachers using each unit.

Since most evaluation activities were to be conducted in off-contract time, the NETC

board approved paying each participating teacher a stipend of five hundred dollars

($500.00).

Identification of TECHShare Lesson Units: Five units of TECHShare lessons were

identified for the Model Classroom study: elementary math, elementary science, middle

school language arts, middle school math, and Dine language stories. The Content

29

Development Specialists, using the criteria that the selected lessons should constitute a

thematic unit and/or fit the curricula being taught in the five content areas and grade

levels in both New Mexico and Arizona, compiled the units. Table 1 identifies the

lessons selected for the five units and the Curriculum Preparation Studio that was

responsible for developing each lesson. The math units contain more lessons than the

others because certain math lessons are shorter and less substantial than others.

Orientation for the Teachers: In response to lessons learned during the first

implementation of the Model Classroom study, CEER provided a one-day orientation

session for all participating teachers and the Content Development Specialists who would

be supporting them. The orientation was held on a Saturday early in October 2003 at

Dine College in Tsaile, Arizona, as this was considered a fairly central location relative to

the four training centers. Sixteen of the teachers who had been identified for the study

attended the orientation. During the orientation, teachers were introduced to their units of

lessons, provided with hands-on training on how to use a Smartboard® for instruction,

and shown how and where to access the online evaluation instruments. Each teacher was

also provided with:

• A support manual containing all of the lesson teacher guides

• Screen shots from each online lesson

• Two CD-ROMs, one containing all of the lessons used in the study with

supplemental activities and one containing all of the plug-ins required to view the

lessons

• Pre- and post-assessments for each lesson, together with Scantron answer

sheets

• Handouts for offline student activities, and

• A carton of supplies for conducting supplemental activities suggested with

some lessons.

30

Table 1 Model Classroom Study Units & Lessons

Elementary Math

Elementary Science

Middle School Language Arts

Middle School Math

Navajo Language Stories*

Fire as a Force of Life (UNM)

Longwalk Part 1 – Research (NMSU)

Rotational Symmetry (UNM)

Biib! Biib! Biib! (Geraldine)

Squares (ASU) Magician (ASU) Small-Large Squares (ASU)

Parts of a Plant (UNM)

Longwalk Part 2 – Playwriting (NMSU)

Transformation (ASU)

Kiizh Doo Zeedsisgai Baa Hane’ (Melanie)

3-D cubes (ASU) Circles to Spheres (ASU) Identify Cones (ASU)

Rainforest (NMSU) Writing in 10 Stages (NAU)

What is a Radian? What is an Angle? What is a Degree? (NAU)

Dine Dabilii’ (animals)

Rikki Rectangle (ASU) Rikki Shapes (ASU) Triangles (ASU)

Moon Phases (NAU)

The Art of Mystery (UNM)

Angle Hypatia (ASU)

Hastiin Dagha (Biography of Barboncito)

Rikki Lines (ASU) Parallel Lines (ASU) Perpendicular Lines (ASU)

Ehii Binii’ (My Face)

*** Lahii Ayoo Be'edilaah (Funny Monkey) * All stories developed by Dine College

Timeline: Experience with the first Model Classroom study indicated that it would be

prudent to start the study as early in the school year as possible so that teachers would be

less inclined to wait until after standardized testing in the spring to begin working with

their units. Thus, preparations for the study were completed in September and teachers

were ready to begin the second week of October. Although teachers were encouraged to

have everything completed by April 15, 2004, it was necessary to extend the timeline so

that teachers who had volunteered later in the year to replace teachers who had dropped

out of the study could have time to finish. All instruction and evaluation activities were

completed by the last week of May 2004.

31

Results from the Model Classroom Study: Evaluating the Impact on Students (GPRA 8.1.1) Demographics Of Participating Schools

A total of eighteen teachers at fifteen different schools in seven NETC member districts

participated in the Model Classroom study in Year 5. The consortium has 105 member

schools, so the Model Classroom study represents just over14% of the total (an increase

of 4% over Year 4). While this may seem like a small sample, the NETC population is

quite homogeneous, with approximately 86% of the student population identified as

Native American/Navajo. Therefore, although the consortium covers an area of close to

25,000 square miles, there is less diversity between member schools than might be the

case in other parts of the country. All of the schools are designated as rural, and 103 of

the 105 schools are eligible for Title 1 school-wide (the other two schools are

private/parochial and do not qualify). Background information and demographics of the

Model Classroom participants are provided below. Copies of the teacher and student

demographic surveys may be found in Appendix A.

Table 2 identifies the schools and districts that took part in the study. As Table 2

indicates, two teachers dropped out from the study late in the school year; it seems other

commitments required more time than expected and so they were unable to finish the

study as planned. In spite of this, the remaining teachers and students constituted a

geographically representative sample of the NETC population.

When making site visits, CEER was able to visit all but three of the fifteen participating

schools: seven sites were visited in February (Tohatchi Elementary and Ramah High

Schools in the Gallup training center area, and Newcomb Middle School, Newcomb High

School, Kirtland Elementary, Kirtland Middle School and Mesa Elementary in the

Shiprock training center area), two sites in March (Kayenta Middle School and Eagle’s

Nest Intermediate in the Kayenta, Arizona training center area) and three sites in April

2004 (Tohatchi High School and Gallup Junior High School in New Mexico and Tse Ho

Tso Middle School in the Window Rock, Arizona training center area). The evaluators

32

observed teachers and students working with both the TECHShare and text-based lessons

and interviewed teachers and principals.

Table 2 NETC Schools that Participated in the Model Classroom Study

Schools (and districts) in Model Classroom Study

Training Center Service Area

Elementary Math Unit:

Tohatchi Elementary (Gallup McKinley County Schools) Gallup, NM

Eagle’s Nest Intermediate (Tuba City Unified School District) Kayenta, AZ

Indian Wells Elementary (Holbrook Unified School District) Window Rock, AZ

Mesa Elementary (Central Consolidated School District) Shiprock, NM

Elementary Science Unit

Tohatchi Elementary (Gallup McKinley County Schools) Gallup, NM

Eagle’s Nest Intermediate (Tuba City Unified School District) Kayenta, AZ

[teacher dropped out April 2004] Window Rock, AZ

Kirtland Elementary (Central Consolidated School District) Shiprock, NM

Middle School Language Arts

Gallup Jr. High (Gallup McKinley County Schools) Gallup, NM

[teacher dropped out May 2004] Kayenta, AZ

Pinon Accelerated Middle School (Pinon Unified School District) Window Rock, AZ

Newcomb Middle School (Central Consolidated School District) Shiprock, NM

Middle School Math

Kayenta Middle School (Kayenta Unified School District) Kayenta, AZ

Ramah High School (Gallup McKinley County Schools) Gallup, NM

Tse Ho Tso Middle (Window Rock Unified School District) Window Rock, AZ

Kirtland Middle School (Central Consolidated School District) Shiprock, NM

Navajo Language Stories

Tse Ho Tso Middle (Window Rock Unified Schools) Window Rock, AZ

Rough Rock Elementary School Kayenta, AZ

Newcomb High (Central Consolidated School District) Shiprock, NM

Tohatchi High School (Gallup McKinley County Schools) Gallup, NM

Of the eleven principals interviewed, five were in their first year serving as principal at

the school, three had served 2-3 years, and the other two had served twelve years or more

at the same school. These numbers indicate that principal turnover is high in the area,

although some principals tend to move to another school within the area served by the

NETC. According to the principals, student populations at their schools range from 230

33

to 1600 students, with an average of 400 students. All but two of the schools have

populations that are 98% Navajo or higher; the other two have populations around 72%

Navajo. These schools have high numbers of English language learning (ELL) students;

one principal said that 65% of his students were ELL but another said that as many as

90% of her students were classified as ELL. These school-wide demographics are

generally in line with the demographics reported by the Model Classroom teachers for

their students, which are presented below. Principals reported student mobility rates

ranging from approximately 5% to 22%, but often the students are moving within the

same district, or they cycle back and forth between districts as they move from one home

to another (i.e., summer versus winter home) or as students move between living with

their parents and living with their grandparents. One principal reported that 50% of his

students live with their grandparents.

All of the schools have large service areas, with some students bussed in from as far

away as fifty miles or more. With poor or unpaved road conditions, this can mean a one-

way ride lasting one and a half hours. When asked about parent support and attendance

at parent-teacher conferences, the principals estimated that between 35% and 85% of

parents attend, with an average attendance rate of about 50%. Distance and lack of

transportation impact attendance at parent-teacher conferences and school events, and

schools have to ask parents for help directly rather than wait on volunteers because

parents are not accustomed to getting involved with school activities, although this is

slowly changing with more parent education and outreach. Parent involvement is

important for many reasons, one of them being that parent involvement has been

identified as a factor in low student achievement and it is now one of the criteria on

which school performance is judged by the state. Five of the schools visited are currently

in corrective action.

Technology Access and Use in Model Classroom Schools All schools have Internet connections in all of the classrooms, and each classroom has a

minimum of one computer, with most having three to four. In addition, all of the schools

have at least one computer lab with an average of twenty-five Internet connected

computers; some have an additional lab or a mobile “lab” on wheels. All but one of the

34

principals indicated that their school district actively supports and encourages teachers to

integrate technology into the curriculum; one principal noted that technology integration

was a part of her school’s corrective action plan. Only one principal complained that his

district “gives us the machines” but they no longer have a technology teacher or someone

to manage the computer lab because the district lost Title 1 funds. While principals said

that their districts encourage technology integration, only three of them were able to

recall any professional development days in the current school year that included

technology training: in each of these instances, the training was on how to use computer-

assisted learning software such as Accelerated Reader or district supported testing

programs. The only training mentioned that seemed to concern integrating technology

into the curriculum was training at one school on how to use a Smartboard in the

classroom.

In order to get a sense of how seriously committed schools and districts are to integrating

technology, principals were asked whether technology skills and use are part of teacher

evaluations. Six said that there was at least one item on the formal evaluations related to

technology, and two said that they personally included technology skills on their informal

evaluations of teachers. One principal said that he was not in favor of putting technology

competencies on the evaluation until the general skill level at his school improved;

presently, there is too wide a range of abilities. Principals were also asked to estimate the

percentage of their teachers who use technology for instruction on a regular basis;

estimates ranged from 1% to 100% and everything in between, suggesting that there still

exists a wide disparity in technology skills and training of inservice teachers in this area.

Seven of the principals said that they had viewed at least one TECHShare lesson,

although not necessarily one of the lessons that were being used in the Model Classroom

study. When asked what benefits they expected for their teachers and students from

using the TECHShare lessons, some of the principals indicated that simply more

experience with technology would be beneficial to both groups, but added that it was

always a challenge to find the time to work it in. Others commented that using

Smartboards with students helped to increase their level of engagement, and that the

35

visual and tactile features of the boards are particularly well suited to Navajo students’

learning styles, so anything that promoted their use would be good. Two principals cited

the cultural aspects of TECHShare lessons as the primary benefit, with one saying that he

hoped his teachers would learn more about the Navajo culture, since only three of his

thirty-one teachers are Navajo, while all of his students are Navajo. Not surprisingly, a

number also said that they would like to see student achievement improve as a result of

TECHShare, and two principals specifically stated that the lessons would be most useful

if they directly addressed their district’s power performance standards. The TECHShare

lessons will be especially attractive to local school principals if the results from Year 5

Model Classroom pre- and post-tests demonstrate that using them positively impacts

student achievement as measured by the standards, since this is such a strong focus of

school improvement.

Model Classroom Study Demographics

Table 3 presents information on the demographics and stages of adoption of Model

Classroom teachers. The teachers who participated in the study represent a range in both

level of technology adoption and teaching experience, but all are certified in the area they

teach. All eighteen teachers are represented; only their initials are used to maintain their

anonymity.

Table 3 Demographics of Model Classroom Teachers

Teacher Ethnicity Grade level Number of Years Teaching

Stage of Adoption of Technology*

JL Asian 8th 3 6

LB Navajo 9-12th 8 3

LT Navajo 3rd 25 2

BH Navajo 9-12th 13 4

CJ Navajo 4th 2 6

RN Navajo 7th 23 5

MG Caucasian 8th 19 5

SR Hispanic 5th 25 4

SH Caucasian 3rd 5 5

LD Caucasian 8th 6 5

36

RG Caucasian 6th 28 6

EM Caucasian 3rd 2 5

JP Caucasian 8th 19 6

BM African-Amer. 4-6th 10 5

AT Caucasian 4th 1.5 5

MP Caucasian 4th 18 6

DG Caucasian 7-8th 1 5

RB Navajo 7th 5 5 *Stages of Adoption: teachers were asked to rate themselves on a six-point scale, where ‘1’ is the lowest, least skilled level of adoption and ‘6’ reflects the most creative level of adoption. See Appendix A for the survey. As seen in Table 3, half (nine) of the teachers are Caucasian, six are Navajo (eight), one

is Hispanic, one African American and one Asian. Five of the teachers have less than

five years’ experience, while nine teachers have ten or more years’ experience (M = 11.8

years).

Table 3 also shows that two teachers rated them selves a ‘3’ and a ‘2’ on the six-level

“Stages of Adoption” measure, but most rated themselves at higher levels of adoption.

Elsewhere on the survey, four teachers reported that they do not have access to a

computer at home, but only one of these rated them self less than five on the scale. Thus,

although previous studies have indicated that both home use of a computer and home

access to the Internet are “very strong discriminators for high or low stages of adoption”

(Knezek & Christensen, 1999, cited in Christensen & Knezek, 2001), meaning that those

with home access are more likely to rate themselves higher on stages of adoption, this

does not seem to be a sufficient predictor in this case.

Table 4 shows the demographics of students who participated in the Model Classroom

study, as reported by their teachers.

37

Table 4 Model Classroom Student Demographics

Category Number of Students Ethnicity

Hispanic or Latino

13

Not Hispanic or Latino 663

Don’t know

Race Native American/Alaskan

616

Asian/Pacific Islander

Black or African American 2

White or Caucasian 58

Mixed racial group/more than one race

Don’t know

Students with special needs

High Poverty (Title 1 eligible) 676

Limited English Proficient (use local definition)*

253

Students w/Disabilities (each has an IEP) 21

Don’t know 402

Total Students (unduplicated count): 676 *Various definitions were provided by teachers for ‘Limited English Proficient;’ most said that English was not the language spoken at home, but others noted that this designation was determined by a test such as the PHLOTE or IPT.

A total of 676 students took part in the Model Classroom study. As Table 4 shows, the

majority of students (91%) are Native American/Navajo, with the remainder (9%)

comprised of small numbers of Caucasian (58 students 8.6%) and African American (2 or

0.3%) students. One hundred percent of the sample is Title 1 eligible. Also significant is

the fact that just over one third of the students (37%) are known to be limited English

proficient (LEP) or English Language Learners, with Navajo as the primary language

spoken in the home. Given the larger percentage of students that are classified as LEP

across the consortium (over 50%) it is likely that some of the students in the ‘don’t know’

category could also be classified as LEP. Twenty-one students (3%) were known to have

an IEP, or individualized education plan, and so are designated as ‘students with

disabilities.’ Males accounted for 47% of all students, females 53%.

38

Students were in grades 3-12 and, like their teachers, represented a range of technology

skills and familiarity. Students’ competence with technology is not necessarily positively

related to age or grade level, as some third-grade students were judged more competent

by their teacher than some fifth grade students, and this was borne out by evaluators’

observations. Table 5 shows how model classroom teachers rated their students’

computer skills and the reported frequency of computer use at school for schoolwork.

Using a five-point scale, 20% of students were rated as ‘basically novice’ by their

teacher; 64% were rated ‘fairly competent, with some assistance;’ approximately 14%

were rated ‘competent’ and just under 3% were rated ‘very competent, they teach others.’

A five-point scale was also used for rating frequency of student use of computers at

school, with ‘1’ being ‘never or rarely’ and ‘5’ being ‘daily.’ A higher frequency of

computer usage did not necessarily correlate with a higher rating of competence.

Table 5 Model Classroom Student Computer Use

Teacher Grade level Students’ Computer Skills Level

Frequency of Student Computer Use at School

JL 8th Fairly competent, with some assistance

About 2-3 times a month

LB 9-12th Competent, able to work alone mostly


LT 3rd Fairly competent, with some assistance

About 2-3 times a week

BH 9-12th Fairly competent, with some assistance


CJ 4th Fairly competent, with some assistance

Daily

RN 7th Fairly competent, with some assistance


MG 8th Competent, able to work alone mostly


SR 5th Not very competent, basically novice


SH 3rd Not very competent, basically novice


LD 8th Competent, able to work alone mostly


RG 6th Very competent, they teach others


EM 3rd Not very competent, basically novice

Daily

JP 8th Fairly competent, with some About 2-3 times a month

39

assistance BM 4-6th Not very competent,

basically novice About 2-3 times a month

AT 4th Not very competent, basically novice


MP 4th Competent, able to work alone mostly

Daily

DG 7-8th Fairly competent, with some assistance


RB 7th Fairly competent, with some assistance


Almost half (eight or 44%) of the Model Classroom teachers reported having four to five

computers with Internet access in their classroom; six teachers (33%) had access to two

Internet-connected computers in their classroom, three (17%) had just one Internet

connected computer, and one teacher had access to twenty-five because his school had a

mobile lab on a cart that he used in his classroom. It was owing to a similar variability in

classroom access to the Internet during the Year 4 Model Classroom study that it had

been decided to supply Year 5 participants with all of the lessons they would need on

CD-ROM. How the teachers ultimately chose to access the lessons is described in detail

in the Results section below.

Results Of the Pre- and Post-Assessments

In Year 4, pre/post test results from the within-subjects design of the Model Classroom

study were analyzed using a paired samples t test (t(1882) = 36.44, p < .001, R2 = .1936).

The results indicated that students who used TECHShare lessons increased their test

performance across all areas from pre-test (M = 57%) to post-test (M = 79%) by an

average of 22%. The evaluation method in Year 5 was modified to include the random

assignment of participants to one of two comparison groups in order to strengthen the

causal link between TECHShare lessons and student achievement. A total of 676

students participated in the Year 5 Model Classroom study; their test scores were entered

into an SPSS database as percentages, and an analysis of variance (ANOVA) was

computed. The results of the analysis of variance (ANOVA, F(1,1334) = 37.42, p < .001,

R2 = .027) of pre/post test change (Change) on two levels of group assignment

(Experimental and Control) suggest that there were significant differences between the

Experimental Group (M = 18.67, SD = 23.91) and the Control Group (M = 11.44, SD =

40

19.14) with the Experimental Group outperforming the Control Group by an average of

seven percent (7%) across all lessons. Due to the random assignment of participants to

conditions, there is a high degree of confidence that the observed differences were not

attributable to chance fluctuations. Figure 2 illustrates the findings.

Group Assignment

ExperimentalControl

Mea

n Pe

rcen

t Sco

re C

hang

e

20

18

16

14

12

10

19

11

Figure 2 Results of an ANOVA Showing Significant Difference in Pre- and Post-test Gains Between

Experimental and Control Groups

As a potential source of data for triangulation with pre- and post-test results, Model

Classroom teachers were asked to record in their online journal a number of details about

the circumstances and experiences surrounding the use of TECHShare lessons. The

Model Classroom teachers made a total of 73 journal entries concerning twenty-one

separate lessons. One of the journal items asked teachers to estimate the percentage of

their students who were able to show mastery of the content standard attached to a lesson

upon completion of that lesson. Figure 3 shows how the Model Classroom teachers

responded. Approximately 58% of the time, 76-100% of students demonstrated mastery

41

of the content standard, and 36% of the time, 51-75% of students demonstrated mastery.

These findings seem to confirm not only that students were able to learn from the

TECHShare lessons in the majority of cases, but also that the content was challenging.

Figure 3 Teacher Estimations of Student Mastery of Standards

Scores from pre- and post-tests of students in the Experimental group were also

disaggregated to see how students performed on TECHShare lessons in the different

subject areas. As Figure 4 illustrates, the greatest score gains can be seen in the

elementary science unit, followed by middle school math, elementary math, and middle

school language arts.

42

Subject Area

MS mathMS lang artsElem scienceElem math

Mea

n Sc

ore

Cha

nge

Perc

enta

ge

40

30

20

10

0

18

12

31

16

Figure 4 Student Mean Score Gains on TECHShare Lessons by Subject

Teachers were also asked to indicate whether the lessons they used were aligned with the

state standards and benchmarks that they purported to address. In ninety percent of the

cases, teachers indicated that the lesson they used met the standards and benchmarks (see

Figure 5); only one response indicated that the lesson did not meet the standards.

43

Figure 5 Lessons Used Met State Standards and Benchmarks

Since the TECHShare project is closely connected with the NETC’s technology

professional development Challenge grant project, known as ETIP, the evaluators looked

again in Year 5 at whether the technology integration training teachers received as

participants in the ETIP project had any effect on student performance. In contrast to

Year 4, when an analysis of variance (ANOVA) test revealed that students whose

teachers participated in ETIP did perform significantly higher on TECHShare pre- and

post-tests than those students whose teachers were not a part of ETIP, the differences

between the two groups was not significant in Year 5. However, teachers’ responses to

the Stages of Adoption instrument reported previously in Table 3 indicate that most of the

teachers consider them selves to be quite high in their level of adoption, so it could be

that there was more parity in technology skills between participating teachers in Year 5

than there was in Year 4.

The evaluators also wanted to re-examine in Year 5 the effects of culturally relevant

curriculum on students’ learning, since findings from Year 4 had indicated that students

showed significantly greater learning gains on those lessons that contain Navajo content

or role models. An analysis of variance (ANOVA) test was performed to answer the

44

question as to whether (Navajo) culturally relevant, multimedia lessons would produce

greater pre- to post-test gains than low culturally relevant curriculum within a

predominantly Navajo population. The results of a one-way ANOVA suggest that

students in the Experimental group who completed TECHShare lessons that were rated as

possessing High Navajo Cultural Relevance (as measured by the Online Lesson Rubric

instrument) achieved higher score increases from pre-test to post-test (M = 16.06, SD =

21.44) than students who completed TECHShare lessons that were rated as Low (M =

13.31, SD = 22.36) on Navajo Cultural Relevance measures (ANOVA f(1,1334) = 5.14, p

= .023, ή2 = .004). Figure 6 graphically illustrates the potential benefit gained from

culturally relevant multimedia lessons for the target population.

Cultural Relevance

HighLow

Mea

n Pe

rcen

t Sco

re C

hang

e

16.5

16.0

15.5

15.0

14.5

14.0

13.5

13.0

12.5

16.1

13.3

Figure 6 Comparison of Student Performance on High Culturally Relevant Lessons vs. Low-Culturally Relevant Lessons

45

In their online journal, teachers were asked to record 1) whether their students seemed

more engaged using the TECHShare lesson, and 2) whether the Navajo cultural or

linguistic components of certain lessons contributed to their students’ engagement. If the

lesson they were journaling on did not contain Navajo culture, they were asked to

indicate whether they thought their students would benefit more from the lesson if these

components were added. Figure 7 illustrates how they responded to the question on

student engagement generally, and Figure 8 illustrates how they responded concerning

the effects of Navajo components on engagement. Overall, the journal entries confirm

the positive effects of culturally relevant curriculum for this sample of the NETC

population.

Figure 7 Changes in Student Behavior Perceived by Teachers

46

Figure 8 Navajo Cultural Elements Increased Student Engagement

Figure 7 shows that most of the time, teachers found their students to be more engaged

when using TECHShare lessons than otherwise. On about sixteen occasions teachers felt

that their students were engaged ‘about the same’ when using a lesson, and on only four

occasions did they report their students being ‘less engaged.’ Figure 8 shows that

twenty-six journal entries concerned lessons that did not have Navajo cultural content; of

the forty-seven entries that did concern culturally relevant lessons, forty-three entries

(91%) confirm that teachers found that cultural components positively contributed to

student engagement. However, in their comments, teachers observed that many of the

lessons could have had even more of the Navajo culture in them, and some teachers

added cultural activities of their own.

Figure 9 shows how teachers responded to the journal question that asked whether

adding cultural components to lessons that didn’t have them would increase student

engagement.

47

Figure 9 Teachers' Responses to Item on Adding Navajo Cultural Elements

Almost half of the teachers agreed that adding cultural components would increase their

students’ level of engagement. One teacher commented,

“All the lessons need to have more of the Navajo language as part of the lesson.”

In addition to cultural components, teachers were asked to indicate in their journals

whether various multimedia features of the lessons were seen to contribute to student

engagement. Teachers indicated which features they thought contributed by selecting

“all that apply” from a list of common features provided. Figure 10 illustrates their

responses.

48

Figure 10 Multimedia Features Seen to Contribute to Student Engagement

Figure 10 shows that many teachers reported that ‘computer technology,’ ‘animation,’

‘photographs,’ and ‘sound’ were multimedia features that especially contributed to their

students’ engagement. Since research has shown that student engagement is a precursor

to student learning, it is reasonable to suggest that these multimedia features contributed

to students’ learning, especially in light of other research pertaining specifically to Native

American students (Nelson-Barber & Estrin, 1995) that suggests that these students

perform better when facts and ideas are presented in sensory-rich formats.

It is also interesting to note that just over 42% indicated that using a Smartboard® to

present the lesson contributed positively to student engagement. A Smartboard® is an

interactive multi-feature whiteboard connected to a computer; that enables computer

content to be projected for group presentations. One of the features of the Smartboard®

is that it gives users the ability to actively manipulate content on the whiteboard. The

Year 4 Model Classroom study had yielded some intriguing results: Students whose

teachers used a Smartboard® with math instruction tended to show greater pre-test to

post-test gains (M = 20.76, SD = 23.67) than students whose teachers did not use the

Smartboard® to facilitate instruction (M = 11.48, SD = 23.68). Since the sample in this

49

analysis was small, the evaluators wanted to explore the possible effects of Smartboard®

use on learning further in Year 5.

The evaluation plan in Year 5 called for a repetition of the Smartboard® study with

increased sample size, across several subject areas. An analysis of variance test was

performed on the percent of change from pre-test to post-test, across multiple tests,

grades and subject areas (Language Arts, Mathematics, and Science). It was found that

students in the Experimental group whose teachers employed a Smartboard®

demonstrated significantly higher (M = 21.07, SD = 26.89) score percentage increases

than students whose teachers did not (M = 15.47, SD = 23.33) use a Smartboard® during

instruction (ANOVA f(1,404) = 4.32, p = .040, ή2 = .01). Figure 11 illustrates the

results.

These findings are exciting because they appear to bear out much of what has been

reported in the literature regarding culture and learners’ preferred styles of acquiring and

using new information (e.g., Zhang and Sternberg, 20013). Native American learners, for

example, are generally thought to be more collaborative and experiential learners who

seem to perform better when facts and ideas are presented in a global and visual fashion.

Anglo-European learners in contrast tend to prefer a verbal and analytical style (e.g.,

Nelson-Barber and Estrin, 19954). The findings from Year 5 seem to confirm the

tentative findings from Year 4 that suggested that Native American learners could benefit

from the highly manipulative, visual environment that is provided by the Smartboard®,

especially when the teacher encourages collaboration and reflection, and the lesson lends

itself well to manipulations (e.g. drag and drop exercises, graphics construction). One

elementary math teacher said,

“When I do it on the Smartboard®, it’s so much easier for the kids to follow. They can see me model using the mouse and the cursor. We need them in every classroom.”

3 Zhang, L.F., & Sternberg, R.J., (2001). Thinking styles across cultures: Their relationships with student learning. In R.J. Sternberg & L.J. Zhang (Eds.), Perspectives on thinking, learning, and cognitive styles (pp.197-226). Mahwah, NJ: Laurence Erlbaum Associates 4 Nelson-Barber, S., & Estrin, E.T., (1995). Culturally responsive mathematics and science education for Native American students. San Francisco, CA: Far West Laboratory for Educational Research and Development.

50

Smartboard

YesNo

Mea

n P

erce

nt S

core

Cha

nge

22

21

20

19

18

17

16

15

14

21

15

Figure 11 Comparison of Smartboard® Use vs. No- Smartboard® Use in Experimental Classrooms with Primarily Navajo Populations

Further studies are needed – for example, to explore the effects of different types and

levels of manipulation and interactivity on student learning - but this finding suggests that

instruction may be facilitated by matching student learning styles and cultural preferences

with specific content and lesson design. Facilitating instruction in this way could have a

significant impact on students with limited English proficiency and on learners whose

cultural predisposition for learning necessitates a global, collaborative or experiential

approach. Given that many classrooms only have a small number of Internet-connected

computers (the average number among the Model Classrooms was four computers) and

that it can be difficult for teachers to schedule class time in a computer lab, the use of a

Smartboard® could be a very effective means of integrating TECHShare lessons into

instruction.

51

Model Classroom teachers were asked to document other pertinent details about how they

used TECHShare lessons with their students. Figure 12 shows how many times lessons

were taught in a lab or a classroom, and Figure 13 shows whether the lesson was used

‘stand alone’ or integrated into the rest of the curriculum.

Figure 12 Number of Lessons Taught in a Lab vs Classroom

52

Figure 13 Lessons Used Stand-alone vs Integrated with Curriculum

As can be seen in Figures 12 and 13, most lessons were taught in the classroom and

were integrated into the curriculum rather than used stand-alone. Teachers were also

asked to indicate how easily or smoothly the lessons integrated with their curriculum, as

this is important for the promotion of TECHShare lessons as ready-to-use, standards-

aligned resources. Figure 14 shows how easy or difficult teachers perceived it was to

integrate the lessons.

53

Figure 14 Ease of Integration of Lesson with Curriculum

Teachers indicated that the TECHShare lessons they used integrated into their curriculum

‘very well’ or with ‘some adaptation’ in 90% of the cases, the rest of the time the lessons

needed a lot of adaptation or did not integrate well at all. As a corollary to this, the

journal also asked teachers to indicate whether they thought that the lessons they used

could be integrated by other teachers in the field. Figure 15 shows that 92% agreed or

strongly agreed (M = 4.51, SD = 0.73).

54

Figure 15 TECHShare Lessons Could be Integrated by Most Educators

However, teachers also indicated that they encountered some problems or barriers when

using the lessons almost half (47%) of the time. Fifty-two comments were provided:

twenty of the comments indicated that there had been problems with Internet access;

three comments reported that the lessons took a long time to load; three comments

indicated problems with not being able to get all of the students logged into the NETtrain

site at the same time, and three referred to technical problems such as hooking up the

Smartboard or loading plug-ins in the computer lab. Five comments referred to the

content of the lessons themselves, indicating that some of the lessons did not have

enough interaction for the students (e.g., Angle Hypatia) and others were only good for

introducing vocabulary, like some of the Dine language stories.

Summary As in Year 4, the results of the Model Classroom study in Year 5 were positive overall.

The results of objective standards based pre- and post-tests administered to Model

Classroom students showed that students in the experimental group had significantly

greater gains from pre- to post-test than students in the comparison group, which suggests

not only that students did learn from the TECHShare lessons they used, but also that

TECHShare lessons may be more effective in some instances than traditional instruction.

55

The design of the study, which entailed random assignment of teachers to condition

rather than students, did not wholly control for teacher effects or for other variables (such

as length of time spent on instruction) that may have also contributed to student scores.

However, the evaluators are confident that the results of the study are meaningful and

should be used to guide future project activities.

The Year 5 study did confirm findings from Year 4 that suggested that lessons set in

Navajo or localized contexts, or that use Navajo role models, result in greater learning

gains for this primarily Navajo student population. Year 5 data also appeared to confirm

that using a Smartboard® to facilitate instruction in classrooms with largely Native

American populations provides a good fit with preferred learning styles and leads to

improved student learning and academic performance. Using a Smartboard® can also

help to overcome technology access issues in classrooms with a limited number of

computers, so this approach should be considered more strongly by NETC

administrators.

In Year 4, results suggested that students benefit when their teacher participates in

technology professional development, since students whose teacher was a participant in

the NETC’s ETIP professional development project showed significantly greater learning

gains than those whose teacher was not a participant. Participation in ETIP was

examined again in Year 5, but although students whose teacher had been in ETIP scored

marginally higher than other students, the difference was not statistically significant.

Generally, Model Classroom teachers agreed that the lessons integrated well with their

curriculum and supported state standards. Some teachers felt that certain lessons were

not substantial enough and so added activities to supplement instruction. The majority of

teachers agreed that any educator in the field would be able to integrate these lessons into

their curriculum

According to personal interviews with teachers as well as entries from the online journal,

most of the teachers enjoyed using the lessons and agreed that they had something to

56

offer. One teacher noted that using the TECHShare lessons assisted her with meeting the

instructional goals set by her district, as well as building student confidence:

“Since we have to have a variety of modalities in our lesson plans, they [TECHShare lessons] provided this…They also helped with ELL and ESL [English as a Second Language] strategies. I’m still in the mid-range with technology, so this gives them a chance to help me with it which builds their confidence and increased their enthusiasm.’

When asked, all of the Model Classroom teachers said that they would use TECHShare

lessons again. One of the middle school language arts teachers observed,

“I would like to use them again next year. I think it’s relevant to their culture, and that and the Smartboard® really motivate them. It’s not as boring to them as reading history from a textbook. I’m glad that I had the opportunity to do this; I’ve integrated history, science – all across the curriculum in language arts for the students.”

And an elementary teacher who used the science unit responded with,

“Oh yes, I’ll always use them. I have four special education students – two speech and two regular – and using TECHShare lessons helps them to integrate more into the classroom at the standards level. The visuals with words [narration] is good support for them without depending on others.”

Recommendations Recommendations drawn from the Year 5 Model Classroom study are much the same as

those in Year 4 and include:

Encourage all TECHShare Curriculum Preparation Studios to include cultural

components in their lessons. Results from both the Year 4 and Year 5 studies indicate

that students show greater gains on tests of lessons with high cultural context. If the

lesson itself cannot be set in a Navajo or localized context, include an extension

activity that helps students to apply the new learning in a familiar context.

57

Ensure that all lessons provide support for English language learners, such as

narration, clear graphics , legible text, and demonstrations of the concepts, since a

large proportion of the NETC student population is limited English proficient (27% of

the Model Classroom sample population are known to be LEP and numbers for the

NETC population as a whole are larger).

Encourage NETC schools to purchase Smartboards® and to provide training and


using Smartboards® to facilitate instruction with TECHShare lessons resulted in

higher score gains for students in the Model Classroom studies.

Provide some sort of assessment activity, preferably one that can be downloaded and

printed out, that supports and reinforces the instruction in the lesson. Teachers find

more instructional value in lessons that are interactive for the students and contain

some form of assessment that can be used to gauge and document student learning.

Be sure that assessments align with specific state standards and benchmarks to further

support district goals and the goals of the No Child Left Behind legislation.

As part of the implementation evaluation of the project, each year CEER surveyed the

Curriculum Preparation Studios as a means of monitoring progress on various

components of Objective 1, which concerns curriculum development. The results of an

analysis of Year 5 surveys are presented in the next section.

Curriculum Preparation Studio Surveys CEER administered two surveys to the Curriculum Preparation Studios (CPSs): the first

(Survey A) in December and the second (Survey B) at the end of the project, in May

2004. This follows the same procedure used in Years 1-4. The purpose of the surveys

was to assist with monitoring the progress of lesson development at the CPSs, and to

identify ways in which development and distribution could be facilitated and improved at

the studio partner level. Since the studios were ultimately responsible for producing and

delivering the online lessons for teachers and students to use, CEER believed that

58

monitoring the functioning and coordination of the studios provided important evaluative

information for managing and advancing the TECHShare project.

The Year 5 surveys are similar in content and scope to the surveys administered in

previous years, with some minor revisions that reflect the project being in its final year.

Both surveys are comprised of a combination of forced-choice and open-response items:

Survey A focuses on the curriculum development process within and between studios and

internal quality control processes at individual studios, while Survey B is more focused

on task completion, attainment of objectives and lessons learned over the five years of the

project. Both surveys may be found at Appendix B.

CEER administered both surveys electronically as email attachments, as this format is

well-suited to small populations and instruments with open-ended response items that

collect primarily qualitative data. When administering the surveys, CEER sent emails to

the six studio directors requesting that they or a nominee complete the survey by the

deadline (December 18, 2003 for Survey A, and May 21, 2004 for Survey B). A

minimum of two subsequent requests was sent to those studios that had not completed the

survey by the deadline, and additional time was allotted to include more responses. All

six studios completed Survey A and five studios completed Survey B.

A summary of the highlights from both surveys is provided below, with special attention

paid to changes, improvements, or challenges, and the ‘lessons learned’ described by

each of the studios.

Lesson Development: According to information from the surveys and from the NETtrain database of lessons in

development, three of the studios have met their lesson development targets for Year 5,

while the other three anticipate that they will by the project end date of June 30, 2004. In

addition to developing new lessons in Year 5, most studios made use of evaluative

feedback collected on lessons in Year 4 to revise existing lessons and correct minor

technical problems, such as updating links. As part of their new development work, two

studios worked on creating alternative language versions of previously developed lessons

59

and/or added voice over narrations in English, Spanish and Navajo to lessons that did not

have narration. Since a majority of the target population of students is English Language

Learners (ELL), the inclusion of audio and narration in support of the text is a very

important feature that should be included in all TECHShare lessons.

All six studios indicated that they were working with one or more of the Content

Development Specialists (CDSs) in some fashion during Year 5, although the intensity

and frequency of collaboration varies between studios. Some choose to work with the

CDSs throughout the development process, and find that they have become a very useful

and necessary link between the multimedia developers at the studios, the school-based

teachers, and NETC project management. Other partners use the CDSs more in the role

of translator (primarily into Navajo) or Navajo cultural advisor, and feel that this input is

very helpful since they do not have any local consultants or internal expertise. For some

studios, collaboration with the CDSs appears to have broken down somewhat over the

course of the year. Responses to the first survey are all very positive regarding the

working relationship between studios and CDSs, as these comments reflect:

“The collaboration works well because we are clear about our roles.” “We continue to work very closely with [the CDS]. Our relationship with her has been great.”

However, feedback on collaboration with CDSs from the second survey indicates that

certain expectations were not met; two of the four studios responding at the end of the

year expressed frustration with the collaborative process, and a third studio reported that

they did not collaborate with a CDS on lesson development at all. One suggested that the

geographic distance between the CDSs and studios was partially to blame, and that

emails and telephone calls could not take the place of face-to-face meetings or of having

a curriculum consultant on site. The breakdown in communication meant that some

problems with lessons did not get addressed in a timely manner so that too much revision

was required late in the development process. The other studio indicated that they had

problems getting needed support on cultural issues; neither the CDS nor the NETC

60

executive director had responded to telephone calls and emails requesting assistance with

locating resources, although this help had been offered earlier in the year. The third

studio did not report any problems. For the most part, the CDSs have played a very

effective and essential role over the course of the project: their individual classroom

experience and collective liaison efforts with NETC teachers in the field provided an

important link between the studios developing TECHShare lessons and the classrooms

for which they were intended.

One of the studios reported that their role had expanded greatly beyond lesson

development during Year 5, and provided a list of the additional products and services

they provided. For the Model Classroom study, they had created two CD-ROMs, one

that contained the thirty lessons used for the study and one that provided the plug-ins

necessary for viewing and interacting with the lessons. For marketing purposes, they had

collaborated on or been primarily responsible for the creation of roughly twelve products,

including an electronic newsletter, brochures, posters, bookmarks, business cards,

bumper stickers and a promotional CD-ROM. In addition, they had provided staff and

materials for a booth that was set up for the National Indian Education Association

meeting in Washington DC. In describing their efforts, they noted that they worked very

closely with one of the other studios and that they felt this collaboration was an

unmitigated success:

“The coordination of the marketing materials brought [our studio] and [the other studio] working together. We had three short weeks to produce numerous marketing materials and coordinate content. It was an amazing experience to work alongside them and see these materials produced and used.”

In spite of some challenges and additions to the planned workload for some, the studios

were able to meet their production targets for Year 5. Over the course of the five years of

the project, they have produced a total of 1516 modules (246 lessons) that are posted on

the NETtrain web site and ready for use by teachers and students.

61

Communication and Collaboration: Levels of satisfaction with communication and collaboration varied between studios. In

general, the two studios that were also involved in the NETC’s Challenge grant (ETIP)

collaborated with one another more than with the others and expressed greater

satisfaction with communication and collaboration between studios. When asked on

Survey A to describe “successes or challenges related to project organization,” two of the

studios not a part of ETIP commented that this factor impacted their participation in

TECHShare. One studio wrote,

“Communication continues to be the biggest challenge for our studio. Since [we are] not as directly involved in the ETIP/TECHShare partnership our relationship to the group is different.”

-and another studio noted,

“It would have been helpful if we had been integrated with the ETIP project from the first.”

One of these studios is on the Navajo reservation and immersed in the culture; however,

the other studio noted that other factors contributing to their “disconnect” with the project

were their studio’s distance from the reservation and lack of on-site familiarity and

expertise with the Dine culture. To help alleviate this, they noted in Survey A that they

planned to work more closely with the Content Development Specialist who had been

assigned to work with them, but responses to Survey B at the end of the project year

indicate that they did not receive the support they had hoped for and requested.

Other responses to Survey B indicate additional areas where communication appears to

have deteriorated. Two studios mentioned the four-level lesson approval process that had

been instituted to ensure that necessary revisions and additions to lessons under

development were identified early in the process to avoid major and more costly changes

at the end of production. Below are two comments regarding the process:

62

“Developing the TECHShare lessons through the level 4 process is a challenge. It is very cumbersome and time consuming to process your own studio’s work this way, never mind having time to monitor other studio’s work as well. We have not felt very successful using the 4 level process.”

“Another challenge has been to get approval during Level 2 production. We still tend to get too many changes at the end of production.”

Studios suggested that this problem could be ameliorated if additional project resources

were allocated to the review of lessons early in the process. It seems that both lesson

content development and lesson approval would be facilitated by each studio having a

reliable and knowledgeable curriculum development person on-site, rather than being

dependent upon distant project staff with multiple responsibilities.

Project Leadership/Project Organization: Both Survey A and Survey B asked studio partners, “What successes or challenges

related to project organization has your studio experienced meeting project goals?” The

successes cited in response to Survey A were “growth in communication with our

Content Development Specialists and in understanding of cultural issues,” and a recent

visit from program officers from the funding agency that garnered positive feedback on

the project. In regard to challenges, three of the studios cited the challenge of getting the

finished TECHShare lessons used in the classroom. Two suggested that the problem was

partially due to lack of marketing undertaken by the project, and that this should have

been a focus much earlier in the lifespan of the project. The third studio perceived a lack

of commitment to the project on the part of the school superintendents who comprise

NETC administration to be the problem.

Responses to Survey B suggest that these challenges still exist. Three of the five studios

that responded to this survey pointed to the frequent changes in both the project’s

Executive Director and the membership of the NETC board as possible contributing

factors. Two studios noted the complexity of the organization of the project, with both

diverse partners and decision-making agencies which made “inter-institutional”

63

collaboration difficult to maintain. One studio, in expressing their frustration with the

frequent changes in key personnel, opined that these changes had been responsible for the

consortium not developing “viable long term plans.” They suggested that the six partners

needed representation on the NETC board in order to provide them with a voice in

determining policies that affected all partners, and to provide some stability in

management. Another partner expressed the opinion that the frequent changes in

leadership had brought concomitant changes in interpretations of the goal of the project,

“which crippled the project’s ability to do good with teachers and parents.” Successes

cited at the partner level were the development and distribution of high quality marketing

materials. Other successes cited were at the individual studio level, and included things

such as the successful attainment of production targets, integration of archival and

primary sources into lessons, and the development of a streamlined method of producing

standards-based and interactive instructional materials for K-12 students.

Summary In general, feedback from the six Curriculum Preparation Studios over the course of

the five years of the project has provided much-needed insight into the internal

workings of a very complex endeavor. Information on how individual studios

managed lesson development and implemented the project has been collected and

shared in an attempt to streamline processes where possible and disseminate

information on practical and effective strategies. Although at the beginning of the

project, studios at times struggled to maintain their independence and their own

ways of managing and implementing processes, by the end of the project there is

recognition of the need for, and benefits inherent within, more standardization and

collaboration across partners. In Survey B, studios were asked to reflect on their five

years’ experience with project implementation and management at their

organizations to provide valuable “lessons learned” that might benefit others

undertaking projects of similar scope and complexity. Below is a summary of what

the studios shared.

64

Lessons Learned/Recommendations: Work on sustainability from Day One; identify it as one of the project goals and




wanted.




consistent, recognizable and easy to use environment. [Note: project

management tried to establish this early in the project but was resisted by the

studios repeatedly, because they wanted more ‘artistic license’ when developing

their lessons.]





partners.



between.

CEER also monitored lesson/module development progress on the project web site,

where the partners would upload the lessons that were currently under development or

which had been completed. This information is summarized in the next section.

Development of TECHShare Lessons (Modules): Five partners continued to produce lessons in Year 5. The sixth partner, NITI, was

directed to focus on developing supplemental resources for the project, such as Navajo

role model videos and resources for parents. The resources developed by NITI in Year 5

comprised seventy-three modules. Two of the studios, Arizona State University and New

Mexico State University, increased their production targets in Year 5 from fifty modules

65

to seventy; the University of New Mexico planned to complete seventy modules;

Northern Arizona University contracted to produce a minimum of seventy but would

attempt ninety-five, and Dine College contracted to complete twenty modules.

TECHShare lessons are designed to align with the state standards of New Mexico and

Arizona. The studios design lessons starting with their own state’s standards (some

studios are located in New Mexico, some in Arizona) as the foundation, and then match

their lessons with the other state standards at the time the lesson is uploaded to the online

database. The lesson archive on NETtrain has been developed so that multiple state

standards can be attached to the metadata of each lesson; thus, a lesson originally

developed to align with specific Arizona standards can also be tagged with New Mexico

standards. When a teacher accesses a lesson on the site, she or he can see which

standards the lesson addresses in terms of both New Mexico and Arizona. Some lessons

also have the Navajo Nation’s Dine Cultural standards attached, and project programmers

are working to add the state standards of Utah. External reviews of TECHShare lessons

(see the next section in this report on the online lesson evaluation rubric) confirm that the

lessons are aligned with state standards, and findings from the Model Classroom study

reported earlier in this report indicate that the lessons assist students in mastering the

objectives.

TECHShare lesson development is documented and monitored on the NETtrain website,

where a matrix shows the developmental meta-data for each lesson and the number of

modules created in order to complete each lesson. Tables 6-10 show the number of

standards-based modules produced by the five Curriculum Preparation Studios in Year 5.

Table 6 Year 5 Lessons Produced by Dine College

Lesson Subject Grade Number of Modules Various Navajo Language stories

various N/A 15

TOTAL: 15 (of 20)

66

Table 7 Year 5 Lessons Produced by UNM

Lesson Subject Grade Number of Modules

Clockwise Math K-4 5

Ko’ Fire Science 5-8 31

Phrase O’Clock Math K-4 5

Telling Time to the Hour Math K-4 14

Beauty Way: Corn Grinding Culture & Science

unknown 5

Dine Clanship unknown 5

TOTAL: 72

Table 8 Year 5 Lessons Produced by NAU

Lesson Subject Grade Number of Modules Cartesian Coordinate System Science unknown 6

Converting Angles Math unknown 6

Family Language 1-3 19

Maebaah Language 1-3 5

Maebaah Navajo unknown 6

Navajo Fry Bread Science Kindergarten 8

States of Matter Science unknown 9

Tea unknown unknown 6

Webquest: Five Regions unknown unknown 1

Webquest: My Trip to Arizona unknown unknown 5

Webquest: Navajo Code Talkers unknown unknown 1

Webquest: Southwest Tribes unknown unknown 1

Webquest: Three Billy Goats Gruff

unknown unknown 1

Webquest: Whales unknown unknown 1

Webquest: Navajo Leaders unknown unknown 1

Webquest: Butterflies unknown unknown 1

What is a Percent? unknown unknown 9

What is Volume? unknown unknown 5

TOTAL: 91

67

Table 9 Year 5 Lessons Produced by ASU

Lesson Subject Grade Number of Modules Cones, Cylinders, Spheres/Spanish voice

Math 9-12 8

Cones, Cylinders, Spheres/English voice

Math 9-12 8

Cones, Cylinders, Spheres/Navajo voice

Math 9-12 8

Identify Cones/English voice Math 1-3 8

Lines/Congruency with Rikki/English voice

Math 1-3 8

Rectangles with Rikki/English voice

Math 1-3 8

Rectangles with Rikki/Navajo voice

Math 1-3 8

Shapes with Rikki/English voice

Math 1-3 8

Squares with English voice Math 1-3 8

TOTAL: 72

As Table 6 indicates, the lessons produced by the studio at Dine College are different

from the other studios’. Dine College produces all of its modules in the form of Navajo

language stories, with the stories incorporating different content areas such as language

arts, health or science. Since these lessons are in effect “foreign language” lessons

designed for Navajo language learners or bilingual students, they are not given the same

grade-level attribution as the other lessons. Instead, they are rated in terms of the

different levels of language proficiency they address, for example: beginner,

intermediate, or advanced.

According to the available data displayed in Tables 6-10, the five Curriculum Preparation

Studios have completed a total of 320 modules to date for Year 5 (more are anticipated

by June 30, 2004). These modules make up 52 new lessons since Year 4, for a total of

247 standards-based multimedia lessons available on the NETtrain website. This means

68

that the five year total of modules developed by TECHShare is 1516, confirming that the

project has fully met Objective One, which was to produce 1000 modules.

A sample of 187 lessons comprising 1165 modules was reviewed in Year 5 and evaluated

for usability, quality, cultural relevance and alignment with state standards. The results

of these reviews are presented in the section below, and the tables showing detailed

GPRA indicator data can be found at Appendix G.

Online Lesson Evaluation Rubric (GPRA 8.1.2) TECHShare lessons were evaluated using an online lesson evaluation rubric. This rubric

was initially pilot tested in Year 3, and was used consistently throughout Year 4 so that

an exploratory factor analysis and examination of the internal consistency or reliability of

the instrument could be conducted. The exploratory factor analysis revealed that the

rubric does address the three major variables it is intended to measure, i.e., Instructional

Design, Lesson Mechanics, and Culturally Responsive Curriculum. The second type of

analysis performed on the rubric sought to establish an estimate of internal consistency or

reliability of the instrument (see Appendix F for the complete Reliability table).

Cronbach’s alpha was calculated for the three variables; all three were high, with the

lowest being Lesson Mechanics at α = .77 (7 Items, M = 19.35, SD = 2.26), followed by

Culturally Appropriate Curriculum at α = .91 (4 Items, M = 8.65, SD = 3.20), and

Instructional Design at α = .94 (14 Items, M = 37.70, SD = 39.95). These data support

the assertion that the Online Lesson Evaluation Rubric can be said to possess high

reliability5.

Method Two main groups evaluated TECHShare lessons in Year 5: Teachers participating in the

Model Classroom study, and teachers from schools in the project area who were taking

classes with the executive director (see Table 11). A minimal number of reviews (four)

5 One note of caution is needed, however, in that although the reliability study was based on 182 reviews (cases) or uses of the lesson evaluation rubric, some lessons had a low ‘n’ with only one or two reviews so that a comparison of responses to each item for every lesson was not possible.

69

were carried out by project staff. Model Classroom teachers reviewed the lessons in their

unit that they used with their students, and the other reviewers selected lessons at random

to review. A total of 630 reviews were conducted.

Table 10 Number of Reviews by Reviewer Type

Reviewer Type Number of Reviews

Model Classroom Teacher 96

NETC area teachers 530

Project personnel 4

Total 630

The rubric was developed in an online format so that lesson users and reviewers could

link directly to the rubric from the NETtrain site. CEER constructed the rubric

evaluation site so that the rubric, the TECHShare lesson, and the state standards attached

to that lesson are all visible at once. In this way, reviewers can work through the lesson

on the right side of the page and revisit the standards at the top of the page at the same

time as they complete the rubric on the left, thereby making it possible to refer to specific

lesson sections or features when responding to the rubric rather than rely on memory. A

paper version of the online rubric is included at Appendix D. A link to the rubric can be

found on the landing page of each TECHShare lesson on the NETtrain site

(http://nettrain1.unm.edu), underneath the Resources section. A summary and discussion

of the results of the evaluations is presented below.

Results At the beginning of the rubric, several items ask reviewers to provide information about

their Internet access, such as the type of browser they use, the speed of their connection

(if known) and whether or not they have speakers or headphones to receive audio. This

information is important as the quality of users’ Internet access directly affects the quality

of the experience of TECHShare lessons. Table 12 and Figures 16 and 17 below show

the responses to these items.

70

http://nettrain1.unm/

Table 11 Type of Internet Browser

Response Count Percent Microsoft Internet Explorer 561 89% Netscape Navigator 61 9.7% Other Browser 8 1.3%

Total: 630 100%

Figure 16 Reported Speed of Internet Connection

The vast majority of reviewers (89%) used Microsoft’s Internet Explorer browser, and

while most reviewers (73%) did not know the speed of their Internet connection, at least

16% had fast connections via T1/T3 lines, cable or DSL. Figure 17 shows that the

majority of reviewers (87.3%) were able to hear audio, and so could respond to rubric

items concerning narration and sound.

71

Figure 17 Reported Ability to Receive Audio

Items on the rubric concerning the quality of the lessons ask reviewers to evaluate three

domains of interest: Lesson Mechanics, Lesson Instructional Design (including alignment

with state standards), and Culturally Responsive Curriculum. Reviewers used a five-

point scale, ranging from 1 = Not evident to 5 = Clearly Evident. There were seven items

that address Lesson Mechanics; a summary of responses to these items is provided in

Table 13.

Table 12 Summary of Responses to Lesson Mechanics Domain

Rubric Item:

Mean

Standard Deviation

Lesson loads within an acceptable time period 4.28 1.15

Graphics and text are appropriate and clear 4.61 .86

Spelling is correct throughout the lesson 4.64 .82

Grammar throughout the lesson is appropriate and correct

4.67 .79

Lesson navigation is understandable, consistent and age appropriate

4.51 .94

All lesson links are identifiable and intact 4.34 1.18

Lesson uses clear and easy to follow directions and instructions

4.46 1.03

Response scale ranged from 1 = Not evident to 5 = Clearly evident.

72

As Table 13 illustrates, the three sub-components of Lesson Mechanics that received the

highest ratings were spelling, grammar and graphics. The two components that received

the lowest ratings (although still relatively high) concerned the time the lessons took to

load and the functionality of the links within the lessons. The majority of lessons that

were identified as taking a long time to load, or which reviewers could not get to open,

were the Dine language stories that were created in Authorware and need a Macromedia

plug-in to view. Because of these problems, which were also identified in previous

years’ evaluations, the new Dine language stories are being developed using Flash and

Quicktime software, which should make them quicker to download and more user-

friendly. As a short-term solution, CD-ROMs containing all of the existing lessons on

the NETtrain site were distributed to each NETC school in Year 5, but this practice

would be cumbersome to manage and maintain in the long term. Depending on CD-

ROMs for access to the lessons means that every teacher needs to be made aware that

their school has the CDs and what they need to do to be able to use them. Judging by the

Model Classroom teachers, even when CD-ROMs are available most teachers prefer the

ready access of the lessons online rather than having to coordinate loading the CDs on

every computer that is going to be used, or keeping track of the whereabouts of the disks.

Fortunately, download time and plug-ins were not a significant problem for the majority

of TECHShare lessons and Lesson Mechanics were rated highly overall.

The second domain addressed by the rubric is Instructional Design. Fourteen items were

used to address this area; Table 14 provides a summary of reviewers’ ratings.

Table 13 Summary of Responses to Instructional Design Domain

Rubric Item:

Mean

Standard Deviation

Lesson standards and benchmarks are clear 4.58 .99

Specific lesson objectives are in evidence 4.56 1.01

Lesson objectives are clearly related to stated standards and benchmarks

4.56 1.04

Lesson contains instructions to the learner that clearly state objectives

4.47 1.09

73

Lesson includes information on prerequisite knowledge or skills 4.32 1.22

Lesson key points are supported with appropriate media 4.52 1.05

Engages learners through the use of new materials, or reinforces concepts previously taught

4.43 1.11

Demonstrates desired skills or process for learners to master 4.49 1.04

Lesson uses various activities to engage different types of learners

4.39 1.15

Lesson tasks are related to stated objectives 4.55 1.00

Feedback is consistently provided to learners during instructional tasks

4.24 1.28

Learners are provided with formative assessments as they are engaged in learning tasks

4.10. 1.41

Learners are provided with a summative assessment to determine learning outcomes

4.08 1.44

Assessment is authentic and interwoven within lesson 4.15 1.41 Response scale ranged from 1 = Not evident to 5 = Clearly evident.

As Table 14 shows, reviewers gave the five Instructional Design components related to

objectives, standards and benchmarks (items 1-4 and 10) the highest ratings. Most

importantly in terms of GPRA indicator 8.1.2, which is concerned with challenging

content aligned with standards, reviewers indicated that overall the lessons reviewed are

aligned with the state standards they purport to address. The number of modules in the

core content areas found to be aligned with state standards are as follows: math 481

modules, language arts 162 modules, and science 384 modules. For more comprehensive

details on GPRA 8.1.2 see Appendix G).

As in previous years’ evaluations of lessons, the instructional design components that

received lowest ratings are those related to feedback and assessments. These low ratings

may well reflect some of the challenges TECHShare developers have faced over the

years trying to replicate authentic assessment activities in an open online environment.

Currently, all students log into the web site using one universal user name and password.

This means that it is not possible to track individual responses or manage student

performance “intelligently,” so that truly authentic and responsive student-centered

feedback is impossible. In addition, some of the lessons, such as the Dine language

74

stories, are not interactive and do not provide any assessments embedded within the

stories for students and teachers to gauge how well they are learning from the lesson.

Since sixty-one (33%) of the 187 lessons reviewed were Dine language stories, this

would be likely to lead to lower mean ratings on items concerning feedback and

assessment. A sample of the comments that reviewers provided on the instructional

design aspects of the lessons is provided below:

“There were no pre and post test or any kind of assessments with this lesson so I gave a 1 on the questions about the assessment.”

“This is a very good story for introducing one's self, but it doesn't have a lesson plan to go with it. If it had one it would be great book to read and use in the classroom.”

“The various activities students are instructed to do are helpful and fun.”

“This lesson is a good lesson, it just needs the pre and post testing attached to it and it would make a great lesson.”

“Overall I really like this lesson. There are a few areas that I didn't like. There isn't a whole lot of assessment-just the occasional question. There is immediate feedback, however, which is nice. There isn't any explanation given for wrong answers though. Also on the parts where you have to drag and drop I couldn't figure out how to make the lines go on a diagonal. There wasn't a clear set of instructions or feedback.

A number of comments are positive about the lessons generally, but suggest that the

lessons would be made even better with the inclusion of more assessment activities either

within the lessons themselves or to be used as pre- and post assessments. Without these

features, the lessons are seen more as sources of information rather than as instructional

activities.

Table 15 provides a summary of the ratings that reviewers gave to lessons in the domain

of Culturally Responsive Curriculum.

75

Table 14 Summary of Responses to Culturally Responsive Curriculum Domain

Rubric Item:

Mean

Standard Deviation

Presents content with a base in the local knowledge system, including cultural beliefs, values and practices, and the ecology and geography of the Dine' (Navajo) environment

3.89

1.58

Incorporates elements of contemporary life in Dine' communities, along with historical and traditional aspects of Dine' culture

3.87

1.58

Lesson incorporates community knowledge base into curriculum

4.01 1.49

Provides good support for learners with limited English proficiency by including sound, read aloud text, color images, or other devices

4.20

1.34

Response scale ranged from 1 = Not evident to 5 = Clearly evident.

Here, it is important to note that not all TECHShare lessons are designed to have an

explicit Navajo cultural focus. With regard to lesson evaluations, the evaluators defined

a lesson as being culturally relevant if it had explicit Navajo content (for example, was in

the Dine language or dealt with traditional ways of living) or was set in a culturally

relevant context, such as in well-known locations on the reservation. In Year 5, reviews

of Navajo-focused lessons accounted for 56% of all reviews (357 out of 630). This helps

to explain the lower mean ratings and the larger standard deviations that resulted for the

four items that make up the Culturally Relevant Curriculum domain. Whether or not a

lesson is seen to be culturally appropriate and relevant is important because findings from

the Model Classroom study in Year 4 indicated that NETC students, who are primarily

Navajo, benefit from learning from lessons with these characteristics.

The fourth item in the Culturally Relevant Curriculum domain asked reviewers to rate

lessons specifically on how well they support English language learners through the

incorporation of TESOL strategies such as read aloud text, sound effects, and color

coding. As Table 15 illustrates, these supports were generally considered evident,

although there was considerable variation in responses (M = 4.20, SD = 1.34). A look at

76

individual responses to this item reveals that approximately 78% of reviewers rated this a

‘4’ or ‘5’ on a five-point scale, suggesting that the majority of lessons reviewed provided

good support for ELL students.

The rubric concludes with three items under the heading ‘Rating Summary.’ The first of

these three items asks reviewers to indicate their level of agreement with the statement,

“Overall I believe that any educator teaching within this field could integrate this lesson into their curriculum.”

-Figure 18 shows how reviewers responded on a five-point scale ranging from ‘strongly

disagree = 1’ to ‘strongly agree = 5,’ with a ‘neutral = 3’ response option.

Figure 18 Summary of Responses to Item on Lesson Integration

As Figure 18 shows, roughly 90% of reviewers agreed or strongly agreed (M = 4.50, SD

= 0.77) with the statement, indicating that the majority of the reviewers did not think it

would be difficult for teachers to integrate the lessons reviewed into the curriculum. This

suggests that the reviewed lessons do align generally with the curricula being taught in

the target area, which is very important. Reviewers also had an opportunity to comment

77

on their rating, and most comments were very positive, for example:

“This lesson is clear and simple to integrate into any curriculum.”

“I think it’s a good short story for children to enjoy, especially for those who still herd sheep at their grandmother’s.” “Wonderful lesson and great speakers in English and Navajo. Both languages were clearly spoken.”

Examples of comments that suggested some room for improvement or revision include:

“You would have to supplement this unit with more overview.” “This lesson needs much more practice and examples than I saw in the lesson.” “Nice work, but add more illustration of a Navajo eating the food.”

The second to last item asks reviewers to indicate their level of agreement with the

statement,

“Overall, I believe that this lesson would significantly aid the student in mastering the stated objective(s).”

-Figure 19 shows the summary of responses.

78

Figure 19 Summary of Responses to Item on Student Mastery of Objective(s)

Using the same scale as the previous item, Figure 19 shows that 90% of reviewers agreed

or strongly agreed (M = 4.49, SD = .78) that the lesson they reviewed would assist

students with mastery of the objectives. A sample of representative comments is

provided below:

“I liked the text that was highlighted as the lesson was read; this helps those that are learning to read Navajo.” “The students could use this lesson to review or to begin their study of angle relationships.” “It will meet the foreign language standards if you could just add extended activities, a teacher’s guide and an assessment.” “Using the “think about” and key definitions will be very helpful in mastering the objectives.”

The last item on the rubric was open ended and simply asked reviewers for their

‘summary lesson comments;’ reviewers provided 238 comments. The majority of

comments are positive and congratulate the developers on creating interesting and

engaging lessons. Many comments reveal how appreciative teachers in the area are to

79

have resources that are culturally relevant for their Navajo students. Other comments

note some weaknesses with lessons and provide suggestions for how they might be

improved. Table 16 shows some common themes or issues that emerged from a content

analysis of the comments; all of the comments are provided in Appendix D.

Table 15 Content Analysis of Lesson Reviewers' Comments

Topic of Comment # of Comments Good learning activity 127 Generally a good activity with some qualifications and/or recommendations for improvements

35

Not age appropriate for the grade level specified: either too hard or not challenging enough

9

Good culturally relevant learning activity for Navajo students 27 Sound needs improving 5

Summary In general, the reviews of lessons conducted in Year 5 were positive, and the comments

suggest a higher level of satisfaction with the lessons reviewed than in previous years.

However, there are still some areas in need of improvement; for example, in the domain

of Lesson Mechanics, lesson download time and broken links were two areas identified

again as needing attention.

Instructional Design of the lessons was again rated highly in the areas of objectives,

standards and benchmarks, but as in Year 4, lessons were not rated as highly in the

instructional design areas of feedback and assessment. Reviewers indicated that lessons

could be improved to provide more consistent feedback to students, and in providing

formative and summative assessments. Since the NETtrain system does not have the

capacity to track and monitor the performance of individual students, separate pre- and

post-assessments were developed for the lessons used in the Model Classroom study in

Year 5. If lesson development continues, this strategy should be expanded to all lessons

in order to provide teachers and students with assessment activities that can be

downloaded, printed and completed on paper so that teachers have access to important

80

feedback on student learning to inform their instruction and support data-driven decision-

making.

In the domain of Culturally Appropriate Curriculum, those lessons that were designed

specifically for a Navajo audience were rated highly by reviewers. Understandably,

lessons set in the context of Navajo culture and communities are very popular in

classrooms that have an average population of 85% Navajo students. In Year 5,

reviewers also rated lessons more highly for the support they provide for limited English

proficient learners. This may reflect the addition of narrations to lessons that previously

lacked this feature. Since over half of the NETC student population is Limited English

Proficient, and literacy plays such a major role in learning and standardized test

performance, this is an area where TECHShare lessons could make a real difference in

local classrooms and support the goals of the No Child Left Behind (NCLB) legislation.

Furthermore, findings from both the Year 4 and Year 5 Model Classroom studies indicate

that students in the NETC population learn better from TECHShare lessons that are

culturally relevant and appropriate.

Overall, reviewers gave the lessons high overall ratings, with a large majority of

reviewers indicating that any educator in the field could use the lessons and that the

lessons assist learners in mastering the objectives. If lesson development is continued, a

few practical recommendations can be drawn from the evaluation findings.

Recommendations Teachers find more instructional value in lessons that contain some form of

assessment that can be used to gauge and document student learning. Until the

NETtrain system is developed to manage student performance data, provide

downloadable assessment activities that can be used to support the online

instruction.

Ensure that all future lessons developed give strong support to English

language learners, including audible narration of the text, multiple language

versions of the same lesson, and visuals that serve to reinforce content. This type

81

of support is especially critical in lessons developed for the elementary grade

levels.

Keep in mind the grade level of students when designing navigation and

instructions, and make every attempt to keep links within lessons intact.

Continue developing lessons that set the instruction within a familiar context,

whether it is explicitly Navajo or simply customized to the regional culture and

locale. The lessons that have these contextual features are generally very popular

with students and teachers and seem to promote learning.

As the last point suggests, providing culturally relevant curricula for Dine students is a

highly specialized niche that TECHShare is uniquely positioned to fill. TECHShare fills

a void that scholastic publishers are unwilling or unable to fill, yet if left unfilled can

have deleterious effects on the persistency of Native American students (Hale, 2002).

Given that NETC schools have predominantly Navajo student populations, it is

imperative that school administrators find practical solutions to address persistent

problems affecting their students. Results of the online lesson rubric show that

TECHShare lessons are seen as a valuable standards-based resource by many of the

teachers who review them, and results from the Model Classroom study indicate that

students do learn from using the lessons. TECHShare has demonstrated that it can be at

least part of the solution to improve NETC students’ academic performance; now it is

time for NETC administrators and board members to give it the support necessary to

ensure its widespread and consistent use in the classroom.

Over the five years of the project, it has been very clear that one of the keys to getting

TECHShare lessons used in NETC classrooms is the support and dissemination efforts of

the Content Development Specialists (CDSs). Each year the CDSs complete a survey to

provide insight from their perspective on the curriculum development process, where

support is most needed in the field, and how TECHShare is being received by teachers

and students. The results of the Year 5 CDS survey are presented in the next section.

82

Content Development Specialist Survey One of the subcomponents of Objective 1 of the TECHShare project concerns the work

of the master teachers – now known as Content Development Specialists (CDS) – with

ETIP teachers on curriculum development for the TECHShare project. The CDSs, who

joined the project in Year 2, act as a liaison between the teachers in the classroom and the

Curriculum Preparation Studios responsible for digitizing and enhancing lessons intended

for their use.

Five CDS positions were created, with one at each of the four ETIP training centers and

an extra one at the largest training service area in Gallup, NM. For most of Year 5

however, only four of the positions were filled, so that it was necessary for one of the two

CDSs at Gallup to work with the schools in the Window Rock training area. Also, in

November of Year 5, the CDS at Kayenta took on the responsibilities of TECHShare

project director in addition to her CDS responsibilities, so that the CDSs were stretched

rather thin overall for much of the year. In addition to their work on lesson content

development with the studios, the CDSs once again played an important role in setting up

and supporting the Model Classroom study in Year 5 as they had in Year 4. Data

obtained from the CDSs were used in triangulation with data collected from the

Curriculum Preparation Studios (CPSs) regarding the lesson development process and

progress, and with data collected from Model Classroom teachers.

CEER administered the CDS survey as an email attachment, which was mailed out to the

five CDSs after the Model Classroom study had finished at the end of May 2004, with a

request for completion by June 7, 2004. The survey comprised twelve items that

combined forced choice and open-ended formats. Nine items asked the CDSs about their

work on curriculum development, two items asked them about their work with the Model

Classroom study, and one asked them for general comments and concerns. Three of the

four CDSs working in Year 5 completed and returned a survey. A copy of the survey can

be found at Appendix E. A summary of the findings from the survey is presented below.

83

Curriculum Development The Content Development Specialists – or CDSs, as they are called – spent less time in

Year 5 on curriculum development than they had in previous years. All of the CDSs who

responded to the survey reported spending just one quarter of their time on curriculum

development (in the past it had been as much as three-quarters of their time). They

explained that the studio they were assigned to work with had a sufficient quantity of

lessons in development from previous years that their assistance was not needed as often.

Also, as had been the case in previous years, certain studios continued to work on lesson

development independently, and only occasionally called on a CDS for support with

tasks such as assisting with language translations and cultural components.

All of the CDSs reported encountering challenges in communicating and working with

the studios. For some, challenges arose from the lack of face-to-face meetings that were

made impractical by the distances between the studios and the CDSs’ offices. All of the

CDSs were located at least one hundred miles from the nearest studio, many of them at

even greater distances. Although efforts were made on both sides to maintain regular

communication, there were times when this was not sufficient, with the result that some

changes would need to be made to lessons late in the development process. Other

challenges came from ‘too many cooks in the kitchen:’ the studio at the University of

New Mexico was directed to focus on the development of a Navajo Beauty Way lesson

by the project director, who also made a lot of decisions on the content and design of the

lesson that at times contradicted the advice of the CDSs. This inevitably led to frustration

for all involved. The comment below from one of the Content Development Specialists

illustrates the creative tensions that can occur:

“It was my understanding that the studio would be working on Language Arts lessons for K-8 this year, but they were being directed to work on lessons that did not fit this category – especially the Beauty Way lessons. I felt that much of their time was being taken up with activities that were not facilitated by the CDSs – sometimes the CDS assigned to work on a lesson was not even aware of work that the studio was doing on that particular lesson until after the fact….The studio was also expected to put a lot of

84

work into the advertising/marketing materials and that put lesson development on a back burner.”

Similar challenges were noted by the CDSs in Years 3 and 4, so it seems that some

problems have persisted. As with most complex processes, it is most likely a

combination of factors that are contributing to the persistence, including the challenges

inherent in collaborating at a distance, working in teams, and having unclear or shifting

roles and responsibilities. If lesson development is to be continued in the future, these

hurdles will need to be resolved, or some new procedures adopted, in order to keep

development flowing smoothly and resources used most efficiently and effectively.

Work with NETC Teachers In contrast to Year 4, all of the CDSs reported that they did not work with NETC teachers

at all on lesson development. This was in part due to the fact that the studios were not in

need of any new materials, and in part due to teachers not being interested in developing

lessons. One CDS noted that teachers in his area were more focused this year on

managing new online testing software and preparing their students for state standardized

tests in order to avoid their school being placed in corrective action. Another CDS

reported that although she did work with some ETIP team members on lesson

development, these lessons were not submitted to the studios for multimedia development

but were simply used by the teachers themselves. The third CDS described her

experience of working with teachers on lesson development in the past and proffered her

perception of why teachers were not more interested:

“Teachers are interested in sharing their ideas and sometimes don’t mind if someone picks up ideas from units they teach – but they don’t really have the time or the energy to put into development. When the multimedia work has been done on their lessons, they don’t even recognize their own work any more and that’s a turn-off for some. Even though some teachers say that they would submit lessons if there was money involved, they usually don’t follow through. The timing always seems to be bad for teachers – too early in the year, too close to a break, too close to testing, too close to the end of the year…You get the picture.”

85

Fortunately, all of the Curriculum Preparation Studios had sufficient material from which

to develop lessons in Year 5. In the future, it may be a better tactic to survey teachers in

the target population on their needs, and then use an on-site curriculum specialist at each

studio to design the actual lessons.

Model Classroom Study Support CDSs were asked two questions regarding their work with the Model Classroom study in

Year 5; the first asked them to select from a list the three tasks that they assisted Model

Classroom study teachers with most frequently, and the second asked them to report on

the greatest challenges they faced in supporting the Model Classroom teachers. The list

of tasks is presented below to give some idea of the types of support that CDSs might

have been asked to provide to teachers during the Model Classroom study.

• Administering pre- and post-tests • Finding and accessing the selected TECHShare lessons online • Finding and accessing the selected TECHShare lessons on CD-ROM • Integrating the TECHShare lesson into their curriculum • Identifying supplemental activities to use with the lesson • Using the Smartboard® to present a lesson • Using a projection device such as a Proxima, In-focus or Averkey to present the lesson • Filling out online forms such as the Teacher’s Journal or lesson evaluation rubric • Troubleshooting technical difficulties

All three CDSs reported that they had assisted Model Classroom teachers with filling out

the online evaluation forms; this usually meant that the CDS helped the teacher to locate

the link or URL for the form. Two of the CDSs also helped teachers use either a

Smartboard® or projection device to view a lesson. Individually, the CDSs provided

support on integrating a lesson into the curriculum, accessing the lessons on the CD-

ROM, or troubleshooting technical difficulties. In a comment, one CDS observed that his

teachers did not find using the lessons on the NETtrain web site intuitively easy, and so

technical support in this area was frequently requested.

86

The CDSs all encountered different challenges in working with the Model Classroom

study. For one, the greatest challenge was integrating the lessons into the school’s

curriculum, as he explained:

“The lessons either didn’t exactly fit the teacher’s curriculum or they were too abbreviated in their content. Designing lessons that fit with each math or language arts teacher’s curriculum is difficult at best, since each teacher chooses slightly different areas on which to focus or approaches the material on a different timeline.”

For another CDS, the greatest challenge was keeping teachers in the study, and traveling

the long distances to support teachers outside of her training center’s service area. This

CDS also indicated that her teachers seemed to have a harder time in Year 5 staying on

schedule and getting everything finished, as there were often school-wide activities that

would come up unexpectedly. The third CDS felt that her greatest challenge was in

getting the teachers to understand that she was available to help them; most seemed intent

on doing everything themselves and rarely took her up on her offers of assistance.

Overall, the Model Classroom teachers did not seem to require as much support in Year 5

as in Year 4, and this may have been partially due to providing them with an orientation

to the study at the beginning of the school year. This had been recommended by the

CDSs after their experience with the study in Year 4, and the strategy appears to have

paid off in terms of fewer demands on the CDSs over the school year.

Only one CDS offered a final comment; he observed that the teachers in his area seemed

to prefer to use lessons that had been developed by a subject matter expert, rather than

another teacher, which would add further weight to the idea of using curriculum

specialists to develop lessons at the studios with input from local teachers.

Summary and Recommendations The Content Development Specialists have played a key role in the TECHShare project

since Year 2 in support of lesson development and lesson dissemination and use. CEER

surveyed them annually to gain their perspective and insights into the curriculum

87

development process, starting with teachers in NETC schools and ending with the

Curriculum Preparation Studios, and on the conduct of the Model Classroom studies.

Feedback from the CDSs indicates that while they have been able to sustain a fairly

effective working relationship with the studios for the most part, it may be necessary to

revise procedures in order to ensure more efficient development in the future.

Furthermore, NETC teachers – like all teachers - are under such intense pressures and

time constraints that a monetary stipend is not enough to stimulate interest for them to

produce lessons. The perception is that teachers are more concerned that the

supplementary resources available to them will integrate smoothly into their curriculum

with minimum fuss and effort and provide students with a solid instructional activity.

This seems to be one further endorsement for changing the current lesson development

process; evaluation data suggest that it would be more efficient for the studios to use an

onsite curriculum specialist for lesson development and to have the CDSs concentrate

their efforts on working with teachers in the field to promote the use and integration of

TECHShare lessons.

Thanks to lessons learned and applied from the Year 4 Model Classroom, CDSs did not

have to devote as much time and effort to support the study in Year 5. However, the

support that they did provide, both to the teachers who participated in the study and to the

external evaluators, was the sine qua non of the whole endeavor and must be

commended.

NETtrain Web Site Evaluations

Towards the end of Year 4, CEER worked with a committee of NETC members, project

partners and staff to develop an evaluation checklist to collect feedback from users on the

perceived usability and utility of the project web site, NETtrain. Everyone agreed that

the checklist needed to be short and straightforward in order to garner as many responses

as possible, from the broadest range of stakeholders. After a number of drafts and

revisions, the committee settled on a twenty-one-item instrument, divided into five

categories. The five categories are: ‘Look and Feel,’ ‘Site Content,’ Site Sources,’

88

Navigation and Manageability,’ and ‘For Educators.’ The checklist was placed online,

and a link to it appears on the main landing page of the web site. Since April 2003, sixty-

one evaluations have been collected, the results of which are presented here. A paper

copy of the checklist can be found at Appendix C.

Findings

Teachers conducted the majority (roughly 66%) of reviews of the web site. Table 17

shows the breakdown of reviewers by type. The ‘other’ category includes reviewers who

did not identify their position or identified themselves as a ‘tutor.’

Table 16 Web Site Reviewers by Type

Reviewer Type Number of Reviews

Teacher 40 Parent 4 Student 4

Ed. Technician 2 Project staff/partner 3

Administrator 1 Other 7

Total 61

Reviewers responded to the twenty-one items by selecting one of three responses: very

good, satisfactory, or needs improvement. If a reviewer selected ‘needs improvement,’

they were requested to provide ideas on how that particular item or feature could be

improved. Table 18 presents the results of the sixty-one reviews, showing the number of

reviewers that selected each response.

89

Table 17 Results from the NETtrain Site Reviews

Very Good

Satis- factory

Needs Improvement

Look and Feel: The site is interesting.

44 16 1

Its purpose is clear and easy to understand. 47 11 3 Site Content: Is useful for teaching and learning.

50 12 1

Contains practical information on the integration of Navajo culture into instruction.

46 11 3

Has links to other sources that are useful for teaching and learning.

48 12 1

Has links to alternative pages/sources for English learners.

46 8 7

Site Sources: It is clear that the site is hosted by the Navajo Education Technology Consortium (NETC).

50 7 4

Information is provided that makes it easy to contact the NETC.

49 10 2

Information is up-to-date (time-sensitive material is current).

47 13 1

It is clear how the privacy of users will be handled.

47 10 4

Navigation and Manageability: There is a site index or map that is easy to locate and use.

44 12 5

All links are clearly labeled and easy to understand.

47 11 3

It is easy to find my way around the site. 43 11 7 Accommodation is made for persons with disabilities.

46 10 5

Instructions for downloading plug-ins are clear. 43 13 5 For Educators: Pages load fast enough to be practical for use in the classroom.

45 9 7

Adequate support is provided for me to use the site to supplement instruction.

46 9 4

The ‘threaded discussion’ feature for each lesson is useful.

47 11 1

Provides useful strategies for addressing a diverse student population.

46 11 2

Support features, such as the tutorials, calendar, online resources, instructions, etc. are helpful to me.

47 12 0

Provides links to useful articles on the theory and effectiveness of online learning.

47 11 1

Note: Where totals do not add up to 61 reflects missing responses

90

As Table 18 shows, most reviewers selected ‘very good’ for each item on the checklist.

Although very few comments were provided, those that were give some insight into the

areas that are found to be most useful or most in need of improvement. Under ‘Look and

Feel,’ one reviewer suggested that it might help if the site had a “catchy phrase” or tag

line associated with it that would also serve to let people know immediately the main

purpose of the site; something to do with Navajo students or the cultural relevance of

much of the resources offered.

Under ‘Site Content,’ seven reviews indicated that there is room for improvement,

particularly in the area of resources for English language learners; one reviewer

commented that these resources are “buried too deep” on the website, and two others

suggested that links to these resources need to be written in both Navajo and English.

Reviewers were generally satisfied with the ‘Site Sources’ element of the web site,

although some felt that more needed to be promoted about the Navajo Education

Technology Consortium. ‘Navigation and Manageability’ was another area where

reviewers saw some need for improvement; one person commented that they had not

been able to get back to the NETtrain site after visiting the Media Gallery, and another

observed that there is no site map provided. At the same time, other reviewers were very

satisfied with the navigation elements, and made comments such as “great” and “thumbs

up.”

The last section on the checklist contains items mostly relevant to educators, such as

whether the site provides adequate support to help teachers integrate the resources into

their instruction, and whether features such as the online tutorials and supplemental

resources are useful. Most features received positive ratings, but some reviewers had

difficulties with slow loading times of pages, which can be an obstacle to in-classroom

use. Only two comments were provided: one concerned lessons that took so long to

download that the browser would “time out,” and the other suggested that some

additional instructions on the main page of the site would help.

91

In summary, reviews of the NETtrain web site were positive overall, with some areas in

need of improvement as noted above. Additional areas where improvement might be

merited are accommodations for persons with disabilities, and instructions for

downloading plug-ins. Given the complexity of the site and the number of resources and

features that it contains, it seems that the developers have done a very good job of

creating a web site that is generally user-friendly and utilitarian.

92

Evaluation Summary and Discussion of the Findings As in Year 4, project activities in Year 5 centered on the implementation of the Model

Classroom study and the ongoing development and refinement of lessons. The

evaluation of the Model Classroom study was modified in Year 5 to include random

assignment of teachers in an attempt to provide stronger evidence of the link between

TECHShare lessons and student achievement. The evaluation was designed to answer

key questions regarding who participated in the study, whether participating students

experienced learning gains and whether some lessons were more effective than others,

and questions concerning the overall quality of TECHShare lessons and the factors that

might contribute to or detract from the lessons’ effectiveness. The evaluation also

continued to monitor project development and implementation by collecting feedback

from the studios on lesson development and from various stakeholders on the utility and

usability of the NETtrain web site.

Generally, the results of the Model Classroom study were positive once again. The

reiteration of the Model Classroom study was an important ‘next step’ for the project as it

provided confirmation for many of the findings from Year 4, as well as allowing for the

demonstration of the integration of TECHShare lessons into instruction with a new group

of teachers. The results of standards based pre- and post-tests administered to Model

Classroom students indicated that students in the experimental group demonstrated

significantly greater learning gains than students in the control group, across all subject

areas. The results also confirmed that TECHShare lessons set in Navajo or localized

contexts result in greater learning gains than others, and that using a Smartboard® to

facilitate instruction with TECHShare lessons can also have a beneficial impact on

student learning for the target population. Using a Smartboard® has the additional value

of helping schools to overcome technology access issues in classrooms that have no or

few computers. Given the challenges that schools face in coping with reduced budgets

and the demands of No Child Left Behind, these findings taken together are not

inconsequential and may go some way towards helping NETC schools – and other

93

districts with large Native American populations – find solutions to their persistent

problems of increasing student achievement and student retention.

Curriculum Preparation Studios again met or exceeded their module production targets

for the year. While collaboration with the Content Development Specialists on lesson

development has been effective in the past, it may be time to revise the process and retain

curriculum and cultural specialists on site at the studios, thus freeing up the CDSs to

concentrate on providing much-needed support in the field and with dissemination.

Evaluations of a large sample of TECHShare lessons indicate that the lessons are aligned

with standards, but some need student activities and assessments added to make them

more valuable as instruction. Reviews also reflect that the studios are conscience of the

need to provide good support for Limited English Proficient students, with more lessons

including narrations or multiple language versions, and this support should continue to be

a strong characteristic of all TECHShare lessons.

Feedback from NETtrain web site users was generally very positive, with only a small

number of users reporting problems with pages being slow to load or finding the

navigation within the site confusing. Reviewers appreciated the resources offered, such

as those on cultural curriculum and the tutorials for teachers, and recommended that more

of the site be bilingual in English and Navajo.

Following is a reiteration of the major findings and recommendations from the evaluation

of Year 5 project activities:

Major Findings Pre/post-test scores: The results of an analysis of variance (ANOVA, F(1,1334) =

37.42, p < .001, R2 = .027) of pre/post test change (Change) on two levels of group

assignment (Experimental and Control) suggest that there were significant differences

between the Experimental Group (M = 18.67, SD = 23.91) and the Control Group (M

= 11.44, SD = 19.14) with the Experimental Group outperforming the Control Group

94

by an average of seven percent (7%) across all lessons. Due to the random

assignment of participants to conditions, there is a high degree of confidence that the

observed differences were not attributable to chance fluctuations.

Other significant findings from the Model Classroom study:

o The results of a one-way ANOVA suggest that students in the

Experimental group who completed TECHShare lessons that were rated as

possessing High Navajo Cultural Relevance (as measured by the Online

Lesson Rubric instrument) achieved higher score increases from pre-test

to post-test (M = 16.06, SD = 21.44) than students who completed

TECHShare lessons that were rated as Low (M = 13.31, SD = 22.36) on

Navajo Cultural Relevance measures (ANOVA f(1,1334) = 5.14, p = .023,

ή2 = .004). This accords with findings from the Year 4 Model Classroom.

o Although students in the Model Classroom study whose teacher

participated in the NETC’s professional development project (ETIP)

showed gains marginally greater than those students whose teachers were

not in ETIP, the difference was not significant as it had been in Year 4.

o Students in the Experimental group whose teachers employed a

Smartboard® during instruction demonstrated significantly higher (M =

21.07, SD = 26.89) score percentage increases than students whose

teachers did not (M = 15.47, SD = 23.33) use a Smartboard® (ANOVA

f(1,404) = 4.32, p = .040, ή2 = .01).

o In most cases, TECHShare lessons could be integrated into the curriculum

in both New Mexico and Arizona with little or no adaptation, and were

considered by teachers to aid students in mastering the standards and

benchmarks.

95

TECHShare lesson development: Over the five years of the project, the curriculum

development studios have met the development targets of Objective One by

completing 1516 modules. These modules make up 246 standards-based lessons that

are available for use by teachers and students from the NETtrain web site.

TECHShare lesson quality: The evaluation of a large sample of TECHShare lessons

over the course of Years 4 and 5 confirms that the majority of lessons contain

challenging content aligned with state standards. A total of 1165 modules were

evaluated in Year 5: Of these, 1027 fall within the GPRA categories of Language

Arts, Math and Science, and all of these modules were found to be aligned with state

standards.

Web site usability: Reviews of the NETtrain web site against an evaluative checklist

were positive overall. One area identified as in need of improvement was navigation

and manageability, with some users reporting problems with slow downloading of

pages. Given the complexity of the site and the number of resources and features that

it contains, it seems that the developers have done a very good job of creating a web

site that is generally user-friendly and utilitarian.

The Content Development Specialists continued to play a critical role in the project,

although the focus of their work in Year 5 shifted away from new lesson development

towards providing essential support to teachers – especially Model Classroom

teachers – to integrate TECHShare lessons into their instruction.

Recommendations Recommendations drawn from Year 5 evaluation findings include the following:

Continue to encourage all TECHShare Curriculum Preparation Studios to include

cultural components in their lessons since data from both Year 4 and Year 5 have

indicated that students show greater gains on tests of lessons with high Navajo

cultural context. If the lesson itself cannot be set in a Navajo context, include an

extension activity that helps students to apply the new learning in a familiar context.

96

Ensure that all future lessons developed give strong support to English language

learners, including audible narration of the text, multiple language versions of the

same lesson, and visuals that serve to reinforce content. This type of support is

especially critical in lessons developed for the elementary grade levels especially in

light of the fact that a large proportion of the NETC student population is limited

English proficient (at least 37% of the Model Classroom sample population were

LEP).

Encourage NETC schools to purchase Smartboards® and to provide training and


using Smartboards® to facilitate instruction with TECHShare lessons resulted in

higher score gains for students in the Model Classroom studies.

Work on sustainability from Day One; identify it as one of the project goals and




wanted.




consistent, recognizable and easy to use environment.





97

partners.



between.

Provide downloadable assessment activities that can be used to support the online

instruction. Teachers find more instructional value in lessons that contain some form

of assessment that can be used to gauge and document student learning. Be sure that

assessments align with specific state standards and benchmarks to further support

student achievement and the goals of the No Child Left Behind legislation.

Evaluation findings indicate that the TECHShare project has met the objectives it set for

itself, but more work needs to be done on dissemination and marketing if it is to more

substantially address the goal of increasing achievement for the large population of

NETC students (estimated at approximately 47,000). Fortunately, results from both

iterations of the Model Classroom study suggest that TECHShare is on the right path and

project partners and management should capitalize on TECHShare’s unique ability to

provide lessons that blend the qualities of culturally relevant curriculum and technology-

based multimedia for the benefit of student learning. These findings should be

disseminated nationally as they are likely to be of interest to the broader field of

educators working with indigenous populations.

Evaluation Strategy and Lessons Learned CEER employed a science-based evaluation strategy in Year 5, which entailed the

random assignment of teachers to either the experimental or control group in the Model

Classroom study. Although not as strong a strategy as randomly assigning students to

condition, it was the most practical solution to conducting evaluation research in the

authentic environment of the classroom. To avoid claims that participants in the control

98

group were denied potential benefits of the treatment, CEER designed the study so that

after the initial random assignment to condition, teachers (and their students) would

alternate between conditions so that all of the participants were able to receive the

treatment (i.e., TECHShare lessons) at least half of the time. As in Year 4’s Model

Classroom study, in Year 5 CEER again used a convergent mixed methods design,

whereby data were collected from multiple perspectives and in multiple formats to enable

the triangulation of data and so control for many of the confounding variables present in

the uncontrolled environment of the classroom. Also as in Year 4, concerted efforts were

made to include participants that were representative of the broader NETC population;

these efforts were facilitated by the homogenous demographics of the population which

reveal a student population that is 85% Navajo, and all located at schools designated as

rural and eligible for Title 1 funds.

In the course of Year 5, CEER learned a number of valuable lessons. In response to

lessons learned in Year 4, CEER conducted a full day orientation for the teachers

participating in the Model Classroom study before it started, so that everyone received

the same instructions and had an opportunity to ask questions and discuss issues that the

evaluators and project staff had not anticipated. Most, but not all, of the teachers were

able to attend. In spite of this and the ready support offered by the Content Development

Specialists, two of the teachers found it necessary to drop out of the study late in the

school year. Substitutes were identified, but this meant that they had not attended the

orientation and they required additional time to complete all of the activities. Thus,

although the study went smoothly for the most part, there was still some lack of

consistency in the implementation of the study that may have affected results. Also,

without someone observing and recording each and every lesson being used in the

classroom in both the experimental and control conditions, which is neither feasible nor

affordable, there is no way of accurately accounting or controlling for teacher effects.

Even though text based versions of the TECHShare lessons had been provided to

teachers, there was still a lot of room for variation in how both the control and

TECHShare lessons were implemented. In spite of these limitations, the evaluators are

99

confident that the study resulted in meaningful findings that can be used to guide future

project activities.

CEER has only recently learned that the TECHShare project has received authorization

from the funding agency for a no-cost extension of the project. Since actual project

activities have yet to be determined, CEER is not able to propose what evaluation

activities may be appropriate in the future.

100

Products Recommended for Wide Dissemination Over the course of the five years of the TECHShare project, the studios have developed

257 lessons that are available on the NETtrain website for teachers and students in grades

K-12. The lessons are primarily in the core content areas of math, science, and language

arts, but there are also some in Navajo language, arts, health and physical education,

career and workplace skills, and social studies. Many of the lessons specifically address

the learning needs of Navajo and Hispanic students, groups traditionally underserved, and

provide strong support for limited English proficient learners. To view a complete list of

lessons, go to http://nettrain1.unm.edu

101

http://nettrain1.unm.edu/

References Christensen, R. (1997). Effect of technology integration education on the attitudes of teachers and their students. [Online]. Unpublished doctoral dissertation, University of North Texas, Denton, TX. Available at: http://courseweb.tac.unt.edu/rhondac Christensen, R. and Knezek, G. (2001). Instruments for assessing the impact of technology in education. In Computers in the Schools, The Haworth Press, Inc. Vol. 18, No.2/3. Fouts, J. (2000). Research on computers in education: Past, present and future. Report prepared for the Bill and Melinda Gates Foundation. Seattle Pacific University, Seattle, WA. Hale, L. (2002). Native American Education: A Reference Handbook. ABC-CLIO, Santa Barbara, CA. Hezel and Associates (1999). An overview of educational telecommunications and distance learning in the USA. Excerpts from the full report Educational telecommunications and distance learning: The state by state analysis, 1998-99. Syracuse, NY: Hezel and Associates. Nelson-Barber, S., & Estrin, E.T., (1995). Culturally responsive mathematics and science education for Native American students. San Francisco, CA: Far West Laboratory for Educational Research and Development. Schacter, J. (1999). The impact of education technology on student achievement: What the most current research has to say. Milken Exchange on Education Technology Articles: www.milkenexchange.org Sivin-Kachala, J. (1998). Report on the effectiveness of technology, 1990-1997. Software Publisher’s Association. Swisher, K. and Tippeconnic III, J. eds. (2000). Next Steps: Research and Practice to Advance Indian Education. Charleston, WV: Clearinghouse on Rural Education and Small Schools.

102

http://courseweb.tac.unt.edu/rhondac

Appendix A

Model Classroom Study Instruments

103

Navajo Education Technology Consortium (NETC) TECHShare Project

Model Classroom Study

Informed Consent Form

You are being asked to participate in the Model Classroom demonstration project of NETC’s TECHShare Project. We hope you will find this to be a positive educational experience for you and your students. The evaluation portion is being conducted by NETC Project Evaluator, Rebecca Zittle of the Center for Educational Evaluation & Research (CEER). The purpose of the evaluation portion is to gather information to review the impact of TECHShare’s activities and resources. This will provide information about the project’s effectiveness in achieving its goals. As a Model Classroom demonstration project participant, you will be responsible for responding to several instruments throughout the year that will provide information about your use of technology and the TECHShare lessons. All information obtained from you will be kept confidential. All the information derived from the evaluation will be reported in terms of group results, with no individual names attached. All Model Classroom evaluation participants will have access to the identity of the other Model Classroom teachers. Your individual responses will not be made available to anyone and will not be used in evaluation of your teaching competency. Your agreement to participate in this evaluation is appreciated. If at any time there are questions that you are uncomfortable answering, please let Rebecca Zittle know your concerns. You will not be required to respond to any such items. The goal of this evaluation is to ascertain information about the project and its success, not to evaluate teachers or their students. If you desire further information about this study and / or your participation in the study, please contact Rebecca Zittle at (580) 355-5246 ext. 105, or by email at: [email protected]

I consent to participate in the evaluation described above. I understand that responses to questions and other information about me will be kept confidential. I agree to have my identity shared with other model classroom demonstration project participants in online evaluation documents. I also understand that my participation is voluntary and I may withdraw from the study at any time without prejudice to me. Furthermore, I understand that, if my concerns regarding evaluation are so substantial that I decide to withdraw from the participation in the study altogether, I have the option of withdrawing as a Model Classroom teacher, with all the benefits withdrawn that are related to this position.

____________________________________ _____________________ TEACHER’S NAME DATE SIGNATURE

mailto:[email protected]

Model Classroom Study: Year 5 Teacher Interview Protocol

This interview guide is to be used when interviewing teachers taking part in the Model Classroom study for the TECHShare project. Before conducting the interview, review the teacher’s online journal entries and background information survey responses. Explain to the teacher that you are interested in finding out how well TECHShare lessons work for teachers and students, and that any feedback he/she could provide to help make the lessons more effective and useable would be greatly appreciated. Try to ask each question as written, and avoid questions that can be answered yes/no. If an interviewee does answer a question with yes or no, follow up with a probing question (e.g., “Can you give me an example?) to draw out as much detail as possible. Try to promote a relaxed atmosphere during the interview. Fill out in advance: Teacher’s Name:___________________ Date: _____ School:_______________ Grade level(s) taught: ________ Content area(s): ________________________

1. How many of the lessons in the TECHShare unit have you finished with your students so far?

2. In general, how well did the TS lessons work for you and your students?

3. How appropriate was the reading level of the lessons for your students? Did they need a lot of assistance in reading the text of the lessons? (Was font size good? Level of difficulty? Instructions OK?)

4. Did you have any problems with the pre- and post-tests? –If so, what were they? (reading level, alignment with content, alignment with standards).

5. Do you think that practice with these tests will help/helped your students with state standardized tests? –In what way?

6. What do the TECHShare lessons contribute to your instruction? Probe: Do they add anything that your students wouldn’t have otherwise?

7. Do you think that using the lessons helped your students master the state standards?

8. Will you continue to use TECHShare lessons once you finish the unit? Probe: Why/ why not? – Do you have concerns about using these lessons?

9. Do you feel you have the support of your administration to use these lessons? -How about support for integrating technology in general?

10. Which of the following are you currently using? __ Online attendance __ Grade books __Web pages publisher __ PDAs __ CAI (e.g., Accelerated Reader, MathStar, etc.) ________ __ Research using Internet __ Creative software (e.g. PowerPoint for Stu. Presentations, drawing programs, Word Processing etc) __ Email within school __ Lesson plan software __ Other: _____________

11. Have any other teachers expressed an interest in TS lessons? b) Any parents aware of your using these lessons?

12. Any other comments or suggestions? (When finished, thank the teacher for their time.)

106

TECHShare Model Classroom: Principal Interview Protocol

Name: _____________________ School District: _______________________ School: ___________________________ Elem / MS / HS (circle one) Principal’s ethnicity (guess): ________________________

1. How long have been the principal at this school? ________ Can you tell me a little about the history of your school? How many students do you serve? _____

2. What are the demographics of your school? (approx.)

3. What size is your service area, and how far are students bussed in?

4. Is your school currently in corrective action? ___ Yes ___ No

5. a) Do you have a large turnover in students each year? b) How about teachers?

6. Is there strong support for your school from the community? a) What is parent attendance like at teacher-parent conferences? b) Is your school board pro-technology?

7. a) How long ago did your school get connected to the Internet? ____ b) Are all of the classrooms connected? ___ Yes ___ No c) If yes, how many computers with Internet connections in each? ____

8. Do you have a computer lab? ___ Yes ___ No If yes, how many computers connected to the Internet? ____

9. Does your district actively support and encourage teachers to integrate technology into their instruction? – How?

10. Are technology skills and use part of teacher evaluations?

11. Do any of the professional development/inservice days this year include technology training? –If yes, what kind?

12. How are your teachers currently using technology? For example: __ Online attendance __ Grade books __Web pages __ CAI (Accelerated Reader, MathStar, etc.) ____________________ __ Research __ Performance (Stu. Presentations, graphics, etc) __ Internal email communications __ Lesson plans __ Other:

13. Approximately what percentage of your teachers regularly use technology for instruction?

14. Have you personally viewed or tried any of the TECHShare online lessons? -Do you think this is a resource your teachers and students need?

15. What benefits do you expect from teachers and students using TECHShare lessons?

16. Have you observed or been informed of any changes in teachers or students who have used the TECHShare lessons? Please give examples:

108

TECHShare Year 5 Model Classroom Study Observation Guide Observer’s Name: ________________________

This guide is to be used while observing a teacher in a classroom using a TECHShare lesson with students. Observers should pay special attention to the instructional strategies used by the teacher, amount of support needed by and provided to students, and the overall usability of the activities for different skill or comfort levels.

Teacher: ___________________ Grade: ________ School: ____________________ Content Area/Topic: ______________________ No. of students: ______ Date & Time: ______________ Duration of Observation: _________hrs/mins. Lab ____ or Classroom _____ Title of lesson(s) being used: _______________________________ TECHShare ____ or Control lesson ____

1. For the following items, place a check in the box that most closely represents what you observe. Add comments in the margin if necessary to elaborate:

YES

NO

N/A

a. The teacher seems prepared for the lesson (has previewed it, has a plan).

b. A projection device, such as a SmartBoard, Proxima or averkey, is used to present the activity to the whole class

c. Students work through the lesson individually (one student per computer).

d. There is evidence (review, tie-in) that the lesson is being integrated in the curriculum, not just “added on”.

e. The additional Cultural Activities are used or referred to.

f. The pre-test is done immediately before the lesson.

g. The post-test is done immediately after the lesson.

2. For the following items, place a check in the box that most closely represents what you observe. Add comments in the margin if necessary to elaborate:

Agree

Neutral

Disagree

Not applicable

a. The teacher encourages the students to work collaboratively.

b. The students appear to be engaged by the content of the lesson.

c. Students are able to read and understand lesson text and instructions.

d. Students need a lot of guidance to work through the lesson (navigation, links, sequence).

e. Students appear to have the technology skills needed to complete the lesson

Agree

Neutral

Disagree

Not applicable

f. The teacher appears to be enthusiastic about using the lesson

g. The teacher appears to have the technology skills needed to complete the lesson

h. The teacher appears to be knowledgeable about integrating technology effectively

3. Problems or challenges observed while the lesson was being used: Check all that apply __ technology not working __ technology not hooked up/ready __ teacher has no ‘Plan B’ __ no tech support available __ accessing lesson on CD __ lesson load time __ plug-ins not installed __ plug-ins not working __ navigation problems __ sequencing of lesson confusing __ feedback on quizzes not appropriate or clear 4. Reactions, events observed while the lesson was being used (e.g., kinds of questions asked by teacher and students, types of interaction taking place, etc.): Check all that apply __ teacher asks rote/close-ended questions __ content is clearly review for students __ students ask for clarification/directions __ teacher asks probing, open-ended questions __ students make connections to previous knowledge __ teacher and/or students go beyond content given __ teacher does not interact with students re content

5. General observation of classroom climate (circle one): The classroom or lab was….a) on-task quiet b) bored/subdued quiet c) on-task noisy d) chaotic noisy 6. Approximately how much time was spent on the lesson, and was it completed?

110

Year 5 Model Classroom Student Demographic Survey

This survey is to be completed by teachers participating in the TECHShare Model Classroom study with their students. The purpose of the survey is to document the characteristics of students who are participating in TECHShare learning activities. The funding agent for the project, the U.S. Department of Education, requires this information. A separate survey should be completed for each unique class or group of students that participates in a TECHShare lesson. Give completed surveys to your TECHShare Content Development Specialist, or send to CEER at the address at the top of this form. 1. Your Name: _____________________________ 2. School: ________________________________ 3. Principal: ________________________ 4. School District: _________________________________ 5. City, State, & Zip: ____________________________________________________ A. School Information

1. Is your school considered (Please check one): Pre-K ___ Elementary ___ Middle/Junior High ___ High School ___ K-12 ___ 2. Is your school public or private? (Please check one) Public ___ Private ___ 3. Is your school a Chapter 1/Title 1 *school? (*school-wide, at least 50% of students qualify for free/reduced price lunch program) (Please check one) Yes

___ No ___ 4. Is your school’s location considered to be rural, suburban, or urban? (Please check one): Rural ___ Suburban ___ Urban ___

B. Student Information In the table below, please list each student that is participating in TECHShare lessons this school year. For each student, please indicate his / her gender, ethnicity (whether they are Hispanic/Latino or not), the race that they identify with, and whether or not they are considered by your school to be Limited English Proficient / an English Language Learner, and if they have an Individualized Education Plan (IEP). Please provide best estimates if exact information is not available. Do not include the same student in more than one Demographic Survey. Continue on the reverse side of survey.

Gender Ethnicity Race Special Needs Student’s Name M / F Hispanic Not

Hispanic Don’t know

Native Amer.

Asian AfricanAmer.

Cauc-asian

Asian Don’tknow

*LEP / ELL

IEP Don’tknow

Ex 1. Martin Ortega M X

Ex 2. Lucy Begay F X X X

*Please provide the definition your school uses for ‘LEP” at the bottom of the survey

Model Classroom Teacher Background Information 1. Your Name: ________________________________ 2. Date: ___________ 3. School: _________________________ 4. District: ____________________ 5. City, State and Zip: _____________________________________________ 6. Content area you teach: _________________________________________ 7. Grade level(s) you teach: _____________ 8. Number of years teaching: ____________ 9. Are you certified in the area you teach? Yes / No 10. Are you a member of an ETIP team? Yes / No 11. How do you describe yourself? Select one:

African American Asian White/Caucasian

Native American Hispanic Pacific Islander Other _________ 12. Access to computer at home (select one):

I do not have a computer at home I have a computer at home I have a computer at home with Internet access

13. For what purposes do you use a computer at home? Select all that apply:

I do not have a computer at home For personal uses To prepare school material To prepare lesson plans and/or tests To use email To search the Internet

14. How many computers do you have in your classroom? ___________ 15. How many of those computers have Internet access? ___________

16. . Stages of Adoption of Technology: Please read the six stages below and indicate your current level of technology use and integration in teaching by selecting the one statement that most closely matches your practice:

Stage 1 Awareness - I am aware that technology exists but have not used it, perhaps I'm even avoiding it.

Stage 2 Learning the Process - I am currently trying to learn the basics. I am sometimes frustrated using computers. I lack confidence when using computers.

Stage 3 Understanding and Applications - I am beginning to understand the process of using technology and can think of specific tasks in which it might be useful.

Stage 4 Familiarity and Confidence - I am gaining a sense of confidence in using the computer for specific tasks. I am starting to feel comfortable using the computer.

Stage 5 Adaptation to Other Contexts - I think about the computer as a tool to help me and am no longer concerned about it as technology. I can use it in many applications and as an instructional aid.

Stage 6 Creative Applications to New Contexts - I can apply what I know about technology in the classroom. I am able to use it as an instructional tool and integrate it as an instructional tool. I am able to integrate technology into the curriculum. 17. In general, how competent are your students at using computers for schoolwork other than drill and practice, such as creating a presentation or web searching? Select one answer that best describes them:

Not at all competent Not very competent, basically novice Fairly competent, with some assistance Competent, able to work alone mostly Very competent, they teach others

113

18. How often do your students use computers at school for schoolwork? Select one answer that most closely matches your students' use:

Daily About 2 to 3 times a Week About 2 to 3 times a Month Once a month Never or very rarely

19. What do you think technology can contribute to teaching and learning? ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ ________________________________________________________________ ________________________________________________________________ 20. Why did you choose to participate as a Model Classroom teacher? ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ ________________________________________________________________ ________________________________________________________________ 21. Any additional comments? ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ ________________________________________________________________ ________________________________________________________________ 22. Please provide a valid email address and phone number where you can be reached by the evaluators in case of need for clarification or additional information: ________________________________________________

114

TECHShare Model Classroom Journal

Directions: Thank you for taking the time to complete your Model Classroom journal. Please fill out the journal after you have finished each lesson in your unit. It is better to do it as soon as you finish the lesson while it is still fresh in your mind. Since you will receive a stipend for completion of the journal, it is necessary that you complete the journal on your off-contract time. Use the journal to record how each lesson was used, and whether you observed any changes as a result of using TECHShare lessons with your students. The information you provide is extremely important, as it will tell us about the quality and utility of these lessons as a resource for teachers like you. All information will be kept confidential, and will only be reported collectively: no individual will be identified. We greatly appreciate your thoughtful, detailed responses. 1. Name:_________________ Date:______ 2. School:____________________ 3. TECHShare Lesson Unit you are using (check one):

elementary math elementary science mid level math mid level language arts Navajo language lessons

4. Which TECHShare lesson would you like to journal about? ____________ ________________________________________________________________ 5. I taught this lesson in a lab in my classroom other __________ 6. How many computers did you use when teaching this lesson? _________ 7. Did you use a Smartboard or other presentation device (e.g., Proxima, In-focus, Averkey) to present the lesson to the whole class?

Yes No

8. If you used a Smartboard, did any of your students get to touch or work on the Smartboard during the lesson?

Yes No Did not use a Smartboard

Comments:

9. This lesson took ______________ (hours) (days) (weeks) to complete. 10. Did you use this lesson as a stand-alone lesson, or did you integrate it with other activities and materials of your own? Select one: stand-alone integrated with other activities 11. While using this lesson, did your students seem more engaged than usual (e.g., more time on task, more motivated, more responsible about completing assignments, more collaborative, etc.), less engaged (e.g., bored, frustrated, confused, lost), or about the same? Select one answer:

More engaged Less engaged About the same

Comments: 12. Did the Navajo cultural or linguistic elements of this lesson make it more engaging for your students?

Yes No Not Applicable

Comments: 13. If you answered ‘not applicable’ to question #12, do you think that adding elements of Navajo culture and language to the lesson would make a positive difference to your students’ learning?

Yes No Not Sure Not applicable

Comments:

116

14. In your opinion, did the lesson you used address the state standards and benchmarks it was supposed to address? If you answer ‘no’ or ‘not sure,’ please explain why not in the space provided.

Yes No Not sure

Comments: 15. What percentage of students were able to show mastery of the attached content standard after completing the lesson? Select one:

0% 1%-25% 26%-50% 51%-75% 76%-100%

16. What features of the lesson added to student engagement? Select all that apply:

computer technology animation photos sound video internet activities games using a Smartboard

Comments: 17. How did the lesson integrate with your planned curriculum? Select one:

Very well Needed some adaptation Needed a lot of adaptation Not well at all

18. Did you encounter any problems or barriers (e.g., technical, time allocation, implementation, system support, student interest, etc.) in using the TECHShare lessons? If you answer ‘yes’ please provide details below:

Yes No

Comments:

117

19. Finally, do you agree or disagree with the statement:

"I believe that any educator teaching in this field could integrate this lesson into their curriculum."

Select one answer. If you disagree, use the space below to explain why:

Strongly agree Agree Neutral Disagree Strongly Disagree

Thank you very much; we appreciate your time and input.

118

SAMPLE TESTS FOR MODEL CLASSROOM STUDY

Pre- Test for Parts of a Plant (Elementary Science unit) Please use a #2 pencil on the answer sheet provided

1. Which of these is not part of a plant?

A. root B. stem C. thorax D. leaf

2. Find the missing word in this statement: A plant’s roots take in water and _______ from the soil.

A. worms B. nutrients C. crops D. light

3. A carrot is an example of which of the following?

A. flower B. root C. stem D. leaf

4. If a scientist wanted to find out how tall a plant grows each day, the scientist would—

A. Give the plant a half-cup of water each day. B. Put the plant in a sunny place each day. C. Measure the plant with a ruler each day. D. Put the plant on a scale each day and weigh it.

5. Look closely at the diagram of a plant below.

The primary purpose of structure 3 is to—

A. absorb water B. make food C. protect flowers D. produce seeds

6. Which does a plant not need in order to grow?

A. light B. water C. soil D. milk

7. Choose the plant part which BEST answers the question. What part holds the plant in the soil?

A. roots B. leaves C. stems D. flowers

120

8. What will probably happen if the plant is turned around so that it faces the opposite direction?

A. It will stand up straight. B. Its leaves will fall off. C. It will bend back toward the sunlight. D. Its leaves will turn yellow.

9. Nine bean plants were grown in varying amounts of light. What conclusion can be drawn from the graph?

A. Bean plants grow best in low light. B. Bean plants grow best in high light. C. Bean plants grow best in moderate light. D. Bean plants grow the same in all light.

121

10. The picture below shows a flowering plant.

If all of the flowers are picked off the plant, the plant will not be able to—

A. grow taller B. produce seeds C. make its own food D. absorb nutrients from the soil

122

Angles, Radian and Degree Pretest (Middle School Math unit)

Please use a #2 pencil on the answer sheet provided

1.

2. Using a “best guess estimate” select the degree measure of the angle shown above.

A. 1250 B. 3000 C. 550 D. 350

3.

4.

. Two rays which meet at _____________endpoints is known as an angle. . common

relative different

D. opposite

5

AB. C.

124

. 6

7.

125

8. The picture above is an example of a(n) ______________ angle.

A. right B. straight C. obtuse D. acute

. The picture above is an example of a(n) ______________ angle.


9

10. The picture above is an example of a(n) ______________ angle.


126

Po r 3-D Cubes, Circles to Spheres, Identify Cones (Elementary Math unit)

1. Ho ces does this box have?

st- Test fo

w many fa

A. 10 faces B. 8 faces C. 6 faces D. 4 faces

2. Which of the shapes below is a sphere?

A. B.

C. D. 3. Which of the following has the same shape as a cone? A. An egg

B. A party hat C. A basketball D. A can of soup

Go on

4. In this figure, how many small cubes were put together to form the large cube?

A. 7 B. 8 C. 12 D. 24

A. sphere

B. C. cone D. cylinder

6. The squares in the figure below represent the faces of a cube, which has been cut along some edges and flattened. When the origin ube was face was on top?

5. A baseball is most like a—

cube

al c resting on face X, which

A. A B. B C. C D. D

128

7. Christina is making a game. She wants to put numbers on a cube like this one.

How many numbers can she use if she puts one number on each face of the cube?

C. 6 D. 8

8. Which of the following is shaped like a cone?

A. 4 B. 5

A. B.

C. D.

129

9. What solid figure would this pattern make if it were folded on the dashed lines?

A. Triangular prism

be prism

uare

A. square B. circle C. cone

B. CuC. Rectangular D. Sq

10. A sphere is a 3-dimensional—

D. triangle

130

Reading, Writing and Comprehending

___ eacher:_________________

Period:___________________ Score (Out of 100 possible) ______

OST PEOPLE OF MESA VER E by Elsa Marston

ed peacefully on the mesa for 800 years. Then they disappeared.

In the dry land of southwestern Colorado a beautiful plateau rises. It has so many ees that early Spanish explorers called it Mesa Verde, which means "green table." For bout eight hundred years Native Americans called the Anasazi lived on this mesa. And

then they left. Ever since the cliff houses were first discovered a hundred years ago,

ent ones." When they first settled there, ro the Anasazi lived in alcoves in the walls of the high canyons. Later

the level land on top, where they built houses of stone and mud mortar. As tim y constructed more elaborate houses, like apartment buildings, with sev living close together. The Anasazi made beautiful pottery, turquoise jewelry, fine sashes of woven hair, and

askets woven tightly enough to hold water. They lived by hunting and by growing corn nd squash. Their way of life went on peacefully for several hundred years. Then around 1200 A.D. something strange happened, for which the reasons are not uite clear. Most of the people moved from the level plateau back down into alcoves in

the cliffs. The move must have made their lives difficult because they had to climb back up to the plateau to do the farming. But it seems the Anasazi planned to stay in the canyon walls, for they soon filled the alcoves with amazing cliff dwellings. "Cliff

alace," the most famous of these, had more than two hundred rooms. For all the hard work that went into building these new homes, the Anasazi did not ve in them long. By 1300 A.D. the cliff dwellings were empty. Mesa Verde was

ained a ghost country for almost six hundred years. Were the people driven out of their homes by enemies? No sign of attack or fighting, or even the presence of other tribes, has been found. Archaeologists who have studied the place now believe there are other reasons. Mesa Verde, the beautiful green table, was no longer a good place to live. For one thing, in the second half of the thirteenth century there were long periods of cold, and very little rain

Pre-test (Middle School Language Arts unit) Name:___________________

ate:_________________DT

THE L D

The Anasazi liv

tra

scientists and historians have wondered why. Anasazi is a Navajo word meaning "the anci

a und 500 A. D.,y moved to thee passed, theeral families

ba q

P lideserted and rem

fell—or else it came at t from examining the wood used in the clif and bad growing

vived drought and bad weather before, so there must have

re land on the mesa top had to be farmed in order to feed had to be cut to clear the land and also to use for houses

nd fuel. Without the forests, the rain began to wash away the mesa top. roblems that happened about eight hundred years

ago? The Anasazi built many low dams across the smaller valleys on the mesa to slow own rain runoff. Even so, good soil washed away, and the people could no longer raise

enough food. As the forests dwindled, the animals, already over–hunted, left the mesa for mountainous areas with more trees. And as the mesa "wore out," so did the people. It appears that the Anasazi were not healthy. Scientists can l tudying the bones and teeth found in burials. The mesa dwe and their teeth were worn down by the grit in corn meal, a main part of their diet. As food beca heir twenties. Women died very young, and few so close together in the cliff houses, where everyone was hungry and worried, the people must have suffered from

s

orgotten.

at

It is es of America's past.

he wrong time of year. Scientists know thisf dwellings. The growth rings in trees show good

seasons. But the people had surbeen another reason. As the population grew, mothe people. That meant that treesa How do we know about erosion p

d

earn a lot about ancient people's health by sllers had arthritis,

me scarce, people grew weaker. Not many lived beyond t babies survived. Living

emotional strain. They probably quarreled often. In the end the Anasazi must have given up hope that things would get better. Familiepacked up and went away. Of course, the "ancient ones" did not simply disappear. They moved southeast to another area and mingled with other peoples. After a while their heritage as the people of the Mesa Verde was f In time the trees grew back and the plateau became green once more. But, for the Anasazi it was too late. Although they respected nature and tried to farm wisely, land thwas used too hard could not support them forever. Yet in their cliff houses and crafts the "ancient ones" left us a superb monument. truly one of the most fascinating pictur

1. After reading this article, what do you think is the most important information about the Anasazi?

132

2. There were three major moves made by the Anasazi. Below is listed one of those moves. Explain the possible reasons that were suggested in the article for this move.

500-1200 A.D. - The Anasazi moved from the alcoves to the top of Mesa Verde.

3. If you had lived with the Anasazi at Mesa Verde, would you have preferred living on the top of the mesa or in the cliff houses built into the alcoves? Explain your preference by using information from the article.

e article? Explain why you would

4. If you could talk to the author of this article, what is one question you could ask her about the Anasazi that is not already answered in thwant to know this information.

133

5. Which idea from the text about the Anasazi do the photographs above support? (Circle most correct answer)

A) They were able to create many useful objects. B) Farming was probably their major source of food. C) Wood seems to have been their primary building material. D) Their life became much easier when they moved into the cliff dwellings.

134

6. Imagine that you are living with the people of Mesa Verde during the 1200's when they

left the mesa. Some of your friends and neighbors do not want to leave the area. Based on information in the article, what would you tell these people to convince them to leave?

7. The Anasazi's life before 1200 A.D. was portrayed by the author as being (Circle most correct answer)

A) dangerous and warlike B) busy and exciting C) difficult and dreary D) productive and peaceful

8. The title and photograph on the first page of the article are probably meant to make the disappearance of the Anasazi seem to be: (Circle most correct answer)

A) a personal tragedy B) a terrible mistake C) an unsolved mystery D) an important political event

9. Some people say that the Anasazi's success as a civilization may have actually caused their own decline. Using information in the article, explain why you agree or disagree with this statement.

135

10. There were three major moves made by the Anasazi. Below is listed one of those

v 1300

mo es. Explain the possible reasons that were suggested in the article for this move.

A.D. - The Anasazi left Mesa Verde.

136

Appendix B

Curriculum Preparation Studio Surveys A & B

TECHShare Project Year 5 Curriculum Preparation Studio Survey (A)

To assis is brief surv fy issues that need to be addressed. Please elaborate on your answers wherever possible. All responses will be reported collectively; no one will be identified. Thank you very much for your participation. For items requiring a Yes or No response, please type an ‘X’ in the appropriate space. For open-ended items, type in your response, using as much space as you like. Please return your survey as an email attachment to [email protected] or fax it to: (580) 353-2168.

Please respond by December 18, 2003.

1. Name: _________________________ 2. Organization: ______________________ 3. What are your studio’s objectives for Year 5? (e.g., Number of new modules you expect to complete, number of lesson revisions/modifications you plan to undertake, cultural component additions, narrations, or any other activities you plan to engage in for TECHShare.) 4. Has the feedback you received from lesson evaluations conducted during Year Four been useful to your studio? ___ No ___ Yes ___ Not Sure 5. a) Please describe how your studio is using information from the reviews to inform revision and new lesson development: 5. b) Any suggestions for how the feedback process could be improved?

t with monitoring the progress of project implementation, please answer they. Your feedback will be used to keep the project on track and to identi

CEER (580) 355-5246 x.105 1318 SW Lee Blvd., Lawton OK 73501 ________________________________________________________________________________________________

6. a) A Curriculum Development Specialist (i.e. Leni Rivera, Karina Roessel, Nate Southerland & Moni Short) has been assigned to each of the studios. Please describe how your studio is working with the Curriculum Development Specialist

. What successes o has your studio xperienced meeting project goals? i.e., TECHShare project management, NETC oard policies and procedures, local TECHShare management, other rganizational areas:

. Any additional comments or suggestions:

hank you very much. If you have any questions or concerns, please call Rebecca Zittle at (580) 355-5246 x.105 or email: [email protected]

this year: b) What are your expectations for this collaboration? 7 r challenges related to project organizationeBo -Comments or suggestions?

8 T

139

TECHShare Project Year 5 Curriculum Preparation Studio Survey (B) FINAL

T

o assist with improving project implementation, please answer this brief survey. Your edback will be used to identify issues pertinent to the implementation and anagement of large projects such as TECHShare.

n your response, using as your responses, citing

oncrete examples wherever possible. All responses will be reported collectively; o one will be identified. Your participation is appreciated.

. Name: _________________________ 2. Organization: ______________________

ork for the TECHShare project has progressed at our organization this year. Be sure to describe any changes in processes that ave produced positive results or any notable accomplishments you would like to ighlight.

. a) A Content Development Specialist (Leni Rivera, Karina Roessel, Nate outherland & Moni Short) was assigned to work with each of the partners. lease describe how your studio/organization collaborated with the Content evelopment Specialist(s) this year:

b) Do you have any suggestions for improving this type of collaboration?

fem All of the items below are open-ended items; please type imuch space as you like. Please be as specific as possible in cn Please return your survey as an email attachment to [email protected] or fax it to: (580) 353-2168.

Please respond by May 21, 2004.

1 3. Please describe how your wyhh

4SPD

5. What successes or challenges related to project organization has your organization experienced meeting project goals? i.e., TECHShare project manamana

6. Reflecting on your experience with project implementation and management at

essons learned’ that you would share with others embarking on a similar roject? –What would you do the same, and what would you do differently?

7. Any other comments or suggestions?

hank you very much. If you have any questions or concerns, please call Rebecca Zittle at (580) 355-5246 ext.105 or email: [email protected]

gement, NETC Board policies and procedures, local TECHShare gement, other organizational areas:

-What are your recommendations?

your organization over the last five years, what do you consider the most valuable ‘lp

T

141

Appendix C

NETtrain Web Site Evaluation Checklist

NETtrain Website Evaluation Checklist Instructions: Please provide information on your name and position below. Then, for each component of the website listed, tell us whether you think it is very good, satisfactory, or needs improvement. If you think something needs improving, please provide specific suggestions in the comment box next to that item. Name: _________________________ Position (select one): Teacher, Administrator., Parent, Developer, Other ___________

Very

Good Satis- factory

Needs Improve-

ment

Please specify changes or additions:

Look and Feel: The site is interesting.

Its purpose is clear and easy to understand. Site Content: Is useful for teaching and learning.

Contains practical information on the integration of Navajo culture into instruction.

Has links to other sources that are useful for teaching and learning. Has links to alternative pages/sources for English learners. Site Sources: It is clear that the site is hosted by the Navajo Education Technology Consortium (NETC).

Information is provided that makes it easy to contact the NETC. Information is up-to-date (time-sensitive material is current). It is clear how the privacy of users will be handled. Navigation and Manageability: There is a site index or map that is easy to locate and use.

All links are clearly labeled and easy to understand. It is easy to find my way around the site. Accommodation is made for persons with disabilities. Instructions for downloading plug-ins are clear.

For Educators: Pages load fast enough to be practical for use in

the classroom.

Adequate support is provided for me to use the site to supplement instruction.

The ‘threaded discussion’ feature for each lesson is useful. Provides useful strategies for addressing a diverse student population.

Su port features such as the tutorials, calendar, online resources, instructions, etc. are helpful to me.

p

Provides links to useful articles on the theory and effectiveness of online learning.

144

Appendix D

Facsimile of Online Lesson Rubric

and Summary Comments

Rubric for the Eval HShare Lessons Lesson ID#: ____

REVIEWER’S NAME: IF YOU WORK AT A SCHOOL, PLEASE IDEN ______________________ DO YOU HAVE SPEAKERS OR HEADPHONES ATTACHED TO YOUR COMPUTER AND TURNED ON? ___ YES ___ NO WHAT WEB BROWSER ARE YOU CURRENTLY USING? (SELECT ONE): ___ NETSCAPE ___ INTERNET EXPLORER ___ OTHER What type of Internet connection did you use to view the lesson? (select one): ___ T1-T3 line ___Cable Modem/DSL ___ ISDN ___ 28.8 to 56K modem ___ 14.4K modem or slower ___ Don’t know

HOW MANY STUDENTS HAVE YOU USED THIS LESSON WITH? ______ DOES THIS LESSON HAVE A TEACHER GUIDE? ___ YES ___ NO DOES THIS LESSON HAVE AUDIBLE NARRATION OF THE WRITTEN TEXT? ___ YES ___ NO ___ YES, BUT NOT COMPLETE (PLEASE EXPLAIN):

For categories 1-3 below, please rate the lesson on a 5-point scale, with ‘1 = Not Evident’ being the lowest rating and ‘5 = Clearly Evident’ being the highest rating. Put a check mark in the appropriate column:

1. Lesson Mechanics

uation of TEC

___ Lesson Title: __________________________________

_________________________________________________

TIFY THE SCHOOL & DISTRICT: _______________

1

Not Evid-ent

2 3 4 5

Clearly Evident

1.1. Lesson loads within an acceptable time period .......................................... 1.2. Graphics and text are appropriate and clear .............................................. 1.3. Spelling is correct throughout the lesson.................................................... 1.4. Grammar throughout the lesson is appropriate and correct .......................

1.5. Lesson navigation is understandable, consistent, and age appropriate.....

1.6. All lesson links are identifiable and work

1.7. Lesson uses clear and easy to follow directions and instructions ..............

1.8 If you responded with less than a '5' to any item in the Lesson Mechanics category, descri ed or cha

be which items were not "clearly evident" and what you think needs to be improvnged. Please be specific about such things as spelling errors, broken links, bad

grammar, etc. that you notice:

2. LESSON INSTRUCTIONAL DESIGN

1

Not Evid-ent

2 3 4 5

Clearly Evident

2.1. Lesson standards and benchmarks are clear..............................................

2.2. Specific lesson objectives are in evidence ..................................................

2.3. Lesson objectives are clearly related to stated standards and benchmarks..

2.4. Lesson contains instructions to the learner that clearly state objectives.....

2.5. Lesson includes information on prerequisite knowledge or skills................

2.6. Lesson key points are supported with appropriate media ...........................

2.7. Engages learners through the use of new materials, or reinforces ……………………. concepts previously taught ……

2.8. Demonstrates desired skills or process for learners to master ...................

ent types of learners .......... 2.9. Lesson uses various activities to engage differ

……2.10 Lesson tasks are related to stated objectives…………… …………

2.11 Feedback is consistently provided to learners during instructional tasks………….

2.12 Learners are provided with formative assessments as they are engaged in learning tasks…………………………………………

2.13 Le rmine learning outcomes………………………………………

arners are provided with a summative assessment to dete

2 uthentic and interwoven within lesson……………………………..

.14 Assessment is a

2.15 If you responded with less than a ‘5’ to any item in the Instructional Desig tegory, please

eds to be improved

3.

Cultura for English Language Learners

n cadescribe clearly which items were not "clearly evident" and what you think neor changed:

lly responsive curriculum and Support

1

Not Evid-ent

2 3 4 5

Clearly Evident

3.1. Presents content with a base in the local knowledge system, including cultural beliefs, values and practices, and the ecology and geography

of the Dine’ (Navajo) environment ..............................................................

3.2. Incorporates elements of contemporary life in Dine’ communities, along with historical and traditional aspects of Dine’ culture................................

3.3. Lesson incorporates community knowledge base into curriculum .............

3.4. Provides good support for learners with limited English proficiency by including sound, read aloud text, color images, or other devices...............

3.5 If you responded with less than a '5' to any item in the Culturally Responsi urriculum category, pleadescribe clearly which items were not "clearly evident" and any changes or improvements that you would like to e

a

this lesson into their curriculum. e t g

Disagree Strongly Disagree

stated objective

Strongly Disagree

. S

ve C se

s e:

R ting Summary

verall, I believe that any educator teaching within this field could integrate 4. OS lect one:

S rongly Agree A ree

Neutral

5. OS

verall, I believe that this lesson would significantly aid the student in masterilect one:

ng the e(s).

Strongly Agree Agree Neutral Disagree

6 ummary Review Comments:

Summary Comments from Year 5 Reviews of TECHShare Lessons Using the

==Overall the lesson is very good. It will su ited English.

= r

==he story was fun and short. Could use this story to do sequencing, such as what

ral aspects needs to be included. =======================================

g and culturally sound! =======================================

==================================== , easy to relate to!

================================== l done!

=========================== ERY well done!

liant in every way! Couldn't get enough of it! This is learning at its

================================== well done. Immense detail.

=========================== It was a nice lesson, but needs to have a culturally relevant to the Dine culture.

===================== Great lesson. ======================================== From what I did access on this lesson site I think that students would be engaged in the history and biography of TC Cannon. ======================================== I believe that this lesson is one of the most relevant lesson for the children. It just needs to be developed some more to make the students understand why this is one of our sacred mountains. ======================================== I really like the way this lesson is put together. Even though at the beginning it is less attractive, but as you go into the lesson itself, it's quite interesting. It has a nice background to just like the ocean. ======================================== Overall, this was a good lesson except for the Beauty way part needs to be changed. ======================================== Very enjoyable...would have liked more graphics during haiku creation, words of different syllables in different colors, or tones accompanying choices...

Online Rubric ======================================

rely help the students with lim= ====================================== I eally like the lesson. The students will love this!

====================================== Thappened first, next, and last because it’s easy to listen to and short. ========================================

ultuC=Entertainin=Enjoyable and easy to relate to! ====All too realistic======Very wel=============V======================================== Absolutely bril

! finest!!======Extremely=============

===================

===I really enjoyed the activities. You di b.

mputer. Children young As teachers we should be reading

o. When computers had to uld get

attached to the lessons. That was great and many

===============

================

===============

=========================== s!

==========================

story read out loud

ere great. Just want to mention that I have to go into each lesson to get the n" page.

l be nice to see.

uld be an educational and enjoyable endeavor.

ry.

an teach our children to

===================================== d an excellent jo

======================================== I really like the pictures of the monkey. I enjoyed the story, but I honestly think this lesson would be better read from an actual book than a coenough to enjoy this story also enjoy being read to. with our students so we can ask questions and talk as we gread to our children for us I think that's when they become babysitters. A child womuch more out of this story if it was read to them by a real person. ======================================== I really enjoyed the other sites that are ideas. Thank you =========================I would recommend this lesson to another teacher. ========================This lesson was a good lesson. I liked it. =========================This is a good lesson. =============This lesson is a great example of solving real life problem==============It tells of how to do a acrostic ======================================== Have the ======================================== Balii is the little girl’s dog. ======================================== The story is about rusty and Miil ======================================== The lessons wgrade level. I think it should be listed on the "My Nettrai======================================== Again, Some sort of Navajo stories wil======================================== Excellent culturally relevant lesson/activities. Just takes too long to load each section. ======================================== I think it wo======================================== Very creative math lesson for Spanish speakers/readers. ======================================== Very enjoyable, well-illustrated sto======================================== I liked the lesson because it talks about our history and how the Navajo Code Talkershelped win the war by using our Navajo language. This cappreciate their language and speak it. ========================================

150

I liked the lesson and the activities that went with it. They are colorful and fun to work

e I find

ngaged throughout.

t just apart of history but of science so it can be used in many ways.

ails. I thought the dictionary was very ill have difficulty pronouncing

ve the words pronounced audibly. This would ematic unit.

======================== age appropriate. It was fun and

an understanding of the was simple and fun!!

================== is can help them to later

es later in life.

ecially for Navajo students.

dren and grandparents to communicate and caring on

rn to make corn meal ch ways to use the cornmeal.

ajo to describe many of the

ar in description and directions

tudent with different cultural back ground with the

rbiage ed to hear the vocabulary of that field. Otherwise

on, and to listen to. ======================================== I feel that children enjoy doing art and taking on a collage approach allows them to usother resources such as magazines or their personal pictures and do their own work.that children would prefer to use stickers, cut out magazines and pictures versus havingthem to draw it themselves, eventually this will lead them to draw on their own. ======================================== I really enjoyed this lesson. I believe that it will keep the students e======================================== I felt the lesson was put together very well. The lesson plan is grade appropriate and interesting. It is no======================================== I was impressed with the lesson and the graphic dethelpful. However, students who are not native speakers wthe words correctly. A suggestion might haalso be helpful to Anglo teachers using this lesson for a th================Overall I thought this was a great lesson. It was grade andthere were several activities that the students could do to get concepts. It was great. I would use this within my class. It======================This lesson would aid students in learning root words. Thidentify and use word strategi======================================== Very informative lesson, esp======================================== This lesson is a good base for chiltheir relationship. ======================================== I would recommend this to an educator or anyone who wants to leabut I think it needs to be descriptive and tell the ways whi========================================This lesson is too much information and needs to use Navterms. This lesson would be good to use for an end culminating activity to share and an end assessment tool. ======================================== I liked this lesson very interesting and enjoyable. Very clein both Navajo and English. Good graphics. ======================================== I believe this lesson plan would assist sdifferent types of restaurants available in this lesson plan.======================================== It is a good animated presentation for the kids today, a little too much high level vebut it is a scientific concept and kids neit is a good short lesson on cells.

151

======================================== This lesson is better suited for upper grade to high school level.

ds. The only problem I had and

d said it was hard but also that

I encountered is e.

o are focusing on the topic,

======== ent. I enjoyed the story myself very good for the students to

informational story because I learned some Navajo terms myself. I am Otherwise, great information for

ay have, its prevention, etc.

s in Navajo is a good learning experience for them. Not all children

nd earning something they

anning the lesson. The song is

======================================== The lesson is good because you can actually view the image changing from a 2- to a 3-dimensional image. I'm a visual learner and think that students who are visual learners will appreciate it. ======================================== The information provided is appropriate for mid school kithat could be just my computer is it took time to load the book. ======================================== It is a good lesson. My niece, 8 years old tried, and thought it was a good lesson. ======================================== I had my nephew do the lesson first. He is in 8th grade anit was very interesting. He liked the lesson. I enjoyed, like he did. going to the links and learning more. ======================================== This is a very involved interactive lesson on pulleys. The only problemloading the lesson pages. It's probably my connection her======================================== GOOD PHOTOS!!! ======================================== I would recommend this book to teachers and students whGrandparent Day. ================================I thought this story is very relevant to our home environmas my listeners did. Writing page numbers in this story issee. One of my students has noticed it. ======================================== Overall, I like this just wondering what the Standards and Benchmarks are. all of us in my classroom, especially for my Dine Club students. ======================================== This lesson will give students of what sort of disease a rat mThe story has good drawing which are relevant with the story. ======================================== Introducing food itemwill learn these terms at home. ======================================== Some of my students listen to this story and sort of get an idea that this young boy, Treii was lazy. I constantly remind them of being responsible awant by giving time back to whoever they get things from, ex. mom, dad. ======================================== I like this lesson and the Teacher Guide, it is helpful in plbeautiful as well. ========================================

152

This is a cute story of a goat and teaches how to take care of oneself with proper

integrate listening and reading.

o

hem laugh. Except the illustrations need to be redone with funnier all age levels

re figurative language in the This was not appropriate for

dents

ts HShare, though they did have some trouble with the "language" on

d not seem to have been

===========

elped them a lot.

icate and that they

about the animals that we know as pets and food.

h for evention.

n’t relate a sphere to their Navajo Culture.

grooming. ======================================== This is a very good lesson for students in a Navajo as a second language class. ======================================== Very good story to======================================== MY favorite lesson so far. TC Cannon is my favorite Native artist, so I was pleasantly surprised to find this lesson! Wow! Kudos to the creators! ======================================== We need more books on Narbona, Manuelito, Peter MacDonald, great Miss NavajNations, etc ======================================== This book my students' favorite. They learned to read it and felt confident in reading aloud. It made tpictures. We need more books at this level appropriate at======================================== Make future readers read with enthusiastic voices and put mostories. Use cartoonish drawings to make students laugh.my 13-14 year old stu======================================== As a model class activity, I felt the students demonstrated understanding of the concepbetter with the TECthe tests. Also, some questions on the TECHShare test dicovered enough in the lesson that they knew the answer. Specifically, # 10? I don't remember that one being mentioned, but maybe we all missed it... =============================This lesson was fun for the students. ======================================== The students enjoyed the lesson and was rather simple for the fifth grade students. My LD students were able to handle the directions and audio/visuals h======================================== Overall, the lesson was great. My students really enjoyed the lesson and we learned a lot about sheep dogs. They were surprise of how dogs can communrealized that dogs are important to take care of. ======================================== Enjoyed the lesson and learning======================================== This lesson (story) is short and clear. For the 1st grade level I would use this lesson for health awareness of Hantavirus. For 2nd and 3rd I would use this lesson for health awareness and disease prevention. I would also use this lesson for 4th to 6tcomprehension related to health promotion and disease pr======================================== The students could========================================

153

It was a bit difficult to hear the community speakers. Overall the lesson makes onof the effec

e aware ts of a forest fire and how to prevent them. The tasking is engaging for all. I

forward to going to the lab and

e nd improve their lives by

ent well. Just a little problem with understanding part of the story. Where Otherwise all was great.

ture, , I wasn't sure

t e slightly different in content, but it is very confusing.

.

ause the story was so short for my students and we

eir like them in the morning. I

be helpful to visual learners.

this

is bored

t illustrations.

appreciated the additional resources (websites). ======================================== Of all the lessons I have done with the students, they lookworking on the lessons. ======================================== Very good example of how children actually react in the real world. A lot of feelings arexpressed. This could be used to teach them not to be lazy alistening to what they are told to do. ======================================== The lesson wthe story talks about the corn or the chicken having teeth. ======================================== Several students asked about the turtle...they felt they had seen turtles in Navajo Culbut they weren't sure of its relevance to Moon Phases. Not being Navajoeither. ======================================== I'm not sure what the difference is between this lesson and the one marked Day and NighK-2. They seem to be mayb======================================== Through these teachings, she will learn about the importance of a corn in her culture. ======================================== The lesson is great, but needs improvement on navigating the links easier. ======================================== I thought this lesson was great. My class really enjoyed this lesson======================================== The lesson was an average lesson becwent through it so quickly. ======================================== The lesson overall was great the students enjoyed reading it and it really related to thown knowledge. Then we were also saying that it sounds really like Garrity's lesson and she is a great friend of mine. ======================================== The content of this lesson is well done. It would======================================== The only thing I would add would be examples of a story at each stage. ======================================== I really liked the graphics in this lesson, it just seems to be a little of everything-in a badway. If the science concepts were really reinforced in the experiment and the story lesson would be great. Also the story is read way too fast like the person readingand just wants to be finished. ======================================== A good lesson for all young children because its short and grea========================================

154

This story was well written and relates to students in school growing up with their grandparents. I thought it had humor and the values of the Dine message was very clear. Great story! ======================================== Overall this lesson is a great way to teach our children the geometric shape and the illustrations and directions were simple for this grade level.

ped

es, more development and a better way to flow through the lesson.

ss is lesson only gave a definition

==================== e students master one lesson

er and her son. However, the +.

to use this with my students

mary, deciphering codes are all great ideas that you can is fieldtrips, visits, and

dents and n to our country.

========================================

======================================== This lesson needs to have more substance. Even for an introduction it is weak. It needs more examples and student involvement. The graphics are very good but the lesson is not developed. ======================================== This is such a new concept to students even in high school that this lesson needs much more to it. More examples, more things for students to interact with. It is not develoenough to be of use to a student except as a review. ======================================== I love the graphics they just do not teach much in this lesson. Students need more exampl======================================== I will use this lesson in the future as the first day in teaching angle relationships. The one thing that was not useful is the part on transversals. They are useful only if they croparallel lines. I had to teach that part after the lesson so thof transversal but did not show the students how useful they can be. ======================================== These are some things I liked about the lesson: Video lesson guide is easy to follow materials (flag) using sources from the internet ====================I liked the idea of the lesson in two sections. This will help that a time. ======================================== This story is a good story for conversation between the mothage appropriate could be used for older students' such as 3======================================== Good rainforest information and site links. I may be ablewhen we study rainforest animals. ======================================== I feel that this lesson is important in so many areas. The teacher has a lot to cover. The amount of time allocated for this lesson can be adjusted so that the students will be able to get more information. Researching, using timelines, sequencing, using the cause/effect relationships and writing a sumuse to integrate into this lesson. Another important point interviews by the codetalkers. I feel that this lesson is very valuable to our stuthey should be made aware of the Codetalkers contributio

155

The grade level for this lesson is high. I feel that the lesson should be brought downthe age appropriate level so the students will understand the life cycle of a butterfly. Using additio

to

nal resources such as: "The very hungry caterpillar" and/or puppets would is level. Dittoes are not helpful

ided ow the life cycle works.

ds some follow up and teacher's guide.

have done. This is very nicely done.

king of fire.

eds a little bit of

====== explains a typical young girls life in the morning as she goes to school. The

son very well. ============================

dents a lot about their leaders. We also did an extension students really enjoyed that

wrote one of the stories too, and I see that most of classmates stories are

ould have liked.

count to 5 then there is a serious problem. You're this lesson.

the sound.

also be very useful. Students need hands on materials at thfor visua or auditory learners. Although a quiz and a cut and paste worksheet is provI feel that the students need to get a better understanding of h======================================== Great concept. Needs some follow-up activities. ======================================== Good story. Nee======================================== Good story. Needs some follow up activities to check for comprehension. ======================================== I was impress with the animation of all the work youWe need more materials like this! ======================================== Very impressive and wonderful. The students enjoyed the watching the ma======================================== With out the sound, there was not much to do with the lesson. The sound would really help. Perhaps if the words are highlighted as it reads could be very good. ======================================== This is very well done, good humor and good sound. The grammar nework, it has repetitive words. ==================================Lessonstudents could relate to this les============Overall, the lesson taught my stuof learning about more leaders in our Navajo History. Mylesson. Thank you very much for putting these lessons together for us to teaching in the classroom. Ibeing used too. Hagoone' ======================================== I like the lesson however, in some areas weren't quite what I w======================================== Again, for a five year old this lesson would be ok, but for someone in third or fourth grade, if they can't spell lady bug oraverage 3rd or 4th grader should be past the objectives in ======================================== I'm not sure if the Navajo culture was incorporated in the lesson. ======================================== As a quilter, I can see numerous ways this could be used in the curriculum! ======================================== I really enjoyed this lesson. My class will be studying plants next week and I could see where they would enjoy this. I wished that it was a little quicker to boot up ========================================

156

I am impressed that a balanced view and opposing viewpoints were included in this lesson. Students need to see a wider picture and not just take the role of victims of their

====

and sound effects. Great

=============== ents' talent and need to create

m. However, I was unable to because each section did not

a lot from this lesson. The activity on

t.

PREVENTION e taken into consideration while

jo

past. ======================================== This was a very informative and interesting lesson. I think it would help students of all cultures. I especially liked the multi-cultural aspect with the pictures of various cradleboards from different tribes. ====================================The game was the most engaging part. ========================================Brilliant lesson, very informative, Great links! ======================================== This lesson was well written and fun as far as the picturesinformation on the 4 sacred mountains =========================I really like the "talk" section, because it brings in the studsomething artistic. ======================================== I particularly enjoyed the extension activities with this lesson. They gave some great ideas for reaching every type of learner. ======================================== It would have been nice if I was able to see all the sections of the lessons that way I would have commented on theload even after making several attempts. ======================================== This weather lesson is a great lesson. I learnedmaking a barometer was neat too. ======================================== Making patterns Describing patterns Extending patterns Recognizing patterns Creating patterns These are all excellent skills for children of all ages to master. ======================================== I think this lesson was an important lesson that all students should read and learn abouThe only thing missing is cultural relevancy for our Navajo students. This lesson needs some Dine' language, especially for the vocabulary words. ======================================== This lesson is very valuable for our students not only during FIREWEEK. I feel that all the activities and websites should bintegrating this lesson into your curriculum. ======================================== I think our young children will really enjoy this lesson because its easy to understand andthe illustrations are something they will all relate to, even though there is not any Navaculture in it.

157

======================================== Some of the activities that were able to be loaded had valuable information for the students to know.

l. Students can dye wool using

be of use when they complete it. This will allow e long ago. Sequencing and

tudents realize what the Navajos had to endure. other memorabilia's is also

I was

strations.

g

K because the audio is slow and I could not ery slow.

tain things we do at certain time of the day.

ow the lesson assignments are broken up in introduction, vocabulary, re parts and the presentation. This gives students a chance to explore

of learning. The pictures of this lesson are simple and easy to read

=========================== adleboards is an excellent

d

eas

======== d explain in

esson in my class.

Using hands on materials for visual learners is very usefuherbs and plants and make a 'cardboard rug' in class. ======================================== Students will find their "journals" tothem to investigate the hardship their people had to endurlocating areas on the map will help the sA field trip to an archive to view the photos, clothing, andhighly recommended. ======================================== The lesson sounded very interesting when I read the lesson title as the standards butunable to download it and view it. ======================================== I believe I am repeating myself. But the lesson needs to add more emotions, humor, feelings, and colorful illu======================================== Good work but the illustration needs to promote Navajo healthy eating. A Navajo eatinyogurt, Salad, Drinking milk, etc... ======================================== Nothing showing on this lesson therefore no comment. ======================================== Lesson is a bit too complicated for Grade understand what it was saying. Load the lesson was also v======================================== Lesson could elaborate with on cer======================================== I like the part of hacitivity, create pictuin a variety of ways and fun to learn, too. =============Getting Ready: Showing of the geographic area of the tribal cridea! ======================================== This lesson was very appropriate and demonstrates a lot of connection between home anacademics through state standards. ======================================== This lesson is very interesting not only for regular class room but also for special arsuch as Navajo culture class like I am teaching, I can use this in my classroom. ================================Overall, the lesson is good. The only thing is to put some Navajo words ansome parts in the Navajo language. ======================================== I can do this l

158

======================================== I liked the lesson because it had a lot of activities for students to do, vocabulary wowere there for stud

rds ents to find if they don't know a meaning of the terms, it had other

== ou for that information.

======================= are of because you never

ts to learn more about these disasters.

or reptiles.

s

is lesson to see if it will appear. I am really interested in the lesson.

=================================== narration. Just took too long to

n. Need more lessons like this.

t any

ng, communication, spelling, reading, and math ter completing this

subject areas that were integrated into this lesson which was nice. ======================================This is the first time I heard about Smart Board. Thank y======================================== Very good story and our youth need this lesson. ======================================== Nizhoni =================I thought this lesson is appropriate for all students to be awknow when something (natural disasters) might happen or where. The lesson had a lot of other resources/websites for studen======================================== Maybe to break up the lessons into segments-now is a big time span-this is an on-linelesson. ======================================== Overall the lesson is good, but need to be more careful on what you have your students do, especially learning about insects======================================== overall, very good, maybe can be used at middle school.======================================== I think my children would enjoy learning about cradleboards and they would greatly improve their fine motor and eye hand coordination ======================================== I think can be revised and used effectively then using several examples ======================================== I like that the lesson beginning with a short video to grab the students' attention. It makeyou want to explore and curiosity usually drives today's youth to learn something new. ======================================== I will keep opening th======================================== I like this lesson, very visual and very simple to do-self directed =====Great, fun lesson on congruent rectangles. Cute and goodopen pages. ======================================== This lesson included all standards, NM, AZ, Navajo Natio======================================== This lesson of sharing and living with others is clear and simple. It is a story thachild could relate to. It is fun and funny, too. ======================================== The lesson integrates language, writi(graphing). I feel that the students will take pride in themselves aflesson.

159

======================================== I thought it was nice to have the glossary with this lesson. Overall, the lesson was prettycute. ========================================

d speaker/reader of Navajo.

===== very neat. I especially enjoyed

ties.

re Even the video worked fine.

=====

es.

promote Navajo literacy. s were incorporated

ries were great.

I think it would be a very enjoyable lesson for an advance======================================== Really enjoyed it. Nice illustrations, fun story. ===================================I thought this was a good lesson. The video was short butthe activi======================================== Nice animation! ======================================== Wanted more! ======================================== I really enjoyed this lesson because every activities and sound worked. The pictures weperfect. ======================================== Again, can't really evaluate! ===================================Very well done! ======================================== enjoyable! ======================================== good summary of self to others ======================================== Cute lesson. Wish there was narration and maybe some Navajo referenc======================================== Very well done!!! ======================================== Great lesson if the audible narration was clear. ======================================== I like these stories because they model pronunciation and However, they would be more educational if questions or activitiewith them. ======================================== Great job!! ======================================== This could be a good basis for a lesson and the teacher guide helps, but it is not an interactive, complete presentation by itself. ======================================== I enjoyed this lesson. The photos and oral sto======================================== Great Navajo role model. ========================================

160

I really enjoyed the lesson. I had four of my students try the lesson and they really liked

===================================

ation about the author was

nt comprehension.

informative, fun activities to engage the students.

ative lesson. Peppy! Wonderful real photos.

nts talk about their cultural and family background not just

===========================

=============================== for students.

================= o do, good illustrations and

ivities and links.

aphic design, good photos.

tation- Nice origami link!

the presentation as it graphically explains multiplication in ways we older

Song".

don't know whether we ing.

===================== believe will be ok to teach.

However, If I have Dine students in my class I would not teach the lesson or I may

it. =====Awesome!!!!! ======================================== I am not a fan of Tony Hillerman but the books and informprofessionally done. ======================================== Wonderful lesson. Clear definitions and many specific examples. Activities are fun and do assess stude======================================== Beautiful lesson, ======================================== Exciting, inform======================================== Good idea to use visuals, vocal and manipulating objects for students to grasp the concepts. ======================================== I like the idea of having studefor Native Americans students. =============Nice story, nice illustrations. =========Nice story, very nice drawings...should be very enjoyable=======================Very complete and well-linked lesson! Lots for students tmovies! ======================================== Good variety of act======================================== Nice bilingual narration, nice activities, good gr======================================== Especially attractive graphics and cute animation. Love the cookie activity!!! ======================================== Nice presen======================================== Really likedfolks didn't get! ======================================== Well done! Especially liked the "Long Walk======================================== The content is pretty good although the video didn't work so I actually learned to make tea...all those utensils are confus===================This lesson in to teach ants through experiments, which I

161

briefly look at ants in general through pictures, but not in depth. If the majority of students were Dine, I would not teach this any ant lesson.

rward to seeing some actual photos of the mountains and people living

le and basic, but could be more in depth for third and fourth grade

to follow. It teaches students to tell time. The pictures s lesson should tell time in the

sequence the story.

on materials and internet sources are helpful. The activities will help the

ual learners is very important. the students have mastered the

really reminded

n has good graphics but not enough examples for students to master the all and the additional

uter.

.

============================

== ol math.

n students in the

es)

======================================== I was looking fonear the mountains. ======================================== This lesson was simpstudents in the Dine culture and the Dine Language. ======================================== This lesson is very simple and easyshould relate to Dine children. However, I believe that thiDine language as well. Great lesson. ======================================== Lesson could be incorporated with an animal unit. Students will surely enjoy the story and maybe they could======================================== The hands students better understand the concepts. ======================================== Using hands on materials for visFormative and Summative Assessments are used to see ifobjectives. ======================================== This lesson is very interesting and my students really loved the story. It them of one of the student in the classroom. ======================================== This lessoconcepts or even understand what it is. The text is too smworksheets did not work on my comp======================================== I like the pictures of real people. ======================================== I will certainly have my students listen to this when I get the time to show them======================================== I enjoyed this lesson, best of all. ======================================== Nicely presented. I am pleased with this lesson. ============Good job with the video and the written text. ======================================This lesson has fancy graphics but very little middle scho======================================== Nicely done with good photos of children that relates to our owclassroom. ======================================== Nice job I see I could relate this lesson to other subjects. (Math/Science/Social Studi

162

======================================== Nicely presented. I like the animations when I click certain designs.

:

========================= keeping ones hear healthy for

== tories integrated with the

s.

This is nice.

nding of this lesson.

motivate students to write.

sread the word north. She/he said Northern

================ ughout the lesson's

================ older, so that may be why. It

nal Writing Process." Using this lesson tivational factors, I used

nts come up with their ents. Over all, it was a pretty

=============================== nts really enjoyed and loved it. -

Summary Comments from Year 5 Lesson EvaluationsNice work for students to use with website research. ===============Would relate to caring for one's ear and the importance ofgood hearing. ======================================I sure like to see some Navajo or other Native American slesson. i.e. Stories of the milky way, coyote throwing the stars into the universe. etc... ======================================== Audio was the only problem I had ======================================== Lesson could use more animation for excitement. ======================================== I had a little trouble with downloading but I go the article======================================== I like the having access to the dictionary.======================================== Nice job but audio needs to work properly for effective understa======================================== Nice way to======================================== I liked the chefs or waitress/waiter meeting you as you click. I was expecting each nationality to speak their own languages. ======================================== Would be nice to have a scratch pad to figure out the problem. ======================================== There a part where the reader or narrator miWestern ========================The students were very responsive and were involved thro======================================== I really liked this lesson! ========================This lesson is a bit simplistic and boring. My students arewould be great for K-2. ======================================== I think my students are so used to the "Traditiokinda confused some of my students. I think instead of mosomething they are interested in their daily lives. A lot of my studepast experience or stories they heard from their grand- pargood lesson. =========This is one of the best lessons I have ever seen. My studeThey really liked "animation maps & picture gallery"

163

======================================== My students loved this lesson! Please don't miss it!!!!

164

Appendix E

Content Development Specialist Survey

TECHShare Project Year 5 Content Development Specialist Survey

The purpose on the Model Class hare project. CEER is gathering this information as part of the overall evaluation of Year 5 project activities. Please provide as much detail as possible so that “lessons learned” about successful practices and/or challenges may be documented. Please complete and return the survey via email by Monday June 7, 2004. Name: ________________________ Your Work Location: ________________ When responding to the items below, please respond with reference to your work since July 1, 2003. Curriculum Development: 1. Which Curriculum Preparation Studio (CPS) did you work with primarily? 2. What percentage of your time (approximately) was spent on lesson development? Select one answer closest to your experience:

___ 25% (quarter time) ___ 50% (half time) ___ 75% (three quarters time) Comments: 3. Was the communication you had with developers at the CPS effective for coordinating curriculum development? ___ Yes, most of the time ___ Yes, but it could be improved ___ No, it is not effective Comments: 4. Were there challenges in working with the CPSs? Please elaborate on your response below: ___ Yes ___ No

of this survey is to collect information regarding your workroom Study and on curriculum development for the TECHS

___ Not sure Comments: 5. How often on average did you work with NETC teachers on developing a

sson for TECHShare? Select one answer: le ___ About once a week ___ 2-3 times a month ___ Once a month

___ Once a quarter ___ Less than once a quarter Comments:

. Which teachers did you work with this year on lesson development? Select

___ NETC teachers generally (including ETIP and MC teachers) r

. Below is a list of different tasks that you might have assisted teachers with the

6one answer: ___ ETIP team teachers

___ Model classroom study teachers ___ I did not work with teachers on lesson development this yea Comments: 7when helping them develop a lesson for TECHShare. Place a check mark bythree tasks that you assisted with most often. Feel free to add other tasks nosted t abora

t te below.

dules ___ arks ___

ng materials for modules (graphics, clip media) ___ fying supporting resources (links, books, software) ___

Developing embedded assessments ___ Developing supplemental offline activities ___

li hat you helped with or to el Clarifying instructional goals for mo

/benchm Identifying state standardsConverting instruction to online multimedia format ___

LocatiIdenti

Comments:

167

8 you work with . Did more teachers this year on lesson development than last ear?

es ___ No ___ Not Sure

omments:

the teachers in your district(s) in evelo Share? Select one answer:

ested

)

odel Classroom Study: 0. Below is a list of tasks that you may have assisted Model Classroom achers with. Place a check mark by the three

y ___ Y C 9. From your experience, how interested are d ping curriculum for TECH ___ Not interested ___ Somewhat inter ___ Very interested ___ It depends (Please explain M1te tasks that you assisted with most often. Feel free to add other tasks not listed that you commonly helped with, oto elaborate b

r elow.

st-tests ___ sroom lessons online ___

___

Identifying supplemental activities to use with the lesson ___ the SmartBoard to present a lesson ___

Using a projection device such as a Proxima, In-focus or Averkey to resent the lesson ___

Filling out online forms such as the Teacher’s Journal or lesson evaluation bric ___

Troubleshooting technical difficulties ___

orking with the Model

Thank you very much for your time and input. If you have any questions or concerns about this survey, please contact Rebecca Zittle at: [email protected]

Administering pre- and po Finding and accessing the model clas Finding and accessing the model classroom lessons on CD-ROM Integrating the TECHShare lesson into their curriculum ___ Using p ru Comments: 11. What were the greatest challenges you faced in wClassroom study? 12. Any other comments or concerns?

168

Appendix F

iscellaneous Statistical Analyses Results

Lesson Evaluation Rubric Reliability Table

R E A L Y S I S - S C A L E (A L P H A)

1. oads within acceptable time 2. and text are appropriate and cl 3. LMSPELL Spelling is correct thru'out lesson 4. LMGRAMMA Grammar thru'out lesson is appropriate a

Lesson navigation is understandable, con w

.6199 178.0

5. 6. 7.

ean Variance Std Dev Variables

Scale Scale Corrected Mean Variance Item- Alpha

if Item if Item Total if Item

7360 3.8338 .3586 .7852 MGRAPHI 16.5169 3.8782 .5867 .7310 MSPELL 16.4888 4.1948 .4921 .7510 MGRAMMA 16.4888 4.2400 .4356 .7590 MNAV 16.6067 3.7767 .5713 .7315 MLINK 16.6573 3.6842 .4927 .7492

4 .6344 .7161

eliability Coefficients

N o

M

Reliability: Lesson Mechanics L I A B I L I T Y A N LMLOAD Lesson l LMGRAPHI Graphics 5. LMNAV 6. LMLINK All lessons links are identifiable and 7. LMDIRECT Lesson uses clear and easy to follow ins Mean Std Dev Cases 1. LMLOAD 2.6180 2. LMGRAPHI 2.8371 .4401 178.0 3. LMSPELL 2.8652 .3740 178.0 4. LMGRAMMA 2.8652 .3889 178.0 LMNAV 2.7472 .4849 178.0 LMLINK 2.6966 .5705 178.0

178.0 LMDIRECT 2.7247 .5289 N of Statistics for M SCALE 19.3539 5.0887 2.2558 7 Item-total Statistics Deleted Deleted Correlation Deleted LMLOAD 16.LLLLLLMDIRECT 16.6292 3.545 R

f Cases = 178.0 N of Items = 7

169

Alpha = .7747

Reliability: Instructional Design R E L I A B I L I T Y A N A L Y S I S - S C A L E (A L P H A) 1. STANDRDS Lesson standards and benchmarks are clea 2. IDOBJECT Specific lesson objectives are in eviden

app

ven w

66.0 0

1. IDFEEDBA 2.5482 .7266 166.0 2.5723 .7077 166.0 2.5060 .7687 166.0

166.0

H A)

Mean Variance Item- Alpha if Item Total if Item Deleted Correlation Deleted

.9392 IDOBJECT 34.8253 37.0541 .5615 .9405

3. IDSTANDA Objectives are clearly related to standa 4. IDINSTRU Lesson contains instructions to the lear 5. IDPREREQ Lesson includes information on prerequis 6. IDMEDIA Lesson key points are supported with 7. IDREINFO Engages learners thru use of new materia 8. IDDEMONS Demonstrates desired skills or process f 9. IDLEARNI Lesson uses various activities to engage 10. IDTASKOB Lesson tasks are related to stated objec 11. IDFEEDBA Feedback if consistently provided to lea 12. IDFORMAS Learners are provided with formative ass 13. IDSUMASS Learners are provided with summative ass 14. IDAUTHEN Assessment is authentic and interwo Mean Std Dev Cases 1. STANDRDS 2.8133 .4749 166.0 2. IDOBJECT 2.8735 .3996 166.0 3. IDSTANDA 2.8313 .4753 166.0 4. IDINSTRU 2.7831 .5291 166.0 5. IDPREREQ 2.6084 .6676 166.0 6. IDMEDIA 2.7410 .5386 166.0 7. IDREINFO 2.7349 .5523 1 8. IDDEMONS 2.7108 .5727 166. 9. IDLEARNI 2.6386 .6528 166.0 10. IDTASKOB 2.8193 .4578 166.0 1 12. IDFORMAS 13. IDSUMASS 14. IDAUTHEN 2.5181 .7280 N of Statistics for Mean Variance Std Dev Variables SCALE 37.6988 39.9451 6.3202 14 R E L I A B I L I T Y A N A L Y S I S - S C A L E (A L P Item-total Statistics Scale Scale Corrected if Item Deleted STANDRDS 34.8855 36.2232 .6117

170

IDIDINSTRU 34.9STANDA 34.8675 36.0066 .6508 .9383

157 35.0474 .7371 .9360 DPREREQ 35.0904 34.2888 .6664 .9380 IDMEDIA 34.9578 34.9376 .7409 .9359 IDREINFO 34.9639 34.5441 .7849 .9347

EMONS 34.9880 34.5332 .7553 .9353

TASKOB 34.8795 35.4884 .7787 .9357

R E L I A B I L I T Y A N A L Y S I S - S C A L E (A L P H

loca d his edge nts,

4

Scale Scale Corrected

if Item if Item Total if Item Deleted Deleted Correlation Deleted

LTENVI 6.5385 5.6548 .8571 .8539 .8506

I

IDDIDLEARNI 35.0602 33.8388 .7479 .9355 IDIDFEEDBA 35.1506 32.8802 .7845 .9345 IDFORMAS 35.1265 33.6384 .7073 .9370 IDSUMASS 35.1928 33.2232 .6919 .9381 IDAUTHEN 35.1807 32.6823 .8089 .9337 Reliability Coefficients N of Cases = 166.0 N of Items = 14 Alpha = .9409

Reliability: Cultural Component A) 1. CULTENVI Presents content with a base in the 2. CULTHIST Incorporates elements of current an 3. CULTCOMM Lesson incorporates community knowl 4. LEPSUPT Provides good support for LEP stude Mean Std Dev Cases 1. CULTENVI 2.1124 .9222 169.0 2. CULTHIST 2.0414 .9345 169.0 3. CULTCOMM 2.1598 .9086 169.0 4. LEPSUPT 2.3373 .8583 169.0 N of Statistics for Mean Variance Std Dev Variables SCALE 8.6509 10.2643 3.2038 Item-total Statistics Mean Variance Item- Alpha CUCULTHIST 6.6095 5.5728 .8654 CULTCOMM 6.4911 5.7038 .8606 .8530 LEPSUPT 6.3136 6.8832 .5871 .9445 Reliability Coefficients

171

N of Cases = 169.0 N of Items = 4 Alpha = .9065

172

Appendix G

GPRA Indicator Data

173

TECHShare Participant Demographics

Year 5 Indicator 8.1.1 Experimentwise:

Total number of students assessed =4746

Total number of lessons7 = 62 Control Group= 31 lessons Experimental Group = 31 lessons Number of completed student-lessons (Pre/Post) = 1335 Total Student Increase = 411 Total Students Decrease = 30 Total Student No Change = 19 Drop from study = 14 Total number of students / Total number of lessons completed

Number of students demonstrating increased performance in Reading = 411 Total student lessons in cell

/ Number of students reaching outcome8

Number students demonstrating decreased performance in Reading Total student lessons in cell

/ Number of students reaching outcome

Number students demonstrating no change in Reading Total student lessons in cell


Number students demonstrating increased performance in Math Total student lessons in cell


Number students demonstrating decreased performance in Math Total student lessons in cell


Number students demonstrating no change in Math Total student lessons in cell


Number students demonstrating increased performance in Science Total student lessons in cell


Number of students demonstrating decreased performance in Science Total student lessons in cell


Number of students demonstrating no change in Science Total student lessons in cell


6 Total number of students in study = 686. This report does not include 226 students in the Dine’ Language lessons. Data was not available at publish

175

Ethnicity

Hispanic / 12 / 9 4 / 3 4 / 0 Latino

4 / 1 5 / 5 5 / 0 5 / 0 0 / 0 0 / 0 0 / 0

Non Hispanic / Latino

462 / 451 143 / 119 143 / 18 143 / 6 207 / 187 207 / 12 207 / 8 101 / 97 101 / 0 101 / 4

Don't Know

Race

American Indian / Alaska Native

421 / 411 139/115 139 / 18 139 / 6 181 / 161 181 / 12 181 / 8 91 / 88 91 / 0 91 / 3

Asian

Native Hawaiian / Pacific Islander

White 51 / 48 8 / 7 8 / 0 8 / 1 31 / 31 31 / 0 31 / 0 9 / 8 9 / 9 / 1

Multi-r acial

African American

0 0 / 0 0 / 0 0 / 0 0 / 0 0 / 1 / 1 1 /01 1 / 0 2/ 1

Don't Know

Special Populations

High Poverty9 147 / 122 147 / 18 147 / 7 212 / 192 212 / 12 212 / 8 101 / 97 101 / 0 101 / 4 474/ 460

Limited English Proficiency10

212 / 209 123 / 99 123 / 18 123 / 6 61 / 55 61 / 5 61 / 1 28 / 28 28 / 0 28 / 0

Disabilities11 0 / 0 0 / 0 0 / 0 13 / 12 13 / 0 13 / 1 7 / 7 7 / 0 7 / 0 20 /20

Unknown 23 / 22 23 / 22 23 / 1 4 / 3 4 / 1 4 / 0 58 / 54 58 / 0 58 / 4 95 / 85

7 Does not reflect Dine’ Language lessons 8 Averages across multiple lessons 9 High poverty defined as school that is eligible for Title I school-wide program 10 Local definitions 11 Defined as having Individualized Education Program (IEP) under Individuals with Disabilities Act (IDEA)

Challenging Content Aligned w h State Standards low shows the means for the Instru gn items related to

m rks. The scale used ranged from 1 = ‘not evident’ to 5 = ‘clearly evident.’ The 187 lessons reviewed were rated highly on their alignment with state standards and benchmarks

GPRA 8.1.2The graph bTECHShare

itctional Des

and benchme lessons’ align

iaent with standards

Survey Questions

Q18Q1Q13b3

Mea

n

5.00

4.00

3.00

2.00

1.003cQ1 a

4.5044.578 .564.5

Key to Graph Survey Questions: Q13a: Lesson standards and benchmarks are clear

ectives are in evidence lated to stated standards and benchmarks

Q18: Overall, I believe that this lesson would significantly aid the student in mastering the stated objective(s)

e survey items were also calcu ed sons by subject area. Tabod the subject areas of math,

science and language arts for grades K-12 were found to have challenging content aligned with state standards. A total of 187 lessons comprising 1165 modules received

Of these mod s, 1027 were in ath, science and aining 138 m dules were

Q13b: Specific lesson objQ13c: Lesson objectives are clearly re

Means for th17 below shows the results, indi

e sam lat m

forule

less in

le cating how many

630 reviews.language arts; the rem(foreign language) and social studies.

ule the subject areas of m in subjects such as Dine language o

Table 18 Number of Modules in Core Content Areas Aligned with State Standards

Q13a: Q13b: Q13c:

Lesson standards

and benchmarks

are clear

Specific lesson

objectives are in

evidence

Lesson objectives are clearly related to

stated standards

and benchmarks

Q18: Overall, I

believe that this lesson

would significantly

aid the student in mastering the stated

objective(s)

Number of Modules

SUBJECT

Language Arts

M = 4.72 N = 75 SD = .745

M = 4.76 N = 75 SD = .694

M = 4.78 N = 74 SD = .603

M = 4.56 N = 75 SD = .620

162

Mathematics

M = 4.70 N = 169 SD = .791

M = 4.69 N = 168 SD = .854

M = 4.65 N = 166 SD = .933

M = 4.38 N = 169 SD = .926

481

Science

M = 4.75 N = 142 SD = .696

M = 4.70 N = 141 SD = .819

M = 4.75 N = 139 SD = .790

M = 6.64 N = 142 SD = .623

384

Note: the ‘N’s refer to the number of reviews the modules received.

177

Appendix H

Year 5 Evaluation Budget

Evaluation Budget for Year 5, July 1, 2003 – June 14, 2004 Personnel Running Totals

Director Consulting Professionals $18,000

Sub Total P $83,000

Travel

Travel $7,000 Site Visits $7,200

Sub Total Travel $14,200 $97,200

Equipment

Desktop Data Computing $0 Printer $0

Scanner $0 Software $494

Recording (still / audio / video) $0 Equipment/Software leasing $3000

Sub total Equipment $3,494 $100,694

Other

Postage $100 Telecommunications $0

Office Supplies $1,000 Printing / Reports $200

Office space / utilities $3,126 External Accounting Services $2,880

Subtotal Other $7,306 $108,000

Total Annual Costs $108,000 Payment Points Invoices for this project will be submitted on a timely basis. It is requested that accrued costs be reimbursed on a monthly basis, with the monthly reimbursement periods ending on the last Saturday of each month. Budget Justification Year 5

Personnel

• Director. The Evaluation Director’s fee is based on current market costs to

obtain the appropriate degrees and experience necessary to direct and carry out

Year 5 $65,000

ersonnel $83,000

179

an evaluation of a project the scope of Star Schools, over a period of 12-months.

Costs for FY 2003 remain the same as for the previous years at $65,000.

• Consultants. To minimize direct costs and overhead, supplemental staff will be

hired on an “as needed” basis. Expenditures associated with this category include

professional consultants needed for consultations within narrow or specialized

fields of expertise relating to the project (valued at an average of $500/day);

experienced (val 0.00 / hour), and associated costs for field site

visits; data processors to aid in the handling of large datasets (valued at $8.00 /

hour); and casual office help as needed (valued at $8.00 / hour.) Total costs for

at $18,0 0; this marginally higher figure reflects the

anticipated ne ditional support with data collection, statistical analysis,

ted to t entation of the experimental design and

.

Travel

• oject D and consultant(s)will travel extensively

throughout the area covered by the grant (approx. 25,000 square miles) for the

ment, data collection and to attend

meetings as required by the Principal Investigators of the project. It may also be

necessary to travel to Washington D to attend meetings held by the funding

federal agency. Costs for FY2003 are estimated at $7000.00, the same as for the

previous year.

y for

and data collection. These costs are estimated at

2003.

Equi e

•

out the evaluation. During Year 1, two computers (for the Director and consulting

personnel ued at $2

FY 2003 are estimated 00.0

ed for ad

and interpretation rela he implem

summative evaluation

Travel. The Pr irector

purpose of project planning and develop

C

• Site Visits. The director and assigned consultant(s) will

travel to schools in the target area participating in the Model Classroom Stud

the purpose of site visits

$7200.00 for FY

pm nt

Desktop. Data processing equipment is necessary to carry

180

p

co grades

and maintenance are necessary; the estimated FY2003 cost is $0.00.

•

•

ees. Estimated costs for FY2003 are $3000.00.

Postage tage,

•

• s, Printing / Reports. Office supplies and related materials are

essential for office management and evaluation material development. Materials

and supplies for evalu

oks,

report writing and dissemination, etc. The total cost for

FY2003 is estimated to be $1200.00.

Office space / Utilitie

rofessionals) were procured for the construction of measurement tools and the

llection and statistical analysis of data relating to the project. Periodic up

Printer, scanner, software, recording equipment. This

equipment is needed to design, collect, organize and disseminate evaluation

activities. Note that (with the exception of recording equipment and software

updates) these are one-time costs. The combined costs for FY2003 are estimated

at $494.00.

Equipment / Software leases, licensing. The costs

associated with this category relate to miscellaneous equipment needs (e.g.,

rentals for fieldwork) and software and measurement (online surveys, tests and

instruments) licensing f

Other

• . Costs related to postage (reports, paper copies of surveys, return pos

misc., etc.) for FY2003 are estimated at $100.00.

Telecommunications. Costs related to communications for FY2003, including

support for a web presence, are estimated at $0.00.

Office Supplie

ation include but are not limited to expendable materials

used in conjunction with the production of presentation media, such as CD-

ROMs, zip drive diskettes, tapes, diskettes, printer toner/cartridges, notebo

pens, and materials for

• s. Based on an average office rent of approximately

$1200.00 / year, average utilities of approximately 1900.00 / year and start-up

181

furniture costs of approx. $1300.00 for the life of the project. Rent and utilities

alone for office space for FY2003 will remain at $3126.00.

• External Accounting Services. CEER is contracting with the Area Prevention

Resource Center of La t for

wton to provide accounting support services. The cos

accounting services for FY2003 will remain at $2880.00.

182

year five evaluation report: netc techshare star · pdf fileyear five evaluation report: netc...

Documents