evaluation report of on the cutting edge · evaluation report of on the cutting edge february 5,...

63
Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa, Virginia Data collection and analyses conducted by the Cutting Edge Evaluation Team of Ellen Iverson, Evaluation Director Randahl Kirkendall, Evaluation Specialist Monica Bruckner, Evaluation Assistant Science Education Resource Center, Carleton College, Northfield, Minnesota Principal Investigator Liaison to the Evaluation Team Cathy Manduca, Director Science Education Resource Center, Carleton College, Northfield, Minnesota

Upload: phamanh

Post on 24-Jun-2018

216 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Evaluation Report of On the Cutting Edge

February 5, 2010

Submitted by

John McLaughlin, External Evaluator

McLaughlin Associates, Lanexa, Virginia

Data collection and analyses conducted by the Cutting Edge Evaluation Team of

Ellen Iverson, Evaluation Director

Randahl Kirkendall, Evaluation Specialist

Monica Bruckner, Evaluation Assistant

Science Education Resource Center, Carleton College, Northfield, Minnesota

Principal Investigator Liaison to the Evaluation Team

Cathy Manduca, Director

Science Education Resource Center, Carleton College, Northfield, Minnesota

Page 2: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 2 of 63

Contents

Executive Summary 3

2009 External Evaluation of On the Cutting Edge Executive Summary 7

Introduction 7

Internal Evaluation 7

External Evaluator Data Collection 8

Summary 12

The Cutting Edge Structure for Learning 14

An Integrated Professional Development Program 14

Cutting Edge Workshop Participation 15

Cutting Edge Website Contributions and Use 17

Website and Workshop Use Together 19

Faculty Response to Cutting Edge 20

The Cutting Edge Impact on Teaching 32

Changes in Teaching 32

Shifting how Faculty Think About Teaching 40

Impact of Cutting Edge Experience on Student Learning 44

The Cutting Edge Impact on the Geoscience Education Community 45

Networking 45

Contributing to the Geoscience Education Community 49

Result of the Cutting Edge Impact on the Geoscience Education Community 54

Appendix: Evaluation Methodology 56

Large-scale survey of Geoscience Community 56

Surveys of all Cutting Edge Workshop Participants and Website Users 57

In-depth Small Samples of Interviews 59

Report on Statistical Findings 60

Limits of this Evaluation 61

References Cited 63

Page 3: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 3 of 63

Executive Summary

This report summarizes the evaluation of the On the Cutting Edge Faculty Professional

Development program (http://serc.carleton.edu/NAGTWorkshops) from 2002 to 2009. On the

Cutting Edge (Cutting Edge) is a comprehensive program of workshops and related web-based

resources to support faculty professional development at all stages of their careers. The project is

funded by the National Science Foundation Directorate for Education and Human Resources

Division of Undergraduate Education as part of the Course, Curriculum, and Laboratory

Improvement program (CCLI) and managed collaboratively by PIs Heather Macdonald (College

of William and Mary), Cathy Manduca (Carleton College), Dave Mogk (Montana State

University) and Barb Tewksbury (Hamilton College. The Cutting Edge proposal responded to

Program Solicitation NSF 05-559, specifically to increase faculty participation in the cycle of

educational innovation, to provide support in developing teaching skills and knowledge, to

develop leaders in professional development and innovation, and to develop networks and

communities in which a culture of educational innovation can thrive and grow. The methods,

tools, and limitations of this evaluation are described in the appendix.

The Cutting Edge program works via a learning structure built from an integrated workshop

series and a website of teaching resources. Cutting Edge developed models for both face-to-face

and virtual workshops ranging from one to six days in length and offered 56 workshops between

2002 and 2009. 1751 faculty, post-docs, and graduate students participated in one or more

workshops. More than 20% of the participants came to two or more workshops for a total of

2246 workshop participant-seats. Approximately 25% of the geoscience faculty in the U.S. and

more than half the departments in the United States have participated. The reach of the

workshops was in proportion to the different types of departments, with the exception of two-

year colleges which were underrepresented. Cutting Edge workshop participants included a

higher percentage of women and members of underrepresented groups than in the overall

population of U.S. geoscience faculty. End-of-workshop evaluations indicate a high level of

participant satisfaction with a median of 9.1 on a 10-point scale.

The Cutting Edge online collection of teaching activities and other resources supported those

who attended the workshops, provided a venue for sharing teaching resources, and extended the

reach of Cutting Edge to those who had not attended a workshop. The web collection includes

over 4000 pages, 33 topical collections, and more than 1200 community-contributed teaching

activities. In 2008, more than half a million users visited the website and approximately 12,000

visited 10 or more pages in a single session. Approximately 25% of users are geoscience faculty

with the remaining 75% comprised of faculty from other disciplines, K-12 teachers from

geoscience and other disciplines, undergraduate and graduate students, postdocs, and others.

Approximately 30% of U.S. geoscience faculty use the website to find teaching ideas and

Page 4: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 4 of 63

materials, to compare their own teaching to that of others, to learn about teaching methods or

geoscience topics, and to obtain information about career planning. Users report that the website

increases their confidence in trying new teaching methods.

Impact on Participants Teaching

Comparison of faculty who have participated in the On the Cutting Edge program, either using

the website, or by attending a workshop and making use of the website, to faculty who had not

participated in the On the Cutting Edge program, shows that participants were more likely to

converse with other faculty about teaching, make more use of educational research, and more

likely to seek information about others‟ instruction (often through the website, Fig. I).

Participants were more likely than non-participants to have changed their teaching methods in

the past two years, and specifically, they were more likely to have decreased the amount of time

lecturing and to have added group work or small group activities. Participants reported more use

of in-class questioning, small group discussion and in-class exercises. Participation also appears

to have impacted the evaluation strategies and assessment methods that faculty used. In many

cases, impacts were more pronounced for those who both attended a workshop and made use of

the website (Figs. II and III). Qualitative data indicate that workshop participants underwent a

shift in their teaching philosophy to an approach that was more focused on student-centered

learning. This change in philosophy was foundational to subsequent learning and changes in

teaching practice.

Figure I. Cutting Edge website use and changes in teaching.

70%

37%47%

32%25%

35%25%

54%

21%

32%

15% 15%22%

17%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Made changes in teaching

methods used in intro courses over past two

yrs

Spent less time lecturing

Increased questioning of

students during lecture

Added group work/small

group activities

Spent more time on class/small

group discussions

Changed assessment

tools/strategies

Added assignments

Impacts of CE Participation on TeachingWeb

No Web

Page 5: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 5 of 63

Figure II. Cutting Edge website use, workshop participation and changes in teaching.

Figure III. Cutting Edge website use, workshop participation and use of teaching strategies.

15%

22%25%

32%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

Added group work or small group activties

Spent less time lecturing

Changes made in last two years by Cutting Edge exposure

No Cutting Edge exposure

Website only

Website and Workshop

40%43%

42%

17% 19%

53%

28%34%

65%

44%50%

0%

10%

20%

30%

40%

50%

60%

70%

Posing questions answered by entire

class2

Small group discussion or think-

pair-share

In-class exercises

Teaching Strategies Used Weekly or Nearly Every Class by Cutting Edge Exposure

No Cutting Edge exposure

Website only

Website and Workshop

Page 6: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 6 of 63

Impact on the Geoscience Education Community

On the Cutting Edge has increased networking and interaction on topics of teaching throughout

the geoscience community. Workshops provided valuable opportunities to network with other

faculty and with experts in education and cognitive science. Workshop participants talked more

to colleagues about teaching after their workshop experience; they interacted with colleagues

across the nation in addition to those on their campus, and they made use of the website to find

out what others were doing. An important subset of participants built new networks for career

planning and made use of the website as a career development networking tool.

Increased interaction and networking supported ongoing scholarly work in geoscience education.

Participants specifically attribute 40 successful grant proposals to participation in the On the

Cutting Edge program. Cutting Edge participants were more likely than others to contribute to

the community discussion of teaching both through presentations and publications, as well as by

sharing teaching materials on-line.

There also appears to be a clear dose response of reported changes at the department level in

response to the number of faculty participating in workshops. The 2008 Participant Survey found

that participants from departments in which at least 3 department members had attended a

workshop were 3 times more likely to report that the workshop had changed their department‟s

collective approach to teaching compared to participants coming from departments in which 2 or

fewer members had attended a workshop.

The impact of the program clearly extended beyond the geoscience community. The project PIs

contributed to national conversations on faculty professional development. An area not yet

evaluated is the impact that Cutting Edge has on faculty participants who are not from the

geosciences as they also develop new insights into teaching and expand their professional

networks. This population represents 75% of website users.

In sum, as a result of the On the Cutting Edge program:

Education is now an accepted topic for discussion and work among geoscience researchers from

all types of institutions;

Research networks have become more supportive and conducive to education research;

A national perspective on geoscience teaching has been created and is being used to benchmark

teaching norms and behaviors at the institutional and individual levels;

Geoscience faculty members are connecting with new colleagues from different institutions and

different disciplines;

There is a culture of sharing experiences, successes, and new ideas in education and teaching

practice; and

The sphere of influence and leadership roles for education specialists in geoscience have

expanded.

Page 7: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 7 of 63

2009 External Evaluation of On the Cutting Edge

Executive Summary

John A. McLaughlin

Introduction

On the Cutting Edge (CE) is a comprehensive program of workshops and related web-based

resources to support geoscience faculty in their professional development at all stages of their

careers. Cutting Edge specifically aims to increase faculty learning about the cycle of educational

innovation, to provide support in developing teaching skills and knowledge, to develop leaders in

professional development and innovation, and to develop networks and communities in which a

culture of educational innovation can thrive and grow.

The evaluation of the CE project is the responsibility of an evaluation team whose members are

Ellen Iverson, Randahl Kirkendall, and Cathryn Manduca. John McLaughlin was the external

evaluator and supervised the program evaluation design, implementation, analyses, and

reporting. In addition, the external evaluator collected evidence through in-depth interviews with

key stakeholders, participants, and observers of the CE project. The purpose of this report is to

summarize the external evaluator‟s view of the project performance.

Internal Evaluation

As noted above the impacts of emerging from the CE project were projected to occur in two

domains: a) individual workshop participants and website users, and b) geoscience community

impacts following from individual changes and applications. This section presents the external

evaluator‟s observations based on his review of the annual CE evaluation report.

Individual Impacts

One of the most powerful evaluation tools employed by the CE evaluators was a survey of the

total geoscience faculty community. The survey was conducted by an independent survey group

and allowed a comparison of faculty members who participated and did not participate in CE

activities – workshop or web site. In addition to this survey, the team used interviews, surveys,

observations, and document reviews to inform them of CE performance.

It is clear from a review of the evidence produced by these multiple methods that participants in

CE activities changed the way they think about teaching, how they approach planning and

assessment of learning, and how they deliver learning in their classrooms. One of the key aspects

of this shift in teaching practice was toward student-centered learning -- changing from „what do

they teach‟ to what are their students learning‟. In addition, evidence demonstrates that

Page 8: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 8 of 63

participants increased their level of teaching confidence. They increased their knowledge and

skills/tools related to their planning, teaching, and assessment and they increased their

pedagogical expertise.

Community Impacts

The evidence found in the various data collection strategies indicates that Cutting Edge has

played an important role in creating an environment that is characterized by the enhanced use of

established networks and new information pathways in a way that has overcome the traditional

barriers between research and education, across disciplines, and among different institutional

types. This, in turn, has nurtured and encouraged faculty members to learn from each other and

share their experiences with new teaching methods and technologies.

Participants became more likely to take part in the cycle of innovation in which they developed

and tested new teaching approaches and tools in their classrooms, shared successes with

colleagues in their institution, and contributed new learning to the CE web and the geoscience

literature through publications and presentations.

Further, participants were more likely to learn form others rather than self-invent. They were

more motivated to adopt new ideas from colleagues and to seek resources from the Internet,

including CE. They were more likely to attend professional workshops and seminars that focused

on pedagogy. These changes in approach to course design and assessment occurred for both

introductory and advanced courses.

External Evaluator Data Collection

The external evaluator provided technical consultation to the CE staff members on the internal

evaluation design, collection, analyses, and reporting strategies. In addition, he collected data

from participants in the emergent CE Leadership Development program and a number of well-

known geoscience leaders across the nation. This section summarizes the findings from those

interviews.

Leader Interviews

The Leadership Development component of the CE project is best characterized as emergent.

Aas the CE project advanced, the need to expand the offerings through the workshops and

website grew, and thus CE needed to develop other leaders to meet the need. In addition, the

Leadership program was initiated to enable a broader impact on the community at large. Ten

participants in the Leadership program were interviewed by phone. It should be noted that CE

was just beginning a formal Leadership development program that was designed to include a

formal induction workshop and continuous coaching by the PI‟s. The results of these interviews

were meant to contribute to the design work. Those interviewed had participated in a less formal

induction into their leadership roles.

Page 9: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 9 of 63

When asked why they appreciated the CE workshops in comparison to others they had attended,

they interviewees noted that the CE workshop presenters did not use the „expert‟ model but “had

a genuine desire to share their experience.” The workshops were applied in nature and focused

on shared learning. “It was the right mix of getting the ideas and being given the time to produce

something.” “It is like the feeling I get after a talk when I realize that the students are really

listening and I have learned something from them too.” “I got turned on to pedagogy and how it

works. There are all these tools and techniques out there – different ways of teaching and

learning – in all of the workshops it is about engaging people and being engaged by the

material.”

Next, faculty participating in the Leadership Development Program were asked how their

involvement in CE influenced their career path. Influences were deep, lasting and ranged from

changes in their pedagogy and their impact outside of their classes. In addition, their

participation increased their teaching confidence. “The Cutting Edge serves as the model of the

way instruction should be going. I have gotten a lot more active in issues of pedagogy.” “Early

on it cemented my feelings that I wanted to teach and that there were many ways to do that – and

that I could be creative.” “I did have some anxiety about transitioning back to the teaching world.

The reason I went to the Cutting Edge is because I knew I needed a jump start and it did that.” “I

was able to make major curricular changes in our program that was designed based on the

principles that I learned through Cutting Edge colleagues and putting the audience/students‟

needs first. It is truly a student centered program.” “I am a department chair now and I have

encouraged my faculty to invest time in attending these workshops and to invest time in

adjusting their curricula.”

Participants also gained an appreciation for what makes a successful CE workshop. It is

important to get the right mix of people. “It is useful to get experiences from a heterogeneous

group of people – when you are teaching about something topical it is useful to get segments of a

population that is going to be responsible for teaching that.” Starting with the end in mind was

also viewed as important. “Getting the goal right, keep a level of respect for people‟s time, and

make sure the group dynamic is positive with high energy.” Organization from planning to

implementation is critical. Skills, logistics, encouraging people to interact & perform during the

workshop, the amount of work PI‟s have gotten people to do in preparing and following up on

the workshops.” “You need to have a program that mixed the old and new stuff with reflection.

If they do not have the time or space for reflection – the ideas will not be implemented.”

When asked how they were inducted into their leadership roles most suggested that the

experience was incremental – like boiling a frog! “They ask you to present during a session, they

recognize you are doing something special and they want you to share it with others. It is baby

steps. You can volunteer to present – you don‟t have to and the feedback is helpful. This builds

Page 10: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 10 of 63

your confidence. Then I was asked to co-facilitate with experienced presenters who I was

comfortable with. There is a clear place as to your piece.”

When asked how they were selected, participants suggested that they had demonstrated

leadership potential through their participation in CE workshops. “They recognized that I “got it”

and they recognized that I was able to see its potential. I was going to run with the ball. That

was really all they “did”. They did not use me as an example or include me in the conversations.

I am a little extroverted and was able to convey my enthusiasm. They were paying attention.”

“PI called me and said, “I think you would be good at this.” “It seems well thought out – given a

small topic at first and pair with people who knew what they were doing. Doing joint things

where there is someone else in charge and you are there to assist – which teaches you how to do

it. Later pairing experience and less experienced leaders together – co-leading. Finally, I was

leading and coordinating workshops. There is always support.”

All of the ten interviewees indicated that they intend to continue their participation as CE

Leaders. Many have actually taken the initiative to design and offer workshops outside of the CE

organization but still focused on pedagogy in geoscience undergraduate and graduate learning. It

was clear from the interviews that the Leadership development program was beginning to

emerge as an effective way to extend the CE impact.

Gateway Interviews

The external evaluator interviewed a number of nationally recognized leaders in the geoscience

community to determine their impressions of the impact that CE has had on the broader

community and the extent to which the need addressed by CE was still relevant. In the spring of

2009, CE PI‟s identified six recognized leaders in the geoscience field who were knowledgeable

of CE but were not closely affiliated with implementation. The CE external evaluator conducted

the interviews; one person was not available for an interview.

Interviewees had a very broad perception of the CE project that extended beyond the workshops

and website. They viewed CE as having an impact on the geoscience community at large.

“Reaching out to the broaden geoscience community. Providing an opportunity for the

geoscience community to have some sort of common binder from a number of different

perspectives. Making sure geoscience communities are viewed strongly at their institutions.”

“My perception is trying to prevent geoscientists from having to re-invent the wheel from all

aspects by allowing us to communicate with each other in a variety of ways (online, in person) to

help improve the state of university work in Earth Science.”

Interview participants were asked to identify the impact of program activities (workshops,

website). They identified CE as a valuable information resource for geoscientists and the

broader scientific community. The website provides a wealth of information for geoscience

Page 11: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 11 of 63

educators and is rapidly becoming a primary resource for geoscience faculty. One participant

reported: “They speak of CE like it is one of their fingers – one of their regular tools. The people

I work with seem to take it for granted like it is a needed tool.” In addition, CE has improved the

image of teaching geoscience among geological scientists. As a result, interest in the Education

Division of the Geological Society of American (GSA) has grown substantially during the

annual conference. Although one participant had not seen a specific impact of CE on the broader

geoscience community, the individual indicated that CE has had an impact on the teaching and

career development education in a variety of scientific societies.

Leaders recognized the importance of building a community of practice within the geoscience

field. Workshops for students and faculty, and community building were noted as the two major

objectives of CE. The significance of community building was described as: “The community is

a pathway by which we achieve something – what are we trying to achieve? I would still say it

is a savvy, smart, skilled group of people in the geoscience community that is extremely

important for the researchers. Mainly it is the people.”

A second national leader noted: “What is interesting about our relationship with CE, it has

enabled two communities to work parallel and in unity (research and teaching). We have used

Barbara Tewksbury as a national treasure. This isn‟t just somebody coming from an educational

school, but we have a practicing geologist. It conveys an amount of legitimacy. It also opened

up an avenue for people who are professionally dedicated to open up things like these. They can

join our groups and be taken seriously without that „snootiness‟ of educational faculty. The

bridge has been created to enable our people to walk over it.”

The opportunity to disseminate information via the CE website provided a pathway for

developing connections among practitioners. Once the process was in place and information

became available online the website became common ground for both educators and researchers.

Increasing participation on policy-making boards by members of CE is evidence of a growing

community of practice. One participant agreed stating: “Yes – more so in the areas of

geoscience education. The growing emphasis by NSF has already helped. They realized the

need for scientists to get the word out, not just doing the work. They have created educational

programs within what used to be purely scientific disciplines. When they look around to see who

to do to, they naturally go to CE.”

All national leaders agreed that there is a long-term need for CE and that the National Science

Foundation should continue funding the project. CE serves the NSF well in supporting research

and education. CE serves as a clearinghouse of information for geoscience where practitioners

can access timely information quickly. At least one individual judged the website as a critical

resource even more important than the workshops. The CE program is unique in addressing the

Page 12: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 12 of 63

needs of the scientific community, and should be expanded to other disciplines. One participant

noted, “There isn’t a group to develop younger scientists other than CE.” Another said,

“Absolutely – I think the rotating workshops (most of my exposure is from this) to different

disciplines every year. When they did it in geophysics, we had never had anything like that – we

could come together with groups of like-minded professionals. By the time it rotates back to

geophysics there will be all new participants, research, and points of view.” Another referred to

CE as a vital resource to the community when he said, “There are all these ideas out there. How

do you choose? Where do you go? You go to the CE website. You know your colleagues share

the activities and ideas. In this world of ever expanding information, there is a need for accurate

and reliable information.”

Finally, one national leader noted that there are no other alternative projects that address the

same need. “Yes – if they do not, I don‟t know who will. The agency desires we all are able to

carry out research with a real impact and that education is a real part of what we do. They

should be interested in funding CE. I know they don‟t typically invest long term, but they

should. I don‟t know of other funding sources out there - for example foundations. The other

question – is there a way to make it self-sustaining, which is not realistic since they are trying to

make it accessible to everyone.”

Summary

The external evaluator examined all evidence collected through the CE internal evaluation

strategies and collected additional evidence through in-depth interviews with key stakeholders

and participants in the emergent Leadership initiative. The internal evidence is convincing with

respect to the impact CE is having on its workshop and web site participants. There is sufficient

evidence that participants are changing their philosophy of teaching, their approaches to

planning, delivering, and assessing their instruction. Further, their comments indicate that

participation increases their pedagogical confidence. Impacts extend beyond the individual

participants. When compared to non-CE participants they are active participants in the cycle of

innovation.

Participants in the emergent Leadership development program are very supportive of their

experiences. They reported a deep understanding on what makes the CE experience work and

noted the importance of PI continued support. It is clear that CE has been successful in

developing a set of leaders who can design and deliver workshops, contribute to the web site and

impact the broader geoscience community within and beyond their institution.

Finally, national leaders suggested that CE is unique in providing a valuable service to

geoscience professionals particularly those working in the education arena. Leaders in the field

of geoscience recognize the program as the sole source for professional development training

and resources. CE is credited for reducing the gap that exists between research and teaching

Page 13: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 13 of 63

through information dissemination and community building, and increasing interest in the

teaching of geoscience. Interview participants were unanimous in their views that CE merits

long-term support from the NSF.

Page 14: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 14 of 63

The Cutting Edge Structure for Learning

An Integrated Professional Development Program

The Cutting Edge program is based on the concept that, if faculty are engaged in a community

where they can share and discuss their teaching and learn from others within and beyond the

geosciences, then the quality of teaching will rise for individual faculty, ultimately improving

student learning across the country. The Cutting Edge program was designed to develop and

support such a learning community via three elements: an integrated workshop series, a website

of teaching resources, and a leadership development program.

An integrated workshop series

Cutting Edge developed models for both face-to-face and virtual workshops ranging from one to

six days in length and has offered 56 workshops since 2002, including 27 in the last three years.

All workshops were designed to model and promote best practice in pedagogy, to help faculty

stay current in geoscience and educational research advances, and to build community resources

and networks. Workshops of different types engaged faculty with different interests and at

different points in their careers:

Emerging theme workshops accelerated the introduction of new geoscience research or

a new aspect of pedagogy into the undergraduate curriculum. Workshop topics through

2010 include biocomplexity, the deep Earth, discoveries from Mars, the early Earth,

energy, geology and human health, hurricanes and climate change, ocean systems, public

policy, rates and time, the affective domain, assessment of learning, metacognition,

teaching complex systems, teaching with data, teaching with models, teaching online,

teaching with games, urban students and urban issues, visualizations, and web design.

Workshops about teaching a core geoscience topic brought together faculty who teach

a topic that is part of the geoscience major to explore issues in teaching this topic and to

share teaching resources. The phrase 'Teaching X' is used to refer to these workshops.

Past workshops included topics such as geomorphology, geophysics, hydrogeology,

paleontology, petrology, sedimentary geology, and structural geology.

Annual workshops that supported faculty at various career stages included a

workshop for early career geoscience faculty, a workshop for graduate students and

postdoctoral fellows on preparing for an academic career, and a workshop on effective

course design (including course "redesign").

Page 15: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 15 of 63

Large workshops disseminated best practices in teaching (e.g., Teaching Introductory

Geoscience; Teaching with Visualizations, Models, and Online Data; Teaching in the

Field).

Workshops held in conjunction with meetings of professional societies and research

consortia brought a pedagogical dimension to research meetings for faculty who would

not otherwise attend a "teaching" workshop (e.g., workshops at AGU, GSA, AMQUA

and Goldschmidt meetings).

Follow-on workshops repurposed resources developed for a particular workshop and

brought them to a new audience (e.g., Student Motivation and the Affective Domain;

Pursing an Academic Career).

A website of teaching resources

One of the hallmarks of On the Cutting Edge has been the extensive and widely used online

collection of teaching activities and other resources. The collection supported those who attended

the workshops, provided a venue for sharing teaching resources, and extended the reach of On

the Cutting Edge beyond those attending a workshop. The web collection includes over 4000

pages, 33 topical collections, and more than 1200 community-contributed teaching activities.

A leadership development program

In order to increase the number of leaders prepared to offer professional development for

geoscience faculty, On the Cutting Edge provided leadership experiences and targeted leader

training for more than 100 co-conveners and facilitators. In addition, it provided additional

opportunities for several individuals to develop substantial modules of topical web resources and

for several others to design and lead follow-on workshops that built upon and repurposed

resources developed initially for other Cutting Edge workshops.

Cutting Edge Workshop Participation

The 56 workshops in the Cutting Edge series have attracted a wide range of participants: 1751

faculty, post-docs, and graduate students from the full spectrum of geoscience disciplines

participated in one or more workshops. More than 20% of the participants came to two or more

workshops for a total of 2246 workshop participant-seats. The 1751 workshop participants

included 1376 faculty (out of an estimated 5,600 geoscience faculty in the U.S.) and 375

graduate students and post-docs. Participants came from all 50 states and from 467 different

institutions. As figure 1 shows, 8% of the faculty participants were from two-year colleges; 34%

from undergraduate-only departments; 15% from departments offering MA/MS degrees as their

highest degree, 39% from departments offering PhDs, and 4% classified as other, which included

scientists in research institutions and government agencies. Workshops that addressed the

teaching of a specific core geoscience specialty (e.g. Teaching Structural Geology) drew

Page 16: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 16 of 63

participants representing 10-15% of departments teaching these specialties. For comparison, the

AGI Directory of Geoscience Departments lists 757 geoscience departments, with approximately

18% as two-year colleges (134), 33% as four-year institutions (250), 14% offering a masters

degree (109), 33% offering a doctorate (247) and 2% classified as other (17).

Figure 1: Cutting Edge participation by highest degree granted by department.

Thus, Cutting Edge workshops made significant inroads into the geosciences community,

reaching approximately 25% of the geoscience faculty in the U.S., and more than half the

departments in the United States. The faculty reach of the workshops was in proportion to the

different types of departments, with the exception of two-year colleges, where the reach was

disproportionately small compared to the percentage of faculty in two-year colleges.

By two measures, the Cutting Edge workshops participants were more diverse than that of

geoscience faculty in the U.S. overall. While less than 15% of geoscience faculty were women

(AGI,2009; Fig. 2), and only 2% of recent PhD graduates in the geosciences were racial/ethnic

minorities (NSF 04-590), Cutting Edge workshops attracted 47% women and 6% minorities

underrepresented in geoscience (Black/African American, Hispanic/Latino, Native American,

Pacific Islander), more than double the percentage in the faculty population. Several people with

disabilities have also participated in Cutting Edge workshops.

8%

34%

15%

39%

4%

Cutting Edge Participation by Highest Degree Granted by Department

2 Year College (110)

4 Year Degree granted (468)

MA/MS Degree granted (206)

PhD Degree granted (537)

Other (55)

Page 17: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 17 of 63

Figure 2: Workshop participation gender compared to breakdown among all geoscience faculty.

Cutting Edge Website Contributions and Use

The Cutting Edge website aims to support workshop participants and extend the reach of the

program by providing a venue for learning about and sharing teaching resources. An important

goal has been to involve the geoscience community in developing the website in order to make it

an accurate, living, and up-to-date representation of current understanding in each topical area.

The website content is developed through the workshop process including major contributions by

workshop conveners, speakers, and participants. The project has successfully completed a

section of the website for each workshop topic. Together these comprise more than 4000 pages

of content in 33 topical collections with more than 1200 community-contributed teaching

activities reflecting extensive involvement of the community in generating an on-line resource

for use by geoscience faculty.

The website is heavily used both within and beyond the geosciences and usage has grown

consistently since it was launched. In 2008, more than half a million users visited the website

and approximately 12,000 visited 10 or more pages in a single session. For comparison, there are

47%

15%

53%

85%

0%

20%

40%

60%

80%

100%

Workshop participants (including students)

Total geoscience faculty, by gender

Workshop Participation by Gender

Women

Men

Page 18: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 18 of 63

5,600 U.S. based geoscience faculty members. Traffic to the website in 2009 reflects a 27%

growth in visitors over that of 2008. (Table X).

Table 1: Annual Number of Cutting Edge Visitors

Year Cutting Edge

Visitors

2002 1,500

2003 14,400

2004 46,600

2005 241,000

2006 301,000

2007 400,000

2008 550,000

2009 700,000

A picture of website use can be drawn from qualitative interview and survey data combined with

sampled web log analysis and overall web statistics. Users reported visiting the website to find

ideas about teaching and to compare their teaching to what other colleagues were doing in

specific teaching situations. Analysis of the web logs of website user visits reveals that

approximately 40% of all “deep session” visitors (those seeing 10 or more pages) browsed

teaching activities, 30% visited visualizations, and 20% explored a pedagogic approach. The site

also appeared to be useful to those interested in career development with more than 10% of

session visits including pages from one of the Career collections.

The sheer numbers of users, as well as data from multiple pop-up surveys confirms that the site

extends the reach of the Cutting Edge program beyond workshop participants. Approximately

25% of users are geoscience faculty with the remaining 75% comprised of faculty from other

disciplines, K-12 teachers from geoscience and other disciplines, undergraduate and graduate

students, postdocs, and others. Using these data we estimate that 3,000 of the 12,000 deep

sessions users are geoscience faculty. Overall web usage statistics suggests that 60% of the

visitors to the website were from within the United States, which applied to the 3,000, provides

Page 19: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 19 of 63

an estimate that 1,800 of the 5,500 (33%) U.S. geoscience faculty were regular users of the

Cutting Edge website. This estimate is comparable to results from the 2009 Faculty Survey

where 30% of responding faculty indicated that they had used the Cutting Edge website. 2009

faculty survey responses also indicate that 45% of those who report using the website have also

attended a Cutting Edge workshop.

Figure 3. Geoscience Users of the Cutting Edge website.

Website use and workshop participation together

Data from the 2009 Faculty Survey revealed that faculty members who participated in a Cutting

Edge workshop were more than four times as likely as non-participants to have reported ever

using the Cutting Edge website (87% vs. 20%). And faculty members who reported ever using

the website were fifteen times more likely than non-users to have participated in a workshop

(45% to 3%). Thus, there is a very high correlation between website use and workshop

participation. The following table shows the frequency of website use and workshop

participation among the respondents to the survey.

Table 2: Frequency of website use and workshop participation among respondents to 2009 Faculty Survey

Ever used the CE website Never used the CE website

Any CE workshop participation 288 44

No CE workshop participation 358 1443

12%

27%

61%

Of Those Who Teach Geoscience2007-2008

Student

K-12 Teacher

College/University Faculty

Page 20: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 20 of 63

It should be noted that because website use among workshop participants is very high (87%), any

of the findings in this report that include workshop participation should be interpreted with the

consideration that website use may also be a factor. As already demonstrated, a sizable number

of U.S. geoscience faculty members are using the website and the majority of them (55%) had

not participated in a workshop. Thus, the primary mechanisms by which the structure for

learning operated were via the website alone and the website with workshop participation.

Faculty Response to Cutting Edge

The evaluation found that faculty members who use the Cutting Edge website or participate in a

Cutting Edge workshop (which includes website use) show an increase in their learning from

others and in their use of online resources for teaching. These behaviors are the means by which

faculty with Cutting Edge exposure seem to build upon and use their Cutting Edge experiences

to make the teaching changes that will be discussed in the next section.

Increased learning about teaching from others

In general, faculty members with Cutting Edge exposure were associated with an increase in

learning about teaching from others, via different sources. For example, the findings of the 2009

Faculty Survey indicate that when designing a new activity for an introductory course, Cutting

Edge website users and workshop participants were more likely than non-users and non-

participants to look on the web for others‟ activities for introductory courses and majors courses,

and to read education research papers about the methods they were considering for introductory

courses and majors courses (Figs. 4 and 5). Additionally, Cutting Edge website users were more

likely than non-users to look for activities in texts, lab manuals, or instructor guides (Fig. 4).

Regarding the sources of their learning, the 2009 Faculty Survey found that Cutting Edge

website users and workshop participants were more likely than non-users and non-participants to

learn about new teaching methods from professional meetings or workshops, from publications,

from discussions with colleagues in other institutions, and from on-line resources (Figs. 6 and 7).

Cutting Edge website users were also more likely to learn from discussions with colleagues on

campus.

The specific behaviors indicating increased learning from others reported in the 2009 Faculty

Survey preferentially by faculty members with Cutting Edge experience included:

Seeking advice on teaching Both Cutting Edge website users and workshop participants

were more likely than non-users and non-participants to seek advice on teaching from

colleagues at the campus teaching/learning center, colleagues outside of their institution

who they knew from their geoscience research, colleagues outside of their institution who

they met through their interest in teaching, and nationally known leaders on education

(Figs. 8 and 9).

Page 21: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 21 of 63

Utilizing research. When designing a new activity for an introductory course or majors

course, Cutting Edge website users and workshop participants were more likely than non-

users and non-participants to indicate in the 2009 Faculty Survey that they read education

research papers about the methods they are considering (Fig. 10 and 11).

Motivation from others Reflecting on the last time they made a substantive revision to a

course in the 2009 Faculty Survey, Cutting Edge website users were more likely than

non-users to be motivated by a great new idea from a colleague, a publication, or the

web; and finding the ideal activity on a website or in a publication and adopting it

wholesale (Fig. 12). Workshop participants were more likely than non-participants to be

motivated by a great new idea from a colleague, a publication, or the web; and being

inspired by attending a workshop or other professional development activity (Fig. 13).

In looking at the additive effects on motivation of workshop participation to website use,

the data show that the percentage of faculty members who reported that they were

motivated by attending a workshop or other professional development opportunity that

inspired them to make changes doubled from 13% for those with no Cutting Edge

experience to 26% for website users. The percentage then jumped to 60% for website

users who also participated in at least one workshop (Fig. 14). Conversely, being

motivated to make substantive course revisions because the content needed to be updated

was less likely for Cutting Edge website users than for faculty members with no Cutting

Edge experience (52% vs. 68%). And this percentage falls again with the addition of

workshop participation, down to 34% (Fig. 15).

Looking at what others have done and making comparisons Cutting Edge website

users and workshop participants were more likely than non-users and non-participants to

consider what their colleagues were doing when revising an introductory course or a

course for majors (Fig. 16 and 17).

Page 22: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 22 of 63

Figure 4: Sources of teaching activities for Cutting Edge web users and non-users.

Figure 5: Sources of teaching activities for workshop participants vs. non-participants.

81% 79%

57%63%

32% 29%

54% 52%46% 46%

15% 13%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90% Teaching Activity SourcesWeb

No Web

INTRO INTRO INTROMAJORS MAJORS MAJORSLook on web to see what

activities others have developed

Read education research papers to learn about methods considered

Look for activities in texts,lab manuals, or instructor's

guides

78%71%

34%27%

60% 59%

19% 16%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%Teaching Activity Sources

WorkshopNo Workshop

Look on web to see what activities others have developed

Read education research papers to learn about methods considered

INTRO INTROMAJORS MAJORS

Page 23: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 23 of 63

Figure 6: Comparison of sources for learning about new teaching methods between website users and non-

users.

Figure 7: Comparison of sources for learning about new teaching methods between workshop participants and

non-participants.

73%

50%56%

62% 65%

35%27%

48% 47%37%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Professional meetings/workshops

Publications Discussions with other colleagues on

campus

Discussions with other colleagues in other institutions

On-line resources

Sources of Learning about New Teaching MethodsWeb

No Web

83%

46%

61% 63%

40%32%

50%42%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Professional meetings/workshops

Publications Discussions with other colleagues in other

institutions

On-line resources

Sources of Learning about New Teaching Methods

Workshop

No Workshop

Page 24: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 24 of 63

Figure 8: Places web users vs. non-users turn to for teaching advice.

Figure 9: Places where workshop participants vs. non-participants turn to for teaching advice.

24%

54%

35%

11%

17%

43%

14%

4%

0%

10%

20%

30%

40%

50%

60%

Colleagues at the campus teaching/learning center

Colleagues outside my institution whom I know

from my geoscience research

Colleagues outside my institution whom I met through my interest in

teaching

Nationally known leaders on education

Sources of Teaching Advice for Website UsersWebNo Web

24%

55%

40%

14%18%

45%

17%

5%0%

10%

20%

30%

40%

50%

60%

Colleagues at the campus teaching/learning center

Colleagues outside my institution whom I know

from my geoscience research

Colleagues outside my institution whom I met through my interest in

teaching

Nationally known leaders on education

Sources of Teaching Advice for Workshop ParticipantsWorkshop

No Workshop

Page 25: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 25 of 63

Figure 10: Comparison between web users/non-users for likelihood for reading education research papers to

learn about teaching methods.

Figure 11: Comparison between workshop participants/non-participants for likelihood for reading education

research papers to learn about teaching methods.

32%29%

15% 13%

0%

5%

10%

15%

20%

25%

30%

35%

Intro Courses: Read education research papers about considered

methods

Majors Courses: Read education research papers about considered

methods

Likelihood of Reading Education Research Papers to Learn about Teaching Methods

WebNo Web

34%27%

19% 16%

0%

5%

10%

15%

20%

25%

30%

35%

40%

Intro Courses: Read education research papers about considered

methods

Majors Courses: Read education research papers about considered

methods

Likelihood of Reading Education Research Papers to Learn about Teaching Methods

WorkshopNo Workshop

Page 26: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 26 of 63

Figure 12: Comparison of activity sources for website users and non-users.

Figure 13: Comparison of activity sources for workshop participants and non-participants.

47%38%

15% 15%

29% 25%

9% 7%0%5%

10%15%20%25%30%35%40%45%50%

INTRO MAJORS INTRO MAJORS

Learn From, Share With, and Use Others' Experiences, Results, and Ideas

Web

No Web

Motivated by new idea from colleague, publication,

or web

Found & adopted activitywholesale from web/

publication

47%

16%

56%62%

33%

8%17% 17%

0%

10%

20%

30%

40%

50%

60%

70%

INTRO MAJORS INTRO MAJORS

Learn From, Share With, and Use Others' Experiences, Results, and Ideas

Workshop

No Workshop

Motivated by new idea from colleague, publication,or web

Inspired by attending workshop/professional development activities

Page 27: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 27 of 63

Figure 14: Percentage of faculty members reporting that attending a workshop or other professional

development opportunity motivated them to revise their introductory course by level of Cutting Edge exposure.

Figure 15: Percentage of faculty members reporting that the need to update content motivated them to revise

their introductory course, grouped by amount of Cutting Edge exposure.

13%26%

60%

No Cutting Edge exposure Website only Website and Workshop

Motivated to make a change due to attending a workshop or other

professional development opportunty

68%

52%

34%

No Cutting Edge exposure Website only Website and Workshop

Motivated to make a change due to content needing to be updated

Page 28: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 28 of 63

Figure 17. Percent of respondents who consider

what colleagues are doing when revising a course;

compares Cutting Edge workshop participants vs.

non-participants.

31% 31%

24%21%

0%

5%

10%

15%

20%

25%

30%

35%

Intro Majors

Consider What Colleagues are Doing when Revising a Course

Web

35% 34%

25%22%

0%

5%

10%

15%

20%

25%

30%

35%

40%

Intro Majors

Consider What Colleagues are Doing when Revising a Course

Workshop

No Workshop

Increased Use of Online Resources for Teaching

In interviews, Cutting Edge website users described a variety of behaviors that were related to

seeking ideas for teaching including:

browsing through topical modules to find ideas when they were conceiving a class for the

first time,

browsing through activities related to a specific topic,

looking for examples of using a specific teaching method,

looking for new ways to teach, and

looking for data that could be used to teach a particular topic.

Of the 30 website users interviewed, 25 described using the website to find ideas for teaching.

These users described using the website as a resource to learn in depth about a particular

teaching method (e.g. think-pair-share, teaching metacognition), teaching tool (e.g. Google

Earth, rubrics), education or geoscience topics (e.g. early earth, affective domain, or assessment).

Figure 16: Percent of respondents who consider

what colleagues are doing when revising a course;

compares Cutting Edge website users vs. non-users.

Page 29: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 29 of 63

Nine of 30 interviewees reported using the site to learn about a new field, to strengthen a content

area of weakness, or to learn more about pedagogy. Comparable findings came from the 2009

Faculty Survey. When asked about their use of on-line teaching resources within the past two

years, 19% of respondents indicated that they learn about the methods they will be using and

45% learn about the content they will be teaching.

The 2009 Faculty Survey shows that Cutting Edge website users and workshop participants were

more likely than non-users and non-participants to have used on-line teaching resources in the

past two years to surf for ideas for teaching, to download materials to use in class, to find

materials for students to use in assignments, and to learn about the methods they will be using

(Fig. 18 and 19). In web user interviews conducted from 2005 to 2009, users reported using the

website to help them to incorporate teaching activities from a given topic, to get information

about course design, to prepare to teach a course for the first time, to prepare to teach a course

ahead of time, and to change or revamp a course.

Figure 18: Uses of online teaching resources by Cutting Edge website users in last two years.

70%85%

64%

31%

52%

77%

51%

17%0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Ideas for teaching Download materials to use in

class

Find materials for students to use in

assignments

Learn about teaching methods

Uses of Online Teaching Resources in Last Two Years

Web

No Web

Page 30: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 30 of 63

Figure 19: Uses of online teaching resources by Cutting Edge participants in last two years.

Responses to the 2008 participant survey confirmed these findings with 61% of respondents

indicating that seeking ideas for teaching was the most valuable aspect of their use of the Cutting

Edge website. When asked about their use of on-line teaching resources within the past two

years, 55% of respondents indicated that they surfed for ideas for their teaching, 78%

downloaded materials to use in class, and 53% found materials for students to use in

assignments. Faculty primarily adapted these ideas for use in their teaching; however, users also

reported using materials wholesale in their courses, particularly if they were teaching out of field.

These descriptions(with the names changed), drawn from the qualitative data (interviews and

open-ended survey questions), exemplify the types of use typically described.

David, an Early Career 07 participant, puts time into preparing his lectures to

emphasize important points, make them more interactive, and keep the

students involved, using techniques such as incorporating science news and

popular media. He went to the Cutting Edge website several times to mine it.

Dot, an Early Career 08 participant, incorporated the use of web-based

practice materials to prepare students for assessments. She administered a

mid-term survey to assess how her contributions and the website influenced

their learning and had them reflect on their effort and performance in class.

She used the Cutting Edge website frequently for lecture content, practice

problems, and ideas for revamping lab activities.

74%85%

64%

30%47%

75%

48%

15%0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Ideas for teaching Download materials to use in

class

Find materials for students to use in

assignments

Learn about teaching methods

Uses of Online Teaching Resources in Last Two Years

Workshop

No Workshop

Page 31: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 31 of 63

The website seemed to be a valuable resource for faculty to explore and learn pedagogy and

educational innovations in depth, which impacted their teaching and supported their ability to

participate in educational innovation in community discussions.

Using Cutting Edge to make comparisons

Evidence that website users were searching for and looking at the contributions made by others

comes from phone interviews in which 12 of 30 interviewees reported using the website to find

out what others were doing or how they were teaching. Users reported a wide variety of

behaviors that can be described as norming: trying to determine how their own behavior

compares to others in the community. Others reported behaviors that included looking at syllabi

and activities to determine if their course covered similar content to those taught by others;

looking at activities to determine how their approach to a specific topic compared to that of

others; looking at activities to determine the level of sophistication or effort that others were

expecting from their students; and looking at examples of teaching methods to determine if

methods were in widespread use or being used to teach the topics that they teach.

These behaviors of checking on what others were doing were also reported in follow-up

interviews specific to workshop Action Plans. The following example is how one workshop

participant (with the name changed) had been using the website in this way.

Sally, an Introductory Geoscience participant, went to the Cutting Edge

website every time she wanted a change to see if someone else had

implemented the change and may have suggestions for a better way to do it.

She used the website to add Think-Pair-Share, Jigsaws, and group experiences

to facilitate student participation.

An interesting finding from the 2009 Faculty Survey was that Cutting Edge website users and

workshop participants were less likely than non-users and non-participants to report that the use

of on-line resources had increased their own knowledge of a particular topic (61% vs. 73% for

website; 60% vs. 71% for workshop). The Cutting Edge programming may be more beneficial

for teaching methods than for geosciences content.

Page 32: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 32 of 63

The Cutting Edge Impact on Teaching

On the Cutting Edge aimed to change faculty members‟ teaching practices in order to positively

impacting student learning. Making such changes requires first a change in attitude toward

student-centered approaches to teaching, followed by new knowledge and skills about

appropriate teaching methods. Knowledge and skills in the absence of a student-centered

approach to teaching is unlikely to lead to change (Barr and Tagg, 1995, Pellegrino et al., 2001).

The findings from this evaluation suggest that Cutting Edge programming develops both a

student-centered approach and the knowledge and skills needed to implement appropriate

teaching methods. Participation is associated with changes in classroom teaching that include

course design and goal setting, use of lecture, use of activities, use of tools, and use of student

assessments. Further examination indicates that these changes are grounded in an immediate and

sustained shift in teaching philosophy and confidence that shapes a faculty member‟s approach

to teaching.

Changes in Teaching

Website users, including both those who have participated in Cutting Edge workshops (~45%)

and those who have not (~55%) report impacts to their teaching. In the 2009 Faculty Survey,

Cutting Edge website users were more likely than non-users to report having changed the

teaching methods used in their introductory courses in the past two years. Specifically, they were

more likely to have spent less time lecturing, increased the questioning of students during

lectures, added group work or small group activities, spent more time on class discussions or

small group discussions, changed assessment tools or strategies, and added assignments (Fig.

20). As it did with Cutting Edge website users, the 2009 Faculty Survey found that Cutting Edge

workshop participants were more likely than non-participants to report having made any changes

in the teaching methods used in their introductory courses in the past two years.

Page 33: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 33 of 63

Figure 20: Impact of website use on use of various teaching practices in the classroom.

Figure 21: Impact of workshop participation on use of various teaching practices in the classroom.

These data are born out in telephone interviews. 45 of 54 past workshop participants in 2005 and

21 of 30 website users in 2008 and 2009 noted that participation had a major impact on their

teaching. Interviewees described specific changes in teaching techniques and activities in their

70%

37%

47%

32%

25%

35%

25%

54%

21%

32%

15% 15%

22%17%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Made changes in teaching

methods used in intro courses over past two

yrs

Spent less time lecturing

Increased questioning of

students during lecture

Added group work/small

group activities

Spent more time on

class/small group

discussions

Changed assessment

tools/strategies

Added assignments

Impacts of CE Participation on TeachingWeb

No Web

71%

41% 48%30%

38% 37%

57%

25%35%

16% 18% 24%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Made changes to teaching method

in intro course over the past two

yrs

Spent less time lecturing

Increased questioning of

students during lecture

Added group work/small group

activiites

Spent more time on class/small

group discussions

Changed assessment

tools/strategies

Impacts of CE Participation on TeachingWorkshop

No Workshop

Page 34: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 34 of 63

courses resulting from their participation in the Cutting Edge program. Similarly, in the 2009

Action Planning Follow-up Survey, 35 out of 43 respondents who participated in an Introductory

Geology or Early Career workshop described specific pedagogical changes that they attributed to

their workshop participation, and 20 made statements about the impact of these changes on

teaching effectiveness. Eleven respondents reported course changes as a result of their workshop

experience. Seventy-one percent of respondents to the 2008 Participant Survey described a

change to their teaching, 85% indicated that as a result of the program they changed their

teaching to incorporate more active learning techniques in their classes, and 87% reported at

least one course was changed as a result of their participation.

As discussed earlier in this report, the use of online resources is an important part of the Cutting

Edge structure for learning. The charts in figures 22 and 23, show that when asked in the 2009

Faculty Survey how the use of on-line resources has positively impacted their teaching within the

past two years, Cutting Edge website users and workshop participants were more likely than

non-users to report that it increased the variety of methods that they use, increased their skill

with a particular teaching method, increased their confidence as a teacher, and increased their

ability to assess student learning.

Figure 22: Impacts of using online teaching resources for web users vs. non-users.

71%

27% 22% 25%

51%

18% 16% 12%0%

10%

20%

30%

40%

50%

60%

70%

80%

Increased variety of methods used

Increased skill with teaching method

Increased confidence as a teacher

Increased ability to assess student learning

Impact on Teaching BehaviorWebNo Web

Page 35: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 35 of 63

Figure 23: Impacts of using online teaching resources for workshop participants vs. non-participants.

Areas of particular note are the changes that occurred in course design and goal setting, the use

of lectures, the use of activities, the use of teaching tools, and the use of student assessments.

Changes in course design and goal setting

In the 2008 Participant Survey, 83% of respondents stated that the workshop affected the way

they use goals and objectives to design their course(s). Similarly, in the 2005 Impact Survey, 234

workshop participants (half of whom used the website for course design) were asked to indicate

the degree to which the Cutting Edge program (workshop or website) had impacted the goals and

objectives that they set for their course(s). Sixty-three percent indicated that it caused them to

make changes and an additional 26% indicated that it raised their awareness in the area. For the

56 respondents that checked participation in one of the course design workshops, 51 (92%)

reported that the workshop experience caused them to make changes with the remaining 5

respondents reporting that it raised their awareness.

To get a better picture of the ways in which Cutting Edge impacts goal setting, we can look at the

2009 Faculty Survey results regarding introductory course design. In designing their most recent

introductory course or activities for it, Cutting Edge website users were more likely than non-

users to address student anxiety about specific activities (54% vs. 39%), while Cutting Edge

workshop participants were more likely than non-participants to address student comfort in the

intellectual environment of the course (76% vs. 64%). As reported earlier, Cutting Edge website

users were more likely than non-users to consider the following factors to be of major

importance in setting goals for their most recent introductory course: developing problem solving

skills, developing interpersonal skills, and increasing student awareness of the utility of

geoscience in addressing important problems. And Cutting Edge workshop participants were

72%

30% 28% 26%

55%

19% 16% 14%0%

10%

20%

30%

40%

50%

60%

70%

80%

Increased variety of methods used

Increased skill with teaching method

Increased confidence as a teacher

Increased ability to assess student learning

Impact on Teaching Behavior

Workshop

No Workshop

Page 36: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 36 of 63

more likely than non-participants to consider developing problem solving skills and interpersonal

skills to be of major importance.

Changes in classroom activities

Less use of traditional lectures

Findings from the 2009 Faculty Survey reveal that compared to non-users, Cutting Edge website

users were less likely to report using traditional lecture often. When asked about teaching

changes made in their introductory courses in the last two years, website users with no workshop

experience were more likely than faculty with no Cutting Edge experience to report that they

spent less time lecturing (32% vs. 22%) and added group work or small group activities (25% vs.

15%). As shown in Figure 24, these percentages increase further with workshop participation.

Overall, Cutting Edge website users were more likely to report “often” use of:

posing questions that are answered by individual students,

posing questions that are answered simultaneously by the entire class,

small group discussion or think-pair-share, and

in-class exercises (Fig. 25).

While Cutting Edge workshop participants were neither more or less likely than non-participants

to use traditional lectures in their introductory courses, they were more likely to report “often”

use of:

posing questions that are answered simultaneously by the entire class,

small group discussion or think-pair-share, and

in-class exercises (Fig. 26).

This question was also analyzed for possible combination effects of adding workshop

participation to website use. As shown in Figure 27, adding workshop participation to website

use increased the likelihoods of using these techniques often by 12 to 16 percentage points.

Page 37: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 37 of 63

Figure 24: Percentage of faculty members reporting changes made in their introductory courses demonstrating

the association with website use and website use with workshop participation.

Figure 25: Impact of website use on use of lecture in the classroom.

15%

22%25%

32%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

Added group work or small group activties

Spent less time lecturing

Changes made in last two years by Cutting Edge exposure

No Cutting Edge exposure

Website only

Website and Workshop

40%43%

78%71%

59%

35%41%

87%

59%

42%

17% 20%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Traditional lecture Posing questions answered by

individual students

Posing questions answered by entire class

Small group discussion or

think-pair-share

In-class exercises

Cutting Edge Impact on Use of LecturesWebNo Web

Page 38: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 38 of 63

Figure 26: Impact of workshop participation on use of lecture in the classroom.

Figure 27: Percentage of faculty members reporting use of teaching strategies in their introductory courses

demonstrating the association with website use and website use with workshop participation.

Increased use of interactive activities

Both web usage statistics and workshop surveys indicate that Cutting Edge influences users and

participants to make changes in their in-classroom activities that increase student involvement.

61%

39%48%44%

19% 23%

0%

10%

20%

30%

40%

50%

60%

70%

Posing questions answered by entire class

Small group discussion or think-pair-share

In-class exercises

Cutting Edge Impact on Use of LecturesWorkshop

No Workshop

42%

17% 19%

53%

28%34%

65%

44%50%

0%

10%

20%

30%

40%

50%

60%

70%

Posing questions answered by entire

class2

Small group discussion or think-

pair-share

In-class exercises

Teaching Strategies Used Weekly or Nearly Every Class by Cutting Edge Exposure

No Cutting Edge exposure

Website only

Website and Workshop

Page 39: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 39 of 63

Web statistics reveal that 40% of deep sessions (sessions seeing more than 10 pages) included

viewing activity pages. Over half of the participants attending an Early Career workshop (16 of

27) and almost a third of the Introductory Geology participants (5 of 16) reported in the 2009

Action Planning Follow-up Survey that they had made changes intended to increase student

involvement. Specifically mentioned were the use of demonstrations, group work, ice-breaker

activities, in class assignments, in-class discussions, interactive lectures, lecture tutorials, peer

review exercises, reflective analysis, student presentations, student projects, Think-Pair-Share,

and web-based practice materials.

In their most recent course, Cutting Edge website users and workshop participants (in the 2009

Faculty Survey) were more likely than non-users and non-participants to report using problem

solving activities that addressed a problem of national or global interest (72% vs. 62% for

website users; 75% vs. 63% for workshop participants). Additionally, Cutting Edge website

users were more likely than non-users to report using problem solving activities that students

posed and solved on their own (16% vs. 10%), and in which students developed a geologic

history of a field (45% vs. 33%).

Use of tools to support teaching methods

In addition to providing information and methods for improving teaching, Cutting Edge also

provided tools, such as visualizations, to support website users and workshop participants in

making those teaching changes. In the 2009 Action Planning Follow-up Survey, 11 of 54

respondents reported without prompting that they used tools or resources from the Cutting Edge

website in their teaching such as visualizations, animated demonstration, and clickers. Feedback

from workshop participants suggests that they valued learning about new tools and resources

(particularly being introduced to tools by people who developed the tool or actively use it). In

reviewing end of workshop summaries and road checks for 11 workshops conducted between

2006 and 2009, evidence that participants valued learning about new tools and resources

surfaced in all 11 of these reports. Participants valued the exposure to new tools and resources

and seeing how they could use them in their own teaching.

While finding visualizations was not a major theme in interview and open-ended survey

responses (for example, only six of the 30 website phone interviews), quantitative survey data

and the web use statistics repeatedly show that using the website to find images for use in

teaching was a common type of use. In consecutive pop-up surveys in 2005 and 2006, a little

over a quarter (26% and 28%) of the website users indicated that they came to the site for

visualizations. Visualization collections On the Cutting Edge website, which through March

2009 were identifiable within the Visualization module, were visited in 30% of all the deep

sessions in 2008 and the Visualization module was the most popular single area of the site. Deep

session visits with substantial use of visualization collections show that users were looking

through large numbers of visualizations on a single topic and branching to pages that discussed

Page 40: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 40 of 63

the use of those visualizations. On average, users in these sessions saw 7 pages in the

visualization collection and also saw 23 pages from elsewhere on the site.

Changes in assessment

The Cutting Edge exposure may change the way faculty conduct assessments and increase the

breadth of assessment methods that they use. Twenty-five percent of respondents to the 2008

Participant Survey wrote about specific Cutting Edge assessments that they used and discovered

either through workshops or on the website. The 2009 Action Planning Follow-up Survey found

that 16 of 43 respondents who attended an Introductory Geology or Early Career workshop

described making changes to their assessments or rubrics following the workshop.

In 2007 interviews conducted with 25 Cutting Edge workshop participants and 14 non-

participants at the AGU annual meeting, the participants reported a broader range of assessments

that included observing levels of student engagement, conversations with students, course

evaluations, as well as quizzes, reports and exams. The non-participants reported assessments

that only included quizzes, reports, and exams.

Shifting How Faculty Approach Teaching

I got turned on to pedagogy and how it works. There are all these tools and

techniques out there. In all the workshops, it is about engaging people and

being engaged by the material.

– Leadership interview participant

Surveys and interviews found that the Cutting Edge experience, whether via the website or

workshops, was associated with changes that reflected increases in student-centered learning

approaches to teaching. The basis for this shift was most evident in changes in the teaching

philosophy of faculty members, but evidence also indicates that Cutting Edge helped faculty

members to increase their confidence in using new teaching methods.

Changes in teaching philosophy

An indication that Cutting Edge helps to shift the way faculty think about orienting their teaching

comes from the 2009 Faculty Survey questions that asked respondents to reflect on what

motivated them to make a change the last time they made a substantive revision to a course.

Cutting Edge website users were more likely than non-users to report that they were motivated to

make the change for an introductory course because they adopted a new philosophy for their

teaching (31% vs. 18%). In the case of Cutting Edge workshop participants, they were also more

likely than non-participants to report that they were motivated to make the change because they

Page 41: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 41 of 63

adopted a new philosophy for their teaching, irrespective of whether it was for an introductory

course (31% vs. 20%) or a majors course (32% vs. 21%).

At the same time, Cutting Edge website users and workshop participants were less likely to

indicate that the need to update the content was the motivation for making a revision whether for

an introductory course (44% vs. 68% for website users; 37% vs. 64% for workshop participants)

or a majors course (46% vs. 71%; 35% vs. 68% for workshop participants).This shift away from

a content focus and attributing a new philosophy for making change is evidence that adopting a

student-centered approach to teaching leads to actual (self-reported) changes in teaching practice.

Suggesting a possible shift in their view of the effectiveness of student-centered teaching

methods, the 2009 Faculty Survey found that Cutting Edge website users and workshop

participants were more likely than non-users and non-participants to indicate that student

learning had been impacted in their most recent introductory course by providing opportunities

for low-stakes practice before high-stakes assignments, activities, or exams; by including

activities that allow students to get to know one another; and by using group projects, field work,

or other activities that promote teamwork and collaborative learning (Figs. 28 and 29).

Figure 28: Affective domain strategies having largest impact on student learning for Cutting Edge website users

vs non-users.

62%54% 56%

66%39% 42%

0%

10%

20%

30%

40%

50%

60%

70%

Practice before exams Activities to get to know each other

Teamwork/collaborative learning

Affective Domain Strategies Having Largest Impact on Student Learning

Web

No Web

Page 42: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 42 of 63

Figure 29: Affective domain strategies having largest impact on student learning for Cutting Edge workshop

participants vs non-participants.

Survey results from different sources indicate that the shift toward student-centered learning

experienced by Cutting Edge workshop participants was both immediate and sustained. In a 2004

participant survey in response to an open-ended question asking about the impact of the

workshop on their teaching, 42 out of 95 participants indicated that they thought more critically

about teaching. Of 274 past workshop participants responding to the 2008 Participant Survey,

91% indicated that they shifted their thinking from “What do I teach?” to “What are my students

learning?” Less than 5% reported that the workshop did not shift their thinking in this way. In

2007 face-to-face interviews, Cutting Edge workshop participants offered unsolicited feedback

that the workshops influenced their philosophy to be more student-centered. In 2005, 70% of

past workshop participants (54 in the sample) articulated this change in attitude in telephone

interviews using words such as “eye opener,” “seismic shift,” and “new twist.” At that time, four

percent of the participants were early in their career and did not feel they had a shift in their

attitude because of the research nature of their tenure-track positions.

An example of this shift occurring as a result of a workshop comes from the 2008 Metacognition

workshop embedded assessment, in which the 29 participants were asked before and after the

workshop about the features that they looked for in a strong teaching activity. The number of

participants indicating that they looked for aspects of the importance of students‟ prior

knowledge, beliefs, or self-regulation increased from 3 before the workshop to 25 after the

workshop.

70%59% 56%

48%41% 45%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Practice before exams Activities to get to know each other

Teamwork/collaborative learning

Affective Domain Strategies Having Largest Impact on Student Learning

Workshop

No workshop

Page 43: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 43 of 63

Evidence of a sustained shift was found in the action plans developed at workshops, and end-of-

workshop survey write in comments from 2007 and 2008 workshop participants. The contents of

those plans and comments indicated a prevailing change in participants‟ focus from content and

material development to student learning. When a sample of these participants were surveyed

one to two years out from the workshop as to how the changes they anticipated at the workshop

were working, their responses indicated that the Cutting Edge workshop had played an important

role in facilitating the adoption of a new philosophy of student-centered learning.

I used to approach course and lesson development based on delivering

material. Now I feel like I am actively assessing their grasp of concepts and

terms continually, rather than only on exams and graded assignments.

– Action Plan Follow-up Survey Respondent

The workshop had a powerful impact on my thoughts of teaching philosophy,

highlighting the need for different approaches to get students to learn the

material, including group work, and hands-on/inquiry-based approaches to

subject matter that is normally covered in a (dry) lecture-based format.

– Action Plan Follow-up Survey Respondent

Increases in confidence

In the 2009 Faculty Survey, both Cutting Edge website users and Cutting Edge workshop

participants were more likely than non-users and non-participants to report that using online

resources had increased their confidence as a teacher (22% vs. 16% for website users, 28% vs.

16% for workshop participants). In interviews, website users reported that having access to the

resources on the website gave them more confidence in trying new things, more comfort in

teaching topics outside of their sub-discipline, and greater confidence in their own teaching

ability and content. In particular, users noted that knowing the resources on the website had been

used by other colleagues made it a trusted source, such as in the following quote.

I would say that it has built confidence for me, and I guess, mainly, the

confidence to try new things and things that I may not have, you know,

necessarily gone and done. It sort of provides an avenue of, you know, okay,

here is an idea. This is something you can try in your classroom. This has been

done. It has been successful. If it isn’t successful, this is what you can do.

- Website user interviewee

Page 44: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 44 of 63

Workshop participants were not asked directly about changes in confidence resulting from their

workshop experience; however, their responses to other questions and their unsolicited

comments would indicate that confidence was enhanced for some of them. In reviewing end of

workshop summaries and road checks for 28 workshops between 2006 and 2009, 21 contained

strong evidence that the participants gained confidence in their ability to teach, their attitude

towards their career, and the material that they use in teaching. The following comments are

characteristic of how participants expressed their increase in confidence.

I've been resisting the move to incorporate more energy in my courses because

of not really knowing the best way to get started. I now have plenty of ideas

and it no longer seems such a daunting task.

- Energy ’09 workshop participant

This workshop has given me the tools I need to really move forward; complete

the transition to a ‘senior’ faculty member; and succeed in becoming the

educator and scientist that I want to be.

- Early Career ‘09 workshop participant

Impact of Cutting Edge Experience on Student Learning

The evaluation for Cutting Edge did not collect data that would directly measure any direct

impacts on student learning. However, anecdotal evidence was revealed in the course of data

collection that would indicate student learning had been impacted by the teaching changes that

faculty members learned from Cutting Edge.

In the 2008 Participant Survey, 61% of respondents indicated that they had seen changes in

content understanding in students as a result of changes they made to their teaching.

Additionally, 45% indicated changes in students‟ skill development, 45% indicated changes in

students‟ attitude toward science, and 47% indicated changes in students‟ motivation to learn.

In 2005 telephone interviews, 33 of 54 workshop participants reported increases in student

participation, student engagement, and student comments for course evaluations that reflected

improved learning. As reported earlier in this report, the 2009 Faculty Survey found that Cutting

Edge website users and workshop participants were more likely than non-users and non-

participants to indicate that strategies that are consistent with a student-centered learning

approach had the largest impact on student learning.

Page 45: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 45 of 63

The Cutting Edge Impact on the Geoscience Education Community

To complete the picture of participation in the cycle of educational innovation, the evaluation

looked at the possible impact of Cutting Edge on the broader geoscience education community,

specifically on faculty networking and their contributions which result in new knowledge that

will be accessible to others via the structure for learning.

The findings suggest that just as Cutting Edge exposure is associated with more learning from

others and greater use of the website, it is also associated with an increase in networking and

contributions to the geoscience education knowledge base.

Networking

One of the best parts about this workshop was having so many colleagues to

talk with about teaching. This is what made the experience so much better

than just a compilation of new teaching materials.

- Geomorphology 2008 Participant

Ninety-eight percent of website users and 99 percent of workshop participants reported in the

2009 Faculty Survey that they corresponded with colleagues about teaching over the past two

years. Cutting Edge experience was associated with much higher rates of faculty communication

on a variety of teaching issues. Cutting Edge website users and workshop participants were more

likely than non-users and non-participants to frequently communicate with colleagues about new

ideas in pedagogy, specific assignments or activities, problems encountered in teaching,

classroom management issues, and how to assess student learning (Figs. 30 and 31).

Additionally, website users were more likely to frequently communicate with colleagues about

course design (51% vs. 43%).

Page 46: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 46 of 63

Figure 30: Topics of frequent conversation with colleagues for web users and non-users.

Figure 31: Topics of frequent conversation with colleagues for workshop participants and non-participants.

Cutting Edge Workshop participation and networking

Workshop participation appears to have influenced behaviors related to networking with others.

In the 2008 Participant Survey, 87% of Cutting Edge workshop participants reported that they

talked more to their colleagues about teaching after their workshop participation than before.

53%

68% 72%

41%

62%

30%

50%57%

29%

46%

0%

10%

20%

30%

40%

50%

60%

70%

80%

New ideas in pedagogy

Specific assignments or

activities

Problems I encounter in my

teaching

Classroom management issues

How to assess student learning

Topics of Frequent Communication with ColleaguesWebNo Web

53%67%

77%

40%

60%

35%

54% 59%

31%

49%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

New ideas in pedagogy

Specific assignments or

activities

Problems I encounter in my

teaching

Classroom management issues

How to assess student learning

Topics of Frequent Communication with ColleaguesWorkshopNo Workshop

Page 47: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 47 of 63

Frequently mentioned in every end-of-workshop summary for 28 workshops between 2006 and

2009, was that one of the most valued aspects of the workshops was the opportunity to

collaborate, network, or to form support groups. In 2009 end of workshop surveys, 91% of

workshop participants indicated that they planned to use the Cutting Edge website to share what

they knew with colleagues and 85% listed specific people whom they planned to contact. For

example, in the 2009 workshop, Teaching About Energy in Geoscience Courses: Current

Research and Pedagogy, 33 of 34 participants said they had formed networks of people that they

would contact in the future and most of these individuals identified members of their network by

name. In 2005, 38 of 54 telephone interviewees indicated that the ability to network with others

was valued and shaped their learning. Twenty percent of those responding to the 2008

Participant Survey indicated in write-in comments that networking (as a means of community

connection) was the most important impact of the program.

Providing an introduction to networking

Some workshops included time for peer review and feedback activities. This provided an

introduction for some, and the opportunity for all, participants to engage in a structured

networking activity around teaching. Evidence that participants valued the activity review and

associated discussion surfaced in 6 of 11 summaries for workshops that offered participant peer-

reviewed activities conducted from 2006 to 2009. The following is an example of a typical

comment on the value of peer review and feedback in the workshops.

The review was a learning experience and confidence builder – I noticed that

others need to make the same adjustments and encounter similar problems

that I do. I will continue to modify and tweak my labs to stimulate critical

thinking and student interest.

- Geomorphology 2008 Participant

Participants valued having their peers critically review their activities and learning from the

group discussions. The activity review and associated discussion at the workshops not only

improved the quality of the activity collection but also helped them to think more critically about

their teaching and what constituted a “good activity.”

Cutting Edge workshops also provide opportunities for geoscientists to interact with experts in

education research and psychology. A review of end of workshop summaries for four emerging

theme workshops specifically related to education research conducted between 2005 and 2009,

found that participants in these workshops on education topics, metacognition, affective domain,

and assessment valued interactions with education researchers and psychologists, as well as

workshop content related to education. As the following comment illustrates, participants valued

Page 48: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 48 of 63

the perspectives and insights that educational researchers and psychologists brought to the

discussions on geosciences education.

I am thrilled to find myself in a community of geoscientists interested in

understanding how students learn. The other very positive aspect of this

workshop is the inclusion of educational psychologists and their willingness to

work with our community to increase student success.

- Metacognition 2008 Participant

Expanding the scale of professional networks

In addition to providing networking opportunities and increasing communications about

teaching, Cutting Edge workshops may also influence the extent of professional networks. In

2007 interviews conducted at an AGU meeting, 16 of 24 (67%) Cutting Edge participants

characterized their teaching network on a national scale compared to only 4 of 14 (29%) non

Cutting Edge respondents, and 10 of 24 (42%) Cutting Edge participants characterized their

teaching network as broader than their discipline compared to only 4 of 14 (29%) non Cutting

Edge respondents (Fig. 32).

Figure 32: Extent of professional network for Cutting Edge workshop participants vs. non-participants.

Cutting Edge website use and networking

While the interviews with 30 website users found that they did not seem to think of the Cutting

Edge website as a networking tool, some of them did report that they would look for content or

material from specific individuals and some had made initial contact with colleagues due to

finding them on the website. Additionally, data cited earlier from the 2009 Faculty Survey

67%

42%

29% 29%

0

10

20

30

40

50

60

70

National Broader than discipline

Pe

rce

nt

of

me

mb

ers

su

rve

yed

fo

r p

arti

cip

ants

/no

n-p

arti

cip

ants

Scale of Professional NetworkCE Participants

Non-Participants

Page 49: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 49 of 63

demonstrates that Cutting Edge website users were more likely to engage in networking

activities and to use the web for sharing information than were non-users. This sharing and

networking was much more likely to be reported by website users who were also workshop

attendees. Career development was a common use of the website reported by website users.

The Cutting Edge website as a career development networking tool

A sub-group of users reported that they used the website to support their career planning and

networking with other faculty. In phone interviews, 16 of 30 interviewees reported using the

website for professional development or networking purposes. In the 2007-2008 pop-up survey,

22% of survey respondents indicated that career planning was the motivation behind their visit.

These users were geoscience faculty, students, and others. In interviews, faculty described using

the site to identify experts to contact with questions, to share information, and to seek

information for career planning.

Contributing to the geoscience education community

There are several indicators from the 2009 Faculty Survey that point to a relationship between

Cutting Edge and making contributions to the geoscience community. Cutting Edge website

users and workshop participants were more likely than non-users and non-participants to have

presented research on teaching methods or student learning at meetings within the past two

years, published about educational topics within the past two years, shared materials from their

courses in the last two years via talks, and publish materials from their courses in journals. And

when they had done something that was particularly successful in class, Cutting Edge website

users and workshop participants were more likely than non-users and non-participants to tell

colleagues, and publish a paper about their activity (Figs. 33 and 34).

Page 50: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 50 of 63

Figure 33: Comparison between web users and non-users for community contributions beyond Cutting Edge.

Figure 34: Comparison between workshop participants and non-participants for community contributions

beyond Cutting Edge.

In interviews conducted in 2005 and 2007, 29 of 79 workshop participants expressed either an

affirmation of their attitude toward teaching or change in attitude toward teaching that motivated

27%

16%

31%23%

71%

8%10%6%

14%6%

54%

4%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Has presented research on

teaching methods or

student learning at meetings

within the past two years

Published about educational

topics 1 or more times within the past two years

Share course materials via

talks

Published in journals

Tell colleagues whom I know

will be interested

Publish a paper about my activity

Community Contributions By Website UsersWeb

No Web

29%

16%

30%22%

70%

9%13%8%

17%9%

57%

4%0%

10%

20%

30%

40%

50%

60%

70%

80%

Has presented research on

teaching methods or

student learning at meetings

within the past two years

Published about educational

topics 1 or more times within the past two years

Share course materials via

talks

Published in journals

Tell colleagues whom I know

will be interested

Publish a paper about my activity

Community Contributions By Workshop ParticipantsWorkshopNo Workshop

Page 51: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 51 of 63

them to contribute to the geosciences education community. Their new attitude or newly

invigorated attitude inspired participants and enhanced their ability to join conversations about

teaching and geosciences education on their campus and nationally. Participants reported that

they were motivated or encouraged to present at meetings and at other campuses, to consider

publishing on education, and to pursue grants related to education. While participants reported

that their inherent interest in teaching may have eventually led them down this path, their

participation in the workshops accelerated and motivated this interest.

Evidence of other scholarly output resulting from Cutting Edge workshops comes from a few

sources. Over a quarter of the respondents to the 2008 Participant Survey indicated that they

could attribute presentations (27%) and posters (26%) at regional or national meetings to their

Cutting Edge participation. Slightly less than a quarter of these survey participants attributed

contributions to websites (24%), department presentations (21%), and proposals submitted (22%)

to their participation. In the 2005 Impact Survey, 116 of 230 participants (50%) listed specific

presentations, publications, grants, or grant proposals resulting, either directly or indirectly, from

their participation in the program. An example of how quickly this sharing can occur comes from

the end of workshop survey for the 2007 Affective Domain workshop when three weeks after the

workshop, 9 of 18 participants reported that they had already disseminated what they learned.

Six of them submitted and latter reported that they received grant funding for continued work in

this area.

Conversations started at workshops around topics as diverse as metacognition and teaching

structural geology have continued with new audiences at professional society meetings in

sessions led by workshop participants. From 2003-2007, alumni of Cutting Edge workshops

made over 1100 scholarly contributions (presentations at meetings, journal articles, book

chapters) documenting what these faculty have done in their own settings to support geoscience

education reform (bibliometric survey based on educational topics as defined in Salisbury, 2008).

Forty individuals attribute their successful grant awards to participation in a Cutting Edge

workshop and participants frequently comment on the value of personal connections made

through the workshops.

Sharing on the Website

The 2009 Faculty Survey found that making contributions to the website is still not a highly

prevalent activity among faculty. However, when Cutting Edge website users and workshop

participants had done something that was particularly successful in class, they were much more

likely than non-users and non-participants to report that they added their activity to an online

collection (8% vs. 1% for website users; 11% vs. 2% for workshop participants). While the

percentages are small, the magnitude of the difference compared to non-users and non-

participants is a good sign for the potential to continue to impact this behavior and raise the

norm. An indication of the potential Cutting Edge role for online contributions comes from end

Page 52: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 52 of 63

of workshop surveys in 2009, in which 91 percent of participants indicated that they planned to

use the Cutting Edge website to share what they knew with colleagues.

Research and Collaboration

In the 2005 Impact Survey, 116 of 235 participants (50%) listed specific presentations,

publications, grants, or grant proposals resulting, either directly or indirectly, from their

participation in the program. Over a quarter of the respondents to the 2008 Participant Survey

indicated that they could attribute presentations (27%) and posters (26%) at regional or national

meetings to their Cutting Edge participation. Survey respondents also credited their workshop

participation for contributions they made to websites (24%), department presentations (21%),

and proposals they submitted (22%). Forty individuals have attributed successful grant proposals

to participation in a Cutting Edge workshop.

Cutting Edge use and participation is associated with more scholarly activity specifically focused

on pedagogical research. Fifty-nine of the 274 respondents (22%) to the 2008 Participant Survey

attributed proposals that they submitted to their participation in a Cutting Edge workshop and 40

(15%) indicated that they were awarded grants that they also attributed to their workshop

participation. As an outgrowth of the 2007 Affective Domain workshop, 11 of the participants

proposed and received funding for the collaborative research grant GARNET (Geoscience

Affective Research Network) which examined the connection between instruction, student affect,

and geoscience learning outcomes From the 2004 Teaching with Visualizations workshop, 4 of

the participants formed a collaboration with the Spatial Intelligence Learning Center bringing

geoscience education into one of their funded working groups.

Interestingly, the 2009 Faculty Survey found that Cutting Edge website users and workshop

participants were less likely than non-users and non-participants to have published any articles

about their research within the last two years (80% vs. 88% for website; 77% vs. 85% for

workshops). However, they were more likely than non-users and non-participants to have

presented research on teaching methods or student learning at meetings within the past two years

(27% vs. 10% for website; 29% vs. 13% for workshops); and they were at least twice as likely to

have published about educational topics within the past two years (16% vs. 6% for website; 16%

vs. 8% for workshops) (Figs. 35 and 36). Additionally, when they had done something

particularly successful in class, they were more likely to tell colleagues (71% vs. 54% for

website; 70% vs. 57% for workshops), many times more likely to add their activity to an online

collection (8% vs. 1% for website; 11% vs. 2% for workshops), and twice as likely to publish a

paper about the activity (8% vs. 4% for website; 9% vs. 4% for workshops). So while Cutting

Edge may not impact the publication of research in general, it does appear to be associated with

the publication of research specific to education and teaching. While the research article

publication percentages are still fairly high for Cutting Edge website users and workshop

participants, the finding that they are lower than non-Cutting Edge faculty seems counterintuitive

Page 53: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 53 of 63

and it does not seem plausible (nor desirable) that Cutting Edge would dampen the publication of

research in general. Thus, other factors or characteristics of website users and workshop

participants that might affect publication behaviors should be explored in subsequent

evaluations. It is worth noting that the relative percentage increase over non-Cutting Edge faculty

in their likelihood to specifically disseminate their classroom successes is much greater than the

decrease in their publications overall.

Figure 35: Comparison between web users and non-users for community contributions beyond Cutting Edge.

27%

16%

31%23%

71%

8%10%6%

14%6%

54%

4%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Has presented research on

teaching methods or

student learning at meetings

within the past two years

Published about educational

topics 1 or more times within the past two years

Share course materials via

talks

Published in journals

Tell colleagues whom I know

will be interested

Publish a paper about my activity

Community Contributions By Website UsersWeb

No Web

Page 54: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 54 of 63

Figure 36: Comparison between workshop participants and non-participants for community contributions

beyond Cutting Edge.

Result of the Cutting Edge Impact on the Geoscience Education

Community

Impact at the local and departmental level

In telephone interviews, 42 of 54 2005 workshop participants highlighted outcomes from the

workshop characterized by developing outreach programs for local community education, pre-

service in-service science teacher programs, leading regional workshops, or publishing in the

area of geoscience education. Fifteen of those participants described specific department-wide

results from their participation in the program such as curriculum overhaul for majors, teacher

preparation, and better cross-disciplinary and cross-institutional alignment. Thirty-seven others

found their departments supportive of the changes but that changes were not necessarily adopted

across the department. Two noted their department was hostile and not supportive. There appears

to be a clear dose response of reported changes in the department to the number of faculty

participating in workshops. The 2008 Participant Survey found that participants from

departments in which at least 3 department members had attended a workshop were 3 times more

likely to report that the workshop had changed their department‟s collective approach to teaching

compared to participants coming from departments in which 2 or fewer members had attended a

workshop (48% to 16%).

29%

16%

30%22%

70%

9%

13% 8% 17% 9%

57%

4%0%

10%

20%

30%

40%

50%

60%

70%

80%

Has presented research on

teaching methods or

student learning at meetings

within the past two years

Published about educational

topics 1 or more times within the past two years

Share course materials via

talks

Published in journals

Tell colleagues whom I know

will be interested

Publish a paper about my activity

Community Contributions By Workshop ParticipantsWorkshopNo Workshop

Page 55: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 55 of 63

It was unclear from the interviews with website users whether use of the Cutting Edge website

had impacted community and department norms. However, reports from website users that

Cutting Edge content was integrated into on-campus department in-services and faculty meetings

were positive signs that the Cutting Edge reach was being extended.

The broader impact

Cutting Edge provided a venue for collegial exchange and communication that included the

sharing of teaching experiences and learning from others teaching experiences. The contributions

made by Cutting Edge website users and workshop participants extended beyond the platforms

provided by Cutting Edge to bring new knowledge into the community, nurture research

activities, and expand the networks used for professional exchange regarding teaching practices

specifically and pedagogical issues in general. The result has been a shift the geosciences

education community in which:

Education is now an accepted topic for discussion and work among geoscience

faculty from all types of institutions;

Research networks have been developed to support education research;

A national perspective on geoscience teaching has been created and is being used

to benchmark teaching norms and behaviors at the institutional and individual

levels;

Geoscience faculty members are connecting with new colleagues from different

institutions and different disciplines;

There is a culture of sharing experiences, successes, and new ideas in education

and teaching practice; and

The sphere of influence and leadership roles for education specialists in

geoscience have expanded.

Page 56: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 56 of 63

Appendix: Evaluation Methodology

We used a tiered evaluation framework that combined mixed-method approaches (Fig. A). We

used small samples of in-depth interviews related to both the workshop and website to inform

larger surveys of all participants as they exit workshops and periodically through the web. The

results of these interviews and surveys also informed the design of a larger survey which we

administered in 2004 and administered again in spring 2009 to the larger geoscience community.

Figure A. Tiered Evaluation Framework

Separate summary reports of the interviews and surveys conducted and results can be viewed at:

http://serc.carleton.edu/NAGTWorkshops/evaluation.html

Large-scale survey of geoscience community

A Faculty survey was conducted by Statistical Research Center of the American Institute of

Physics as a baseline in 2004 and administered again in 2009. The survey focused on what

teaching methods were being used by geoscience faculty, how they learned about new content

and teaching methodology, and how the shared what they knew with their colleagues. The

2009 survey was sent electronically to 5107 geoscience faculty and received 2537 completed

responses (50% response rate). Analysis of survey results as the data pertained to the On the

Cutting Edge program focused on three populations: workshop participants who use the website

(as 87% of workshop participants who responded to the survey also used the Cutting Edge

website), Cutting Edge website users who have not attended a workshop, and survey respondents

who had neither attended a Cutting Edge workshop nor used the Cutting Edge website. The

number of respondents who had attended a Cutting Edge workshop and had not used the website

was too small (44 of 2537 completed responses) to analyze.

Page 57: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 57 of 63

Surveys of all Cutting Edge workshop participants and website users

Workshops participants

Three strategies were used to collect a consistent set of evaluation data for all workshops: daily

road checks, end-of-workshop surveys, and evaluator observations. In addition for some

workshops, action plans and embedded assessments were completed by all participants.

Road Checks – administered at the close of each workshop day. The purpose was to obtain a

quick scan of the participant reactions to the workshop and provide real-time formative

evaluation for use in modifying the program. In addition to the basic demographic information

the road check asked participants what aspects of the workshop seem to be working and not

working for them; what needed to be improved; and, their general satisfaction with the workshop

at that time.

Road check surveys were analyzed immediately and results were provided to the workshop

leaders so that when necessary, mid-course corrections were made in the workshop. Brief

feedback was provided to participants the following day so that they could see the impact of their

comments. Road check surveys were also used to gather information about what was learned in

that day to better identify the causal linkages between the workshop and its impact on

participants.

End-of-Workshop surveys – administered at the end of each workshop. The purpose was to

provide leaders with insights into not only what worked and did not work with the workshop that

would guide improvements for future workshops, but also indications of what might be added to

the workshop agenda. The surveys also helped identify immediate impacts – changes in

knowledge, skills, and attitude as well as anticipated behavioral application that could be

attributed to the workshop.

In addition to demographic information, participants rated the degree to which expected

outcomes (workshop objectives) were achieved for them and the usefulness of key strategies or

activities that were used in the workshop. Participants also responded to open-ended questions

about what aspects worked well/did not work well, what topics might be added/dropped, how

they changed, what they planned to apply, and what they planned to share after the workshop.

Information was also collected about the participant‟s use of the website to prepare for the

workshop as well as any other use. Last, participants gave over-all ratings for the given

workshop.

End-of-workshop surveys were analyzed and along with the road check data, observations, and

interviews summarized for workshop leaders. Quantitative data was averaged by demographic

and write-in comments were recorded and reported in aggregate. The qualitative write-in

Page 58: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 58 of 63

comments were coded by workshop type and used to better understand what participants learned

and how they anticipated using this new knowledge, skills, or attitude.

Observations and interviews –conducted throughout the workshops by the external evaluator or

others serving in the evaluator role, but also included the observations and input from workshop

leaders and/or facilitators. The purpose of these observations or interviews was to better gauge

how well the workshop was meeting the expectations of participants as well as to understand

which aspects of the workshop catalyzed learning outcomes and take note of any emerging

outcomes.

Observations and interviews were discussed with leaders at regular intervals throughout the

given workshop to provide real-time assessment in addition to the written road checks. These

observations were focused on particular aspects identified by leaders as well as captured

unanticipated events/outcomes. These observations and interviews were summarized with the

end-of-workshop data in the end-of-workshop summary report.

Action plans - completed by participants at selected workshops in 2007-2009. The purpose of

these action plans was to help participants articulate how they would put what they learned into

practice and identify any obstacles for change. An email survey was conducted with a sample of

participants one or two years after the workshop from four of these workshops: Teaching

Introductory Geoscience (2008), Teaching with New Geoscience Tools (2008), Early Career

(2007) and (2008). The purpose of this survey was to solicit feedback on teaching changes that

workshop participants had made since their workshop, their use of the Cutting Edge website in

making those changes, and the role that action planning played in helping to facilitate those

changes.

Embedded assessments – a range of assessments was conducted as pilot evaluation strategies at

selected workshops. The purpose of these embedded assessments was to attempt to move

beyond the end of workshop surveys as self reported measures. As part of the Early Career

workshop, participants generated posters that describe plans for both teaching and research. In

addition to the posters participants identified what aspects of the plans they attributed to their

workshop participation and the value of having them reviewed by colleagues. As part of the

Teaching X workshops, participants submitted teaching activities and then participated in an

activity review session using a specific rubric. As part of the evaluation they reported on the

value of this review process in terms of their own activity as well as how they think about

teaching. In selected workshops participants were asked prior to the workshop and again after

the workshop what features they look for in a strong teaching activity. These responses were

qualitative coded for emerging learning themes.

Page 59: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 59 of 63

On-line participant surveys administered in 2004, 2005, and 2008. The purpose of the on-line

survey was to characterize how the different types of Cutting Edge workshops impacted

participants‟ professional development, teaching practice, and courses as well as their

observations of impacts on student learning. The survey also aimed to illuminate other ways that

the program impacted faculty such as leadership opportunities on campus, regionally, and

nationally and means of disseminating their knowledge (attributed to program). Of the roughly

1000 potential participants, 274 participants chose to respond (approximately 27% return rate).

The array of workshops covered and the range of answers to questions provides a more

representative picture than the response rate would suggest.

The 2008 On-line Survey collected responses from participants by workshop. The questions

covered the participants‟ thoughts on the impact of the workshop on their professional

development, philosophy and practice of teaching, how they had shared the knowledge, skills, or

attitudes acquired from the workshop with colleagues, and how they had used the website. It

also asked them to share the most valuable impact related to their workshop experience.

Website users

Web statistics reports produced monthly to better understand the number of users of the website

and the resources viewed.

Pop-up surveys conducted on the serc.carleton.edu website in 2006 and again in 2007 to better

identify the demographics of the website users. The pop-up survey was administered to all users

website who visited 4 or more pages. The survey identified the types of users (student, K-12

teacher, and faculty) as well as for what the website was being used (images, data, teaching

activities). Of the 2105 responses, 657 responded from a Cutting Edge website page. This

sample was 1% of the estimated users of the Cutting Edge website for the period the survey was

active.

Awareness poll conducted from November 2007 to January 2008. The email survey was

administered to a randomly generated sample of 1054 geoscientists. The survey had a 44%

return from email. To reduce the opportunity for bias with non-respondents, the survey was also

administered by telephone to 6% of the non-respondents. The telephone survey found an even

higher percentage of awareness to the On the Cutting Edge program. The confidence level is

95% with a 4.4% margin of error.

In-depth small samples of interviews

Community Interviews conducted by Ellen Iverson and John McLaughlin at the December 2007

meeting of the American Geophysical Union. The 39 participants were a mix of Cutting Edge

participants (25) and geosciences faculty who had never attended a Cutting Edge workshop (14).

The purpose of the interviews was to better understand the nature of the cycle of educational

Page 60: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 60 of 63

innovation in geosciences and the specific roles that Cutting Edge activities and resources play in

supporting, strengthening and broadening the cycle of education innovation for individuals and

the geoscience community. In addition, we wanted to start to understand the structure of

subpopulations within the geosciences community with respect to these questions. We were

particularly interested in understanding the faculty who defined/aligned their professional life

and network to the following subpopulations: 1) Science research 2) Education research 3)

Community of geoscience educators (practice of teaching) 4) Isolated faculty.

Workshop Participant Telephone interviews conducted from May to September 2005 by Ellen

Iverson and John McLaughlin. The purpose of the phone interviews was to determine the suite

of important impacts for the various types of workshops. In particular the object was to explore

impacts of new knowledge, networks, and leadership skills on teaching practice, professional

planning, and leadership in community. The 54 telephone interviews were from past (2002-

2004) workshop participants.

Leadership Interviews conducted by John McLaughlin via telephone in March-April of 2009.

The nine participants included past conveners of Cutting Edge workshops, contributors to the

Cutting Edge website, and participants in a co-hort of leaders for regional workshops. The

purpose of the interviews was to understand the role the program is playing in developing

leadership for the geoscience education community.

Geoscience Leader Interviews conducted by John McLaughlin via telephone in July-August

2009. In the spring of 2009, the Cutting Edge Principal Investigators identified six recognized

leaders in the geoscience field who were knowledgeable of the CE but were not closely affiliated

with implementation. John McLaughlin, the external evaluator conducted the interviews; one

person was not available for an interview. The purpose of the interviews was to collect

information related to: the perceptions of the program by leaders in the field, program impact on

geoscience teachers and the broader geoscience community, the long-term need for CE services,

and suggestions for addressing changing needs in the geoscience field.

Web User Telephone Interviews conducted by Linda Goozen and Randy Kirkendall from

September 2008 to April 2009 and by Ellen Iverson in November 2005. The goal of these

phone interviews was to define the types of uses of the Cutting Edge website and its impact on

teaching.

Reporting of Statistical Findings

The responses to the 2009 Faculty Survey were analyzed for differences between Cutting Edge

website users and non-users and between Cutting Edge workshop participants and non-

participants using simple 2x2 chi square tests of association. The results from those analyses

were included in this report if the p-value using Pearson‟s method was less than .01.

Page 61: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 61 of 63

For the purposes of exploring possible additive effects due to adding workshop participation to

website use, questions that asked about specific teaching behaviors that faculty members

reported using in their introductory and majors courses in which the 2x2 chi square comparisons

for both website use vs. teaching behavior and workshop participation vs. teaching behavior

were statistically significant (at p<.01) were selected for further analysis. The results of these 2x2

chi squares are reported throughout this report. The tests for website associations were

irrespective of workshop participation and the tests for workshop participation were irrespective

of website use.

In order to determine if there was a combined effect for adding workshop participation to

website use, three more chi square tests were run for each selected question.

Website use only (no workshop) vs. neither website use or workshop participation.

Website use only vs. combined website use and workshop participation.

Combined website use and workshop participation vs. neither website use or workshop

participation.

For reported additive effects in this report, all three of the above comparisons were significant at

the level of p<.05.

Limitations of this Evaluation

Workshop participation and website use are highly correlated with each other. Among the

respondents to the 2009 Faculty, Survey Cutting Edge workshop participants were much more

likely than non-participants to have used the Cutting Edge website at all (87% vs. 20%) and

seven times more likely to have used it on at least a monthly basis (43% vs. 6%). And Cutting

Edge website users were much more likely to have participated in a Cutting Edge workshop

(44% vs. 3%). From a programmatic perspective, this is good news because the workshops

emphasize the use of the website as a tool for continuing the learning and making the teaching

changes presented. However, it presents challenges in trying to determine the separate effects of

workshop participation and website use. In fact, the effect of workshop participation only could

not be determined because of the very few number of workshop participants who did not use the

website (44 out of 2537 responses). Thus, any reported effects of the workshops may inherently

include some undetermined level of website effect.

The small number of workshop participants who did not use the website also meant that

interaction effects could not be sufficiently tested and the only combination effect that could be

tested was of workshop participation in addition to website use.

In addition to the 2009 Faculty Survey results that demonstrated a significant difference between

faculty with Cutting Edge exposure and those without it, there were a number of findings from

Page 62: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 62 of 63

the survey in which a statistical difference did not exist. Thus, there are a number of attitudinal

and behavior measures in which Cutting Edge may not have had any impact.

For the evaluation results that rely on small sample sizes or low response rates, it is possible that

the reported effects simply reflect that workshop participants and website users who experienced

the measured effects (i.e., shifts in attitude, changes in teaching practice) were more likely to

respond to the questions and that those who did not experience the effects. This may be true,

however, it is reassuring that the results were consistent across the different evaluation findings

using different tools. Additionally, even if all non-responders were not affected by Cutting Edge,

the magnitude of the measured effect would still be large. For example, the 2008 Participant

Survey which found a greater than 90% shift in attitude had a response rate of 27%. So it is

possible that participants who experienced the shift were more likely to respond than participants

who did not experience the shift. And yet, even if all non-responders did not experience this shift

(an unlikely scenario), there were still 25% of the total 1000 potential responders who reported

that they did experience a shift in attitude.

While faculty may report using educational innovations, their behavior in the classroom may be

different or not in line with the intent of the teaching method. This evaluation did not directly

measure any changes that might have occurred in the classroom. All reported effects on teaching

behavior were based on self-reported data.

Similar to the limitation with respect to demonstrating any effects on teaching behavior, the data

cannot demonstrate that student learning was impacted. While student learning is the ultimate

goal, it was not a direct goal for this project and was not measured for the evaluation. All the

same, the teaching innovations developed, disseminated, promoted, and taught by Cutting Edge

were based on best practices and research that indicated that when the innovations are adopted

and implemented, student learning will be positively impacted.

Page 63: Evaluation Report of On the Cutting Edge · Evaluation Report of On the Cutting Edge February 5, 2010 Submitted by John McLaughlin, External Evaluator McLaughlin Associates, Lanexa,

Page 63 of 63

References Cited

American Geological Institute, 2009, Status of the Geoscience Workforce 2009:

http://www.agiweb.org/workforce/reports.html

Barr, R. B. and Tagg, J., 1995, From Teaching to Learning - a New Paradigm for Undergraduate

Education: Change, v. 27(6), p. 12.

Pellegrino, J. W., Chudowsky, N. and Glaser, R., 2001, Knowing What Students Know: The

Science and Design of Educational Assessment: National Research Council (U.S.). Committee

on the Foundations of Assessment.