NLII Focus Session

Transformative Inklings and the Mystery of “Faculty Buy-In.”

Gary Brown, DirectorThe Center for Teaching, Learning, & Technology

Washington State University

The Center for Teaching, Learning, & Technology

CTLT — Who Are We?

• Staff of 15 technology, assessment, and learning specialists.

• Work with about 100 WSU faculty each semester

•Design

•Develop

•Deliver

•Assess

Student Enrollments in WSU Online Learning Spaces

0100020003000400050006000700080009000

1000011000120001300014000150001600017000

1998 1999 2000 2001 2002

The Goals, Activities, and Practices

• Invite faculty to join in the process of formulating the assessment.

• A series of three short, online surveys designed to provide formative assessment: one instructor survey and two student surveys.

The 3 Survey Goals

1. To help faculty gather useful data about their teaching goals, values and strategies…

2. To help faculty learn how those strategies compare and influence their students’ goals, values, and learning behaviors…

3. And to examine the interaction between goals and practices that faculty might share with each other

—and thereby establish a transformative culture of evidence.

GAPs: Encouraging Participation• Brown Bags - informal luncheons to help initiate the conversation on

assessing teaching and learning.

• We emailed each faculty approximately 10 times to encourage participation…

• Student surveys were linked to course sites of instructors who answered the instructor survey.

• We analyzed and returned results promptly to each participating faculty member...

• Regular updates were sent to instructors with their class results and overall results.

• We offered and have co-authored papers with faculty to help enhance their professional development.

• We have talked with chairs, deans, and the assessment coordinator’s office to encourage departments to begin the dialogue with the faculty and to include results as part of their teaching portfolios.

• We wore out our shoes crossing campus to meet with faculty and invite their contributions to the GAPs instruments…

Participation in GAPs

563

10655 16

0

300

600

Invited faculty Faculty who tookinstructor survey

Faculty w/ studentresponses

Faculty w/ >50% studentresponse rate

Sample Findings• Significant mismatch between faculty and student goals• Online courses are significantly more effective than video based courses.• Designed courses scored significantly higher on all principles of good practice• Faculty motivation predicts perceptions of efficacy of online learning

Faculty who want to keep abreast of the scholarship of T & L report that online learning is at least as effective as traditional classroom learning

Faculty who teach online primarily for the money report that online learning is not as good as the traditional classroom

Buy-in?

The Assessment Gold Standard—Participants Who Used Data to Transform….

563

10655 16 2

0

300

600

Invited faculty Faculty who tookinstructor survey

Faculty w/ studentresponses

Faculty w/ >50%student response

rate

Faculty who usedthe data

However….• One of the two programs endured a budget related, faculty

initiated inquisition….. Demonstrated the program improved freshman retention Demonstrated the program was cost effective Demonstrated the program improved student grades, including

“special admits” expected to struggle. Demonstrated the “creative use of technology” and gains in

critical information literacy Demonstrated evidence of good practice. Demonstrated, by virtue of GAPS, systematic, formative data

gathering and responsiveness to that data…• GAPS, unlike traditional student evaluations, shows HOW this

came about….

Which has resulted in….• The faculty committee that initiated the inquiry was commended by administration

for its attentiveness to issues of quality….• They have subsequently been invited to examine other programs… • Which has shown signs of transforming the CTLT assessment group into a collegial

partner…• We are seeing the inklings of a culture of evidence sprouting in a rough, research

dominated terrain…

The Boyer Commission• Construct an inquiry-based freshman year

• Build on the freshman foundation

• Remove barriers to interdisciplinary education

• Link communication skills and course work

• Use information technology creatively

• Make research-based learning the standard

• Educate graduate-students as teachers

• Culminate in a capstone experience

The Learning Context: Student Performance @ WSUCum GPA by Admissions Quartiles

1st Quartile 2nd Quartile 3rd Quartile 4th Quartile

English

GenEd

Math

FS

0

1

2

3

4EnglishGenEdMathFS

Dimensions of Critical Thinking1. Identifies and summarizes the problem/question at issue (and/or the

source's position).

2. Identifies and presents the STUDENT’S OWN perspective and position as it is important to the analysis of the issue.

3. Identifies and considers OTHER salient perspectives and positions that are important to the analysis of the issue.

4. Identifies and assesses the key assumptions.

5. Identifies and assesses the quality of supporting data/evidence and provides additional data/evidence related to the issue.

6. Identifies and considers the influence of the context * on the issue.

7. Identifies and assesses conclusions, implications and consequences.

Critical Thinking and Measures of Growth

Scant _____________________________________ Substantial

Identifies and summarizes the problem/question at issue (and/or the source's position).

Does not identify and summarize the problem, is confused or identifies a different and inappropriate problem.

Does not identify or is confused by the issue, or represents the issue inaccurately.

Identifies the main problem and subsidiary, embedded, or implicit aspects of the problem, and identifies them clearly, addressing their relationships to each other.

 

Identifies not only the basics of the issue, but recognizes nuances of the issue.

Critical Thinking Study—Results

• Gains in courses when rubric is used—when the faculty in this project integrated the WSU Critical Thinking Rubric into their instruction and assessment, evidence of student gains in critical thinking increased dramatically.

• Gains from freshmen to junior years—Critical thinking was significantly higher among juniors than among freshmen. But even the writing of juniors had only a mean of 3.1 on a 6 point scale.

• Comparisons to WSU’s writing assessment—As critical thinking scores went up, writing placement scores and portfolio exam scores went down.

CT Growth in Entom 401

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

Problem Position Other Views Context Evidence Assumptions Implications CT Avg Score

CT Dimensions

Entom Spr 99

Entom Spr 00

Critical Thinking Gains

With CT

W/O CT

Additional Findings, Observations, & Implications

• The dimension of least gain was in students’ abilities to articulate their own viewpoints.

• The greatest gains by juniors reflect improved abilities to analyze issues from multiple perspectives.

• Financial incentives have increased faculty adoption of the rubric.

• The faculty questionnaire revealed a focus on grading over fostering critical thinking for broader life-long learning.