Advanced Methods in Using and Doing the Scholarship of Teaching & Learning (SoTL)

Regan A. R. GurungBen J. & Joyce Professor of Human Development & Psychology

University of Wisconsin - Green BayFaculty College

May 28th- 31st, 2013

Some Reflection

1. Name2. Discipline3. Your

Biggest classroom frustration.

4. What do YOU want from this workshop?

“The essence of skillful teaching lies in the teacher constantly researching how her students are experiencing learning and then making pedagogical decisions informed by the insights she gains from the students’ responses.”

—Stephen D. Brookfield

The Approximate Plan

Contemporary Issues in SoTLThe Big PictureMajor Research DesignsThreats to Validity of SoTLKey Variables in the Study of

LearningEvidence-Based Teaching in Higher

Education

A Teaching Hierarchy: Where are YOU?

SoTL

Going Through Motions

Sincere Teaching

Scholarly Teaching

Gurung & Schwartz (2009) adapted from Richlin (1993)

Systematic, Intentional

ModificationsKNOWLEDGE BASE

ABOUT TEACHING/ LEARNING

Assess Success

Publication

Peer Review

Scholarly Teaching

Reflect on

Teaching/Learning

Share results

Presentation

SoTL: Intentional, systematic reflections on, and modifications to teaching and learning, resulting in peer-reviewed products made public.

The Place of SoTL

Aubrey Stoll: http://500px.com/photo/9827809

SoTL is

..the systematic study of teaching and learning, using established or validated criteria of scholarship, to understand how teaching (beliefs, behaviours, attitudes, and values) can maximize learning, and/or develop a more accurate understanding of learning, resulting in products that are publicly shared for critique and use by an appropriate community. (Potter and Kustra, 2011, p. 2)

The Evolution of SoTL: First Wave

Pedagogical research is scholarship too!!!

Greeks to James …….to …Boyer Hutchins & Shulman (1999)

Second Wave Learning To Think

▪ Donald (2002)

Decoding disciplines▪ Identify bottlenecks

▪ Pace & Middendorf, (2004)

Signature Pedagogies ▪ Teach students your discipline’s habit of mind

▪ Gurung, Chick, & Haynie, (2009); Shulman, (2005)

Threshold Concepts▪ Teach students fundamental/troublesome

concepts▪ Land, Meyer & Smith, 2008; Meyer & Land, 2003

Third Wave: Beyond Boundaries

Infiltrate the Mainstream Run interference Look at the Big Picture Catalyze SoTL Use

Infiltrating the Mainstream

Goal: Make SoTL even more visible. Where do you publish your SoTL?

▪ See Weimer, 2008 Where can you publish it? Where is your ‘Commons’? Break new ground

▪ SoTL tracks▪ SoTL journals

Running Interference

Goal: Facilitate more SoTL Connect with higher administration Become higher administration Network of department chairs Mentoring & Advocacy

Look to the Big Picture

Goal: Partition out Influences on Learning Models of Teaching and Learning

▪ What’s on your notepad? Empirical data: Meta-meta-analyses

What Works in the literature?

Traditional lecture Active learning Service learning Problem-based learning

Group learning Mentoring Cooperative learning Discovery learning

Inductive learning Learning by example Inter-teaching Desirable difficulty

Learner centered Curriculum Centered On-line teaching Clickers

Powerpoint Overheads Chalktalks Teachable moments

Universal design of instruction CAP Model Multiple Intelligences Kolb’s learning styles

Journaling Reflective practice Reciprocal teaching Uncoverage

Concept maps question generation Film strips Laboratory-based instruction

Video clips Role playing Modeling Programmed instruction

Keller method Skill practice Guided practice Collaborative learning

Apprenticeship Situated learning Authentic assessment Formative assessment

Classroom research techniques Book reports Class Discussion Small group discussion

Think-pair-share Peer instruction ConcepTests Panel of experts

Brainstorming Case studies Worksheets Guest speakers

Student debates Jeopardy Portfolios Posters or Bulletin boards

Flashcards Research papers Interviewing Lecture with discussion

Oral reports Study abroad Mock convention Textbook assignments

Just-in-time teaching Jigsaw method Wikis Team teaching

Socratic method Modules Podcasts internships or practicums

Writing Across the Curriculum Web enhanced independent research independent study or tutorials

faculty-student research Reaction papers Reflection papers Brain-based education

student teaching letter writing observation immersion learning

peer review Concept grids Self-paced Instruction Learning Communities

What Doesn’t Work in the literature?

Traditional lecture Active learning Service learning Problem-based learning

Group learning Mentoring Cooperative learning Discovery learning

Inductive learning Learning by example Inter-teaching Desirable difficulty

Learner centered Curriculum Centered On-line teaching Clickers

Powerpoint Overheads Chalktalks Teachable moments

Universal design of instruction CAP Model Multiple Intelligences Kolb’s learning styles

Journaling Reflective practice Reciprocal teaching Uncoverage

Concept maps question generation Film strips Laboratory-based instruction

Video clips Role playing Modeling Programmed instruction

Keller method Skill practice Guided practice Collaborative learning

Apprenticeship Situated learning Authentic assessment Formative assessment

Classroom research techniques Book reports Class Discussion Small group discussion

Think-pair-share Peer instruction ConcepTests Panel of experts

Brainstorming Case studies Worksheets Guest speakers

Student debates Jeopardy Portfolios Posters or Bulletin boards

Flashcards Research papers Interviewing Lecture with discussion

Oral reports Study abroad Mock convention Textbook assignments

Just-in-time teaching Jigsaw method Wikis Team teaching

Socratic method Modules Podcasts internships or practicums

Writing Across the Curriculum Web enhanced independent research independent study or tutorials

faculty-student research Reaction papers Reflection papers Brain-based education

student teaching letter writing observation immersion learning

peer review Concept grids Self-paced Instruction Learning Communities

Entwistle (2009)

Fink’s Conceptualization of College and University Teaching

Knowledge

Course Design

Teacher-StudentInteraction

CourseManagement

Beginning of the Course

Fink, L. D. (2003).

Lowman’s Two-Dimensional Model of Effective College Teaching

Dimension 1: Intellectual Excitement Clarity of Presentations (what is presented) Emotional Impact on Students (way material is presented)

Dimension 2: Interpersonal Rapport Awareness of Interpersonal Nature of the Classroom Communication Skills that Enhance Motivation and

Enjoyment of Learning and that Foster Independent Learning~

Lowman, J. (1995).

Extended Lowman-Keeley et al. Model

StudentTeacher Knowledge LearningTechnique

Social Context(rapport)

Social Context(rapport)

Intellectual Excitement

Intellectual Excitement

Topic, Content, andLearning Goals

Level of Student Understanding

Characteristics of the Teacher

Post-event Reflection

Manipulate

Monitor,Manage,Manipulate Monitor Manipulate

Student-Teacher Rapportand Classroom Atmosphere

In-the-Moment Reflection Pre-event

Reflection

Form of Assessment

TeachingStrategies

Characteristics of the Learner

LearningStrategies

The TACOMA Model of Teaching (Chew et al., 2009)

Background, preparation, and individual characteristics

Understanding the ways that humans learn

Classroom design, technology, and institutional priorities

Content difficulty, relevance, organization, and accuracy

Teaching technique, teacher behaviors, andstudent learning activities

Desired results of teaching, short- and long-term goals, and assessment practices

Groccia’s (2012) 7-Component Model From St. Clair, K. L., & Groccia, J. E. (2012). Change to social justice education: A higher education strategy. In Skubikowski, K., Wright, C., & Graf, R. (Eds.). Social justice education: Inviting faculty to transform their institutions. Sterling, VA: Stylus.

Innovations & Imperatives

Making SoTl accessible National SoTL infrastructure

▪ (Poole, Taylor, & Thompson, 2007) Building on MERLOT SoTL Electronic Repository The psychology of teaching: An

empirically based guide to picking, choosing, & using pedagogy

The MetaSearch ProjectTackle cross-cutting questions.

▪ What are the processes most linked to learning?

Innovative Strategies

Making Learning More Visible: Lesson Study

Blooming Biology Tool (Crowe, Dirks, & Wenderoth, 2008)

Question-Eliciting-Questions (Dickman, 2009)

The Question of Impact

HOW do you use SoTL?Change course design?Modify assessments?Tell students about SoTL results?

Main uses (McKinney & Jarvis, 2009; Meyers, 2009)

The Big Picture

• How will you do it?

• Did it work?

• Present• Publish

• What’s YOUR question?

• What will YOU do?

• What are students’ learning?

• How can you do better?

• What’s been done? Reflect &

Review

Focus &

Change

Assess &

Evaluate

Share &

Respond

Reflect & Review

• How are students’ learning?

• What can you do better?

• What’s been done? Reflect &

Review

The Context

Course

Goal

SLOAssessme

nt

SLOAssessme

nt

Goal

SLOAssessme

nt

SLO Assessment

PEDAGOGY

Unpacking Pedagogy

Pedagogy

ContentText

Readings

Methodology

Lecture

Discussion

AssignmentsPapers

Service Learning

Oral Presentation

DebateLab

Backward Design and Action Research

Backward Design (Fink, 2003; Wiggins & McTighe, 1998)

1. Articulate learning goals first/determine learning objectives (outcomes)

2. Determine assessment methods/techniques

3. Select pedagogical / teaching strategies

Action Research

Three-Column TableEnsure course activities and assessment aligned with LO’s

Landrum, 2012

Landrum, 2012

Landrum, 2012

Establish Clear Learning Outcomes

(Gurung & Landrum, 2012)

Focus & Change

• What’s YOUR question?

• What will YOU do?

Focus &

Change

Focus: What’s Your Question?

Someone teaches Something to Someone else Somewhere (Schwab, 1973)

Teacher Scrutinize your assignments

Material Textbook evaluations

Students How do students study?

ContextOnline, hybrid, face to face

Think about One problem you encounter in your

coursessuch as:A student behavior you would like to change

A learning objective you want to better achieve

Change: What will you do?

Flip your classroom Engage problem-based learning in a class Add a case study approach Introduce service-learning components Teach without a textbook Have students construct learning

portfolios Increase the amount of writing, music,

visuals, or reflection used in class

Think about : One problem you encounter in your

courses

What solution might you use to address the problem?

Disciplines Vary in Methodology

Learning To Think▪ Donald (2002)

Signature Pedagogies ▪ Teach students your discipline’s habit of mind

▪ Chick, Haynie, & Gurung (2012)▪ Gurung, Chick, & Haynie (2009)

Think about

One problem you encounter in your courses

What solution, might you try to address one of these problems?

How will you assess the success of your solution? What evidence will you collect?

A key distinction

Quantitative Vs. Qualitative Methodology vs. Analysis

How will you do it?Research Design

Describe

Qualitative Quantitative

Correlate Compare

Pre-Post Groups

Across semesters

Within semesters

Within classes

Did it work?

Watch Classroom Observation Content Analysis

▪ Develop a coding scheme (categories, rubrics)

▪ Units of analysis (words, turns-at-talk)▪ Rater Training & Reliability

Ask Survey Focus Group Protocol Analysis

Lesson Study (Cerbin, 2012)

How Do You Know It Worked?

Retention over the termJournal evidenceStudent discussion

increasesStudent preparation

improvesStudent evaluations

improvePortfolio showcasing

student workClassroom assessment

techniques

Assessment ChoicesQuantitativesurvey scorescourse exam,

project, paper scores

frequencies of multiple choice test item responses

standardized scales and tests

counts (participation, web requests, office visits)

measures of time use

institutional research data

Qualitative

performances interviews focus groupsstudent

projects term papers essay items exams

reflective statements journals reports of others

Assess & Evaluate: Key Methodologies in SoTl

• How will you do it?

• Did it work?

Assess &

Evaluate

How Do You REALLY Know It Worked?Statistical Significance (SPSS;EXCEL)Correlational r (ranges from -1 to 1)T-Tests; Analysis of Variance (ANOVA)

Did it Work? Correlations

Use of Pedagogical Aids

(Gurung, 2003; 2004)

Does using Pedagogical Aids Help?

(Gurung, 2004)

Does Online Studying help?

Online

0.18

0.15

0.19

Correlations to Exam Score

Connect MyPsychLab Psychportal

Comparing Group Differences?

Compare Classes

Ethical counterbalance

Compare Classes

Intervention

Random Assignment

Class

Section 1

Novelty

Test

Nothing

Test

Section 2

Nothing

Test

Novelty

Test

Comparing Groups Between Classes

Compare Groups

Ethical counterbalance

Compare Groups

Intervention

Random Assignment

Class

Group1

Novelty

Test

Nothing

Test

Group 2

Nothing

Test

Novelty

Test

Comparing Groups Within Class

Thinking Like a Historian (Voelker, 2012)

Qualitative to Quantitative

Voelker Results

Validity of Measurement

Construct Validity of Surveys and Polls

Choosing question formatsWriting well-worded questionsEncouraging accurate responses

Choosing Question FormatsOpen-ended questionsForced-choice formatLikert scaleSemantic differential format

Writing Well-Worded Questions

Leading questionsDouble-barreled questionsDouble negatives (vs. negatively worded items)Question order

Encouraging Accurate Responses

Response sets (three types) Yea-saying/ nay-saying Fence sitting

Faking good or badSaying more than we can knowMeasuring subjectivity vs. objectivity

Good surveys (or not): How great is this workshop?

Your Name: What do you look like?

Yesterday we talked about validity. Did I do a good job of explaining it?

a. Absolutely! b. Completely c. Very much yes d. yes e. Mostly yes

Is this your favorite workshop and do you have two legs? Yes/No

Do you favor reducing the overwhelming amount of homework you are forced against your will to do? Yes/No

If you weren’t to advocate not doing your homework, would you also not advocate not increasing the amount of reading you don’t feel is too much already? A. No B. Not no

Do you enjoy being alive? Yes/No DO you think humans should keep having children? Yes/No Will you give me a chilly pepper on ratemyprofessors.com? Yes/No

anonymityScale choice

Doubl

e

barre

l

Loaded

Double NegativeYea

saying response

set

What are the violations?

Fence sitting possible

No labels on all items

Too Manychoices

Threats To Validity: The Dirty Dozen

Design confoundSelection effectOrder effect

MaturationHistoryRegression to the mean

AttritionTestingInstrumentation

Observer biasDemand characteristicsPlacebo effects

Important Considerations

Individual results may vary.

Ethical Pedagogical Research

To Publish or Present---

Get Institutional Review Board (IRB) clearance

▪ Gurung (2012)

Share & Respond

• Present• Publish

Share &

Respond

Key Variables in Learning

What the Data Says: Predicting Learning

John Hattie (2009)

800+ meta-analysis

50,000 studies

240+ million students

Elementary, secondary, &

tertiary

Identifying what matters

Percentage of Learning Variance

StudentsLecturers

Home

Peers

TEOs Others

Effect on Learning over time?

Decreased EnhancedZero

0 .22

Typical Effect Size

1.0

.40

The Winners ...Rank Influence Studies Effects ES

1 Self-reported grades/SC of ability 209 305 1.44

2 Piagetian programs 51 65 1.28

3 Formative evaluation of own teaching 30 78 .90

4 Micro teaching 402 439 .88

5 Acceleration 37 24 .88

6 Classroom behavioral 160 942 .80

7 Comprehensive interventions for learning disabled students 343 2654 .77

8 Teacher clarity na na .75

9 Reciprocal teaching 38 53 .74

10 Feedback 1287 2050 .73

Strategy #13 Example ES

Organizing & transforming Making an outline before writing a paper .85

Self-consequences Putting off pleasurable events until work is completed .70

Self-instruction Self-verbalizing the steps to complete a given task .62

Self-evaluation Checking work before handing in to teacher .62

Help-seeking Using a study partner .60

Keeping records Recording of information related to study tasks .59

Rehearsing and memorizing Writing a mathematics formula down until it is remembered .57

Goal-setting/planning Making lists to accomplish during studying .49

Reviewing records Reviewing class textbook before going to lecture .49

Self-monitoringObserving and tracking one’s own performance and

outcomes .45

Task strategies Creating mnemonics to remember facts .45

Imagery Creating or recalling vivid mental images to assist learning .44

Time management Scheduling daily studying and homework time .44

Environmental restructuring

Efforts to select or arrange the physical setting to make learning easier .22

Learning has multiple determinants E.g., Predicting GPA Correlations

Individual results may vary. Robbins, Lauver, Le, Davis, Langley, & Carlstrom (2004)

Major Student Characteristics

Cognitive: ACT, SAT, Highschool GPANon-Cognitive:

Conscientiousness (Noftle & Robins, 2007)

Intrinsic Motivation (Komarraju, Karau, & Schmeck, 2009)

Self-Efficacy Academic discipline

Why Is This Important for SoTL?

Learning

Instruction

Effort

X

A measure of Learning Styles ?

Mental learning models

Learning orientations

Context

Richardson, 2011

86

Dunlowsky et al. (2013)

87

88

Dunlowsky et al. (2013)

“The growth of any craft depends on shared practice and honest dialogue among the people who do it. We grow by private trial and error, to be sure – but our willingness to try, and fail, as individuals is severely limited when we are not supported by a community

that encourages such risks.” —Parker J. Palmer

On Reading

Helen Regueiro Elam explains in “The Difficulty of Reading” (1991: 73), American culture does not take well to

the idea of difficulty. Our penchant is for one-step, one-stop solutions to problems, and we expect and demand in all areas of life, including reading, an ease of achievement that is antithetical to thought itself. . . . Difficulty is there to be overcome, disposed of, certainly not to become the invisible partner of our daily lives.

Compare Pre-Post: Close Reading

Chick, Hassel, & Haynie (2009)