Anatomy of Course Redesign:

How to know what works by removing the guesswork.

Dr. Mike Simmons | Dr. Ron Carriveau 1

2

INTRODUCTIONThis session focuses on the procedures and data collection used to provide meaningful information for use by faculty in making valid instructional and course redesign decisions.

3

Why We Did What We Did:

• Course redesign goals and challenges with which institutions have difficulty– Linking student learning to Student Learning

Outcomes– Instruction and assessment providing evidence of

student outcome attainment – Demonstrating how assessment results are used to

make instructional changes

4

Background

• Background– Five years of QEP implementation–Providing evidence of student outcome

attainment (NOT GRADES)–UNT process and methodology used to

address these goals and challenges

Next Generation Course Redesigntm

• 350 + sections• 16,000 + students• 30 + courses

5

6

NextGen Outcome Based Model

• Outcome attainment measures • Instructional methods • Foundation – Assessment– Intended student learning outcomes– Instructional methods used

The NextGen Outcome based assessment model

Align instruction to

assessment

Develop instructional strategies that tell how opportunities will be provided to help students achieve.

Develop outcome statements that tell what students are to achieve.

Develop assessments to measure the degree to which students are achieving and have achieved.

Use outcome statements to develop test items

Align instruction to outcome statements Use assessment

results to inform and improve instruction, assessment, and outcomes

Source: Carriveau, R.S. (2011). Connecting The Dots: Writing Student Learning Outcomes and Outcome Based Assessments. Fancy Fox Publications, Denton, TX.

The Three Level ModelItem 1. 2. 3. 4. 5. 6. 7. 8. 9.10.11.12.13.14.15.16.17.18.19.20.

Specific OutcomesLO 1.1.1

Specific OutcomesLO 1.1.2

Specific OutcomesLO 1.1.3

Specific OutcomesLO 1.2.1

Specific OutcomesLO 1.2.2

General OutcomeGLO 1.2

General OutcomeGLO 1.2

Goal1

Source: Carriveau, R.S. (2011). Connecting The Dots: Writing Student Learning Outcomes and Outcome Based Assessments. Fancy Fox Publications, Denton, TX.

9

RATIONALE FOR ATTAINMENT

Anatomy of Course Redesign: How to know what works by removing the guesswork.

10

Why Use Attainment Values?• Valid measure of student attainment of learning

outcomes as the basis for instructional changes and course redesign

• Provides superior measures to the use of grade distributions, percentages, and other traditional achievement measures which lack the ability to address student learning outcomes at the course level and provide little to no evidence for course improvement

%age Item Correct1. 872. 903. 654. 585. 636. 527. 668. 779. 8410. 9311. 9612. 8813. 8214. 8815. 9016. 8017. 9218. 8119. 8120. 82

Specific Outcome1.1.1

Avg = 81

Specific Outcome1.1.2

Avg = 60

Specific Outcome1.1.3

Avg = 88

Specific Outcome1.2.1

Avg = 85

Specific Outcome1.2.2

Avg = 85

General Outcome1.1

Avg = 81

General Outcome1.2

Avg = 85

Calculating Outcome Attainment Values (p.83)

Goal1

Avg = 83

Source: Carriveau, R.S. (2011). Connecting The Dots: Writing Student Learning Outcomes and Outcome Based Assessments. Fancy Fox Publications, Denton, TX. Pg 83

Item Difficulty SummaryL M H15 25 10

30% 50% 20%

Suggested Difficulty ScaleH M L

Below 60% 60-79% 80-100%

Outcome Attainment Goal GLO sLO AV

1 81 1.1 81 1.1.1. 66 1.1.2. 87 1.1.3. 80 1.1.4. 75 1.2 93 1.2.1. 93 1.2.2. 73 1.2.3. 852 83 2.2 81 2.2.1. 73 2.2.2. 85 2.2.3. 81 2.2.4. 883 81 3.1 81 3.1.1. 71 3.1.2. 81 3.1.3. 84 3.1.4. 80

Fixing Items

Item # N=50

09 Fall Item %

10 Fall Item %

11 Spring Item %

49 59 57 5231 55 48 569 33 84 84

64 61 89 8810 70 75 8254 43 38 6211 23 10 1436 65 76 6060 37 27 6327 51 63 6429 97 74 8139 66 92 8315 32 67 7421 38 87 8724 54 71 6337 32 66 6052 17 51 352 35 33 21

33 50 86 8214 71 30 2126 65 36 2430 33 54 5348 46 68 774 43 28 356 52 40 67

Outcome Attainment

Applying Attainment Measure to a Marketing Class Learning Activity

Experiential Activity: OBSERVING MARKETING AND GENDER STEREOTYPINGThe purpose of this assignment is to examine the marketing of toys and sports equipment as well as advertising images of boys and girls in play and sports contexts. The focus will be on memory capabilities of adults compared to the memory capabilities of children. Class will be divided into three groups. Each group will be assigned a specific task to research and will post their findings online.

Marketing Experiential ActivityItem n=10

Item % sLO

sLO AV GLO

GLO AV

23 83 1.2.3 1.218 67 1.2.3 1.222 88 1.2.3 1.2

4 38 1.2.3 69 1.2 692 94 1.3.1 92 1.3

13 98 1.3.2 1.314 86 1.3.2 1.316 85 1.3.2 91 1.3 9142 56 2.1.2 2.132 50 2.1.2 53 2.1 53

1.2.3

Students will be able to detect differences between sub cultural ‐market segments’ attitudes toward brands.

1.3.1

Students will be able to recognize consumer sub cultural market ‐segments’ VALs.

1.3.2

Students will be able to detect differences between consumer sub‐cultural market segments’ attitudes toward brands.

2.1.2

Students will be able to relate how self identity may impact consumers ‐on consumption choices.

Examples of Statements That Can Be Made Using the Three Level Model

• The class as a whole met the criterion on four out of five specific learning outcomes (sLOs).

• The class met the criterion on both general outcomes (GLO level) and the associated Goal.

• An improvement goal is to raise the general outcome (GLO level) criteria to

82%.

• Our improvement goal is to increase Goal 1 attainment by 5 points within a year.

15

INFORMATION FOR INSTRUCTIONAL DECISIONS

Anatomy of Course Redesign: How to Know what works by removing the guesswork.

16

Information for instructional decisions

• Need An Analytics (Many Sources of Data) Approach To Make Good Instructional and Course Redesign Decisions– “Analytics is quickly becoming a term that gets

slapped onto any existing product.” G. Siemans• This is academic (course level analytics) - key

difference from learning (SIS or system analytics)

1st International Conference on Learning Analytics and Knowledge

17

Learning Analytics

Penetrating the Fog: Analytics in Learning and Education – Educause

“learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs.”

18Source: George Siemens, http://www.learninganalytics.net/

Obtaining Information About Students

Formative information • Monitoring learning• Measuring achievement on the construct scale• Determining the degree to which student is learning• Determining the degree to which student meets outcome expectations

Pre-Post Information• Measuring gains• Measuring growth• How far did individual move• How far did group move

Summative Information• Making judgments • Assign proficiency level

PRECOURSE DEMOGRAPHIC PROFILE1. Gender Male Female

2. Race African American Asian Caucasian Hispanic Indian Non-Resident Alien

3. Classification Freshman Sophomore Junior Senior Post Bac Masters Doctoral

4. Hours Worked per Week Currently do not work Less than 10 hrs per week 10 to 20 hrs per week 21 to 30 hrs per week greater than 30 hrs per week

5. Residence Resident Hall City of Denton Denton County Outside of Denton County

6. Age Below 19 19 to 20 21 to 22 23 to 25 26 to 30 31 to 40 41 to 50 Greater than 50

Decision Questions

Does this year differ significantly from last year?

Would seeing this demographic information prior to starting the class cause you to want to make changes to the instruction and course. design?

How might you make changes in instructional design to accommodate the profile of the class you are getting?

CHARTS SHOWING PRE-COURSE PROFILES

Classification Counts with Percentage

145

113

48

161

3124

10

34

0

20

40

60

80

100

120

140

160

180

Freshman Sophomore Junior Senior

Count

Percent

RESIDENCE

110

70

115

27

0

20

40

60

80

100

120

140

Resident Hall City of Denton Denton County Outside DentonCounty

AGE of STUDENTS

62

118

82

42 4028

102

020406080100120140

Below19

19 -20 21-22 23-25 26-30 31-40 41-50 50 +

Types of Normed Data Used by UNT

• Entrance Exams- SAT- ACT- TOEFL- Acuplacer Math Placement

• Core Curriculum Evalaution- CAAP- CLA- CBase

Traditional Measure Student Success for a ClassMGMT Student Success Rates

SuccessS U

Format INSTRUCTOR Count Percent Count Percent TotalFTF 1 Semester 08-Fall 213 0.87 32 0.13 245

09-Spg 238 0.90 25 0.10 263

09-Fall 267 0.91 28 0.09 295

10-Spg 241 0.90 26 0.10 267

2 Semester 10-Spg 0 0.00 1 1.00 1

3 Semester 05-Fall 147 0.75 48 0.25 195

06-Fall 135 0.70 59 0.30 194

06-Spg 155 0.71 62 0.29 217

07-Fall 191 0.77 56 0.23 247

07-Spg 159 0.73 59 0.27 218

08-Spg 175 0.80 44 0.20 219

09-Fall 215 0.81 51 0.19 266

4 Semester 05-Fall 160 0.76 50 0.24 210

06-Fall 194 0.89 25 0.11 219

06-Spg 155 0.77 47 0.23 202

07-Spg 170 0.78 48 0.22 218

2815 0.75

NextGen 5 Semester 07-Fall 173 0.82 39 0.18 212

08-Fall 188 0.87 29 0.13 217

08-Spg 208 0.83 43 0.17 251

09-Spg 234 0.91 25 0.10 256

10-Spg 253 0.94 15 0.06 268

25

NextGen Information

• Attitude Toward Course Survey• Learning Environment Preference Survey • Course Format Preference Survey• Student Evaluation of Teaching Effectiveness• Tests• Engagement

Student Preferences for N-Gen Course Format Versus Traditional FTF Format for a COMM course

Table 1 shows the results of the Student Format Preference surveys administered at the end of each semester. Provided are the counts and percentages of student preferences for course format plus the preferences by the categories of Successful (grade of A,B,C) and Unsuccessful (D,F,W,I). The values in parentheses are percentages.

COMM 1010Preferred N-gen

PreferredFTF

TotalNumber Success

Total Number Un-successful

Success preferred N-Gen

Successpreferred FTF

Un-success preferred N-Gen

Un-success preferred FTF

2009 Spring (n=595) 334(.56) 261(.44) 564(.95) 31(.05) 318(.56) 246(.44) 16(.52) 15 (.48)

2009 Fall (n=507) 232(.46) 275(.54) 472(.93) 35(.07) 216(.46) 256(.54) 16(.46) 19(.54)

2010 Spring (n=386) 181(.47) 205(.53) 380(.98) 6(.02) 177(.47) 203(.53) 4(.67) 2(.33)

Student CommentsStudent comments as to why they preferred an NextGen format versus an FTF format were also collected, and the current semester comments are sent to you in a separate email.

Student Preferences for NextGen Course Format Versus Traditional FTF FormatStudent Comments

Student comments as to why they preferred an NextGen format versus an FTF format were also collected. After conducting a study of student responses over a two year period, it was found that the following categories emerged. Categories with descriptions for reasons why students chose NextGen or Traditional FTF.

Format

Reason Category

Description NextGen

Pace

Students liked that they could control the rate at which they absorbed information.

Flexibility

Students liked that they could do assignments whenever and wherever they wanted. Learning

Students found it easier to learn content when it is internet based. Practice

Students liked that there were more opportunities to practice and learn

FTF

Manage

Students needed a structure so that they wouldn’t procrastinate.

Learning

Students found it easier to learn content when format is FTF.

People

Students felt that they needed the face to face (lecture) interaction.

Technical

Students had difficulties with computers , network, and technology used

SURVEY RESULTSSTUDENT ATTITUDE TOWARD COURSE SUBJECT (SATCS)

N = 61 Class enrollment = 99 +/- Pre (SD) Post (SD)Diff (* is sig at .05) (SD)

1 This subject is worth knowing. + 4.29 (0.56) 4.33 (0.66) +0.05 (0.67)

2 I like this subject. + 4.00 (0.55) 4.14 (0.73) +0.14 (0.57)*

3 Knowing this subject makes me more employable. + 3.90 (0.70) 3.81 (0.98) -0.10 (1.09)

4 This subject is easy to learn. + 3.71 (0.64) 3.95 (0.67) +0.24 (0.70)

5 This subject should be required for all students. + 3.29 (1.00) 3.48 (0.93) +0.19 (0.87)*

6 This is a difficult subject for me. - 2.35 (0.67) 1.95 (0.76) -0.40 (0.82)

7 Learning this subject requires a lot of hard work. - 2.86 (0.79) 2.33 (0.86) -0.52 (1.08)

8 Knowing this subject is valuable to me. + 4.14 (0.66) 3.81 (0.81) -0.33 (0.91)

9 This subject makes me feel anxious or uncomfortable. - 1.81 (1.81) 1.52 (0.51) -0.29 (0.64)

10 This subject does not fit into my overall educational needs. - 1.90 (0.77) 2.10 (0.89) +0.19 (0.51)*

11 This subject is interesting. + 4.25 (0.55) 4.05 (0.61) -0.20 (0.70)

12 This subject is difficult to understand. - 2.24 (0.70) 2.00 (0.89) -0.24 (0.89)

13 This is a complicated subject. - 2.45 (0.89) 2.15 (0.88) -0.30 (1.08)

14 I know a lot about this subject. + 3.00 (0.80) 3.35 (0.81) +0.35 (0.81)*

15 This subject is relevant to my personal goals. + 3.80 (0.83) 3.75 (0.97) -0.05 (0.89)*

16 I can learn this subject. + 4.25 (0.44) 4.25 (0.55) 0.00 (0.65)

17 This subject is useful to my everyday life. + 4.00 (0.55) 3.90 (0.77) -0.10 (0.70)*

18 I will have no application of this subject in my profession. - 2.05 (0.87) 1.95 (0.87) -0.10 (1.00)

19 I am scared by this subject. - 1.48 (0.51) 1.52 (0.60) +0.05 (0.59)*

20 I want to learn more about this subject. + 3.95 (0.74) 3.90 (0.70) -0.05 (0.74)*

21 This is a fun subject. + 3.90 (0.77) 4.05 (0.67) +0.14 (0.66)*

Learning Environment Preferences Survey (LEP) ReportCourse:

Semester:

1. What to Learn 2. How to Learn3. How to Think 4. How to Judge

LEP Summary Table

Semester Adm Category 1 Category 2 Category 3 Category 4 CCI

Spg 09 (n=35) Pre 48% 27% 13% 12% 288(36.6)

Post 44% 29% 14% 13% 292(44.7)

Difference -4% +2% +1% +1% +6Spg 10 (n=57 )

Pre 41.2 (20.2) 25.6 (11.1) 18.1 (15.3) 15.1 (10.9) 307.0 (47.0)

Post 44.2 (20.5) 24.4 (11.6) 16.0 (12.3) 15.4 (10.9) 303.0 (45.0)Difference +3.0 -1.2 -2.1 +0.3 -4.0

Fall 10 (n = 35) Pre43.21 (17.42) 28.18 (12.63) 12.91 (10.99) 15.64 (10.99) 301.05(40.38)

Post 38.03 (18.68) 37.77 (10.91) 16.85 (12.77) 13.48 (8.97) 305.71(42.65)

Difference -5.18 (1.26) 9.59 (-0.08) 3.94 (1.78) -2.16 (-2.02) 4.66 (2.27)Spg 11 (n= 46)

Pre 43.85(22.24) 24.96(12.25) 18.39(17.68) 12.78(10.82) 300.13(52.16)

Post 42.22(25.50) 24.20(14.02) 17.96(15.97) 15.72(11.52) 307.13(55.52)Difference

-1.63 -0.76 -0.43 2.93 7.00Fall11 (n=24)

Pre 44.71(24.68) 26.54(16.41) 16.08(17.30) 12.75(9.72) 297.00(51.80)

Post 41.54(25.25) 28.63(13.92) 12.79(14.15) 17.08(12.05) 305.25(55.08)Difference

-3.17 2.08 -3.29 4.33 8.25

31

Engagement – Course Level

• CLASSE is a pair of survey instruments that enable one to compare what engagement practices faculty particularly value and perceive important in a designated class with how frequently students report these practices occurring in that class.

• CLASSEStudent is the survey instrument completed by each student enrolled in the designated class, while CLASSEFaculty is the survey instrument completed by the faculty instructor of the designated class.

http://assessment.ua.edu/CLASSE/Overview.htm

32

Exploring Other Analytics

• Big Data– McKinsey Global Institute defines big data as

“datasets whose size is beyond the ability of typical database software tools to capture, store, manage and analyze.” (James Manyika, “Big Data: The Next Frontier for Innovation, Competition, and Productivity,” Executive Summary, McKinsey Global Institute, May 2011)

• Learning management system analytics

33

34http://research.uow.edu.au/learningnetworks/seeing/snapp/

index.html

35

INSTRUCTIONAL ANALYTICSUsing information to improve instruction

36

Scenario Which of the following statements best represents your conclusion about the particular online activity?A. I created a great learning environment but I need to fix that dang 45% correct item that didn’t work last year either. B. I need to consider redesigning the online activity so that students will do better on attaining the sLO.C. I will make sure that next semester I spend more review time on what the 45% test item was covering.

You just got your end of course results on student attainment of outcomes. •For a particular online activity that was designed to help students achieve a high degree of attainment on a particular sLOs, five test items were used. •Four of the test items showed above 80% percent correct and one was at 45% correct, which of course lowered considerably the average the attainment value for the sLO. •Your attitude toward the course topic survey showed a slight positive result•70% of the students who took the format preference survey said they prefer NextGen to traditional FTF. •There was also high positive agreement on the engagement survey between what you thought was important and what students saw happening in the class.

37

Information and Contacts

• Mike Simmons, Ph.D.– mike.simmons@unt.edu

• Ron Carriveau, Ph.D.– ronald.carriveau@unt.edu

http://nextgen.unt.eduhttp://clear.unt.edu/Twitter: @nextgeneducatehttp://www.slideshare.net/simmonsweb