1

Metrics and Money:The Process and Politics of

Accountability

Stephen Daigle, Ph.D, California State UniversityMichael Large, Ph.D, Social and Behavioral Research Institute, CSU San MarcosPatricia Cuocco, California State University

2

Overview

• ITS and the Measures of Success (MOS)

• MOS Research Agenda

• Findings

• Questions (and, we hope, answers)

3

Integrated Technology Strategy

• Early 1990s – Presidential Interest in Using Technology As a Strategic Tool Driven By:– Immense Growth in Technology, – Antiquated Legacy Systems, – Increasing Demand for Access– Fiscal Constraints

• Mid 1990s – Develop Integrated Technology Strategy Framework– Iterative Process, Constituent Input and Approval

4

ITS – The Process

• 1996 Accepted by Board of Trustees• Living Framework – Not Written Plan• 2003 – Technology Decisions Still

Governed by ITS

5

What Is ITS

• Outcome Driven– Excellence in Learning and Teaching– Quality of the Student Experience– Administrative Quality and Productivity– Personal Productivity

6

ITS – the Icon

7

ITS – How It Fits

• Personal Productivity - Attained Through Infrastructure Initiative – Minimum Baseline at All CSU Campuses

• The Infrastructure Is Critical If ITS Benefits Are to Extend to All Students, Faculty and Staff

• The Infrastructure Requires Money ($250M) = PROBLEM

8

Getting the Infrastructure• Legislature Challenged CSU to Be

Creative• CSU Was Too Creative • Public/private Partnership Made

Legislature Uncomfortable• Californians Passed a Bond• Presidents Decided Infrastructure Had

Priority Over New Buildings

9

Getting to “Yes” *

• Bond Expenditures Require Legislative Approval• Legislature Wanted to Tie Investment in

Infrastructure to “Learning Outcomes”• Not What ITS Is About• Infrastructure Enables ITS Initiatives Which, in

Turn, Enable Outcomes

Apologies to Fisher, Ury and Patton

10

Negotiating Accountability

• Worked With Legislature – Agreed on What Could Be Measured

• Developed 10 Year Process for Measures of Success

• Framework • Baseline• Changes to the Baseline

11

Types Of Data Collected

• System• Campus• Individual (Student, Faculty, Staff)• External

12

Scope Of Data Collection

• 23 Campuses; 1,000 Miles• 400,000 Students• 35,000 Faculty And Staff• 10 Year Commitment• Institutional Comparisons Across Time• S, F, S Comparisons Across Time

13

Technology User Survey Samples

• Students: Campus, Class Level, Ethnicity; 23 X 5 X 9 Cells

•  Faculty: Campus, Rank, Discipline; 23 X 4 X 8 Cells

• Staff: Campus, Job Classification; 23 X 7 Cells

14

CATI LOGISTICS(COMPUTER ASSISTED TELEPHONE INTERVIEWING)

• No Self-select As With Mail• About 100 Questions; 20-30 Minutes• Importance Of Skip Facility• Instant Database Creation

15

CATI (Continued)

• Standardized Context Provided (E.G., High Speed Network)

• Trained Interview Staff, Monitoring, Evaluation

• Refusals Less Than 2 Percent; Still Over Sample

• $75 To $100k Per Survey, But Systemwide

16

User Outcomes

  Hardware Software Network Support Training

ACCESS          

USE          

SATISFACTION          

17

Examples of Metrics• Institutional

– Library Cost Avoidance– Smart Classrooms– Data Center Savings– Training and Support Spending

• Individual– Computer Ownership– Network Use– Help Desk Satisfaction– Use of Administrative Systems

18

Findings and Data

19

Findings

Hardware Software Network Support Training

Access

Use

Satisfaction

20

Hardware

• Access– Students’ access to hardware

• Use– Percent of Faculty Requiring Computer Use

• Satisfaction– Faculty satisfaction with Teaching in Computer La

bs

21

Software

• Access– Faculty access to software

• Use– Percent of Students Using Components of Their St

udent Information System• Satisfaction

– Staff Satisfaction with Software

22

Network

• Access– Students’ access to wireless networking

• Use– Staff use of network from off-campus

• Satisfaction– Staff satisfaction with e-mail

23

Student Access to a University-provided Computer

Access to a University-Provided Workstation

YesNo

Perc

ent

100

80

60

40

20

0

Administration

2001

2003

94

6

94

6

24

Percent of Faculty Requiring Students to Use a Computer

Required Students to Use Computer

YesNo

Perc

ent

100

80

60

40

20

0

ADMINISTRATION

2000 AY

2002 AY

85

15

64

36

25

Faculty Satisfaction with Giving Instruction in a Computer Lab

SURVEY ADMINISTRATION

2002 AY2000 AY

Mea

n Sa

tisfa

ctio

n10.0

9.0

8.0

7.0

6.0

5.0

4.0

3.0

2.0

1.00.0

Technical Support

Pedagogical

Effectiveness

7.87.4

6.76.4

26

Faculty Access to University-provided Software Needed

Access to University-Provided Software

YesNo

Perc

ent

100

80

60

40

20

0

ADMINISTRATION

2000 AY

2002 AY

94

6

96

27

Percent of Students Using Their Campus Student Information System

Administration

20032001

Perc

ent U

sing

Stu

dent

Info

Sys

tem

100

80

60

40

20

0

Information about

Registration

Information about

Grades

Information about

Financial Aid

Information about

Billing

Degree Progress

Information

37

22

40

20

41

29

88

71

86

67

28

Staff Satisfaction with University-provided Software Available

Satisfaction with Software

Completely Satisfied

987654321Not at All Satisfied

Perc

ent

40

30

20

10

0

29

21

25

12

55

29

Percent of Students with Laptops and Wireless Capability

Wireless NetworkingOwns a Lap-Top

Perc

ent

100

90

80

70

60

50

40

30

20

10

0

17

36

30

Staff Use of University Network from Off-campus

Accessed Network from Off-Campus

YesNo

Perc

ent

70

60

50

40

30

20

10

0

Year of Study

2000

2002

60

40

5050

31

Staff Satisfaction with E-mail Services

Q4B1B Satisfaction with Campus E-mail Services

8.74 2224 1.476

8.86 2123 1.344

8.80 4347 1.415

ADMIN Year of Study1 2000

2 2002

Total

Mean N Std. Deviation

32

Importance of Providing Electronic Access to Course Instruction

Any Time and Place

Academic Rank

Assistant ProfessorAssociate Professor

Professor

Mea

n Im

porta

nce

10.0

9.0

8.0

7.0

6.0

5.0

4.0

3.0

2.0

1.00.0

5.15.05.0

33

Comparison between Faculty and Students in Perceived Importance of Providing “Any-time, Any-place” Instruction

Q1A3 Importance of Providing Electronic Access to Course Instructionfor Students at Any Time and Place

5.07 3167 3.040

8.11 3086 2.156

GroupFaculty - 2002

Students - 2003

Mean N Std. Deviation

34

Uses of the Data

• Accountability• Description of the Population of Interest• Change Over Time• Subgroup Comparisons• Cross-group Comparisons

35

Conclusion

• Expensive – but You Get What You Pay For: Valid

• Reliable• High Level of Confidence• Negotiate – Don’t Be Passive – Be Proactive• Accountability Provides Cover• Infrastructure Is a Utility and a Prerequisite• Strategic Planning Is Dynamic – Change/add

36

Look For Yourself

http://its.calstate.edu/documents/Data_Collection/I_Reports_MOS/Measure_of_Success.shtml

37

Copies of the Presentation

http://its.calstate.edu