The Effect of Quality Matters™ Training on Faculty Perceptions of Their Ability to Design, Develop, and Deliver Online Courses

James M. Wrightjwright@kennesaw.edu November 6, 2008

Agenda

Problem Statement Purpose of the StudyPurpose of the StudyResearch QuestionsMethodsResultsDiscussionLimitations ImplicationspFuture Research

Download Paper and Presentation at:http://bcoe.kennesaw.edu/facultyonline/

Numbers are like people; torture them enough andthey'll tell you anythingthey'll tell you anything.

Problem Statement

Kennesaw State University (KSU) is making great strides to develop and promote online learning. p p gPerhaps the faculty may not know how to develop pedagogy for this new medium, or

nderstand hat makes a good learning understand what makes a good learning experience. A significant factor for embracing new g gtechnologies may be the faculty’s perceptions and attitudes towards distance learning.

Purpose of the Study

Using the theoretical framework of social cognitive theory and self-efficacy (Bandura, g y y ( ,1997; Schunk, 1994; Ormrod, 2007). The purpose is to increase online learning self-efficacy through the introduction of the Quality efficacy through the introduction of the Quality Matters™ framework.

The objective of this study is to determine if there is linkage between faculty training and faculty perceptions of their ability to be faculty perceptions of their ability to be successful online teachers.

Research Questions

1. Can training with the Quality Matters™ framework positively increase faculty perceptions of their ability to design, develop, and deliver online courses?

2. Is there a relationship between faculty perceptions of their ability to design, develop, and deliver online courses and demographic variables like age, gender, ethnicity, and faculty rank?

Quality Matters Framework

KSU is implementing the Quality Matters™ (QM) f k (M l dO li I 2006) t framework (MarylandOnline, Inc., 2006) to ensure excellence in their online courses. The QM framework is a series of rubrics and The QM framework is a series of rubrics and tools for peer-review and improvement of online instruction.

Rubric Parts:1. Course overview and introduction2 L i bj ti

5. Learner engagement6 C t h l2. Learning objectives

3. Assessment and measurement4. Resources and materials

6. Course technology7. Learner support8. Accessibility

Methodology

Participantsn = 17 pre-test, n = 14 post-testp , p

Measures Background

Kinuthia (2003) from Ga. State, developed an instrument to examine faculty participation in web-based instruction. Demographic information and many of the questions regarding the computer competencies.competencies.Miltiadou and Yu (2000) designed a valid instrument, the Online Technologies Self-Efficacy S l (OTSES) t t d lf ffi i li Scale (OTSES), to study self-efficacy in online learning. The section of the instrument focuses on the perceptions of online learning were based on p p gthe OTSES.

Measures

Questionnaire sections:1. Demographic & professional experienceg p p p2. Computer competencies3. Perceptions of online teaching

Given twice on pre and post testsGiven twice on pre and post tests

Five-point Likert scaleUsed Survey Monkey to collect resultsy yInformed consent agreement on instrument

Sample Questions

Procedures

Participants volunteered and attended a face-to-face training workshop offered on three different days Before the training, the participants were surveyed with a pre-test questionnaire with a pre-test questionnaire After the training the participants were asked to complete a post-test questionnaire. Post-test only asked section three of the questionnaire, perceptions of online learning

Results Procedures

Reversed the scores of the negatively worded questions Computer competency section – calculated index score for each participantPerception of online learning pre test calculated Perception of online learning pre-test – calculated index score for each participantPerception of online learning post-test – calculated p g pindex score for each participantUsed paired sample t test for analysis

Results RQ1

A significant increase from pre-test to post-test was found (t (13) = -6.526, p < .001)

Results in English

Mean - averageStandard Deviation - spread of data from the mean Stand Error Mean - SD divided by the square root of the n (n=14)Confidence interval – 95% confidence – further from zero the bettert – further from zero, less chance happen randomlydf – degrees of freedom n-1Sig value p value less than 001 (smaller the better)Sig value – p value less than .001 (smaller the better)

Results - Humor

Q: Did you hear about the statistician who was thrown in jail?statistician who was thrown in jail?

A: He now has zero degrees of A: He now has zero degrees of freedom.

Correlation & Reliability

Correlation analysis of the pre-test and post-test indexes showed a high Pearson correlation value of g.901 (n = 14)

Cronbach’s alphas for the sections ere in Cronbach’s alphas for the sections were in acceptable range

Computer competency questions alpha = .87 (n=17)Pre-test perception questions alpha = .91 (n=16)Post-test perception questions alpha = .81 (n=16)

Results RQ2

Relationship between faculty perceptions and demographic variables like age, gender, ethnicity, and faculty rankA factorial analysis of variance (ANOVA) was performedperformedA Bonferroini-adjusted p-value of .05 was used to measure age, gender, ethnicity, and faculty rankThere was no significant difference found regarding age, gender, ethnicity, or faculty rank in perceptions of ability to teach onlineperceptions of ability to teach online

DiscussionChanging attitudes is an important first step to creating change - affirms the RQ1The sample was self-selected and possessed a high level of computer skills

47% (n 8) Intermediate Computer Skills 47% (n = 8) Intermediate Computer Skills 41% (n = 7) Experienced Computer Skills

Perhaps these faculty members are the early Perhaps these faculty members are the early adopters of technology within the College of Education (Rogers, 2003). Therefore, it may not be t diffi lt t h th i titoo difficult to change their perceptions.

Discussion

Interesting – no significance because of sex, race, or faculty rank – RQ2race, or faculty rank RQ2Perhaps because of the same selection processpHowever, we need to capitalize on this change in attitude

Limitations

Very small sample size!!

Problems with getting faculty to attend

Limitations

Does a change in attitude produce more online courses?Does the correlation between faculty excitement translate into more classes.

Implications

Practice, Policy & LeadershipThe University must sustain this momentum with The University must sustain this momentum with additional training, resources, and support Vital to have technical support and instructional design support ready and available to the faculty Leaders must facilitate training to improve the f lt kill d id l ti faculty skills and provide release time Tenure and promotion must value online learning as scholarshipscholarship

Future ResearchRepeat of this study with a larger sample

Does the change in faculty attitude translate into more online courses?

Create mechanism for tracking creation of online Create mechanism for tracking creation of online courses

Does QM translate into quality courses? Empirical study to test for quality in the online

l classes.

Q ti ?Questions?