author: awe, jill d. evaluation of organizational …offsite training session called...

1

Author: Awe, Jill D. Title: Evaluation of Organizational Culture Training in a Health Insurance

Organization The accompanying research report is submitted to the University of Wisconsin-Stout, Graduate School in partial

completion of the requirements for the

Graduate Degree/ Major: MS Training & Development

Research Advisor: Jeanette Kersten, Ed.D.

Submission Term/Year: Fall, 2013

Number of Pages: 62

Style Manual Used: American Psychological Association, 6th edition

xI understand that this research report must be officially approved by the Graduate School and that an electronic copy of the approved version will be made available through the University Library website xI attest that the research report is my original work (that any copyrightable materials have been used with the permission of the original authors), and as such, it is automatically protected by the laws, rules, and regulations of the U.S. Copyright Office. xMy research advisor has approved the content and quality of this paper.

STUDENT:

NAME Jill D. Awe DATE: 12-09-13

ADVISOR:

NAME Dr. Jeanette Kersten, EdD DATE: 12-09-13

---------------------------------------------------------------------------------------------------------------------------------

This section for MS Plan A Thesis or EdS Thesis/Field Project papers only Committee members (other than your advisor who is listed in the section above) 1. CMTE MEMBER’S NAME: DATE:

2. CMTE MEMBER’S NAME: DATE:

3. CMTE MEMBER’S NAME: DATE:

--------------------------------------------------------------------------------------------------------------------------------- This section to be completed by the Graduate School This final research report has been approved by the Graduate School.

Director, Office of Graduate Studies: DATE:

2

Awe, Jill D. Evaluation of Organizational Culture Training in a Health Insurance

Organization

Abstract

This study was conducted to evaluate the effectiveness of a modified culture training session in a

health insurance organization. The evaluation method used in this study consisted of levels one

and two of Kirkpatrick’s Four Levels of Evaluation (2007). To perform the level one evaluation,

an online survey was distributed to participants after completion of the training which occurred

in the fall of 2013. The evaluation was done to measure the participant’s reaction to the

modified culture training. The level two evaluations were performed by distributing a paper pre

and post-test to all participants to evaluate what was learned as a result of the training. This

paper discusses the analysis of results for both the quantitative and qualitative data collected

followed by recommendations to the organization and opportunities for future research.

3

Acknowledgements

I would like to take this opportunity to express my gratitude to the people who have been

instrumental in the success of this study. First of all, I would like to thank the Culture Council

from the studied organization that allowed me to perform the evaluation of the modified culture

training. Their support and willingness to participate means the world to me. Also, I would like

to thank the participants in my survey, who have willingly shared their precious time during the

process of evaluation.

I would like to express my gratitude to my thesis advisor Jeanette Kersten, EdD for the

useful comments, remarks and engagement through the learning process of this study. She has

become a valued mentor and friend. I appreciate her patience and guidance throughout this

entire process.

There are several students in the Training and Development program that I would like to

extend a personal thank you to for answering my questions, providing humor when I needed it,

and for becoming friends. Thank you to Dale Kiesow, Jennifer Held, and Michele Hartness for

all the support and fun along the way!

Last and most importantly, I would like to thank my husband and my parents, who have

supported me throughout the entire process. Without their encouragement, I wouldn’t have had

the confidence and motivation to complete this study. I also want to send love to my sweet

daughter Loftyn. I hope mommy has made you proud!

4

Table of Contents

.................................................................................................................................................... Page

Abstract ............................................................................................................................................2

List of Tables ...................................................................................................................................7

Chapter I: Introduction ....................................................................................................................8

Statement of the Problem ...................................................................................................10

Purpose of the Study ..........................................................................................................10

Assumptions of the Study ..................................................................................................10

Definition of Terms............................................................................................................11

Limitations of the Study.....................................................................................................11

Methodology ......................................................................................................................11

Summary ............................................................................................................................11

Chapter II: Literature Review ........................................................................................................13

Organizational Culture .......................................................................................................13

Measuring Organizational Culture .........................................................................14

Organizational Culture Models ..........................................................................................15

Hierarchy model.....................................................................................................15

Market model .........................................................................................................15

Clan model .............................................................................................................16

Adhocracy model ...................................................................................................16

Organizational Climate and Job Satisfaction .....................................................................17

Evaluation of Organizational Culture Training .................................................................18

Level One: Reaction ..............................................................................................19

5

Level Two: Learning..............................................................................................20

Level Three: Behavior ...........................................................................................21

Level Four: Results ................................................................................................22

Chapter III: Methodology ..............................................................................................................25

Subject Selection and Description .....................................................................................25

Instrumentation ..................................................................................................................26

Data Collection Procedures ................................................................................................27

Data Analysis .....................................................................................................................27

Limitations of the Study.....................................................................................................27

Summary ............................................................................................................................28

Chapter IV: Results ........................................................................................................................29

Level One: Reaction ..........................................................................................................29

Quantitative Data ...............................................................................................................30

Qualitative Data .................................................................................................................36

Length of Course Was Too Short ..........................................................................36

Not Enough Group Interaction ...............................................................................37

Inadequate Training Environment .........................................................................37

Lacks Proof of Concept .........................................................................................37

Level Two: Learning..........................................................................................................37

Pre and Post-test Results ....................................................................................................38

Chapter V: Discussion ...................................................................................................................42

Limitations .........................................................................................................................42

Conclusions ........................................................................................................................43

6

Recommendations ..............................................................................................................45

Future Research .................................................................................................................46

Summary ............................................................................................................................47

References ......................................................................................................................................49

Appendix A: Invitation to Conduct Study .....................................................................................53

Appendix B: Consent to Participate in UW-Stout Approved Research ........................................54

Appendix C: Organizational Culture Training Pre-Test ................................................................56

Appendix D: Organizational Culture Training Post-Test ..............................................................57

Appendix E: Approval to use Kirkpatrick Model ..........................................................................58

Appendix F: Email Invitation to take the Evaluation of Culture Training Survey ........................59

Appendix G: Level One Reaction Survey .....................................................................................60

Appendix H: Email Reminder to take the Evaluation of Culture Training Survey .......................61

Appendix I: Email Thank You for taking the Evaluation of Culture Training Survey ................62

7

List of Tables

Table 1: The material covered in the program was relevant to my job ........................................30

Table 2: The material was presented in an interesting way ..........................................................31

Table 3: The instructor was an effective communicator ...............................................................31

Table 4: The instructor was well prepared ....................................................................................32

Table 5: The audiovisual aids were effective ...............................................................................32

Table 6: The handouts will be of help to me ................................................................................33

Table 7: I will be able to apply much of the material to my job ...................................................33

Table 8: The facilities were suitable .............................................................................................34

Table 9: The schedule was suitable ..............................................................................................34

Table 10: There was a good balance between presentation and group involvement ....................35

Table 11: I feel that the workshop will help me do my job better ................................................35

Table 12: List the company’s five core values ............................................................................38

Table 13: How do you apply core values in your daily job function ............................................38

Table 14: What is the natural midpoint on the organizational culture mood elevator ..................39

Table 15: What does the term “blue chip” mean ..........................................................................39

Table 16: Explain the concept of “Be Here Now” ........................................................................40

Table 17: Explain the accountability ladder .................................................................................40

8

Chapter I: Introduction

Company XYZ, who employs over 70,000 employees, set out to actively define their

organizational culture. To do so, Company XYZ partnered with Senn Delaney™, an

organizational culture consulting firm specializing in transforming organizational cultures, to

tailor an organizational culture training program for their leadership team with the intent to

further roll it out to all employees. The goal of the organizational culture training is to ensure

behaviors, decisions and daily interactions are driven by the organization’s core values and are

evident across the enterprise.

Once the training program was developed by Senn Delaney™, executive leaders from

Company XYZ delivered the training program to the staff. Within two years, 9,000 leaders

throughout the enterprise had attended the training. The training is divided into a two-day initial

offsite training session called “unfreezing” which unlocks the trainee from the old organizational

culture beliefs and introduces new thinking (Lewin, 1951). That session is followed by a one-

day offsite “reinforcement” training session three months later to reinforce the concepts learned

in the first session and to provide the trainees the opportunity to discuss successes and failures

they’ve experienced since the “unfreezing” session. These sessions mimic Lewin’s Change

Management Model which consists of three-stages: “unfreezing” (get employees motivated to

change), “change” (communicate change and empower people to think differently) and

“refreeze” (create stability after change is implemented), designed to model the changing

behavior of individuals in a changing situation (Lewin, 1951). The class size restriction for the

training is 30 or less participants. The current two and one day offsite sessions are costly and

time intensive to implement to the remaining 61,000+ employees.

9

A group of six leaders, who were from the same site and had attended the unfreezing and

refresh courses, were motivated to disseminate the organizational culture values to the

employees at their site versus waiting for the organization to complete the roll out enterprise-

wide. The leaders, along with assistance from a human resource manager, created a four hour

organizational culture training session which introduces the organizational culture initiative and

highlights the key organizational culture values the executive leadership team wants to instill in

the employees.

The leaders have no formal background in training but are visible leaders at the site and

have strong communication skills. These leaders have volunteered their time to perform the

training. The class size of the modified training sessions is 15 employees or less. The trainers

select participants from varying departments to ensure learning and collaboration occurs among

the site versus within one given department. Each session is led by two of the six leaders and

each leader holds one training session each month.

The training sessions are largely activity based to provide interaction among the

participants and to provide an opportunity to practice the company’s core values and cultural

behaviors. The activities are also meant to be transferrable so the participant can facilitate the

activity with the team they manage once they have completed the training. Some of the cultural

behaviors included are: taking accountability, prioritizing daily work so activities that will create

the most value for the company are completed first each day, and being present physically and

mentally during meetings and conversations with colleagues. The training also instructs trainees

where to locate additional resources upon completion of the course.

10

Statement of the Problem

Company XYZ was lacking a training strategy to implement a culture program to its

61,000+ employees. The original culture program was held over two days which was time

intensive and not cost effective. A group of six leaders in Company XYZ created a modified,

four hour culture training program in one site but had not yet evaluated the effectiveness of the

modified training.

Purpose of the Study

The purpose of this study was to evaluate the modified training session created by one

site in Company XYZ. The original organizational culture training completed by 9,000 leaders

although effective, was costly and time intensive to disseminate to the rest of the organization in

the original format. The new format created by the six leaders in one site in the organization, is

less time consuming and costly, but it is yet to be determined whether or not it is effective.

This study provided Company XYZ’s executive leadership team the opportunity to

measure the performance of the modified training to ensure it captures enough content to prove

effective in employees understanding the organizations vision and core values, as well as to

determine the scalability of rolling the training out to the remaining 61,000+ employees.

Assumptions of the Study

1. The modified training would be led by leaders in the organization versus led by trainers

in the training department.

2. If leaders in the studied site volunteer to deliver the training, leaders at other sites will

also volunteer to deliver training.

11

Definition of Terms

Organizational culture. The set of values, beliefs, and behavioral norms shared by

members of an organization that provide them with meaning and rules of behavior (Schein,

2004).

Organizational climate. Defined as a shared perception of the work environment that

influences performance (Parker, Boris, & Young, 2003).

Job satisfaction. Defined as "a positive emotional state resulting from the appraisal of

one’s job" (Brief & Weiss, 2001, p. 282).

Limitations of the Study

1. No training budget has been established by the organization.

2. Training participation in the modified training program may be a limited.

3. Participation in the training program has not been determined to be mandatory by the

organization; this may limit the sample size for the study.

4. Sustainability for the program may not occur due to the organization lacking financial

resources and/or motivated leaders to maintain the program.

5. This study may not be generalizable to other sectors of the organization.

Methodology

The methodology of this study included a pre-test for the participants as well as a post-

test to validate if learning occurred as a result of the training. Aside from the pre and post-test,

the study included collecting quantitative and qualitative reaction data using Kirkpatrick’s Model

of Evaluation upon conclusion of the modified organizational culture training to use for

recommendations on how to enhance the training for future sessions (Kirkpatrick, 2006).

12

Summary

Chapter 1 described the business case and organizational need for evaluating the

modified organizational culture training. Chapter 2 will provide a review of the literature related

to organizational culture, organizational climate, and training evaluation models. Chapter 3 will

review the methodology for collecting data and Chapter 4 will explicate data analysis techniques

used in this study. This study will also present the results of this study and provide a discussion

of the findings and recommendations for future study and application in Chapter 5.

13

Chapter II: Literature Review

The organization in this study provided a structured organizational culture training

program for its executive leadership in 2009, however was lacking a training strategy to

implement the remaining 61,000+ employees. The lack of a training strategy created a gap in

communication of the company’s values, vision, and expected behaviors between the executive

leadership and non-leadership staff. A modified training has been created and the effectiveness

for that training has not been evaluated.

In order to determine which evaluation method should be utilized to evaluate the

modified organizational culture training, two models for evaluation were researched. Included in

this literature review are Kirkpatrick’s (2006) four level evaluation model and Brinkerhoff’s

(2003) Success Case Method. Before a decision on which evaluation tool should be used,

research was compiled by Schein (2006) to define organizational culture, Cameron and Quinn

(2006) introduced types of organizational models, and Lehal (2004) to define organizational

climate.

Organizational Culture

Organizational culture has been defined by Schein (2004) as behavioral norms that define

rules in the organization whereby employees share and demonstrate the same values and beliefs.

Trevino and Nelson (1999) also defined organizational culture as shared beliefs and values and

believe it’s the social aspect that holds the organization together. Additionally, Schein (1985)

further defined organizational culture as:

14

A pattern of basic assumptions – invented, discovered, or developed by a given group as

it learns to cope with problems for external adaption and internal integration – that has

worked well enough to be considered valid and, therefore, to be taught to new members

as the correct way to perceive, think, and feel in relation to those problems. (p.9)

According to Deal and Kennedy (2000), strong organizational cultures have developed

rules indicating how people should behave, whereas weak organizations have employees that

waste time trying to figure out what to do. To understand what type of culture an organization

has, it is important to measure it.

Measuring organizational culture. Cameron and Quinn (2006) developed an

Organizational Culture Assessment Instrument (OCAI) consisting of six questions that pertain to

six dimensions of organizational culture (as cited by Yazici, 2011). These dimensions were:

• Dominant characteristics of the organization-what the overall organization is like.

• Organizational leadership style and approach that permeate the organization.

• Management of employees-the style that characterizes how employees are treated and

what the working environment is like.

• Organizational “glue,” or bonding mechanisms that hold the organization together.

• Strategic emphases-what areas drive the organization’s strategy.

• Criteria of success-what determines how victory is defined, and what gets rewarded

and celebrated (p. 22).

The six dimensions were valuable in defining success, values, and environmental

characteristics which were determined as measurements of an organizational culture (Yazici,

2011).

15

Organizational Culture Models

There were four models of organizational culture (hierarchy, clan, market, and

adhocracy) that were determined by using Cameron and Quinn’s (2006) OCAI tool as cited by

Fong and Kwok (2009). The personality of any organization should fit into one of these models

(Fong and Kwok, 2009). The definition of each model as well as examples of organizations that

fit that model is listed below.

Hierarchy model. Cameron and Quinn (2006) consider the hierarchy organizational

culture as the earliest approach, recognized by a formalized and structured working place. Fong

and Kwok (2009) state this model sets up rules, policy, or specialization to provide a stable

environment which increases efficiency, productivity, and generates efficient and reliable

products. Cameron and Quinn’s (2006) perspective was that the hierarchy culture was reflected

by workplaces such as McDonald’s, Ford Motor Company, and the Justice Department where

there were clear lines of decision-making, standardization, and control and accountability was

valued.

Market model. Cameron and Quinn (2006) indicated the major focus of the market

culture was to create a competitive advantage. Fong and Kwok (2009) stated the market culture

focused on management of external affairs, was results-oriented, and customer-based. Cameron

(2007) also referred to this organizational culture as rational and placed an emphasis on stability

during positioning in the competitive market. Cameron and Quinn (2006) stated that a market

culture has leaders who are producers and competitors such as companies like General Electric

and Philips Plastics Medisize.

16

Clan model. Clan organizational culture, referred to by Cameron and Quinn (2006) as

group organizational culture, focused on collaboration, flexibility and internal maintenance.

Fong and Kwok (2009) stated clan organizational culture was the sharing between individuals

through teamwork, loyalty, commitment, and participation of employees. Human resource

processes have been developed as a result (Fong & Kwok, 2009).

Similarly, Cameron and Quinn (2006) stated there was an emphasis on the long-term

benefit of individual development. Cameron and Quinn (2006) indicated in their competing

values framework book, that PeopleExpress Airlines had characteristics of a clan culture

(minimal management levels, job rotation, participation, employee ownership, etc.) which

contributed to high employee morale. Unfortunately, when this organization merged with

Frontier Airlines which was highly unionized, PeopleExpress was no longer in business due to

the culture clash between the two organizations. When two organizations have such a clash,

mergers and acquisitions are less successful due to those cultural differences. It is necessary for

organizations to evaluate culture during the merger and acquisition process (Cameron & Quinn,

2006).

Adhocracy model. Adhocracy organizational culture defined by Fong and Kwok (2009)

was dynamic and creative. This organizational culture has a higher ability to assume risk and

encouraged employees’ initiative and innovation by promoting development of unique products

and seeking new resources (Fong & Kwok, 2009). Hauser (1998) agreed that the adhocracy

organization’s basic orientation is to innovate. Bertrams (1999) also stated that an adhocracy

culture promoted innovation because people are motivated to learn, experiment, and take risks.

17

Adhocracies adapted to the culture which generates creativity (Cameron, Quinn, &

Degraff, 2007). Cameron, Quinn, and Degraff (2007) indicated the difference from adhocracy

culture and those previous mentioned was the centralization of power in the adhocracy culture

moves from individual to individual based on the situation. Some of the organizations Cameron,

Quinn. And Degraff (2007) referenced as adhocracy organizations were software development,

filmmaking, and aerospace.

Organizational Climate and Job Satisfaction

In a study by Zhang and Liu (2010), determined the effects of organizational climate on

organizational variables, the authors also identified that “organizational climate was the

perception of work environment by organization staff members” (p. 196) and as a result “the

subjective evaluation and perception of individuals for this reality could be adjusted and

controlled.” (p. 196)

Zhang and Liu’s (2010) research identified two basics modes of organizational climate,

macro and micro. The macro mode was the perceptions of individuals on the work environment

as a whole. The micro mode examined a particular dimension of the work environment in an

organization. A dimension researched further is job satisfaction. Bhutto et al (2012) performed

a study on the relationship between organizational climate and job satisfaction based on 14

factors. The authors performed the study on 12 branches of banks and concluded that three

factors that positively related to job satisfaction were organizational structure, identity, and

human relations. Equity and empowerment were viewed as negative factors related to job

satisfaction (Bhutto et al, 2012).

The study by Zhang and Liu’s (2010) proved that an employee’s length of service with

the organization, determined how satisfied they were with leadership and the organizational

18

climate. Those employees that had been with the organization the longest were the most

satisfied and effective in their role. Lehal (2004) cited by Bhutto, et al (2012) stated:

Organizational climate has a major influence on human performance through its impact

on individual motivation and job satisfaction. Climate does all this by creating

expectations about what consequences will follow from different actions. Employees

expect certain rewards and satisfaction on the basis of their perception of the

organization’s climate. Individuals in the organization have certain expectations, and

fulfillment of these depends upon their perception whether organizational climate suits

according to their needs or not. (p. 3)

Ashkanasv (2000) believed there was a strong correlation between job satisfaction and

job performance, so organizations must understand that employees' satisfaction will help in

encouraging a positive organizational culture. Lok and Crawford (2004) indicated that

employees who were dissatisfied with their job were less committed to the workplace.

Employees with low job commitment often quit their job, looked for jobs elsewhere or mentally

withdrew from the organization (Lok & Crawford, 2004). Kane-Urrabazo stated "it is crucial

that managers at all levels are aware of their roles and responsibilities in upholding positive

workplace environments that can increase employee satisfaction” (p. 193). Zhang and Liu’s

(2010) agreed that it was the manager’s role to investigate negative perceptions and follow up

with staff by over communicating to alleviate negativity.

Evaluation of Organizational Culture Training

Smidt, Balandin, Sigafoos, and Reed (2009) determined it is necessary to evaluate the

evidence in support of any particular study or training program. Kirkpatrick’s (2006) evaluation

method was designed to indicate whether the desired knowledge, skills, or attitudes were

19

retained by the learner (p. 42). Kirkpatrick’s Model (2006) looked at what effect the training had

on the overall organization. Typically, for this evaluation to occur, Kirkpatrick (2006) stated

business goals should be linked to the results of the training. This would indicate if the

organization’s profitability was impacted as a result of the training.

Alliger, Tannenbaum, Bennet, Traver, and Shotland (1997) indicated that the four-level

model can be used as a tool to benchmark outcomes of learning interventions. The four-level

model has been widely accepted and highly popular (Alliger et al, 1997). Kirkpatrick’s model

was based on four simple questions that translate into four levels of evaluation.

Level 1: Reaction. Evaluating level 1, Reaction, was deemed important by Kirkpatrick

and Kirkpatrick (2007), because participants may have felt that the instructor did not care about

personal perspectives if the trainer does not ask for feedback. This acknowledged the

importance that participants were satisfied with the training and if not, the feedback was used to

enhance the training for future use (Kirkpatrick and Kirkpatrick, 2007). In corporate training, if

there were unsatisfied participants who felt there was little benefit or positive experience could

lead to a loss for the trainer or training department which could limit the future use of the

training or trainer (Kirkpatrick, 1959). Similarly, Smidt et al (2009) stated a post-course

evaluation would evaluate the participant’s impressions of the program. Smidt et el (2009)

found that this type of evaluation gauged the interest, motivation, and attention levels of

participants, but does not measure what participants have learned.

The following guidelines were recommended by Kirkpatrick and Kirkpatrick (2007) to

measure reaction:

1. Make a list of the items where you want feedback. We suggest from eight to fifteen

items.

20

2. Design a form where the reactions can be quantified. The most common forms are on

a 5-point scale using either Excellent, Very good, Good, Fair, and Poor or the well-

known Likert scale, Strongly agree, Agree, Neutral, Disagree, and Strongly disagree.

3. End the form asking for suggestions for improvement.

4. Do not ask trainees to sign or put their names on the forms. In some cases, you may

want to say that a signature is optional in case someone makes a suggestion you may

want to pursue.

5. Try to get a 100% response at the conclusion of the program. If evaluation e-

learning, make it easy to response and stress the importance of a 100% response. If

evaluating an instructor-led program, give them time to fill out the form and lay it on

the back table as they leave. (p. 25)

Kirkpatrick and Kirkpatrick (2007) opined that the ultimate goals for reaction evaluation

says Kirkpatrick and Kirkpatrick (2007) were to ensure the participants were satisfied with the

training so there was not a perception that participant time was wasted, and also to reaffirm with

the participants that the trainer cared about the perceptions of how the training went.

Level 2: Learning. Kirkpatrick and Kirkpatrick (2007) stated that in any training

program, there were three possible objectives:

• For the participants to acquire knowledge related to their jobs

• For participants to learn new skills and/or increase their present skills

• For participants to change their attitudes (p. 47)

21

Kirkpatrick (1996) indicated there should be some measureable gain from at least one of

those objectives. Smidt et al (2009) suggested participants use role-plays or written assessments

to demonstrate skills. The authors believed that type of evaluation allowed participants to

demonstrate their understanding of specific skills and/or knowledge within the learning program.

Kirkpatrick and Kirkpatrick (2007) indicated that this level of evaluation was more difficult and

time consuming than measuring reaction. Kirkpatrick and Kirkpatrick (2007) provided the

following guidelines for evaluation of learning:

1. Use a control group if practical.

2. Evaluate knowledge, skills, and/or attitudes both before and after the program.

a. Use a “paper-and-pencil” or online test to measure changes in knowledge

and attitudes.

b. Use a performance test to measure increase in skills.

3. Get a 100 percent response, if possible.

4. Use the evaluation results to take appropriate action. (p. 48)

As indicated above, a control has been recommended if practical, however Kirkpatrick

and Kirkpatrick (2007) also indicated that using controls can be a challenge unless both groups

are alike in factors such as job duties, education, age, attitudes, experience or the comparison

was not valid. Kirkpatrick and Kirkpatrick (2007) stated that control groups may not be

appropriate unless the organization has experts at selecting control groups.

Level 3: Behavior. Kirkpatrick and Kirkpatrick (2013) explained that evaluating

behavior was the level by which participants applied what they learned upon returning to the job.

Smidt et el (2009) similarly believed that this level attempted to determine whether participants

use their new skills when they returned to the work environment. Kirkpatrick and Kirkpatrick

22

(2007) believed that without level three evaluations, proving business value was a challenge if

the skill was not transferred on-the-job. Kirkpatrick and Kirkpatrick (2007) opined that there

were at least four ways to evaluate level three (surveys and questionnaires, observations and

checklists, work review, and interviews and focus groups). Kirkpatrick and Kirkpatrick (2007)

suggested the following guidelines for evaluating behavior:


2. Allow time for the on-the-job behavior to be performed.

3. Evaluate both before and after the program if practical.

4. Consider a variety of methods. If you use surveys and/or interviews, consider one

or more of the following who have an opportunity to observe the trainee’s

behavior:

• trainees themselves

• immediate supervisor

• direct reports

• peers

• customers

5. Get a 100 percent response or a sampling.

6. Repeat the evaluation at appropriate times.

7. Consider cost versus benefits. (pg. 86)

Level 4: Results: Kirkpatrick and Kirkpatrick (2007) believed this was the most

beneficial level as demonstrated by the results the training has on the bottom line. Smidt et al

(2009) agreed with the importance of measuring financial impacts and add that morale should

23

also be measured as a result of the training. For example, has there been a decrease in staff

turnover?

Kirkpatrick and Kirkpatrick (2007) indicated that one of the challenges of this level was

that executives become impatient and want to measure right away. However, Kirkpatrick and

Kirkpatrick (2007) stressed the importance of reinforcing with executives that it takes time to

impact on-the-job habits and even longer for outcomes to be noticed in full-force. The

guidelines for evaluating results that Kirkpatrick and Kirkpatrick (2007) recommend were:


2. Allow time for results to be achieved.

3. Measure both before and after the program if practical.

4. Repeat the measurement at appropriate times.

5. Keep in mind you are charged with demonstrating Return on Expectations (ROE).

6. Be satisfied with evidence if proof is not possible. (pg 110)

Although Kirkpatrick’s model has been identified as the most popular, there have been

several weaknesses identified with Kirkpatrick’s model. Kirkpatrick’s model has focused more

on the reactions of trainees, leaving it incomplete, and not measuring performance as evidenced

(Rajeev, Madan & Jayarajan, 2009). A competing model has been the Success Case Method

(SCM), which was designed to be a simple way of determining the success of new organizational

initiatives (Brinkerhoff 2003).

Brinkerhoff (2003) developed a 5 step process for planning and implementation of the

success case method.

1. Focusing and planning a success case study

2. Creating an “impact model’ that defines what success should look like

24

3. Designing and implementing a survey

4. Interviewing and document success cases

5. Communicating findings, conclusions, and recommendations (p. 29)

However, the success-case method does not seem to produce a balanced assessment of the total

results of training (Brinkerhoff, 1983). Reports of training need to be appraised according to an

appropriate standard. The Kirkpatrick model has provided one technique for evaluating a

training program. The model can be used to determine if there were positive impacts to

operating costs, staff attitudes, and an increase in staff knowledge Smidt et al (2009).

Bates (2004) reviewed several models and concurs that the Kirkpatrick model was the

model most widely used in organizations. The Kirkpatrick model can be transferred to any

industry and would be considered advancement in evaluation practices in most organizations

(Lin and Chiu, 1997). Kirkpatrick’s model was chosen for this study as it closely aligns with the

objectives of this study.

Chapter 2 provided a review of the literature within the categories of organizational

culture and climate. Chapter 3 will provide an overview of the methodology used to collect data

and the technique used for analysis. Chapter 4 will provide the result of this study. Chapter 5

will provide a discussion of the results and recommendation for future studies.

25

Chapter III: Methodology

The purpose of this study was to do an evaluation of a modified organizational culture

training session in a health insurance organization. The original organizational culture training

was offsite during two days for session one and one day for the second session. The original

format proved to be too time consuming and costly to disseminate to the entire organization. A

four hour modified organizational culture training session was developed by six managers in one

operational site to reduce the time and cost of the original training. The evaluation of training

performed will determine the effectiveness of the modified organizational culture training.

Based on research, Kirkpatrick’s four levels of evaluation are deemed the best method for

evaluating the training.

Subject Selection and Description

The organization supplied an invitation to conduct the survey (see Appendix A). The

sample size for the pre and post-test (level two evaluations) was twelve managers from one

operational site in the organization. The managers were randomly selected from 257 managers

from that operational site using the simple random sample method. The selected participants

were asked to go into a room to take a written pretest to gauge their knowledge of organizational

culture concepts prior to attending the modified organizational culture training (see Appendix C).

After completion of the modified organizational culture training the participants were asked to

take a post-test to measure if their knowledge of organizational culture concepts increased as a

result of the training (see Appendix D). Aside from the post-test, a level one reaction survey was

distributed by email with a link to complete an online survey in the Qualtrics™ system (see

Appendix F). Approval was provided by Kirkpatrick to use the level one reaction survey (see

26

Appendix E). Participants were provided an informed consent form explaining their ability to

withdraw from the survey (see Appendix B).

Instrumentation

The basis for evaluating the modified organizational culture training was testing two

levels of Kirkpatrick’s Four Levels of Evaluation. Electronic questionnaires were developed and

distributed via email using the Qualtrics™ system at UW-Stout for the level one reaction survey

(see Appendix G). The level one-reaction focused on whether or not the trainees liked the

training program. This level measured customer satisfaction, understanding that a negative

reaction will reduce the amount of learning that occurred and a positive reaction may or may not

ensure learning occurred. A one-page form was created electronically using Qualtrics™ that the

trainees responded to both quantitative and qualitative questions. An eight point Likert scale was

developed for each quantitative question ranging from “strongly disagree‟ to “strongly agree‟.

One qualitative question was open ended to capture how the participant felt the training could be

improved (see Appendix G). An email invitation with the link to complete the survey was sent

to participants using the Qualtrics™ system at UW-Stout (see Appendix F). Reminder emails

with the link to the survey were sent to participants using the Qualtrics™ system at UW-Stout

that had not previously taken the survey, five days after the original survey was delivered (see

Appendix H). Once participants completed the survey, thank you notes were sent using the

Qualtrics™ system at UW-Stout (see Appendix I).

Level two (learning) measured what participants have learned from the training. This

measure determines what techniques, facts, and principles were understood and absorbed by the

trainees. To understand to what extent the trainees have improved in their knowledge, skills, and

ability related to organizational culture, a pre and post-test were performed. To ensure

27

participants did not use resources to help them answer questions on the pre and post-test,

participants were provided a written test to take in a room monitored by the researcher (see

Appendix C and Appendix D).

Data Collection Procedures

Prior to the pre-test for evaluation, the purpose of the study and utilization of results were

explained to participants. In addition, further explanation was provided that participation was

voluntary and would have no effect on their employment within the organization. Managers

selected for the pre-test were provided with implied consent and were assured that all surveys

would remain confidential and anonymous only to be viewed by the researcher with

recommendations shared with the Executive Director of the site (see Appendix B). To ensure

anonymity of the surveys, there were not any personal identifiable questions on the survey.

Data Analysis

Data analysis involved both quantitative and qualitative analysis. Quantitative analysis

involved reviewing the completed surveys and analyzing frequency of responses. Qualitative

data analysis identified common themes and subthemes from participant’s answers to an open

ended question. The themes were organized into categories and subcategories, providing

meaning to the participant’s feelings in regard to the training.

Limitations of the Study


2. Training participation in the modified training program may be limited.



28




Summary

Chapter 3 identified the proposed methods for this quantitative and qualitative study. A

description of subjects, sample size, and recruitment methods were proposed. The framework

for data analysis was discussed and limitations identified. Chapter 4 will provide an overview of

results of this study.

29

Chapter IV: Results

The purpose of this study was to evaluate the effectiveness of a modified organizational

culture training session created by one site in Company XYZ. Levels one and two of

Kirkpatrick’s four levels of evaluation (Kirkpatrick, 2007) were used to evaluate the session held

on October 23rd, 2013. During the study, level one evaluated the reactions the participants had of

the training session. Level two evaluated what the participants learned as a result of the training

session.

Level One: Reaction

An online Qualtrics™ survey was distributed to seventeen participants upon completion

of the training session to evaluate participant’s reaction of the session. The survey was

comprised of eleven questions using a Likert scale with response options ranging from strongly

disagree (1) to strongly agree (5) and a twelfth question that was open ended to capture

participant opinions on how to make the training better. A list of the questions from the survey

is below:

1. The material covered in the program was relevant to my job?

2. The material was presented in an interesting way?

3. The instructor was an effective communicator?

4. The instructor was well prepared?

5. The audiovisual aids were effective?

6. The handouts will be of help to me?

7. I will be able to apply much of the material to my job?

8. The facilities were suitable?

9. The schedule was suitable?

30

10. There was a good balance between presentation and group involvement?

11. I feel that the workshop will help me do my job better?

12. Please provide your suggestions for improving the course.

Due to the limited size of the training class, all 17 participants were provided the survey.

There was a 94% response rate with 16 of the 17 participants responding to the survey. The 16

participants that took the survey responded to each of the quantitative questions (questions 1-11).

Only 31% (n=5) of the 16 participants provided responses to the qualitative question (question

12), limiting the number of themes derived from responses.

Quantitative Data

Quantitative data was collected to determine participant’s reactions to the modified

organizational culture training. The first 11 questions of the 12 question survey were

quantitative using a Likert scale. Responses are represented below in tables 1-11 with frequency

of responses for each quantitative question in the survey listed as well as percentages.

Table 1

Question 1 – The material covered in the program was relevant to my job

Question SA A D SD Total Response (n) Mean

The material covered in the program was relevant to my job

9 6 1 0 16 3.50

The participants were asked to rate whether the material covered in the program was

relevant to the job. The question had four possible answers ranging from strongly disagree,

31

disagree, agree, and strongly agree (see table 1). The responses to this question were favorable

with a mean value of 3.50. The favorable responses were represented by 38% (n=6) of

participants agreeing and 56% (n=9) strongly agreeing that the material was relevant to their job.

The favorable response indicates that the program materials do pertain to the participant’s job.

Table 2

Question 2 – The material was presented in an interesting way


The material was presented in an interesting way

5 11 0 0 16 3.31

The second quantitative question was to determine if participants felt the material was

presented in an interesting way. All responses were positive and validated by a 3.31 mean value.

The positive responses were represented by 69% (n=11) of participants agreeing and 31% (n=5)

strongly agreeing that the material was presented in an interesting way (see table 2). The course

material was designed to have a mixture of presentation, open dialogue, and group activities.

Table 3

Question 3 – The instructor was an effective communicator

Question SA A D SD Total Response (n)

Mean

The instructor was an effective communicator

8 7 1 0 16 3.44

32

The volunteer instructor was a manager that did not have formal training to teach this

course. The participants were asked to rate whether the instructor was an effective

communicator. Half of the participants (n=8) strongly agreed and 44% (n=7) agreed that the

instructor was an effective communicator (see table 3). The remaining 6% stated they disagreed

that the instructor communicated effectively. Although the data represents a positive response

with a 3.44 mean value, there may be an opportunity to enhance the volunteer instructor’s

communication.

Table 4

Question 4 – The instructor was well prepared


The instructor was well prepared

7 7 2 0 16 3.31

The majority of participants were evenly divided when asked if the instructor was well

prepared with 44% (n=7) of participants strongly agreeing and 44% (n=7) of participants

agreeing the instructor was well prepared. The remaining 6% (n=1), disagreed the instructor was

well prepared (see table 4). The mean value of 3.31 indicates overall the participants felt the

instructor was well prepared.

Table 5

Question 5 – The audiovisual aids were effective

Question SA A D SD Total Response (n)

Mean

The audiovisual aids were effective

7 6 2 1 16 3.19

33

When the participants were asked if the audiovisuals were effective, 82% of participants

either strongly agreed (n=7) or agreed (n=6) audio visuals were effective (see table 5). Although

the majority of responses were positive with a 3.19 mean value, there were participants that

disagreed (n=2) and strongly disagreed (n=1) indicating the audiovisuals could be enhanced to be

more effective.

Table 6

Question 6 – The handouts will be of help to me


The handouts will be of help to me

3 11 2 0 16 3.06

When participants were asked if the handouts would be of help to them, 69% (n=11)

agreed and 19% (n=3) strongly agreed the handouts would be helpful (see table 6). The

remaining 12% (n=2) disagreed that handouts were helpful to them. With a mean value of 3.06,

there is opportunity to slightly enhance the handouts to be more useful to the participants.

Table 7

Question 7 – I will be able to apply much of the material to my job


I will be able to apply much of the material to my job

7 9 0 0 16 3.44

34

All participants responded positively when asked if the materials covered during the

training session could be applied to their job. Of the positive responses, 56% (n=9) agreed and

44% (n=7) strongly agreed that they would be able to apply most of the material to their job (see

table 7). The mean value of 3.44 indicates the training content was effectively designed to have

applicability to the participant’s job.

Table 8

Question 8 – The facilities were suitable


The facilities were suitable

7 5 4 0 16 3.19

The course was held in a training room that is traditionally used for technical training on

system applications. When participants were asked if they felt the facilities were suitable, the

majority did indicated agree (31%) or strong agree (44%) that the facilities were suitable. There

was a 25% response rate that indicated the facility was not suitable (see table 8). The mean

value of 3.19 indicates there is an opportunity to enhance the training facility to allow

participants a better learning environment.

Table 9

Question 9 – The schedule was suitable


The schedule was suitable

6 10 0 0 16 3.38

35

The next question asked participants if the training schedule was suitable. There was a

62% (n=10) response rate from participants that agreed, and a 38% response rate from

participants that strongly agreed the schedule was suitable (see table 9). The mean value of 3.38

indicates that scheduling training on a Wednesday afternoon from 12:30 p.m. – 4:30 p.m. was

suitable for participants.

Table 10

Question 10 – There was a good balance between presentation and group involvement


There was a good balance between presentation and group involvement

10 4 2 0 16 3.50

When participants were asked if they felt there was a good balance between presentation

and group involvement, the majority of participants (63%, n=10) strongly agreed and 25% (n=4)

agreed there was a good balance between presentation and group involvement (see table 10).

The training was designed to include presentation, group activities and individual reflection.

With 12% (n=2) of participants responding there could have been a better balance between group

involvement and presentation. This was interpreted as an opportunity to balance the presentation

style of the training session.

36

Table 11


I feel that the workshop will help me do my job better

5 9 1 0 15 3.27

The final question asked participants whether the workshop will help them perform at a

higher level. The majority (60%, n=9) agreed job performance would increase as a result of the

workshop. An additional 33% (n=5) strongly agreed the workshop will increase job performance

(see table 11). One participant (7%) disagreed this course would impact job performance.

Overall, the results indicate that the course is effective in helping participants perform at a higher

level.

Qualitative Data

The final question of the survey was qualitative in nature, asking participants their

suggestions for improving the course. The responses were logged and reviewed for common

themes. From the five responses that were submitted by the participants, the following four

themes were identified; (1) length of course was too short; (2) not enough group interaction; (3)

inadequate training environment; and (4) lacks proof of concept. Due to the limited number of

responses no subthemes were identified.

Length of course was too short. Although 100% of participants either agreed or

strongly agreed that the schedule was suitable, the strongest theme from the qualitative data was

participants felt that the course was too short for the volume of content. One participant

indicated the trainers ran out of time so the last two culture values did not get as much attention

37

as the first three. Another participant felt time needed to be monitored better quoting one of the

trainers as saying “no one had made it this far before.”

Not enough group interaction. The training course was designed to have a mixture of

presentation and group activities. Twelve percent of participants disagreed that there was a good

balance between presentation and group involvement. One participant specifically indicated

more group involvement or interaction during the training session was needed.

Inadequate training environment. Participant comments indicated the room was

unorganized which impacted participant’s ability to effectively concentrate. A participant

specifically stated “it makes me feel that I am not valued enough to take the time to have a clean,

organized area to learn in. But I understand that we have to use the rooms that are available to

us”. That same participant stated difficultly reading the screen due to poor lighting and the size

of the room. This participant stated that when concerns were shared with the trainer, the trainer

made an accommodation, moving the participant closer to the front of the room.

Lacks proof of concept. Participants felt there was no proof shared with the class that

the values trained worked. Participants wanted to know how the principles worked and if there

were other companies that had done something similar to compare to. One participant felt

proven results needed to be shared to validate the value of the training initiatives.

Overall, the comments provided by the five participants to the qualitative question

provided constructive feedback. The comments related to the inadequate training environment

can be fixed with minimal effort and no cost. All comments will be discussed in chapter five in

the recommendations section.

Level Two: Learning

38

To evaluate what the participant learned as a result of the training session, a paper pre

and post-test was administered. The pre-test was distributed and monitored to the 17 participants

by the researcher in the training room prior to the start of the course. The researcher returned to

the training room after the session completed to distribute and monitor the post-test to the 17

participants. All 17 participants took both the pre and post-test. The test consisted of six open-

ended questions listed below:

1. What are the company’s five core values?

2. How do you apply the core values in your daily job function?

3. What is the natural midpoint on the organizational culture mood elevator?

4. What does the term “blue chip” mean?

5. Explain the concept of “be here now”.

6. Explain the accountability ladder.

Pre and Post-Test Results

Table 12

Question 1 – List the company’s five core values.

# of Core Values Participant Knew

Pre-Test (n=)

Pre-Test %

Post-Test (n=)

Post-Test %

Pre-Test Mean

Post-Test Mean

1 Core Value 5 29% 0 0%

2 Core Values 6 35% 0 0%

3 Core Values 1 6% 0 0%

4 Core Values

1 6% 0 0%

5 Core Values 4 24% 17 100%

3.38 5.0

39

Question one on the pre and post-test asked participants to list the company’s five core

values. Prior to the course, 24% (n=4) of participants knew all five core values (see table 12).

After completion of the course, 100% of participants could list all five core values.

Table 13

Question 2 – How do you apply core values in your daily job functions?

Response Pre-Test (n=)

Pre-Test %

Post-Test (n=)

Post-Test %

No or incomplete Response 1 6% 0 0%

Correct Response

16 94% 17 100%

The next question asked how the participants applied the core values in their daily job

functions. Majority of participants submitted an appropriate response on both the pre and post-

tests. 94% (n=16) of participants submitted an appropriate response on the pre-test and 100%

(n=17) submitted an appropriate response on the post-test (see table 13). The results indicate that

the participants felt they already applied core values to daily job functions.

Table 14

Question 3 – What is the natural midpoint on the organizational culture mood elevator?


Pre-Test %

Post-Test (n=)

Post-Test %

No or incomplete response 9 53% 0 0%

Correct response 8 47% 17 100%

The third question on the pre and post-test asked if the participants knew what the natural

midpoint on the organizational culture mood elevator is. Majority of participants (53%, n=9) did

40

not complete this question on the pre-test or provided an incorrect response (see table 14). This

indicates there was a need for training on this concept. After completion of the training, 100% of

participants correctly answered what the natural midpoint of the organizational culture mood

elevator, proving the training was effective in explaining this organizational culture concept.

Table 15

Question 4 – What does term “Blue Chip” mean?


Pre-Test %

Post-Test (n=)

Post-Test %



Majority of participants submitted a correct response on both the pre and post-test when

asked what the term “blue chip” meant. There was an 82% (n=14) correct response rate from the

pre-test (see table 15). During the pre-test there were three participants that either incorrectly

responded to this question or skipped it completely. After the training course, all 17 participants

responded to the questioning and were able to explain the term “blue chip”. This proves the

training was effective in explaining the concept “blue chip”.

Table 16

Question 5 – Explain the concept of “be here now”?


Pre-Test %

Post-Test (n=)

Post-Test %



41

The fifth question asked if the participants knew the concept of “be here now”. There

was a 100% (n=17) correct response rate for the pre and post-test (see table 16). This indicates

that there was a good understanding of this concept prior to participants coming to the training

session. The training reinforced the concept which is evident by all participants accurately

responding to the question in the post-test.

Table 17

Question 6 – Explain what the accountability ladder is?


Pre-Test %

Post-Test (n=)

Post-Test %



The final question on the pre and post-test asked the participants to explain what the

accountability ladder is. The majority of participants knew what the accountability ladder was

prior to the course with 65% (n=11) submitting a correct response (see table 17). The training

was effective in providing the remaining participants (n=6) an understanding of the

accountability ladder concept which is proven by the 100% response rate in the post-test.

Summary

Chapter 4 provided an analysis of the results from the level one reaction survey

distributed to the participants who attended the modified organizational culture training session

on October 23rd, 2013. The results included both quantitative and qualitative data. Common

themes were presented from the participant’s comments to the one qualitative question.

Additionally, results from the pre and post-tests were compared and analyzed. Chapter 5 will

provide a discussion of the results, conclusions, and recommendations for this study.

42

Chapter V: Discussion

This study consisted of conducting a literature review of defining organizational culture

and methods for evaluating training to determine which model was most suitable to evaluate a

modified organizational culture training session for Company XYZ. The training evaluations

reviewed included Kirkpatrick’s Four Levels of Evaluation (2007) and Brinkerhoff’s (2003)

Success Case Method.

The modified organizational culture training session was evaluated using level one and

levels two of Kirkpatrick’s Four Levels of Evaluation (2007). Qualitative and quantitative

survey data were analyzed from the level one reaction survey that participants completed online.

Data was presented graphically and themes were presented and explicated by the (n=16)

quantitative responses and the (n=5) qualitative responses to an open ended question.

Additionally, a level two evaluation was completed using a written pre and post-test distributed

directly before and after the training. The results of the two tests were compared and analyzed

graphically and explicated by the (n=17) responses.

Limitations


2. Training participation in the modified training program may be limited.






43

At the conclusion of the study, the only limitation present was the lack of a training budget

being established by the organization. Although the training proved was effective, there were

opportunities identified that could have benefited from a formal budget for the program such as

formal training for the volunteer manager trainer, a more suitable training environment, and

more effective audiovisuals. The other limitations, although not present during this study, would

continue to be a potential concern if this program is disseminated to other sites in the

organization.

Conclusions

Kirkpatrick’s first two levels of his four levels of evaluation were performed and

supported as a result of this study. The level one reaction evaluation proved the importance of

asking participants for their feedback. Kirkpatrick (2007) indicated if participants were not

satisfied with the training, the participant feedback could be utilized to enhance the training for

future use. Overall, participants provided positive feedback in regard to the modified

organizational culture training; however analysis of the qualitative data supports benefit of

asking participants for their feedback.

The study identified that the training environment was not suitable for all learners.

Participants commented concentrating in the messy environment was difficult. Participants also

indicated the audiovisuals could be improved. The level one reaction results indicate that some

participants felt the trainer was not well prepared and did not communicate effectively. Without

providing the level one reaction survey, the trainer may assume the training either was more or

less effective than it really was. These comments as well as additional suggestions will be used

in the recommendations section of this study. The level one reaction survey proved to be an

effective evaluation tool. As indicated by Smidt et el (2009) in chapter two, the participant’s

44

motivation, interest, and attention level are measured, what the participants learned was not. As

a result, Kirkpatrick (2007) recommends conducting a second level of evaluation to determine

what the participant learned as a result of the training. For the purpose of this study, a pre and

post-test was administered to evaluate what the participants learned. At the end of the training

session, all participants scored 100% on the post-test. All but one question showed an increase

in correct responses from the pre to the post-test. The question that did not have an increase in

results had a 100% response rate on the pre and post-test. The most successful increase in scores

was the first question which asked participants what the company’s core values are. In the pre-

test, only 24% of participants knew all five core values compared to 100% of participants that

knew all five core values after the training.

The level two evaluation in this study concluded that participants did learn additional

organizational cultural concepts as a result of the modified training session. This is important in

showing the training was effective, and also agrees with the research in chapter two from Zhang

and Liu (2010) that stated individual’s perceptions of organizational climate can be adjusted and

controlled. Schein (1995) stated that organizational culture can be invented or developed and

taught to employees which teach those employees how to deal with potential problems that may

arise. The modified culture training evaluated in this study, accomplished that by presenting

participants with shared culture concepts and activities that provided the participants with ways

to apply the concept to their job.

Additional research in chapter two performed on Kane-Urrabazo stated "it is crucial that

managers at all levels are aware of their roles and responsibilities in upholding positive

workplace environments that can increase employee satisfaction” (pg. 193). The fact that the

modified training in this study was led by a manager in the organization shows management

45

commitment to shared beliefs and organizational culture concepts. That commitment was also

shown by the manager providing the modified training was a volunteer who is passionate about

teaching employees the shared vision.

Recommendations

The study’s results show that participants did learn the organization’s five core values as

well as the culture concepts presented in the training. Considering the post-test showed all

participants (n=17) correctly responded to all of the questions, there is value in the training and it

should be continued to be rolled out to additional employees in the studied site as well as in other

sites in the organization. The results of the study showed that 12% (n=2) of participants felt the

trainer was unprepared and one participant felt the instructor was did not communicate

effectively. Before future dissemination of the modified training to other sites, a train the trainer

certification program would be beneficial in which all volunteer managers would need to attain

before presenting the modified culture training to future participants. This would allow for

consistency in training and better prepare managers to train.

A comment by a participant in the level one evaluation indicated that it might be good for

an on-line pre-course to be provided to participants prior to attending the modified culture

training. This participant reported that would allow for more activities by eliminating the need

for the trainer to spend so much time in the beginning of the training explaining the initiative.

That recommendation would be easy to implement as there is already a cultural value online

course that has been created by the Learning and Development team at Company XYZ. That

course could become a pre-requisite for the modified training course. There were also comments

that participants felt the trainers ran out of time. By utilizing the pre-requisite course for an

46

overview will allow additional time for the culture concepts this will in turn allow participants to

feel less rushed.

Results of the study showed that 25% (n=4) of the participants felt the training facility

was not suitable for learning. The qualitative responses indicated the room was messy and was

too large so the audiovisuals were difficult to view. Based on participant comments, it may be

beneficial to limit the number of participants to 10 to 15 participants so a smaller training room

or meeting room could be utilized. The room should also be cleaned prior to the training and

tables should be free of computers so participants can focus on the materials versus the

surroundings.

The final recommendation would be to complete levels three and four of Kirkpatrick’s

(2007) evaluation model. Level three (behavior) could be evaluated in six months by sending

out a survey to the study participants asking if a positive change in job performance has occurred

as a result of the modified training. Participant’s managers could be sent a survey to determine if

the manager observed behavior changes in the participant since the training.

Level four evaluations may be a challenge to implement as it would require additional

time and funding. One way to implement level four evaluations would be a review of turn-over

rates by participants and participant’s employees one year from the study date. Company XYZ

also administers a yearly all employee satisfaction survey. There could be a question added to

the survey asking whether or not the respondent had completed culture training. Responses to

that question could be cross-tabulated to determine if employees that had attended training were

more satisfied with the job and with leadership than those employees that had not completed

culture training.

47

Future Research

Implementation of the recommendations of this study would contribute to future research

to determine what other organizations have done to successfully evaluate effective culture

training. A replication of this study could be completed for each future training class to learn

from participant’s reactions to the training and determine if the recommendations around class

size and more suitable environments had a positive impact to the overall rating of the course.

The culture trainers in the studied site could partner with the managers in another site to

pilot the modified training. The additional site could validate whether employees in that site had

the same reaction to the training. A survey could also be created and sent to the management

level to determine whether there would be volunteer involvement to present the training in

additional sites.

The culture trainers from the studied site could also partner with the Training and

Development team to develop and disseminate an online pre-requisite course. The Training and

Development team has a tracking mechanism in place that tracks which employees have taken

the course and the ability to send evaluations to course participants. The Training and

Development team could also provide additional recommendations on creating and

implementing a train the trainer certification program.

Summary

Chapter 5 discussed the findings of the evaluation of the modified organizational culture

training course at Company XYZ against the review of the literature which included

Kirkpatrick’s Four Levels of Evaluation (2007). The limitation revealed during the study was

the lack of funding for the modified organizational culture training. In order to have controls in

48

place and enhance the training deploymentto other sites in the organization, funding or additional

resources from within the organization may be needed.

The study concluded with several opportunities for additional research; such as,

decreasing the class size, continuing to utilize the level one and two evaluations on future

training classes, improving the training environment, and providing formal training to the

volunteer trainers. The study’s results in aggregate form will be reviewed with the executive

director of the site from Company XYZ to assist in the development of a sustainable modified

organizational culture training session that can be replicated and disseminated to other locations.

49

References

Alliger, G.M., Tannenbaum, S.I., Bennett, W., Traver, H., & Shotland, A. (1997). A meta-

analysis of the relations among training criteria. Personnel Psychology, 50(1), 341-358.

Ashkanasy, N.M., et al (2000). Handbook of organizational culture and climate. London: Sage

Publications.

Bates, R. (2004). A critical analysis of evaluation practice: The Kirkpatrick model and the

principle of beneficence. Evaluation and Program Planning, 27(3), 341-347.

Bertrams, J. (1999). De kennisdelende organisatie. Schiedam, Netherlands: Scriptum The

knowledge sharing organization.

Bhutto, N.A., Laghari, M.K., & Butt, F. (2012). A comparative study of organizational climate

and job satisfaction in public, private, and foreign banks. Asian Social Science, 8(4), 259-

267.

Brief, A.P., & Weiss, H.M. (2001). Organizational behavior: Affect in the workplace. Annual

Review of Psychology, 53, 279-307.

Brinkerhoff, R. (1983). The success case: A low-cost, high-yield evaluation. Training and

Development Journal. Retrieved from http://ehis.ebscohost.com/ehost/pdfviewer?sid

=0eddaa48-ea1f-49eead75-b948c41e843e@sessionmgr114&vid=3&hid=115

Brinkerhoff, R.O. (2003). The success case method. San Francisco, CA: Berrett-Koehler.

Business dictionary.com. (n.d.). Organizational culture. Retrieved from

http://www.businessdictionary.com/definition/organizational-organizational culture.html

Cameron, K.S., & Quinn, R.E. (2006). Diagnosing and changing organizational culture-based

on the competing values framework. San Francisco, CA: Jossey-Bass.

50

Cameron, K.S., Quinn, R.E., & Degraff, J. (2007). Competing values leadership: Creating value

in organizations. Williston, VT: Edward Edgar Publishing.

Deal, T.E., & Kennedy, A.A. (2000). Corporate organizational cultures: The rites and rituals of

corporate life. Cambridge, MA: Perseus Publishing.

Fong, P.S.W., & Kwok, C.W.C. (2009). Organizational culture and knowledge management

success at project and organizational levels in contracting firms. Journal of Construction

Engineering and Management ASCE, 1(1), 1349-1359.

Hauser, M. (1998). Organizational culture and innovativeness of firms – An integrative view.

International Journal of Technology Management, 16, 239-255.

Kane-Urrabazo, C. (2006). Management's role in shaping organizational culture. Journal of

Nursing Management, 14, 188-194.

Kirkpatrick, D. (2006). Evaluating training programs: The four levels (3rd ed.). San Francisco:

Berrett-Koehler.

Kirkpatrick, D. (1996). Revisiting Kirkpatrick’s four-level model. Training and Development,

50(1), 54-59.

Kirkpatrick, D. (1959a). Techniques for evaluating training programs. Journal for the American

Society of Training Directors, 13, 21-26.

Kirkpatrick, D., & Kirkpatrick, J. (2007). Implementing the four levels (1st ed.). San Francisco,

CA: Berrett-Koehler Publishers, Inc.

Kirkpatrick, J., & Kirkpatrick,W. (2013, July). The Kirkpatrick model. Retrieved from

http://www.kirkpatrickpartners.com/OurPhilosophy/TheKirkpatrickModel/tabid/302/Def

ault.aspx

51

Lehal, R. (2004). Research design in relation to study of organizational climate, job satisfaction

and managerial effectiveness. Research Methodology in Management Theory and Cases,

New Delhi, India: Deep and Deep Publishers.

Lewin, K. (1951). Force theory in social service (1st ed.). New York, NY: Harper and Row.

Lin, Y.Y., & Chiu, H.Y. (1997). The evaluation of training and development. Journal of Human

Resources Development, 7, 67-83.

Lok, P., & Crawford, J. (2004). The effect of organizational culture and leadership style on job

satisfaction and organizational commitment: A crow-national comparison. The Journal of

Management, 23(3/4), 321-339.

Nolan, S. (2007). A climate of success creating the right organizational climate for high

performance. Strategic HR Review, 6 (6), 36.

Parker, C.P., Boris, B., & Young, S.A. (2003). Relationships between psychological climate

perceptions and work outcomes. Journal of Organizational Behavior, 24(4), 389-416.

Rajeev, P., Madan, M.S., & Jayarajan, K. (2009). Revisting kirkpatrick's model - An evaluation

of an academic training course. Research Communications, 96(2), 272-276.

Rice, C. (1982). Your team of tigers, getting good people and keeping them. New York, NY:

American Management Association.

Schein, E. (1985). Organizational culture and leadership (1st ed.). San Franscico, CA: Josey-

Bass Publishers.

Schein, E. (2006). Organizational culture and leadership (3rd ed.). San Franscico, CA: Josey-

Bass Publishers.

52

Smidt. A., Baladin, S., Sigafoos, J., & Reed. V. (2009). The Kirkpatrick model: A useful tool for

evaluating training outcomes. Journal of Intellectual & Development Disability, 34(3),

266-274.

Trevino, L.K. & Nelson, K.A. (1999) Managing business ethics: Straight talk about how to do it

right (2nd ed.). New York, NY: John Wiley & sons, Inc.

Yazici, H.J. (2011). Significance of organizational culture in perceived project and business

performance. Engineering Management Journal, 23(2), 20-29.

Zhang, J., & Liu, Y. (2010). Organizational climate and its effects on organizational variables:

An empirical study. International Journal of Psychological Studies, 2(2), 189-201.

53

Appendix A: Invitation to Conduct Study

Company XYZ 2725 Mall Drive Eau Claire, Wl 54701

Jill Awe University of Wisconsin. Stout 712 South Broadway Street Menomonie, WI 5475 1

Dear Jill,

This letter is to provide you with the required letter of interest from Company XYZ, a hcalthcare insurance organization. In an effmt to keep the culture program moving forward at our site, a group of six leaders in partnership with Human Capital developed a four hour cultme training session. Transforming our culture will not happen overnight but by providing employees with the training and tools around our values, concepts and provide ideas of how to keep the culture moving in our site, we believe we can make a difference. Company XYZ is appreciative of your interest in evaluating our organizational culture training program to ensure it is meeting the needs of our employees and proves to be et1~cti.ve.

Your goal is to create two surveys. one which will evaluate the reactions of the employees to the culture training. and the second one will be to measure what the participant leamed as a result of the training. We are grateful for your interest in us and we are excited to learn the results of the stucly. The infonnation compiled by you \viii deterrnine if' this culture training is effective. and if so, it will confirm the value in continuing the program.

Thank you for taking this study on and this letter serves as your consent to proceed vvith this project.

i·s??:l~~ Eric McBride Executive Director

54

Appendix B: Consent to Participate in UW-Stout Approved Research

Title: Evaluation of Organizational Culture Training in a Health Insurance Organization

Research Sponsor: Dr. Jeanette Kersten, EdD 715-232-5229 [email protected]

Investigator: Jill Awe 715- 559-4050 [email protected]

Description: This study is an important project designed to evaluate the effectiveness of an organizational culture training program at UnitedHealthcare. This study is being used for a thesis project in the Masters of Training and Development program at the University of Wisconsin Stout. You will be asked to complete a brief survey at the conclusion of training that will collect data on your opinions and views on the structure of the training, the trainer’s effectiveness, and environment of the training. Risks and Benefits: Participation in the study comes with minimal risks. You may find it distressful to provide your opinions and views related to the organizational culture training. If at any time you feel discomfort discontinue the survey. There are direct benefits to your participation in this study. Your participation will assist the organization in determining the effectiveness and value in continuing the organizational culture training. Your opinions may also result in opportunities to enhance the existing training program. Special Population There are no special populations in this study. Time Commitment and Payment: The survey will require minimal time commitment and is expected to take 15 minutes or less to complete. There is no monetary payment that will provided to you for you participation in this study. Confidentiality: Your name will not be included in any documentation or reporting that is created from this study. It is assumed that there will be no connection between survey results and participants. The results of the survey will only be disclosed to the researcher. Your information will be kept completely confidential and stored in a secure location for 7 years before being destroyed. Right to Withdraw: Participation in this study is entirely voluntary. You may choose not to participate without any adverse consequences to you. You have the right to stop the survey at any time. However, should you choose to participate and later wish to withdraw from the study, there is no way to identify your anonymous document after it has been turned into the investigator.

55

IRB Approval: This study has been reviewed and approved by The University of Wisconsin-Stout's Institutional Review Board (IRB). The IRB has determined that this study meets the ethical obligations required by federal law and University policies. If you have questions or concerns regarding this study please contact the Investigator or Advisor. If you have any questions, concerns, or reports regarding your rights as a research subject, please contact the IRB Administrator. Investigator: Jill Awe, Graduate Student E9810 327th Ave Elk Mound, WI 54739 715.559.4050 [email protected]

IRB Administrator Sue Foxwell, Research Services 152 Vocational Rehabilitation Bldg. UW-Stout Menomonie, WI 54751 715.232.2477 [email protected]

Advisor: Dr. Jeanette Kersten, EdD, SPHR Assistant Professor Chair-People, Process, Culture College of Management 302b Jarvis Science Wing University of Wisconsin, Stout Menomonie, WI 54751-0790 715.232.5229 [email protected] Statement of Consent: By completing the following survey you agree to participate in the project entitled “Evaluation of Organizational Culture Training in a Health Insurance Organization”. _______________________________________________________

56

Appendix C: Organizational Culture Training Pre-Test

The following is a pre-test that was distributed to twelve participants prior to them

attending the modified organizational culture training. The purpose of this test is to measure

what the participant knew about the company’s core values and organizational culture concepts

prior to training.

Q1. What are the company's five core values?

1. 2. 3. 4. 5.

Q2. How do you apply the core values in your daily job function? Q3 .What is the natural midpoint on the organizational culture mood elevator? Q4. What does the term "blue chip" mean? Q5. Explain the concept of "Be Here Now" Q6. Explain what the accountability ladder?

57

Appendix D: Organizational Culture Training Post-Test

The following is a post-test that was distributed to twelve participants after attending the

modified organizational culture training. The purpose of this test was to evaluate what the

participant learned as a result of the modified organizational culture training.

Q1. What are the company's five core values?

1. 2. 3. 4. 5.

Q2. How do you apply the core values in your daily job function? Q3 .What is the natural midpoint on the organizational culture mood elevator? Q4. What does the term "blue chip" mean? Q5. Explain the concept of "Be Here Now". Q6. Explain what the accountability ladder.

58

Appendix E: Approval to use Kirkpatrick Model

From: [email protected] [mailto:[email protected]] On Behalf Of Wendy Kirkpatrick Sent: Tuesday, September 17, 2013 8:36 PM To: Awe, Jill D Cc: Linda Hawk Subject: Re: The Kirkpatrick Model - Approval to Use Dear Jill, We are so pleased to hear that you are planning to use the Kirkpatrick Model for your project. There is no problem to use it; we appreciate it if you attribute it to us in your references and/or footnotes. Best regards, Wendy

On Tue, Sep 17, 2013 at 9:17 PM, Awe, Jill D <[email protected]> wrote:

Dear Kirkpatrick Partners,

I’m a graduate student at the University of Wisconsin-Stout in the Training and Human Resource Development program entering into my final semester. As part of my program, we complete a Plan B project which consists of putting into practice what we’ve learned in the program. I have chosen to do an evaluation of a training session using the Kirkpatrick Model. I’m writing to gain permission to use this model for my project.

Please let me know what steps I need to do to gain that approval.

Thank you,

Jill D. Awe

mailto:[email protected]

59

Appendix F: Email Invitation to take the Survey TO: Organizational Culture Training Participants FROM: Jill Awe – University of Wisconsin, Stout RE: Evaluation of Culture Training DATE: October 23, 2013 This email is your invitation to take the Evaluation of Culture Training survey. I’m a graduate student at the University of Wisconsin-Stout and have been asked by your organization to measure your satisfaction of the culture training program. Your participation is greatly appreciated. Please click on the survey link below to get started: https://uwstout.qualtrics.com/SE/?SID=SV_a3k22yylMmbDue1 The survey will only take 10-15 minutes to complete, please answer all 11 questions which must be completed in one sitting. I would greatly appreciate your participation by Wednesday, October 30th, 2011. Thank you in advance for providing me your reactions to the culture training. Your responses are important and the results of the survey will provide information to measure effectiveness of the program. Sincerely, Jill Awe University of Wisconsin, Stout

60

Appendix G: Level One Reaction Survey

The following reaction sheet was sent electronically to trainees once they completed the

modified organizational culture course. The purpose of the reaction survey is to evaluate the course,

content, environment, instructor, and the relevancy to the job.

In order to determine the effectiveness of the program in meeting your needs and interests, we need your input. Please give us your reactions, and make any comments or suggestion that will help us to serve you.

1 Strongly Disagree

2 Disagree

3 Agree

4 Strongly

Agree 1. The material covered in the program was

relevant to my job.

2. The material was presented in an interesting way.

3. The instructor was an effective communicator

4. The instructor was well prepared.

5. The audiovisual aids were effective.

6. The handouts will be of help to me.

7. I will be able to apply much of the material to my job.

8. The facilities were suitable.

9. The schedule was suitable.

10. There was a good balance between presentation and group involvement

11. I feel that the workshop will help me do my job better.

12. Please provide your suggestions for improving the course. Kirkpatrick, D., & Kirkpatrick, J. (2007). Implementing the four levels. (1st ed., p. 28). San Francisco, CA: Berrett-Koehler Publishers, Inc.

61

Appendix H: Email Reminder to take the Evaluation of Culture Training Survey TO: Organizational Culture Training Participants FROM: Jill Awe – University of Wisconsin, Stout RE: Evaluation of Culture Training DATE: October 28, 2013 Last week you should have received an email inviting you to take the Evaluation of Culture Training survey. This is a reminder to complete that survey. Your participation is greatly appreciated. Please click on the survey link below to get started: https://uwstout.qualtrics.com/SE/?SID=SV_a3k22yylMmbDue1 The survey will only take 10-15 minutes to complete and must be completed in one sitting. I would greatly appreciate your participation by October 30th, 2013. Thank you in advance for providing me your reactions to the culture training. Your responses are important and the results of the survey will provide information to measure effectiveness of the program. Sincerely, Jill Awe University of Wisconsin, Stout

62

Appendix I: Email Thank You for taking the Evaluation of Culture Training Survey

TO: Organizational Culture Training Participants FROM: Jill Awe – University of Wisconsin, Stout RE: Evaluation of Culture Training DATE: October 28, 2013 Thank you in advance for providing me your reactions to the culture training. Your responses are important and the results of the survey will provide information to measure effectiveness of the program. Sincerely, Jill Awe University of Wisconsin, Stout

author: awe, jill d. evaluation of organizational …offsite training session called...

Documents