process evaluation of a diversity training program

(This is a sample cover image for this issue. The actual cover is not yet available at this time.)

This article appeared in a journal published by Elsevier. The attachedcopy is furnished to the author for internal non-commercial researchand education use, including for instruction at the authors institution

and sharing with colleagues.

Other uses, including reproduction and distribution, or selling orlicensing copies, or posting to personal, institutional or third party

websites are prohibited.

In most cases authors are permitted to post their version of thearticle (e.g. in Word or Tex form) to their personal website orinstitutional repository. Authors requiring further information

regarding Elsevier’s archiving and manuscript policies areencouraged to visit:

http://www.elsevier.com/copyright

http://www.elsevier.com/copyright

Author's personal copy

Process evaluation of a diversity training program: The value of a mixedmethod strategy

Halime Celik a,b,*, Tineke A. Abma c, Ineke Klinge a, Guy A.M. Widdershoven c

a Dept. of Health, Ethics and Society, Faculty of Health Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlandsb Simons & Partners Advocaten, Gulpen, The Netherlandsc VU Medical Center, EMGO Institute, Department of Medical Ethics, Amsterdam, The Netherlands

1. Introduction

In healthcare delivery, patients’ experiences and needs areoften reduced to a set of signs and symptoms within a biomedicalframework (Mead & Bower, 2000). However, patients’ symptomsare not purely biomedical (Bird & Rieker, 1999; Hoffman, 2000).Patients’ health and the way in which symptoms are presented areinfluenced by dimensions of diversity such as sex, gender, ethnicorigin, and socioeconomic status (Crenshaw, 1994; Street, 2002).These dimensions influence patients’ healthcare needs. Recenttheoretical developments have addressed the problem of how totake account of these dimensions and their interplay in theprovision of healthcare (Bekker, 2003; Moser, 2006; Pinn, 2003;Van Mens-Verhulst, 2001; Wekker, 2001). Diversity and theprovision of holistic care for patients has also been a special focusof the nursing profession for a long time (Leininger, 2001, 1995;Mahoney & Engebretson, 2000).

In healthcare practices, however, the relevance of dimensions ofdifference is just emerging (Celik & Klinge, 2005). There is, forinstance, not much focus on the fact that a leg fracture is experienced

differently by an athlete than by an office worker. A leg fracture willalso lead to different challenges for women with caring responsibil-ities for children than for single men. Also the necessary medicaltreatment can differ among these patients. Some will prefer rest,while others, who do not have the opportunity to take sufficient rest,will choose pain relief through medication and/or surgery, with itsconsequences. The education of healthcare workers reflects the lackof attention to diversity. Medical students are hardly trained in theconcept of diversity. Studies in the Netherlands, for instance, showserious deficiencies of sex and gender issues in Dutch education forhealth professionals (Verdonk, Mans, & Lagro-Janssen, 2006). Also, itis reported that medical students tend to copy the diversity neutralbehaviour of their teachers and mentors, who are mostly masculineRisberg, Hamberg, & Johansson, 2003. Essed (2005) even speaks ofcultural cloning; only those physicians who copy the behaviour ofthe rest are accepted. Sameness and blankness function as a mode ofbonding, excluding others.

Diversity appropriate care is care in which professionals paydeliberate attention to the special needs of an individual patientfollowing from the interplay between sex, gender, ethnic originand SES and other dimensions of difference. To enhance awarenessof the importance of diversity sensitivity in the provision ofhealthcare, health professionals should first be aware of therelation between dimensions of diversity and patients’ health andhealth demands. According to Pathman, Konrad, Freed, Freeman,and Koch (1996), awareness is the first criterion for change. To

Evaluation and Program Planning 35 (2011) 54–65

A R T I C L E I N F O

Article history:

Received 26 April 2010

Received in revised form 30 June 2011

Accepted 1 July 2011

Keywords:

Gender

Diversity

Implementation

Training

Process evaluation

Mixed-methods

A B S T R A C T

Patients’ health and health needs are influenced by categories of difference like sex, gender, ethnic origin

and socioeconomic status (SES). To enhance awareness of this diversity among patients and to provide

holistic care for them, health professionals should first be aware of the relation between dimensions of

diversity and patients’ health and health demands. This paper presents a formative process evaluation of a

diversity sensitivity training programme for healthcare professionals. The training was implemented in

three healthcare settings (mental healthcare, nursing home and hospital care). Mixed methods were used

to monitor the implementation of the training and its effects after three years. Findings demonstrate that

the training stimulated participants’ awareness, knowledge and critical attitudes towards diversity. Their

motivation and willingness to take action regarding diversity was also enhanced. Yet these developments

were less apparent among nursing home participants who felt less satisfied and did not develop a critical

perspective on this issue. Qualitative data were helpful to explain differences between the settings. By

means of the combination of quantitative and qualitative data, we can conclude that individual learning

was not enough to guarantee a sensitive approach to diversity at the organizational level.

� 2011 Elsevier Ltd. All rights reserved.

* Corresponding author at: Dept. of Health, Ethics and Society, Faculty of Health

Medicine and Life Sciences, Maastricht University, P.O. Box 616, 6200 MD

Maastricht, The Netherlands. Tel.: +31 43 388 24 58; fax: +31 43 388 41 72.

E-mail addresses: [email protected], [email protected]

(H. Celik).

Contents lists available at ScienceDirect

Evaluation and Program Planning

jo ur n al ho m ep ag e: www .e ls evier . c om / lo cat e/eva lp r og p lan

0149-7189/$ – see front matter � 2011 Elsevier Ltd. All rights reserved.

doi:10.1016/j.evalprogplan.2011.07.001


result in actions, awareness should be followed by acceptance,adoption and adherence. Therefore the issue of diversity should bein the scope of practice and an integral part of the repertoire ofprofessionals. It is also important to attend to the professionals’enthusiasm for (or resistance to) change as well as barriers in theinstitutional context (Kotter & Cohen, 2002). Awareness aboutdiversity in healthcare practices can be stimulated by providinghealth professionals with diversity awareness training (Celik,Lagro-Janssen, Klinge, van der Weijden, & Widdershoven, 2009;Verdonk, Benschop, De Haes, & Lagro-Janssen, 2008).

Our aim in this paper is to investigate the extent to whichdiversity awareness training can promote diversity sensitivityamong health professionals. In this article we present the results ofa process evaluation conducted in three healthcare settings(mental healthcare, hospital care, and nursing home care) in theNetherlands. After a brief description of the intervention, wedescribe the process evaluation and the results. In the discussionwe reflect on the limitations of our design and the trainingprogramme. Finally, the lessons learned in the evaluation of thetraining are discussed, followed by a conclusion.

2. Intervention: a diversity training for healthcareprofessionals

At the time when the need for a diversity training programmeemerged, a search indicated that there were no training modulesavailable in The Netherlands. Therefore in cooperation withMOVISIE, the Dutch Center for Social Development, a blueprintfor a diversity awareness training scheme was developed (Celik &van Oosten, 2004). The training was based on an approach todiversity which regards the various categories not as isolated butinterconnected (see Fig. 1).

To fine tune the training to each of the three healthcare settingsand to create favourable social conditions, informal talks withmanagers of the healthcare settings facilities were carried out. Thetraining materials consisted of a manual for tutors (Celik and vanOosten, 2005a) and a manual for participants with a reader (Celikand van Oosten, 2005b). The manuals consisted of four modules withrelevant theoretical and practical information. The training wasprovided in four sessions of 4 hours by two trainees: Nico van Oosten(MOVISIE), an experienced trainer, and the first author of this article.

2.1. Participants of the training

For the selection of the three healthcare practices (mentalhealth, hospital, nursing home) the following criteria were used:(1) the willingness to take diversity into account (motivation); (2) aminimum of eight and a maximum of twelve participating healthprofessionals; (3) no current mergers or intentions to merge thehealthcare practice; and (4) willingness to financially support thetraining (in terms of time for following the training during officehours). The training participants from the healthcare practiceswere selected by the programme manager of the healthcarepractice. In total 31 professionals took part in the training: 8participants from mental health practice; 12 from hospitalpractice; and 11 from nursing home practice. In the mentalhospital the participants were composed of nurses (n = 2), threestaff persons (n = 3), a psychiatrist, a psychologist and a socialworker. In the hospital participants included nurses (n = 6),paramedics (n = 3), managers (n = 2) and a senior secretary. Inthe nursing home several nurses (n = 4), nurse aids (n = 3), amanager and a secretary participated.

Personal characteristics of participants: All participants had aDutch background, except for one psychiatrist in the mentalhospital, who was born in Suriname. The age of the participantsvaried between 25 and 60. Almost 60% of the participants werefemale. Almost all participants had followed a higher education,and some had an academic degree.

Overall one participant dropped out. The reason for drop-outstayed unclear. Four participants (2 from mental healthcare, 1 fromthe hospital, and 1 from the nursing home) missed 1 training module.From 2 participants we did not get complete pre- and post-test data.

2.2. Training content

During the four training modules, participants carried out severalexercises individually, in subgroups and plenary (see Table 1).

The first module began with an exercise to introduce theconcept of diversity. Participants finished the sentence: ‘‘To me,diversity means..’’. The theoretical background of diversity wasdiscussed, its dimensions and its relevance for the institution(individual and organizational level) as well as the possible pitfallsof not considering diversity (Health Canada, 2000). This wasfollowed by the method of ‘Asking the other question’, focussing onother than biomedical questions, in order to raise awareness of allrelevant aspects of diversity (see Box 1). In the exercise ‘Imaginebeing a patient’ participants were asked to look at their

* SES means the Soci o economic state of a patie nt

Gender

Religion

Age

Other dimensi on

SES*

Sexuality

Disabi lit y

Cultu re

Fig. 1. Diversity matrix. *SES means the Socio economic state of a patient.

Box 1. Example of group activity in mental health practice:

‘Asking the other question’

The case of a patient using mental healthcare.

P. has a history of psychiatric illness. After the death of P.’s

partner, P. is admitted several times to a general hospital. The

diagnosis is: depression with a psychotic character. P. also has

a serious form of Parkinson’s disease. Since 1998 P. has been

living at the current address, but unfortunately it is not going

well. Despite good intentions, P. disturbs the neighbours a

great deal, especially due to the habit of cleaning the house in

the middle of the night. Due to Parkinson’s P. falls often. P.

sustained a wound from a recent fall which is not healing well

due to neglect.

Participants responded to the following questions: Which

dimension of diversity is dominant? Are there other important

dimensions? How are these related? And are some dimensions

underused? Participants’ discussed whether the person in the

case was male or female, whether gender mattered, how old

the person was and why these things were not usually sys-

tematically accounted for in their work.

H. Celik et al. / Evaluation and Program Planning 35 (2011) 54–65 55


organization (interior, entrance of the building et cetera) throughthe eyes of patients. Additionally, participants critically reviewed avideo fragment from the Dutch Heart Foundation about the impactof diversity on cardiovascular diseases.

In the second module participants brainstormed about howsensitive they themselves were to diversity. They elaborated on thequestion: What could be better in my practice and for the patient?Furthermore, implementation theories were provided, and thepros and cons of changing routines and attitudes were discussed(Grol, Wensing, & Eccles, 2005). In addition, the formal andinformal managerial support required for innovation was outlined.The strengths and weaknesses of a recent care innovation in theparticipants’ ward were examined as well as possible opportu-nities for and barriers to change.

The third module started with a theoretical discussion aboutmainstreaming gender (Rees, 2001). Mainstreaming, in thiscontext, is a strategy to integrate diversity related experiences,knowledge, and interests into the agenda at all levels of anhealthcare institution (Council of Europe, 1998). A HumanResource Department might, for example, adopt a diversity policyto work towards a more mixed composition of staff. Strengths ofthe setting were articulated, such as politeness, empathy and teamspirit, and their usefulness for practice improvement with regardto diversity sensitive care. Next, participants assessed the gaps(weaknesses) regarding diversity in their own setting andformulated short and long-term targets for improvement.

In the last module participants evaluated policy documents,care protocols and guidelines used in their practice, by lookingthrough ‘Diversity glasses’. Participants were made conscious ofthe possible influence of diversity and evaluated these documentsfrom a non-neutral perspective. Furthermore, the participantsexamined the pros and cons of these documents from a diversityperspective. Finally, the participants prepared a work plan foraction using knowledge gained as a result of the training.

The training programme was based on the Deming cycle forquality (of care) improvement. The original Deming cycle (Langley,Nolan, Nolan, Norman, & Provost, 1996) consists of four phases. For

our purposes, we modified the cycle, expanding it from four to sixphases (see Fig. 2). The first extra phase is the no attention phase:there is no attention to diversity, either because health profes-sionals do not recognize diversity factors or because they thinkthese issues are irrelevant for their practice. The second extraphase is the (some) attention phase: participants have (some)awareness of diversity, however they do not put it into practice.These two phases were followed by the well-known phases ofplanning, doing, studying and acting. Planning is the intention topay attention to diversity in practice, doing is the application ofknowledge to practice, studying is systematically evaluating theresults, and acting is making desired modifications, accordingly.

3. Evaluation design and instruments

This study can be seen as a process evaluation to monitor theimplementation process of the diversity training. Process evalua-tion can be both formative and summative (Linnan & Steckler,2002). This study had a summative goal aimed at evaluating

Table 1Content of the training modules.

Module Aim Content

1. General

introduction

Conceptualization of diversity and achieving

individual engagement

Introduction

Warming-up exercise: what is diversity?

Interactive theoretic discourse about the matrix of diversity, and the levels of

diversity: individual, symbolic and institutional

Apply the ‘Asking the other question’ method to a patient case (see Box 1)

Exercise ‘Imagine being a patient’

Critically review a video consult

Evaluation of module 1

2. Diversity and the

professional

Identify the forces and advantages

of diversity for professionals

Association exercise: How diverse am I?

What could be better in my work?

What were the tensions for professionals in a recent innovation?

Instructions for the assignment How do politics work at the institutional level?


3. Diversity and the

institution

Knowing how to apply diversity in practice.

Analyzing the tensions for change.

Interactive theoretic discussion about mainstreaming gender

Short and long term aims and priorities

What are the strengths of this practice?

Exercise how to make the strengths useful for implementing diversity?

Instructions for the assignment ‘Diversity Glasses’


4. Work plan for

action

Making a work plan to stimulate actions.

The results of module 1–3 were used as input

Looking through ‘Diversity glasses’

Building a work plan for action


Evaluation of training program

Plan

(some) Awareness

Unaware ness

ActStu dy

Do

Fig. 2. Modified Deming cycle.

H. Celik et al. / Evaluation and Program Planning 35 (2011) 54–6556


whether our diversity training raised awareness and whether thetraining was implemented as planned. Our research designcombined qualitative and quantitative methods. Greene, Kreider,and Mayer (2005) argue that mixed-method approaches are morecomprehensive, as they include different aspects and perspectives,and therefore yield results which provide more insight and deeperunderstandings of an issue. A combination of various methods wasused to monitor the implementation process of the training as wellas to determine the extent to which the goals of the training hadbeen realized (enhanced awareness, knowledge, motivation andwillingness to take action in a diversity sensitive manner in thelong and short run, see Table 1). Quantitative methods included asurvey of participants’ diversity awareness and self-reportedknowledge (pre and post) and an assessment of the participants’satisfaction with a questionnaire. Qualitative methods includedparticipant observations of all training sessions, oral groupevaluations with participants during the training and semi-structured interviews with participants as well as managers.Details will be explained below.

3.1. Quantitative methods and instruments

The training was assessed through a survey for diversityawareness. Unfortunately at the time of the study there were novalidated surveys present that fitted our aim, and these are stillmissing. We therefore developed the survey ourselves. Relevantvariables to measure sensitivity to diversity were derived from theprogramme goals, which are grounded theoretically in theliterature on quality improvement processes and the Demingcycle (Plan, Do, Study, Act). As pointed out the programme theoryalso included the notion of awareness as the first step towardssensitivity. Subsequently these variables (awareness; knowledge;motivation and willingness to take actions) have been operatio-nalized in terms of survey questions.

A quantitative pre-test–post-test design with closed questionswas used to evaluate the effects of the training on participants’diversity awareness and knowledge (see Table 2a). We expectedthat the interpretation of these quantitative results would bedifficult, since outcomes might be the result of combined effects of

Table 2bQualitative instruments.

Instrument What Content When

Interviews N = 9 Semi-structured interviews

with nurses and

(nurse) managers

Interview topics: How respondents characterize their

institutions and their treatment plans; behaviours and decisions;

diversity orientation

Before the training

Participative

observation

N = 12 (3*4)

Observation participants

during training

Direct observation During all 12 sessions of the training

Oral (Process)

Evaluation

N = 12

Module 1–4

and the training

Strengths and limitations of the modules and the training and what

could be better.

After every module

Follow-up N = 3 Open interviews with

program coordinators

Interview topics: interpretation of the quantitative results of the training,

long term experiences and initiatives, suggestions for improving the

training and its effects for future purposes.

3 years after the training

Table 2aQuantitative instruments.

Instrument What Content When

Pre-post survey for

diversity awareness

(N = 31)

General questions

regarding demographic data

Name Before and after

training program

Sex

Educational background

Job function

Diversity sensitive

statements

1. In general, if attention is paid to gender, health supply will be better

accommodated to specific health problems; 2. Striving for (cultural)

diversity among employees has a surplus value when the patient population

is diverse as well; 3. It is not relevant to consider diversity issues in policy

(documents); 4. To appoint a diversity expert as staff member or coordinator

is superfluous; 5. The diversity (gender, cultural background, age and education)

in a patient can not significantly effect patients specific health demand; 6. In the

current scope of health care organization diversity should be calculated; 7. Specific

knowledge regarding diversity issues among health professionals is not necessary to

understand patients’ health demand; 8.For attention to diversity in health care,

the availability of financial support is important

Before and after

training program

Participant opinion

about diversity in their

healthcare practice

Deming PDCA cycle: No attention phase: Some attention phase: Planning:

Doing: Acting: is as a result of the evaluation

Before and after

training program

Possible answers: ‘could be much better’, ‘could be better’, ‘good’, or ‘excellent’.

Self assessment of the

knowledge

Determine/assess your knowledge for after the training (on a scale 1–10). After the programme

Participants self

reported attitude for

diversity

Deming PDCA cycle Before and after

training program

Participants read the

literature in the reader?

Read the literature in reader: YES/NO After the programme

Questionnaire for

Satisfaction (N = 31)

Program Satisfaction Were participants’ course objectives met? Were individual lectures and workshops

presented effectively? Did the training provide new insights? Did the training provide

practical and applicable knowledge? Were methods of the training adequate?

After the training



factors that are not directly related to the training such as shortageof staff, work pressure, changes in the patient population andorganizational dynamics. With this in mind, the data werecombined with qualitative results in order to elaborate on,enhance or illustrate the quantitative results (Greene, Caracelli,& Graham, 1989).

The satisfaction of the participants was assessed by aquestionnaire filled out at the conclusion of the training (Table2a). The participants were asked to evaluate the training by givinggrades from 1 to 10 and to provide an explanation for the grade interms of the extent to which the programme fulfilled theirexpectations and whether their course objectives were met.Furthermore, the participants were asked to indicate whether theprogramme content was presented effectively and whether thetraining provided new insights. They also evaluated the practicalapplicability of their knowledge and the training method. Finallywe measured to what extent the participants read the reader withstudy material.

3.2. Qualitative methods and instruments

Nine semi-structured interviews (three in each setting) wereheld before the training to gain an understanding of theparticipants’ expectations of the training and to get an idea oftheir prior experiences (See Table 2b). The topic list for theseinterviews with nurses and (nurse) managers were developed onthe basis of existing literature on diversity and the implementationof innovations in healthcare from recent research (Celik, Abma,Widdershoven, van Wijmen, & Klinge, 2008). Topics includedrespondents’ characterizations of their institutions and treatmentplans; behaviours and decisions of respondents regarding diversi-ty; factors that enable or obstruct attention paid to diversity; anddiversity orientation. We asked the programme coordinator toselect a variety of respondents to gain a broad range of experiencesand meanings. The interviews, which lasted between 40 and60 min, were tape-recorded and transcribed verbatim. Field noteswere taken of contextual information, impressions during inter-views and from the informal talks after the interviews (Miles &Huberman, 1994).

Participant observations were made during 12 sessions of thetraining programme and during informal conversations withparticipants and managers after the sessions. During all modulesobservation notes were recorded by one of the trainers, designatedas the observer, while the other was teaching the module. The rolesof the trainers (observer and moderator) alternated for practicalreasons. Notes of the informal conversations were made inretrospect by the first author. The notes included interpretativecomments on the intentions or attitudes which might underlie thevarious observed behaviours. There are two major limitations tomaking observational notes. First, two observers will see, hear andrecord different things (selectivity). This weakness was limited bythe trainer and the observer spending time discussing the noteswith each other. The second drawback to this method is if anobserver has prior information, observer bias can occur. Thispotential bias can be detected when two persons record whathappened. In our case, a discussion between the trainer asmoderator and observer helped to illuminate differences in theobservation of a session. For the most part, the observations werecomplementary. The advantage of making observational notes wasthat we also gained extra information from observing theunobtrusive actions and interventions, which are signs thatindicate a kind of behaviour (pattern), while it was not explicitlymeant to do so (Lincoln & Guba, 1985).

At the end of the session every module was orally evaluated by adiscussion about the value of the module as perceived by theparticipants and its impact on them in terms of what they had

learned. At the end of Module 1 we asked participants to imaginethat they encounter a colleague later that day, who has notparticipated in the training. The colleague is in a hurry, but asksyour opinion about the training. What would you say (in short) tothis colleague? After Module 2 participants gave a score to thesession using a standard 0–100 ‘thermometer’ scale, after whichwe discussed participants’ experiences and feelings. Module 3 wasevaluated by an open discussion started by asking the participants:How do you feel about this session? Directly after completing thewhole programme, an open group discussion was held in order toevaluate the training. The discussion started with the followingquestion: To what extent do you feel that the objectives of thistraining have been fulfilled? We ended this group discussion withthe following question: Do you have suggestions for improving thetraining for future purposes? Notes were taken of all oralevaluations by both the trainer and observer, and discussedafterwards.

To gain insight into the long-term results we assessedexperiences three years after the last session, by means of threeinterviews with the same programme managers/coordinators ofthe healthcare practices. These open interviews were guided by thefollowing topics: interpretation of the quantitative results of thetraining; long term experiences and initiatives; and suggestions forimproving future training for maximal effect. These interviewswere tape-recorded and completely transcribed.

3.3. Analysis and integration of data

Statistical comparisons of the changes in the self-reportedawareness and knowledge of diversity as measured by the surveywere made using the Wilcoxon signed rank test for two relatednon-parametric samples. The change has been reported assignificant if p < 0.05. The statements component of the surveywas rated with ‘strong agreement’, ‘agreement’, ‘disagreement’ or‘strong disagreement’. For each question the answer representingthe highest awareness was coded with 4 and the answerrepresenting the lowest awareness with 1. Participants’ opinionsregarding diversity dimensions were coded with ‘could be muchbetter’; ‘could be better’; ‘good’; and ‘excellent’. The statementswith a pre-post comparison were analyzed using a t-test for pairedsamples. The change was found to be significant if p < 0.05. Allstatistical data were processed using SPSS. The pre- and post-testof the participants and the degree of satisfaction with the diversitytraining were scored by giving grades from minimum 1 tomaximum 10 to the evaluation questions. Average scores werecompared.

The analysis of the qualitative interviews was done by the firstauthor, and the second author gave feedback on the analyses interms of questions regarding the credibility/robustness of thecategories/themes and the link between the raw data and theinterpretations. After reading the field notes, to get a generalimpression of the data, the interview contents were characterizedand classified. The preordained topics served as a framework forthe content analysis (Hsieh & Shannon, 2005). After analysingseven interviews, conceptual saturation occurred. The remaininginterviews and field notes were used to confirm the analyses andconcept definitions.

The qualitative and quantitative data analyses were not justcomplementary. Greene’s (2007) mixed method approach goesbeyond accumulation of the different facets or dimensions of thesame complex phenomenon. We were gathering data from amultiplicity of sources (triangulation) and integrated these toilluminate the complexity of the diversity training and to get amore insightful and comprehensive ‘picture’ of the phenomenonunder study. We performed ‘crossover track’ analysis (Greene,Benjamin, & Goodyear, 2001), which means that the results from



one method are clustered, summarized, or transformed andintegrated with the other method. Using qualitative data next tothe survey we aimed at complementarity of data interpretationand development (as described by Greene, 2008), meaning thatevidence from one data set was not only corroborated by data inanother dataset, but also enriched and supplemented with moredetail and other information. For example, developing a criticalattitude was not a variable in the survey, but turned up as animportant variable from the qualitative data set.

4. Results

The modules were provided as planned in the mental healthinstitution and hospital. In the nursing home didactic adjustmentswere needed, because these participants had difficulty with thediversity concept, the training programme content and theacademic language used by the trainers. In all settings thebackground of participating health professionals and managersvaried. However, the educational level of the nursing homeparticipants was lower (intermediate vocational education) thanthat of the other mental health and hospital practice participants(academic and higher vocational education). Another differencerelated to the fact that the mental healthcare setting had alreadybeen exposed to the concept of gender, and taken actions toimplement gender sensitivity within their institution. In thissection the findings will be presented. The structure of this sectionis based on the main themes relating to the improvement ofdiversity sensitivity among professionals in the three settings.

4.1. Awareness and knowledge

Awareness and knowledge are required to recognize therelevance of diversity for one’s practice and to develop plans forimplementing diversity sensitive care. Diversity awareness wasassessed by a survey before (baseline) and after the training(outcome). Awareness improved significantly for the mental health(p = .026) and for the hospital (p < 0.005), which suggests that thetraining was effective for this outcome. In the nursing homepractice there were no significant changes (p = .749). Self reportedknowledge concerning diversity also increased as a result of thetraining. Participants from the mental health institution rated theirown knowledge on average as 6.3 (SD: 1.71) on a scale of 1–10before the training and on average as 8.1 (SD: .86) after the training(see Table 3). The reported knowledge of the participants at thehospital showed an even greater increase from 4.1 (SD: 1.30) to 7.4(SD: .70). The increase in self reported knowledge was the lowestamong nursing home participants: from average 5.9 (.99) to 7.3(.67), in relative terms.

We provided all participants with a document containinggeneral information on diversity as well as specific diversityliterature related to their field. This reader was read by 63%, 58%and 36% of the participants in the mental health practice, hospitalpractice and nursing home practice, respectively.

Mental health participants were already familiar with thenotion of diversity before the training. That the highest selfreported knowledge at baseline was (see Table 3) for mental healthpractitioners (6.3 versus 4.1 among hospital and 5.9 among

nursing home participants) can be understood in the light of thegender specific educational programmes implemented in thissector some time before our diversity training. Although partici-pants stated that some exercises were too simple given theirforeknowledge, they reported that the training helped them tofurther acquire know-how regarding diversity; they began to seethat gender was only one dimension of diversity, and that variousdimensions of diversity (age, ethnic background and SES)interrelate. As one of the participants aptly notes:

The client does not exist and is more than a diagnosis. (Mental

Health, Participants)

Participants mentioned that they became more critical about aneutral approach to diversity; and that they felt empowered toquestion organizational routines, as a result of the training. Inaddition, they started to realize that diversity sensitivity should bepart of the whole organization (mainstreaming) and not just adimension of the professional–client interaction.

I have learned that I need to continue to ask questions, at allorganizational levels, about what is obvious. (Mental Health,Evaluation).

The baseline situation at the hospital was quite different.Participants reported that they had no background knowledge (selfreported knowledge score at baseline: 4.1) and mentioned thatdiversity was not considered to be relevant for care giving beforethe training. The following reflects this:

No, everyone gets the same treatment. (. . .) The care given whensomeone walks in is not dependent on diversity characteristics.(Hospital, Interview, nurse HT)

The self reported diversity knowledge and awareness scoreindicates that participants acknowledged the limits of theirexpertise. They thought they could learn something and feltmotivated to follow the training programme. During the trainingparticipants reported that they started to realize that diversity wasimportant in their work. They began to perceive patientsdifferently and to recognize various diversity dimensions andtheir interrelatedness. It also became clear to them that everypatient is unique and has specific health demands.

You have differences between a man and a woman, but alsoamong men you have many different men. What is important isto get an eye for these differences. We work with people. Youcan standardize work processes, but not their application to thepatient. (Hospital, Follow-up, FE)

The survey data suggest that the training in the hospital waseffective, and was the most effective of the three settings inincreasing the level of awareness and knowledge among partici-pants (see Table 3). This knowledge and awareness was alsosustained over a longer period, as will be pointed out later.

Nursing home participants also reported increasing theirawareness and knowledge concerning diversity, but to a lesserextent compared to the other settings. The qualitative data show

Table 3Self reported knowledge of the participants (N = 31) graded on a scale of 1–10 before (pre) and after (post) the training. You need Ns for each setting.

Mental Health practice N = 8 Hospital practice N = 12 Nursing Home practice N = 11

Pre Mean (SD) Post Mean (SD) Pre Mean (SD) Post Mean (SD) Pre Mean (SD) Post Mean (SD)

6.3 (1.71) 8.1 (.86)a 4.1 (1.30) 7.4 (.70)a 5.9 (.99) 7.3 (.67)a

a Knowledge improved after the program.



ambiguity. There are indications that diversity is reasonablyunderstood, as the following quote from a manager illustrates:

With regard to sex, we maintain a segregated policy for maleand female residents. Once in a while it happens that a room ismixed, but that is really sporadic. We try as much as possible totake different customs into consideration for people fromdifferent ethnic origins. With regard to age, we specialize in theelderly. With regard to handicapped residents, we stimulatethem to their potential. (. . .) With regard to socioeconomicstatus, we adjust our communication to the appropriate level.(Nursing home, Interview, CdO, 107)

Yet, during the last module a Babel-like ‘confusion of tongues’occurred as the complexity of the concept was also difficult tograsp, especially for those with lower education. Some even feltthat they were back to where they started. The confusion, they said,was the result of the fast pace and the high academic level of thetraining. The training could be improved, from their perspective, bytailoring it to the cognitive level of the participants. They alsosuggested giving the participants more time to assimilate thematerial.

To improve the training we need more leeway to understandthe concept of diversity and to make it less difficult. (Nursing

home practice, Participants, Process Evaluation)

The responses from nursing home participants generallyindicated that they did not feel the training process was completedsuccessfully since some of the participants had just started tounderstand the complexity of the notion of diversity.

4.2. Critical attitude

In this context, critical attitude is attained once healthprofessionals have the ability to see the blind spots of a neutralapproach to diversity. They will also be less likely to be neutralthemselves and conscious of the possible influence of diversity inhealthcare delivery. The initial opinion of the participants in allpractices was that the attention paid to diversity issues was good.After the training the opinion(s) about diversity among mentalhealth and hospital participants was that it could be better (seeTable 4). The participants indicate that more attention to thisconcept is desirable. The obtained qualitative data indicate thatmental health and hospital participants did become more aware ofthe shortcomings of their standard, neutral approach. Mentalhealth participants mentioned, for example, that the provision oftheir care is not adequately adapted to a culturally diversepopulation.

In the future, it is expected that the target group will change,particularly in that there will be more non-native Dutchpatients. Changes made within the institution should take thistarget group into account. (Mental Health, Participants,‘Imagine being a patient’)

And:

Right now healthcare professionals only concentrate on thedisease of patients, but they also should take diversity intoaccount. (Mental Health, participants, Short and long term aimsand priorities)

In the mental health setting this critical attitude towardsdiversity is still present several years after the training according tothe study follow-up. Programme coordinators promoting diversitysensitivity point out how difficult it is to implement the complexnotion of diversity, but they do not give up.

The difficulty (. . .) is that, on the one hand, gender issues arealready accepted by some who take these issues into accountand apply them without being noticed and, on the other hand,(. . .) others are much less focused on gender and, therefore, it ismuch less in their mind and approach. And if you add diversityfactors to this it only becomes more complex. (Mental Health,Follow-up, CA 37)

Hospital participants also became more critical during thetraining. They mentioned, for instance, that older patients mightfeel disoriented or on unfamiliar ground due to the poor signs(small lettering, not clearly placed, and not well recognizable) inthe organization.

The clinic is not in the clinic building and the signs are bad or ahindrance. The signs should be more visible, clear andrecognizable. (Hospital, participants, ‘Asking the other ques-tion’)

Participants noticed their own learning process, for examplewith regard to terms that had initially no meaning, but graduallybecame concrete.

In the beginning it was difficult to imagine what the termsgender and diversity meant. That’s different now. By lookingthrough the glasses of the patient, I clearly look with differenteyes at the information for and treatment of patients. (Hospital,Participants, Looking through ‘Diversity glasses’)

The follow-up interview with the hospital programme coordi-nator reveals that there still is a sense of urgency to improve thequality of care. She critically reflects on the actual situation, as thenext quote illustrates:

The stance is: the doctor knows what is good for the patient.(. . .). One looks in a particular way at the patient. (. . .) Do yousee the wound or the patient? (. . .) The point is that we need torecognize that things are different. (Hospital, Follow-up, FE)

While mental health and hospital participants see ways tofurther improve the quality of care taking into account diversity(‘could be better’), the nursing home participants appeared toremain constant in their opinion that enough attention is paid todiversity in their setting (‘good’). The development of a criticalattitude among mental health and hospital participants is fosteredby an enhanced awareness and knowledge in these settings; if oneunderstands what diversity means and why it is important, then itis easier to be critical of one’s practice.

4.3. Willingness to take actions

Willingness means a readiness among professionals to takeactions to promote diversity sensitive care giving. The pre- (before)and post- (after) training results of the participants’ attitude

Table 4Opinion participants (N = 31) regarding the attention for diversity before and after

the training.

Mental Health

practice N = 8

Hospital practice

N = 12

Nursing Home

practice N = 11

Pre Post Pre Post Pre Post

Good Could be better Good Could be better Good Good



towards diversity are shown in Table 5. The classifications arebased on the Deming cycle (see Fig. 2).

We found that mental health participants developed awillingness to put diversity into practice. At baseline, participantsindicated that they were motivated to pay attention to diversitycharacteristics in practice. They were in the plan-phase of theDeming cycle. After the training, the participants reported in theinterviews that they started with putting their intentions intopractice. On the basis of these data we decided the participantsprogressed to Deming’s do-phase. Such a development can also beseen among hospital participants, who progressed from the ‘notaware’ to the ‘plan’ phase. Nursing home participants did notreport a change in their willingness to take actions; they werealready quite willing to take action and remained in the ‘do’ phase.

During the training the mental health participants started todevelop a working plan, and intended to establish a diversity studygroup at their institution. This group planned to make a thoroughinventory of whether the treatments took the concepts of diversityinto consideration. Plans for diversity management included thefollowing innovation:

Diversity management is described as a process of creating andmaintaining an environment that enables every person to fullyactualize his or her potential. It is not an end in itself, but avehicle to improve results (Mental Health, Participants).

Realization of diversity management is only possible accordingto the study group if diversity factors of patients are known. Thisinformation was lacking and therefore the following innovationwas planned:

During the intake and procedures for transfer the forms shouldinclude explicit questions about diversity factors such as thepersonal characteristics and background of the patient. Alsothere should be central information with addresses of orga-nizations that can be consulted or that are appropriate forreferral based on diversity factors (Mental Health, Participants).

It turned out that the study group was established, butdisbanded after a year. One of the reasons was that the studyteam was not fully integrated into the management team whichweakens the link to potential policy changes. The person whoembraced diversity left and the new manager did not have thewillingness to invest in the diversity team.

The project Group which we had was discontinued (. . .) stoppedalso because it was not sufficiently imbedded in the manage-ment of the organization. It is disconcerting that in anorganization where there are three signatures from a manage-ment level supporting this group, that one person can stop itafter a change of just one staff member. (Mental Health, Follow-up, CA 104)

Despite this setback, since the team had the willingness to carryon with the integration of diversity in the institution, they madenew plans. The team currently organizes an annual symposium on

gender and also presents an annual gender award to the healthprofessional with special competencies for addressing genderissues in healthcare delivery. Additionally, they intended to makediversity part of the core values of their institution. Recently, theirBoard of Directors was contacted and it reacted positively to thegroup’s plans to integrate diversity at all levels.

In the hospital practice, participants were in the no awareness

phase at baseline. After the training, the participants had progressedto the plan phase. During the training, participants made aninventory of their own information material for patients. Theyfound that the current information for patients was not up tostandard when evaluated according to diversity sensitivity criteria,such as that the information should be assessable and understand-able for each and every patient. The information was not attuned tothe elderly and certainly not to non-Dutch speaking patients, forexample.

After a cursory inventory of our patient information pamphletsit became clear that we did not have information, of any kind,available in any language other than Dutch. This was also thecase for supplementary audio-visual materials. Illiterate andnon- (or poor) Dutch speakers/readers lack this sort ofinformation. (Hospital, Participants ‘Building a work plan’)

Another example of an innovation at the hospital:

What we are doing is sensitizing people, so that theyacknowledge that there are differences between patients (..).We can shout this from the top, but it will only land if there in anintrinsic motivation for change. Therefore we have organized aworking conference for 700 employees. (Hospital, Participants‘Building a work plan’)

After the last session of the training, several encouraginginitiatives in the hospital practice were taken such as mentioningthe subject in annual reports and organizing training programmes.These initiatives helped to trigger professionals continuously toconsider diversity in practice and to see the rewards for patientsand professionals.

A lot has happened since you left (. . .). We continued with thethought that the quality of care will be good if the quality of theapproach is good. (. . .) You see the patient and the differencesbetween patients, attributed to cultural background. For thatreason, we have organized work conferences for nearly 700 co-workers. (Hospital, Follow-up, FE)

These initiatives are grounded in a vision of quality care thatfinds support among managers at the top-level, as indicated in thefollow-up interview.

The participants in the nursing home practice reported thatthey were in the awareness phase, but as shown in 4.1 this could bedoubted from an outsider’s perspective. As participants did notfully comprehend the concept of diversity, they were initially notfully aware of their tunnel vision and blind spots. Later on, in thelatter stages of the training, their awareness had grown; theybegan to realize how complex the concept was, and howcomplicated it was to put it into practice. There was readinessto take action, yet difficulties were experienced in designing a workplan for action. As the following quote shows this is directly relatedto the complexity of the concept.

During the previous module we understood the concept ofdiversity, but now we feel we are back where we started. It isvery complex. (Nursing home, participants, Module 4 ‘Buildinga work plan for action’)

Table 5Participants attitude (N = 31) in Deming’s cycle (no/some attention-plan-do-study-

act) for diversity before (pre) and after (post) the training.

Mental Health

N = 8

Hospital N = 12 Nursing Home

N = 11

Pre Post Pre Post Pre Post

Plan Doa Not aware Plana Do Do

a Attitude improved after the training.



The follow-up indicates that in all settings the readiness to takeaction is impeded by organizational constraints, such as restrictedtime due to the heavy workload, bureaucracy, structural shortageof personnel and lack of support.

4.4. Satisfaction

When assessing participants’ satisfaction at the end of thetraining we found that participants were positive about thetraining (see Table 6). The level of satisfaction among nursinghome participants was, however, significantly lower than thatamong those in the mental health and hospital practice.

Some mental health participants said that their appreciation ofthe training was positively influenced by their previous knowledgeabout gender sensitivity in care giving. Others said that the traininghad an additional value, as diversity encompasses more than justgender.

I was already inspired by an earlier course on sex-specific caregiving. (. . .) I am especially satisfied due to the primacy ofdiversity over its dimensions such as gender. (Mental Health,Participants, Evaluation)

Hospital participants reported that they felt inspired by thetraining, as it was connected to social questions.

The gathering was interesting and inspiring. The subject isconnected to social issues. (Hospital, Participants, Evaluation)

The nursing home participants were less positive; they haddifficulty with the intensive training and grasping the concept ofdiversity given their educational level and limited previoustraining.

5. Discussion

The objective of the present study was to investigate the extentto which a diversity training programme can enhance sensitivity todiversity among health professionals. Our study indicates that thenotion of diversity and maybe more specifically the interplaybetween the different dimensions of diversity like sex, gender,ethnic origin and SES was difficult to grasp for individual healthprofessionals. How to conceive of this interplays itself is a topic ofongoing debate. The dominant analysis of health determinants ischallenged by approaches that try to better conceptualize thecumulative interlocking dynamics that affect human experienceincluding human health. Such an approach is emerging in theoryand not readily transferable to healthcare practices yet. But even atthe level on which our training was implemented the notion ofdiversity–as awareness of the relevance of dimensions of differ-ence for health and healthcare–turned out to be difficult. This wasespecially clear in the nursing home practice where some of theparticipants were less educated. Most likely, the training and theliterature provided were too difficult for them. This is indicative ofa ‘programme failure’ (Linnan & Steckler, 2002) which means, inthis instance, that the training did not activate the causal process

(more awareness and knowledge) that would potentially culmi-nate in the intended goals (readiness for action). In the case of thenursing home practice the training was not fully appropriate and,therefore, less effective. The responses of the nursing homeparticipants therefore require us to adapt critical components ofthe training in such a way that they become accessible to all levelsof education. However, the basic underlying assumption, that thediversity training is an appropriate tool to effect positive changetowards diversity sensitive healthcare, remains tenable. The basicingredients of the training include awareness raising, becomingmore critical of one’s practices, developing feasible plans forinnovations and creating conditions such as managerial support.

Although the training and the introduction of categories ofdifference and their interplay seemed too academic, there might beother reasons why the notion of diversity was not fully taken intoaccount in the three settings. One such factor is related to personal,sociopolitical and contextual characteristics. Literature on processevaluation is rich on how these factors influence the effectivenessand implementation of a programme (Abma, 2006; Guba & Lincoln,1989; Mabry, 1991; Parlett & Hamilton, 1977; Stake, 1975). Thereare ample examples of conflicts, jealousy and envy in organizationsthat hinder the implementation of programmes, or how manage-rial support, creativity and the charisma of a leader foster suchimplementation. In our study we also noticed these influences. Inthe mental health institution a small group of content-drivenprofessionals motivated others to join them in their attempt toimplement diversity. Yet, change of staff hampered their enthusi-asm. In the hospital the middle manager was of utmost importanceto fuel the conversation on diversity. Creativity, enthusiasm andmanagerial support was lacking in the nursing home. The effect ofthe training might as well be linked to the personal characteristicsof the trainers. Both felt a passion for raising sensitivity to diversity,but they were not acting like experts standing above practicetelling others what to do and were open to the experiences ofparticipants. It might have been the case that characteristics of thetrainers (in terms of education and class) matched better with theparticipants in the mental institution and hospital, than with thosein the nursing home. The importance of personal and socio-political factors implies that we should not overvalue the training,and be very careful to consider these factors when implementingsuch training elsewhere.

In line with this observation is the suggestion that diversitytraining for individual professionals is a prerequisite, but notenough to ensure action being taken for diversity sensitive caregiving at the organizational level. Despite the participants’increased insight into diversity, there were still many constraintson implementing the diversity concept. This was not due to a lackof enthusiasm at the individual level. Participants were motivatedand had a particular interest in new ideas. According to Rogers(2003), this is a prerequisite for innovations. Participants also felt a‘sense of urgency’ to change their practice, which in the literature isalso seen as an element necessary for organizational transforma-tions (Kotter & Cohen, 2002). Yet, their sense of urgency was notautomatically shared with other stakeholders in the organization.As a result, diversity remained the sole responsibility of a smallgroup of well-trained health professionals, and was not embraced

Table 6Program satisfaction of participants’ (N = 31).

Mental health Mean (SD) N = 8 Hospital Mean (SD) N = 12 Nursing Home Mean (SD) N = 11

Were participants’ course objectives met? 7.0 (1.5) 7.6 (0.8) 5.9 (0.6)

Were individual lectures and workshops presented effectively? 7.3 (1.6) 7.8 (0.7) 6,1 (1.0)

Did the training provide new insights? 7.7 (1.8) 8.3 (0.8) 6.5 (1.1)

Did the training provide practical and applicable knowledge? 7.6 (1.5) 7.1 (0.8) 6.2 (1.2)

Were methods of the training adequate? 7.6 (1.2) 8.0 (0.5) 6.3 (0.9)



by the other health disciplines and managers. In order to bringabout institutional changes an adjusted policy is required whichtakes into account the need for mainstreaming diversity at allorganizational levels and among all stakeholders (compare: Grol &Grimshaw, 2003; Yano, 2008). Institutional resistance is bestaddressed, according to the literature, by creating a sense ofurgency through real-life stories that evoke emotions, by fosteringcreative individuals, by giving managerial support, and byexpressive communication of small, but important successes.

A diversity training programme per se will aim at awarenessraising of and knowledge transfer of cultural differences toindividual professionals. The range of potential outcomes (ordesired outcomes) of diversity training(s) will however be limitedgiven the fact that such training takes place outside the workplaceand does not take into account the structural barriers toimplementing diversity in healthcare settings. A more narrowunderstanding of diversity – focusing on professionals and theirknowledge – might prove to be in conflict with a moretransformative approach and perspective. Transformation in termsof more social justice and equality includes all levels of thehealthcare organization and larger system, including its culture,structure and working routines. Such transformation can befacilitated if diversity is approached as a dynamic and reflexiveprocess on the core values and normative orientations that guideour healthcare practice in all its facets ranging from thecomposition of the staff to how we approach immigrant patientsso that they feel welcome and empowered.

A methodological point of discussion relates to the fact thatthere is some overlap between the developers of the training, thosewho acted as trainers and those who evaluated it. The third authorof this article was involved in the development of the training. Thefirst author acted as one of the trainers. The third and first authorswere both involved in the evaluation of the training. This raises thequestion of evaluator-bias and whether they were open enough toseeing the shortcomings of the training, and whether participantswould be open enough to share their critique. To anticipate over-identification with the training by the evaluators, the evaluationteam was enlarged by two other persons (second and fourthauthor), who were much more detached and independent, and hadno prior knowledge of or interests in the training. In the literaturethis corrective procedure is known as peer-debriefing (Guba &Lincoln, 1989). To anticipate socially desirable answers ofparticipants a mix of methods was used, also known astriangulation. Despite the shortcomings of over-engagementamong those who developed and executed the training, we wouldlike to emphasize that such involvement also proved to bebeneficial as this was a formative, developmental evaluation. Theevaluator (first author) had a lot of prior knowledge of the trainingas she was the one who was actually involved in all the modules.She knew the participants, had seen their development, and wassensitive to controversial issues. This helped her during theinterviews to go into depth, to probe and to dig deeper.

Critics may also question the status of the self-reportedknowledge on which this study is based. From a post-positivisticparadigm self-reported knowledge is indeed subjective and biased.Working from an interpretive or constructivist paradigm this kindof knowledge is not necessarily negative. It provides an under-standing (versus explanation) of the insiders’ perspective on whatthey have learned about diversity in the training. In combinationwith the qualitative data from the interviews this helps to gain aninsight into the experiences of the participants and the meaningand value they endow on these experiences. The data, for instance,helped to illuminate how participants wrestle with the diversityconcept and the application of diversity in practice. The fine-tuningof the training was an issue followed-up on after three years withthe managers of the settings. This choice was partly made for

practical reasons; some healthcare professionals had already leftthe setting. We expected the managers to have an overview on howthe training had affected the professionals and their workingpractice. In retrospect it might have been worthwhile also tocontact some of the participants in the training to get their view.

Another related question concerns the reliability and theinternal validity of the study. As has been pointed out no relevantvalidated instruments were available at the time of the study. Thevalidity of the survey has been closely considered through anarticulation of the programme theory and relevant variablesunderlying the training. Based on the literature on qualityimprovements and the Deming cycle we have selected awareness,knowledge and readiness for action as the main variables of thesurvey. The reliability of the survey has been reached through theuse of various negative and positive stated questions concerningone variable. A question about a diversity concept in the surveywas asked twice and the participants should score similarly twiceto consider the result as reliable. The correlation values amongthese questions of the pilot survey were tested by using Cronbach’salpha. Unreliable questions in the pilot survey were excludedbefore the survey was used.

It was during the study that some of the shortcomings of thesurveys became visible. First of all we wondered whether theinstruments were subtle enough to measure changes in partici-pants’ thinking as they focused on the variables that were easy tooperationalize. Awareness raising was an important goal of thetraining, but was not corrected for among other items. Therefore,certain outcomes (nursing homes staying in the do-phase) seemednot valid, and were hard to explain. Secondly, the survey did nottake into account other factors that might affect the outcomes ofthe training, such as shortage of staff or personal characteristics.These contextual factors were taken into account in the qualitativestudy. Given these shortcomings later on more emphasis was puton the qualitative data that had been gathered from the start, butwhich had not been integrated with the quantitative data. As suchthis study began as a traditional mixed methods study where dataand methods are not mingled, but just added on to each other. In alatter stage, the data were systematically integrated at the level ofeach setting.

For the qualitative study we used trustworthiness criteria andprocedures to assess the quality/goodness of the interpretations(Guba & Lincoln, 1989). Credibility refers to the degree by whichparticipants in the evaluation can identify themselves with theanalysis and interpretations. In our study we checked thecredibility frequently using the ‘member check’ procedure:respondents were asked whether they recognized the analysesof the interviews. In the follow-up interviews after three years wedeliberately checked whether the coordinators of the trainingprogramme on diversity could approve our analysis of the settings.Furthermore, the combination of methods (triangulation) helpedto sort out various dimensions and perspectives on the effects andimplementation of the training. Since the evaluator was engaged inall the four training modules with the three groups of participantsfor a longer period of time and visited the sites after three years itwas possible to monitor the development of the participants(procedures of prolonged engagement and persistent observation).During the study the evaluator (first author) gained systematicfeedback from her external supervisors (co-authors), who acted aspeer-debriefers. Through these procedures we tried to minimizeevaluator ‘bias’. We add abbreviations, because bias needs not onlyto be interpreted negatively. Foreground knowledge and pre-judices are needed to be able to make relevant observations, tomake distinctions and critically probe in dialogues. It is, however,essential that such prejudices are brought into play and that one isopen to adjusting prejudices that appear incorrect. An indication ofopenness is the learning curve of the evaluator.



The external validity of the study can be questioned since theselection of the settings was based on pragmatic grounds.Generalization beyond these settings to broader healthcarepractice is problematic for that reason. Yet, from an interpretiveparadigm knowledge is always context-bound. A conveniencesample is less troublesome, and requires a detailed description ofthe studied settings in order to allow readers (versus the evaluator)to transfer knowledge from the studied to their own context. Thisis called a naturalistic generalization (Abma & Stake, 2001).

6. Lessons

The design of this study was given in by practical consider-ations. We were happy to find three settings willing to join ourdiversity training programme, and chose not to have any type ofcomparison group and to use a convenience sample. Since therewere no validated instruments to measure diversity sensitivity atthe moment we started, we developed our own instruments incombination with the semi-structured interviews. The evaluatorswere also engaged in the provision of the training, which enabledthem to monitor the process from beginning to end. Initially theparticipant observations were carried out without a clear protocol.As this was not satisfactory later on in the study, we began tosystematically record the observations during the training.Another lesson concerns the follow-up interviews after threeyears. These were initially not planned. Yet, when we analyzed allthe data, we felt the need to check our overall interpretations.Furthermore, we were dissatisfied with the short term outcomes ofthe training. The follow-up interviews provided a lot of informa-tion and were efficient.

Another lesson concerns the combination of quantitative andqualitative data. The large set of methods and instrumentsprovided various sorts of data which quite surprisingly resultedin complementary and more comprehensive results. Manyquantitative outcomes could be better understood and explainedwith the help of the qualitative data. The qualitative data alsoenriched the quantitative data as a new variable was found to beimportant to explain diversity sensitivity, namely the developmentof a critical attitude to one’s practice. Quantitative data helped tovisualize trends which were not so obvious from the qualitativedata, for instance, that the effectiveness of the training varied persetting. What the evaluators had sensed, but had ignored initiallyas being personal observations, became much clearer. We foundalso differences between sorts of data, for example, that mentalhealth participants were quite satisfied. The qualitative datasuggested, however, that the training was sometimes too easygiven the previous knowledge among participants in mentalhealthcare. It is, thus, the combination of qualitative andquantitative data (and not just the accumulation) that has shedmore light on this issue. Yet, we also feel that there is a trade offbetween breath and depth when combining various sorts ofmethods. Our qualitative methods were quite crude and did notreach the depth of understanding that is realized when one onlyuses qualitative methods, like a Grounded Theory approach. Thisisn’t necessarily problematic, but should be deliberately consid-ered when choosing for a mixed methods strategy.

In retrospect we are quite dissatisfied with the results amongparticipants within the nursing home. It seems quite obvious thatthe training was too academic for nurses, and that the organiza-tional circumstances were limiting. A formative process evaluationwould have been more appropriate to support the desiredimprovement of this practice; data from the evaluation couldthen have been used to adjust the training programme to theorganizational constraints and the educational needs of theseparticipants.

7. Conclusion

A formative process evaluation of a diversity training pro-gramme in three healthcare settings demonstrated that such aprogramme can increase awareness, critical attitude, knowledgeand willingness to take action among participants. Yet, the degreeof improved understanding among participants varies across thesettings. The training had a greater impact on the participants fromthe mental health institution and hospital compared to those fromthe nursing home. Qualitative data related to the complexity of thediversity concept, the cognitive level of the training, the pre-understandings of participants and organizational constraints(case load; work pressure) were all helpful in explaining thequantitative differences found between the three settings in thisstudy.

The responses of the nursing home practice urge us to rethinkthe training for those with lower educational levels working in aconstraining organizational context. The responsibility for theimplementation of a complex concept like diversity cannot andshould not be placed upon the shoulders of those with the lowestdegree of education. Implementing diversity sensitive care is firstof all the responsibility of highly qualified professionals, includingmanagement and staff. They can act as role models and supportothers to practice diversity sensitive care.

We recommend further research on this subject to gain moreinsight into the effects of promoting diversity sensitivity in themulti-level and multi-actor context of healthcare practices. Thesestudies should take a few additional outcomes into account, suchas the adoption of the diversity concept and behavioural change.There is also a need to foster inter-organizational learning.Learning networks of participants from various organizationsworking on one highly complicated issue are particularly helpful tojointly discuss problems, to question routines, to pass on creativesolutions and to find inspiration and energy for long-term changesin the direction of diversity sensitive care. Finally, we want to urgehealthcare practices and evaluators to develop and evaluatediversity sensitive care with the communities that it concerns(Burlew, 2003). Diversity sensitive care can be realized bestthrough the participation of minority groups and their culturalbrokers. On an individual level mutual decision making helps toprovide diversity sensitive care. A transformative evaluationapproach, as described by Mertens (1994, 2009), is a preferredmethodology to foster empowerment and transformation of allwho are involved.

References

Abma, T. A. (2006). The Social Relations of Evaluation. In I. F. Shaw, J. C. Greene, & M. M.Mark (Eds.), The Sage Handbook of Evaluation (pp. 184–199). London: Sage Pub-lications.

Abma, T. A., & Stake, R. E. (2001). Stake’s responsive evaluation: Core ideas andevolution. Responsive Evaluation, 92, 7–22.

Bekker, M. H. J. (2003). Investigating gender within health research is more than sexdisaggregation of data: A multi-facet gender & health model. Psychology Health andMedicine, 2, 231–243.

Bird, C. E., & Rieker, P. P. (1999). Gender matters: An integrated model for understand-ing men’s and women’s health. Social Science and Medicine, 48(6), 745–755.

Burlew, A. K. (2003). Research with ethnic minorities: Conceptual, methodological,analysis issues. In G. Bernal, J. E. Trimble, A. K. Burlew, & F. T. L. Leong (Eds.),Handbook of racial and ethnic minority psychology. Thousand Oaks, CA: Sage.

Celik, H., Abma, T. A., Widdershoven, G. A., van Wijmen, F. C. B., & Klinge, I. (2008).Implementation of diversity in healthcare practices: Barriers and opportunities.Patient Education & Counseling, 71, 65–71.

Celik, H., & van Oosten, N. (2004). Gender, diversity and quality assurance in healthcarepractices. Blueprint Training Maastricht, Maastricht University.

Celik, H., & van Oosten, N. (2005a). Mainstreaming diversity in healthcare practices. Thetraining manual. Maastricht: Maastricht University.

Celik, H., & van Oosten, N. (2005b). Mainstreaming diversity in healthcare practices.Participants’ manual and reader. Maastricht: Maastricht University.

Celik, H., & Klinge, I. (2005). Mainstreaming gender en diversiteit in de gezondheids-zorg [Mainstreaming gender and diversity in health care]. Tijdschrift voor Gender-studies, 2, 12–22.



Celik, H., Lagro-Janssen, T., Klinge, I., van der Weijden, T., & Widdershoven, G. (2009).Maintaining gender sensitivity in the family practice: Facilitators and barriers.Journal of Evaluation in Clinical Practice, 15(6), 1220–1225.

Council of Europe. (1998). Gender mainstreaming. Conceptual framework, methodologyand presentation of good practices. Strasbourg: Council of Europe.

Crenshaw, K. W. (1994). Mapping the margins: Intersectionality, identity politics, andviolence against women of color. In F. N. Albertson & R. Mykitiuk (Eds.), The publicnature of private violence (pp. 93–118). New York: Routledge.

Essed, P. (2005). Gendered preferences in racialized spaces: Cloning the physician. In K.Murji & J. Solomos (Eds.), Racialization. Studies in theory and practice (pp. 227–245).Oxford: Oxford University Press.

Greene, J. C. (2007). Mixed methods in social inquiry. San Francisco: Jossey-Bass.Greene, J. C. (2008). Is mixed methods social inquiry a distinctive methodology? Journal

of Mixed Methods Research, 2(1), 7–22.Greene, J., Benjamin, L., & Goodyear, L. (2001). The merits of mixing methods in

evaluation. Evaluation, 7(1), 25–44.Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for

mixed-method evaluation design. Educational Evaluation and Policy Analysis, 11(3),255–274.

Greene, J. C., Kreider, H., & Mayer, E. (2005). Combining qualitative and quantitativemethods in social inquiry. In B. Somekh & C. Lewin (Eds.), Research methods in thesocial sciences (pp. 274–281). London: Sage.

Grol, R., & Grimshaw, J. (2003). From best evidence to best practice: Effective imple-mentation of change in patients’ care. The Lancet, 632, 1225–1230.

Grol, R., Wensing, M., & Eccles, M. (2005). Improving patients’ care: The implementationof change in clinical practice. Edinburgh: Elsevier Butterworth Heinemann.

Guba, E., & Lincoln, Y. (1989). Fourth generation evaluation. Beverly Hills, CA: Sage.Health Canada. (2000). Types of gender bias problems. Health Canada’s gender-based

analysis policy. Ottawa: Health Canada.Hoffman, E. (2000). Women’s health and complexity science. Academic Medicine,

75(11), 1102–1106.Hsieh, H. F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis.

Qualitative Health Research, 15(9), 1277–1288.Kotter, J. P., & Cohen, D. S. (2002). The heart of change. Real life stories of how people

change their organizations. Harvard: Harvard Business School Press.Langley, G. L., Nolan, K. M., Nolan, T. W., Norman, C. L., & Provost, L. P. (1996). The

improvement guide: A practical approach to enhancing organizational performance.San Francisco, CA: Jossey-Bass.

Leininger, M. (1995). Transcultural nursing: Concepts, theories, research and practices.New York: McGraw-Hill.

Leininger, M. (Ed.). (2001). Cultural care diversity & universality: A theory of nursing.National League for Nursing Press.

Lincoln, Y., & Guba, E. (1985). Naturalistic inquiry (pp. 92–109). New York: Sage.Linnan, L., & Steckler, A. (2002). Process evaluation for public health interventions and

research: An overview. In L. Linnan & A. Steckler (Eds.), Process evaluation for publichealth interventions and research (pp. 1–23). San Francisco, CA: Jossey-Bass.

Mabry, L. (1991). Alexandre Dumas Elementry School, Chicago, Illinois. In R. Stake, L.Bresler, & L. Mabry (Eds.), Custom & Cherishing: the Arts in Elementary Schools (pp.137–176). Urbana-Champaign: University of Illinois.

Mahoney, J., & Engebretson, J. (2000). The interface of anthropology and nursingguiding culturally competent care in psychiatric nursing. Archives of PsychiatricNursing, 14(4), 183–190.

Mead, N., & Bower, P. (2000). Patient centeredness: A conceptual framework andreview of the empirical literature. Social Science and Medicine, 51, 1087–1110.

Mertens, D. M. (2009). Transformative research and evaluation. New York, London:Guilford.

Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded source-book. Thousand Oaks, CA: Sage.

Moser, I. (2006). Sociotechnical practices and difference: On the interferences betweendisability, gender, and class. Science Technology Human Values, 31, 537–564.

Parlett, M., & Hamilton, D. (1977). Evaluation as illumination: A new approach to thestudy of innovatory programmes. In D. Hamilton et al. (Eds.), Beyond the numbersgame. London: Macmillan.

Pathman, D. E., Konrad, T. R., Freed, G. L., Freeman, V. A., & Koch, G. G. (1996). Theawareness-to adherence model of the steps to clinical guideline compliance: Thecase of pediatric vaccine recommendations. Medical Care, 34, 873–889.

Pinn, V. W. (2003). Sex and gender factors in medical studies. Implications for healthand clinical practice. Journal of the American Medical Association, 289(4), 397–400.

Rees, T. (2001). Mainstreaming gender equality in science in the European Union.Gender and Education, 13(3), 243–260.

Risberg, G., Hamberg, K., & Johansson, E. E. (2003). Gender awareness among physi-cians—The effect of specialty and gender. A study of teachers at a Swedish medicalschool. BMC Medical Education, 3, 8.

Rogers, E. M. (2003). Diffusion of innovations (5th edn.). New York: Free Press.Stake, R. E. (1975). To evaluate an arts program. In R. E. Stake (Ed.), Evaluating the arts in

education: A responsive approach (pp. 13–31). Columbus, OH: Merrill.Street, R. L. (2002). Gender differences in health care provider–patient communication:

Are they due to style, stereotypes, or accommodation? Patient Education andCounseling, 48, 201–206.

Van Mens-Verhulst, J. (2001). Diversiteit in theorie en praktijk: Een bericht uit defeministische hulpverlening. [Diversity in theory and practice: A message from thefeminist care]. Tijdschrift voor Humanistiek, 2(2), 7–17.

Verdonk, P., Benschop, Y. W. M., De Haes, J. C. J. M. , & Lagro-Janssen, A. L. M. (2008).Making a gender difference: Case studies of gender mainstreaming in medicaleducation. Medical Teacher, 30, e194–e201.

Verdonk, P., Mans, L. J. L., & Lagro-Janssen, A. L. M. (2006). How is gender integrated inthe curricula of Dutch medical schools? A quick-scan on gender issues as aninstrument for change. Gender and Education, 18(4), 399–412.

Wekker, G. (2001). Van monocultuur naar caleidoscoop. Humanistiek, 6, 26–33.Yano, E. (2008). The role of organizational research in implementing evidence-based

practice: QUERI Series. Implementation Science, 3(1), 29.

Halime Celik, PhD, LLM. Her PhD thesis was about gender sensitivity in health carepractices. At the time of the study she was affiliated to Maastricht University at thedepartment of Health, Ethics and Society and she is also working as solicitor at Simonsand Partners Law Practice in the Netherlands.

Tineke A. Abma is Professor Client Participation in Elderly Care at the Department ofMedical Humanities and the EMGO+ Institute for Health and Care Research, VUUniversity Medical Center, Amsterdam. She published on program evaluation, patientparticipation and ethics in elderly care, chronic care and psychiatry.

Ineke Klinge, PhD is Associate Professor at the department of Health, Ethics andSociety, Maastricht University. Ineke Klinge is a biologist by training and specialized inGender Studies in Science.

Guy A.M. Widdershoven is Professor of Philosophy and Ethics of Medicine and Head ofthe Department of Medical Humanities, VU University Medical Center, Amsterdam,and senior researcher at the EMGO+ Institute for Health and Care Research of the sameuniversity. He published on ethics and moral deliberation. He is scientific director ofthe Netherlands School of Primary Care Research (CaRe) and president of the EuropeanAssociation of Centers of Medical Ethics (EACME).


process evaluation of a diversity training program

Documents