impact evaluation of communication as power: listening ...documents.worldbank.org/curated/en/... ·...

38
Impact Evaluation of Communication as Power: Listening, Inquiring, Feedback Heidi S. Zia Meredith K. Stone Maria Bina Palmisano WBI Evaluation Group EG03-68 The World Bank Institute The World Bank Washington, DC March, 2003 Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized

Upload: others

Post on 27-Jul-2020

3 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power: Listening, Inquiring, Feedback Heidi S. Zia Meredith K. Stone Maria Bina Palmisano WBI Evaluation Group EG03-68 The World Bank Institute The World Bank Washington, DC March, 2003

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Administrator
31231
Page 2: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page ii

Acknowledgements

This report was prepared for the Learning Board under the overall guidance of Marlaine Lockheed, Manager, World Bank Institute Evaluation Group. The team was led by Heidi S. Zia.

The team benefited greatly from Sukai Prom-Jackson (HDNED) and Fred Nunes (HRSLO) in designing the questionnaires and providing all course related information for conducting this evaluation. The report also benefited from peer reviews by Jaime Quizon (WBIEG) and Chaoying Liu (WBIEG) Expert document production was provided by Humberto Díaz (WBIEG).

The findings, recommendations, and implications of this impact evaluation were fully discussed with Fred Nunes, designer and team leader of the Communication and Power Workshop.

WBI Evaluation Studies are produced by the WBI Evaluation Group (WBIEG) to report evaluation results for staff, client, and joint learning events. An objective of the studies is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors and do not necessarily represent the view of the World Bank Group. WBI Evaluation Studies are available on line at http://www.worldbank.org/wbi/evaluation/puball.htm.

Vice President, World Bank Institute Ms. Frannie Léautier Chief Learning Officer Ms. Phyllis Pomerantz Manager, Institute Evaluation Group Ms. Marlaine Lockheed Task Manager Ms. Heidi S. Zia

Page 3: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page iii

TABLE OF CONTENTS

Table of Contents....................................................................................................................... iii

Executive Summary.................................................................................................................... v

1. Introduction.......................................................................................................................... 1

Objectives ............................................................................................................................ 1 Background .......................................................................................................................... 1

2. Methodology........................................................................................................................ 3

General Context and Approach ........................................................................................... 3 Evaluation Instruments ........................................................................................................ 3 Response Rates and Representation .................................................................................... 4

3. The effectiveness and Impact of the workshops: The manager’s Perspective .................... 5

Relevance and Effectiveness of the Workshop ................................................................... 5 Quality of Course Design and Delivery of Workshops ...................................................... 6 Impact on Managers: Use of Skills.................................................................................. 6 Impact on Manager’s: Behavioral Changes....................................................................... 8 Facilitators and Barriers to Manager’s use of Communication Skills ................................. 8

4. Impact of the Workshops: Direct Reports’ Perspective ................................................... 11

Who are the Direct Reports?.............................................................................................. 11 Are the Direct Reports Getting Heard, Being Valued and Empowered to learn? ............ 11

5. Recommendations from Managers and Direct reports ...................................................... 15

Manager’s Recommendations on Improving Communication as Power Workshop......... 15 Direct Report Recommendations on Improving Management Communication............... 17

6. Conclusion and Recommendations .................................................................................... 19

Conclusion......................................................................................................................... 19 Recommendations .............................................................................................................. 20

List of Annexes

A. Communication as Power - Level 1 Evaluation Questionnaire ...................................... 25

B. Communication as Power Impact Evaluation - Form A: Questionnaire for Managers and Course Participants .............................................................................. 27

C. Communication as Power Impact Evaluation - Form B: Questionnaire for Staff ............................................................................................................................... 31

Page 4: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page iv

List of Figures

Figure 1. Level 1 Evaluation Questionnaire Results ............................................................... 5

Figure 2. Level 1 Evaluation Questionnaire Results ............................................................... 6

Figure 3. Communication Skills Used by Managers ............................................................. 7

Figure 4. Managers’ Reported Behavior Changes ................................................................. 8

Figure 5. Facilitators of Behavior Change ............................................................................. 9

Figure 6. Getting Heard, Being Valued and Empowered to Learn. ........................................ 12

Figure 7. How to Reinforce the Communication as Power Workshop Learning.................... 16

Figure 8. Single Communication Behavior Manager Should Use More to Support Performance and Productivity (N=30) ..................................................................... 17

List of Tables

Table 1. Being Motivated (N=48) .......................................................................................... 13

Table 2. Enhanced Work Environment (N=48) ..................................................................... 13

Table 3. Manager’s Communication Behavior Improves Staff Behavior (N=47)................. 13

Table 4. How to Improve the Communication as Power Workshop .................................... 15

Table 5. Recommendations for the Management Development Program ............................ 16

Table 6. Other Comments .................................................................................................... 17

Table 7. Single Communication Behavior Manager Should Change (N=24) ...................... 18

Table 8. Single Communication Behavior Direct Report Could Use More Often (N=25) ..................................................................................................................... 18

Table 9. Final Comments on How Manager Needs to Improve (N=15) ............................... 18

Page 5: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page v

EXECUTIVE SUMMARY

The purpose of this evaluation was to determine the impact of the Communication as Power workshop. Is it effective in changing the communication behaviors of the manager participants? If so, what impact did their changes in behavior have on their direct reports? Six months after participating in the workshop, what kind of support do participants request? The two-day Communication as Power workshop utilizes scenarios based on real Bank activities and targets participants who face similar challenges. So the workshops are both highly relevant and extremely motivating. This workshop is specifically designed for managers in a new assignment or role where focusing on communication skills will both facilitate the managers’ growth and the Bank’s progress. The course is offered under the Management Curriculum “Managing Others” series. The Communication as Power workshop sessions are experiential, encouraging learning through peer interaction and self-observation. The communication skills presented are practiced using role plays, so participants get to see themselves both as the manager and the managee during the exercises and experience what it is like to be in each role. The Level 1 feedback collected at the end of each of the three workshops painted a clear picture: the Communication as Power workshop is highly relevant, effectively taught, and motivates participants to continue to use the new skills and knowledge acquired during the course. What is the picture three to six months later? Managers reported that they had changed many of their communication behaviors. They reported consciously using the listening, inquiry and feedback skills that they learned and practiced in the workshop. They reported that their behavior had changed in several areas, including creating an open learning environment, inspiring trust, empowering staff, increasing tolerances of differences and valuing differences to maximize quality. They also reported that their communication changes had the effect of improving their direct reports’ behavior, specifically in showing more self confidence, taking initiative, and being innovative. The managers further reported that while there were few supports for these changes except their own determination and commitment, there were likewise no barriers. The direct reports’ responses, while similar to those of the managers, could not be used to corroborate; since those responding reported to only a small number of the managers, the sample was not representative of the overall group of manager participants. Many of the direct reports who did respond chose not to identify themselves or their manager. Managers made a number of suggestions for improving the workshop. These included scheduling a six-month and nine-month half-day “refresher course,” monthly e-mail reminders, and that both their management and their direct reports could benefit from the

Page 6: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page vi

workshop. Sixty-four percent would recommend it to a colleague (unfortunately, this question was not consistently asked of all participants). Direct reports’ suggestions for what managers could do to improve their staff’s performance were to show more trust, respect, and acknowledgement by sharing more information more often, by listening more, by valuing their work, and by taking an interest in their professional development. Recommendations from the evaluators included (i) making the course available to both more managers and their direct reports—not in the same sessions, (ii) adding a short refresher course six-months later, either live or on tape, (iii) linking the communication behaviors more directly to the two higher level concepts of managing in a matrix environment and approaching conflict with confidence, and (iv) encouraging small groups to continue to work together after the course to discuss and practice their new skills. In summary, the Communication as Power workshop is an effective training course that could have an even more positive impact on the Bank if the training was (i) linked more specifically to the higher level objectives and (ii) if reminder messages or refresher sessions for the positive communication behaviors were scheduled as part of the initial course.

Page 7: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 1

1. INTRODUCTION

OBJECTIVES

1.1 The purpose of this evaluation is to determine the impact of the Communication as Power workshop on the participants and their staffs. The Communication as Power course is one of the two-day workshops instituted by the Bank under the Action Learning paradigm adopted by the Learning Board in the spring of 2001. These workshops bring peers together to participate in small groups focused on a particular topic. By utilizing scenarios based on real Bank activities and inviting participants who face similar challenges the workshops are designed to be both relevant and motivating. The Communication as Power workshop is for managers in a new assignment or role where focusing on communication skills will both facilitate the managers’ growth and the Bank’s progress. The course is one of those offered under the Management Curriculum “Managing Others” series.

1.2 This evaluation study reports on the findings from three sessions of the Communication as Power workshop held during the winter of 2001-2002. The objectives of the workshop are to increase the managers’ capacity to:

• Listen, inquire and give feedback effectively; • Manage complex and sensitive situations; • Approach conflict with confidence.

1.3 As appropriate within the Action Learning paradigm, the workshop uses the participants’ own experiences along with peer feedback, discussion, and reflection to accomplish these objectives.

BACKGROUND

1.4 The two-day Communication as Power workshops comprised a series of modules, with the experience of each previous module necessary for the next. The first task for each group is to produce videos of baseline communication skills which are used to determine where improvement can be made. After an overview of the Learning Process module, the participants practice role play using real World Bank scenarios from HR to illustrate Effective Listening and Effective Inquiry. Day Two starts with a review and reflection session of Day One and follows with the Effective Feedback module. The rest of Day Two is spent in small groups practicing what has been learned in role play, which is once again videotaped. At the end of the day, everyone develops a personal action plan outline to take with them and completes an evaluation questionnaire.

Page 8: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 2

1.5 The workshops are held for small groups of less than twelve. Four facilitators coach the managers through the modules and video taped role play. The sessions are experiential, encouraging learning through peer interaction and self-observation. The communication skills presented are practiced using role plays of actual World Bank scenarios. Participants get to see themselves both as the manager and the managee during the exercises and experience what it is like to be in each role.

Page 9: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 3

2. METHODOLOGY

GENERAL CONTEXT AND APPROACH

2.1 Many training evaluations collect participants’ reactions to a course and ask what they think they have learned during the course at the end of the session, which results in immediate feedback. Such Level 1 “feedback” was collected for each of these three sessions, as well, and was used by those who present the course to improve future sessions. For this evaluation, however, we also wanted to determine if participants changed their behavior once back at their management positions. To assess whether or not the training changed the attitudes or communication behavior of participants, we needed to collect information about what communication skills were being used on the job at a later date. This kind of impact evaluation seeks evidence that increased knowledge and skills have effected a behavior change. Were the communication skills being used by the participants? And if so, was their evidence that the managers’ changes in behavior were noticed by their staff members? We further asked for observations of positive communication skills from the workshop participants’ direct reports. These Level 3 evaluations give us a picture of the real impact these workshops had on positive changes in managers’ attitude and practice of the communication skills.

EVALUATION INSTRUMENTS

2.2 The three Communication as Power workshops that were evaluated were those held November 17-18, 2001, December 17-18, 2001, and February 12-13, 2002. Questionnaires were presented at the end of each session to assess Level 1 criteria: (i) topic relevance, (ii) effectiveness in increasing knowledge and skills in eight specific categories of communication, and (iii) quality of course design and delivery. A copy of the Level 1 feedback evaluation questionnaire appears in Annex A.

2.3 Three to six months after each of the designated workshops, participants were surveyed to assess Level 3 criteria covering changes in the managers’ behavior related to (i) conscious use of specific communication skills of listening, inquiring and feedback in specific areas, (ii) improvements in the behavior and performance of their staff, and (iii) the facilitators or barriers to behavior change. There were also a number of open-ended questions designed to elicit recommendations for improving this workshop specifically and the management development program in general. A copy of the Level 3 impact evaluation questionnaire appears in Annex B.

Page 10: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 4

RESPONSE RATES AND REPRESENTATION

2.4 The three sessions were attended by a total of 31 managers. Both Level 1 “Feedback” data from the questionnaire and Level 3 “Impact” data from the interview were collected from 26 workshop participants for a response rate of 84 percent. Response rates were similar across the three workshops.

2.5 The 26 managers that responded to the questionnaire were 65 percent male and 35 percent female. The majority (73 percent) were grade level GH. While 33 percent were first time managers in the Bank and a few were managers in a new unit, the rest (50 percent) were in new or changing roles as a manager within their same unit. The Part I nationalities represented 65 percent and the Part II nationalities represented 35 percent.

2.6 In addition, in an effort to corroborate Level 3 impacts reported by the managers themselves, questionnaires were sent to all their direct reports (N = 191) three to six months after each of the workshops. These direct report questionnaires asked about the managers’ communication behavior and how they affected the staff member in several different areas: (i) getting heard, (ii) being valued, (iii) empowering to learn, iv) being motivated, v) enhancing the work environment, and vi) improving attitude and performance. A number of open-ended questions asked for the single most important behavior their manager could do to support, enhance, or make their performance more effective. A copy of the questionnaire is in Annex C.

2.7 After repeated contacts over several months, the number of responses from direct reports was 48 for a 27 percent response rate. Response rates were similarly uneven across the three groups with many managers’ direct reports not responding at all and a few managers having five or six responses from their direct reports.

Page 11: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 5

3. THE EFFECTIVENESS AND IMPACT OF THE WORKSHOPS: THE MANAGER’S PERSPECTIVE

RELEVANCE AND EFFECTIVENESS OF THE WORKSHOP

3.1 The workshop was experienced as both highly relevant and effective, with participants giving an average score of 89 percent, that is 89 percent of the participants rated the workshop as 4 (above average) or 5 (high) on a five point scale—well above the WBIEG benchmark of 85 percent. Figure 1 below indicates that four of the ten questions on relevance and effectiveness listed below were ranked in the highest categories (above average or high) by 100 percent of the respondents: relevance, increasing effective communication, feedback skills and having more control in one-on-one situations. All but two questions have 85 percent of the respondents’ highest rating: knowledge of effective decision making was rated highly by only 54 percent and influencing skills for a matrix organization was highly rated by only 65 percent. At the end of all three workshops, the participants indicated that the content was highly relevant, the training they received was for most skills highly effective, and that they intended to put their new skills into practice. The course developers will want to investigate, if they have not done so already, what caused the two ratings to fall below 85 percent.

Figure 1. Level 1 Evaluation Questionnaire Results1

Relevance, Effectiveness, Potential Impact (N=26)

0%

20%

40%

60%

80%

100%

120%

Per

cen

t S

elec

tin

g H

igh

est

Cat

ego

ries

(4

or

5) Relevant to improving performance as a manager100%

Increased knowledge of effective communication100%

Increased skills for giving feedback 100%

Increased skills in one-on-one communication100%

Increased listening skills 96%

Increased inquiry skills 92%

Increased skills for fostering relationships 92%

Will use knowledge and skills acquired in course89%

Increased influencing skills for matrixorganization 65%

Increased knowledge of effective decisionmaking 54%

1 See Annex A, Section I, questions 1 and 2 for actual questions.

Page 12: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 6

QUALITY OF COURSE DESIGN AND DELIVERY OF WORKSHOPS

3.2 Figure 2 below shows that the seven questions in this category are rated in the top categories (above average or high) by more than 90 percent of the respondents except for one: the relevance of the materials used in the case studies (73 percent). The average for questions in this overall category is 92 percent, again well above the WBIEG benchmark of 85 percent. The participants of all three workshops indicated that the course design and delivery was very well done. The one category that fell below the 85 percent benchmark may be due to not setting expectations properly, since the answer to is the training relevant to improving your performance as a manager was rated as above average or high by 100 percent of the respondents. Again the atypical rating on this one question indicates a need for investigation on the part of the course developers and presenters.

Figure 2. Level 1 Evaluation Questionnaire Results2

Quality of Course Design and Delivery (N=26)

0%

20%

40%

60%

80%

100%

120%

Per

cen

t S

elec

tin

g H

igh

est

Cat

ego

ries

(4

or

5)

Opportunity to practicecommunication skills 100%

Overall quality of course designand delivery 96%

Pacing was "just right" 96%

Quality of the plenarypresentations 92%

Facilitation of small groupsessions 92%

Depth of coverage of coursecontent 92%

Relevance of materials in casestudies 73%

IMPACT ON MANAGERS: USE OF SKILLS

3.3 The Level 1 feedback paints a clear picture: the Communication as Power workshop is highly relevant, effectively taught, and motivates participants to use the new skills and knowledge acquired during the course. What is the picture three to six months later?

3.4 The impact evaluation is designed to assess whether the skills addressed in the training are being used by the participants. Three to six months after completing the workshop, participants were interviewed concerning the knowledge and skills presented in the workshop. Specifically, they were asked “to what degree have you consciously used the following communication skills?” While the WBIEG does not have a benchmark figure set for impact evaluations, we will use 50 percent as the cut off. This figure indicates that at least half of the respondents report that they are “highly conscious of applying the particular 2 See Annex A, Section II, question 6 for actual questions.

Page 13: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 7

skill or knowledge acquired during the workshop” as they manage their staff three to six months later.

3.5 Listening Skills: The first cluster of questions concerned listening skills. In Figure 3, below, it is evident that respondents are using certain skills more regularly than others, with the skill gaining the highest percentage (81 percent) that of “use silences/pauses constructively.” Two other skills, “use eye contact well, (60 percent)” and “reflect on and verbalize feelings (58 percent)” are also highly rated. The rest summarizing the content of dialogue and using effective body language, fall below the 50 percent rating which may indicate that more practice of these skills during the workshops is warranted.

3.6 Inquiring Skills: The next cluster of communication skills concerns inquiry. As Figure 3 below indicates, all these skills were consciously used by more than 50 percent of the respondents with the highest percentage of respondents indicating that reinforcing answers (71 percent) was the skill used most. Overall, however, neither the listening nor inquiring skills were consciously used as much as the feedback skills. More attention during the workshop might be paid to questioning assumptions and asking open ended questions, two important communication skills that can reap big benefits.

3.7 Feedback Skills : The highest ratings of all were given to the feedback cluster. Respondents gave this cluster higher ratings than either the listening cluster or the inquiring cluster. As the figure below indicates, only one skill “start with a positive message” was as low as the 50 percent level. The managers believe that the feedback skills acquired during the workshop are being put into practice.

Figure 3. Communication Skills Used by Managers

To what degree have you consciously used the following communication skills since completing the workshop? (N=26)

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

1 Listening Inquiry Feedback

Per

cen

t S

elec

tin

g H

igh

est

Cat

ego

ries

(4

or

5)

Listening

Use silences/pauses constructively 80.8%

Use eye contact well 60%

Reflect on and verbalize feelings 57.7%

Use effective body language 46.2%

Summarize the content of dialogue 46.2%

Inquiry

Reinforce answers 70.8%

Use one question at a time 57.7%

Use open-ended questions 54.2%

Question assumptions 50%

Feedback

Describe the problem and consequences 79.2%

Give clear and specific messages 70.8%

Focus on actionable behavior 70.8%

Seek commitment 60%

Start with a positive messsage 50%

Page 14: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 8

IMPACT ON MANAGER’S: BEHAVIORAL CHANGES

3.8 The next cluster of interview questions asked to “what degree has your behavior changed in the following areas?” This question was designed to get at some of the attitudes that improve communication, rather than specific skills themselves. Figure 4 below indicates that the respondents gave themselves good ratings (>60 percent) in all but one of these areas. And in that specific case, “valuing differences and using them to maximize the quality of the work,” the lack of change probably indicates that the managers think they already do that. The area that is rated very highly by 82 percent of the managers is “creating an open learning environment.”

Figure 4. Managers’ Reported Behavior Changes

As a result of the course, to what degree has your behavior changed in the following areas? (N=23)

0%

20%

40%

60%

80%

100%

Per

cen

t S

elec

tin

g H

igh

est

Cat

ego

ry (

4 o

r 5)

Creating an open learningenviroment 81.8%

Inspiring trust 69.6%

Empowering staff 65.2%

Increasing tolerance ofdifferences 60.9%

Valuing differencs to maximizequality 47.8%

3.9 The next cluster of questions asked “to what degree has the change in your behavior led to improvements in the behavior and performance of your staff?” This, of course, is the heart of the impact evaluation. If the manager changes his behavior, what effect does it have on his staff? The managers report that using these more positive communication skills is bringing about positive change in their direct reports. All four behaviors are rated highly, with the top three “self confidence, initiative, and innovation” all rated at 71.4 percent. The fourth area, “successful completion of tasks” was rated at 57.1 percent.

FACILITATORS AND BARRIERS TO MANAGER’S USE OF COMMUNICATION SKILLS

3.10 The final two clusters of questions for the managers dealt with the facilitators and barriers to using more positive communication skills. Interestingly, the managers rated only one factor highly as a facilitating factor: “their personal commitment to work effectively through others.” As Figure 5 below shows, the rest of the nine factors were all rated at 45 percent or lower.

Page 15: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 9

Figure 5. Facilitators of Behavior Change

To what degree did the following facilitate or enhance your behavior change? (N=23)

0%

20%

40%

60%

80%

100%P

erce

nt S

elec

tin

g H

igh

est C

ateg

ori

es (

4 o

r 5)

Personnal commitment 79.2%

Support from colleagues in the bank45.5%

Staff readiness to be empowered43.5%

Coaching provided via IndividualCoaching iniciative 42.1%

Support from Supervising Manager34.8%

Support form others who attended thecourse with you 31.8%

Follow up provided by the training28.6%

Support from your HR Manager orOfficer 27.3%

Peer support provided by HRSLO PeerLearning 5.9%

Mentoring from an assigned mentor5.9%

3.11 The final ratings question concerned the barriers to the managers changing their communication behavior, “to what degree did the following act as barriers to your behavior change?” Once again, parallel to their taking responsibility for being their own best facilitating factor, the managers indicated that there were no real barriers other than giving a “heavy work load (38 percent)” and “other priorities set with their manager (25 percent)” as reasons, and made no excuses. Even more impressive was that “knowledge and skills of the training not being appropriate in my situation” and “lack of a safe environment for practicing skills” both received 0 percent indicating that these were not factors for any of the managers.

Page 16: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 10

Page 17: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 11

4. IMPACT OF THE WORKSHOPS: DIRECT REPORTS’ PERSPECTIVE

WHO ARE THE DIRECT REPORTS?

4.1 The 34 direct reports that responded to the demographic section of the questionnaire (Annex C) were 33 percent male and 67 percent female; this is the reverse of the percentages for the managers. The majority (44 percent) were grade level GE–GG, with another 21 percent in levels GA-GD. The Part I nationalities represented 63 percent and the Part II nationalities represented 37 percent. Fourteen of the 48 respondents chose not to respond to the demographic questions or to identify themselves by name. We were able to match the direct reports who did provide identification to only eight of the thirty-one managers who completed the impact evaluation, the sample does not represent a cross section of the managers’ direct reports. It is possible, therefore, to make only general comparisons of managers’ and direct reports’ responses.

ARE THE DIRECT REPORTS GETTING HEARD, BEING VALUED AND EMPOWERED TO LEARN?

4.2 The direct reports of all 31 managers were all sent a parallel impact questionnaire approximately three to six months after their manager had participated in the Communication as Power workshop. The first three clusters of questions on the direct reports’ questionnaire addressed “Getting Heard,” “Being Valued,” and “Empowered to Learn” the flip side of the listening, inquiring, and feedback behaviors the managers had practiced in the workshop. Are the direct reports observing the communication changes the managers said they had made? As shown in Figure 6 below, all ratings for all three of these clusters were rated at or above the 50 percent level established as our cutoff earlier. The 50 percent figure indicates that at least half of the respondents rated their managers as “above average or high” on these communication behaviors three to six months after the workshop.

4.3 Getting Heard: The “Getting Heard” ratings are all quite high, ranging from 54 percent to 75 percent. Interestingly, the communication behavior the direct reports rate lowest “summarizes to ensure understanding (54 percent)” is the same behavior the managers themselves rated below 50 percent.

4.4 Being Valued: The second cluster of questions concerned the direct reports response to the managers “attitude” when communicating or “Being Valued.” The table shows that all four aspects listed here are fairly highly rated (58 percent - 67 percent).

Page 18: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 12

These ratings of direct reports are very similar to the managers’ ratings in terms of the degree to which managers think they “inspire trust,” “empower staff” and “tolerate differences among staff” (60-70 percent).

4.5 Empowered to Learn: The third cluster of questions concerned the managers’ “empowering their direct reports to learn,” their willingness to provide learning experiences with feedback, in order to facilitate the direct reports’ professional development. It is interesting to note that three of these behaviors are rated in the low fifties although feedback is an area where the managers, except for summarizing, rated themselves very highly (60 percent-80 percent). Also, only 63 percent of the direct reports rate their manager high in “fostering an environment of innovation” while almost 82 percent of the managers gave themselves a high rating in “creating an open learning environment.”

Figure 6. Getting Heard, Being Valued and Empowered to Learn.

Over the past 3 to 6 months, to what degree do the following behaviors describe your manager's behavior? (N=48)

0%

20%

40%

60%

80%

100%

1.0%

Per

cen

t se

lect

ing

th

e h

igh

est

cate

go

ries

(4

or

5) Getting Heard

Is fully attentive when I speak 74.5%

Promotes open & candid communications 61.7%

Remembers substance of our last conversation 60.4%

Invites my comments and feedback 58.3%

Summarizes content of conversation 54.2%

Being Valued

Respects my opinions 66.7%

Acknowledges my contributions 64.6%

Takes an interest in my professional growth 59.6%

Respects my views even when we disagree 58.3%

Empowered to Learn

Recognize mistakes as learning Opportunities 62.5%

Fosters an enviornment of innovation 62.5%

Provides clear standards of interpersonal behavior 60.4%

Clearly communicates priorities 57.4%

Clearly communicates the priorities for my work 54.2%

Provides specfic feedback for improvement 51.1%

Clearly communicates his/her expectations 50.0%

Getting Heard Being Evaluated Empowered to Learn

4.6 Motivating Staff: The fourth cluster of questions dealt with how the managers’ communication behavior affected the direct reports’ motivation. As shown in Table 1 below, the direct reports’ ratings ranged from 46 percent to 64 percent. All but one is above the 50 percent level, but none is as high as 70 percent. The rating below 50 percent for “takes time to nurture my commitment” is corroborated by a number of the responses to the open-ended questions asked of staff and reported later.

Page 19: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 13

Table 1: Being Motivated (N=48)

Over the past 3 to 6 months, to what degree do the following describe your manager's behavior? % 4 or 5

Being Motivated - My Manager: Encourages desirable behaviors from me. 63.8%

Seeks to achieve mutual agreement 62.5% Empowers me by removing obstacles. 57.4% Uses feedback to motivate me to take actions. 56.3%

Takes time to nurture my commitment. 45.8%

4.7 Enhancing the Work Environment: The fifth cluster of questions asked to what degree the managers’ behavior during the past three to six months had enhanced the work environment of their unit, including improving trust, commitment, valuing differences and being more tolerant. As Table 2 shows below, the direct reports rated the managers fairly high on engendering these positive attitudes, with ratings between 58 percent and 67 percent, but not as highly as the managers’ rated their own behavior in this area (61 percent to 82 percent, see Figure 4 above).

Table 2: Enhanced Work Environment (N=48)

Over the past 3 to 6 months, to what degree has your manager's behavior enhanced the following aspects of the work environment of your unit?

Direct Reports

An open learning environment 66.7%

Tolerance among staff 61.7%

The valuing of differences 60.4% Staff empowerment 60.4% Level of commitment 58.3% Level of trust 58.3%

4.8 Improving Staff Confidence, Attitude and Performance: The final cluster of questions asked the direct reports to what degree the managers’ behavior during the past three to six months had improved their self-confidence, initiative, risk taking, and successful completion of tasks. The direct reports ratings in Table 3 below ranged from 53 percent to 61 percent. The managers’ ratings of the affect of their behavior on the same items were all 10 to 20 percentage points higher (71.4 percent) with the exception of “successful completion of tasks.” The direct reports’ highest rating (61 percent) was for “successful completion of tasks” which the managers rated the lowest (57 percent).

Table 3: Manager’s Communication Behavior Improves Staff Behavior (N=47)

Over the past 3 to 6 months, to what degree has your manager's behavior improved your: % 4 or 5

Successful completion of tasks. 60.9%

Self-confidence. 57.4%

Initiative and drive for results 55.3%

Innovation and risk taking. 53.2%

Page 20: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 14

Page 21: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 15

5. RECOMMENDATIONS FROM MANAGERS AND DIRECT REPORTS

MANAGERS’ RECOMMENDATIONS

5.1 At the end of the survey session, managers were asked a series of questions to elicit recommendations on how to improve the Communication as Power workshop in particular and the management development series in general. The answers to each question were grouped and analyzed by content.

5.2 Eighteen respondents answered “What are your recommendations for improving the Communication as Power workshop?” Their responses fell into five categories as illustrated in Table 4 below. These comments support the original feedback collected immediately after the course was finished. Three to six months after the course, with time for reflection and practice, the managers are still very positive about the content of the course. They are realistic about the practice it takes to change these communication behaviors. They recommend putting some kind of structured follow-up in place, as the primary improvement, whether a formal or informal refresher course or monthly e-mail reminders.

Table 4: How to Improve the Communication as Power Workshop

What are your recommendations for improving the Communication as Power workshop? Need follow-up or refresher course 39.0%

None - Excellent course 22.0%

Both managers and staff need to take it 22.0%

Include other cultural styles for comparison 11.0%

Better match of case material to audience 6.0%

5.3 Nineteen respondents answered “What can you do to reinforce the learning that you had from this workshop?” Their responses fell into six categories as shown in Table 11 below: The most frequent recommendation mentioned was “Review materials frequently and practice, practice, practice—with manager, peers, direct reports, and family. The next two recommendations, schedule follow-up meetings and send monthly e-mails are similar and all are in line with their earlier recommendations; the managers recognize that these skills need to be practiced and suggest ways for doing that.

Page 22: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 16

Figure 7. How to Reinforce the Communication as Power Workshop Learning

What can you do to reinforce the learning that you had from this workshop? (N=19)

0%

10%

20%

30%

40%P

erce

nt C

ateg

ory

Review and practice, practice,practice 36.8%

Schedule follow-up sessions 26.3%

Send Monthly e-mails as reminder15.8%

Handbook with tips and follow-upplan 10.5%

Create a personal learning plan5.3%

Take reflection breaks 5.3%

5.4 Twelve respondents answered “Other comments or recommendations for improving management development?” Their responses fell into four categories as illustrated in Table 5 below: Good communication takes constant effort; Have VP’s and upper management ask their direct reports to attend; Publicize courses repeatedly—need to take same course at different times in career; Ensure consistency and connection between courses. The theme through these categories was that this is a good communication course and more people, both manager and manage, ought to take it. They recommend that it “shouldn’t be the only course” and people should consider taking these courses more than once.

Table 5. Recommendations for the Management Development Program

5.5 Fourteen respondents answered the final question: “Other comments?” Their responses fell into the following four categories as shown in Table 6 below: Would recommend workshop to others; Course well designed and well delivered, facilitators excellent; Making good use of starting with other person’s agenda, but no question for that answer; Now understand the Bank's communication style better which is very different from my cultural upbringing. The “would you recommend workshop?” question was apparently asked at the end of many of the interviews, even though it does not appear on the written interview form.

Other recommendations for improving the management development program?

Good Communication takes constant effort 41.6%

All upper managements' direct reports should attend 25.0%

Need to repeat courses every few years 16.6%

Ensure consistency & connection between courses 16.6%

Page 23: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 17

Table 6. Other Comments

Other comments or recommendations for improving management development?

Recommend to others 64.2%

Well designed and delivered 21.4%

Other person's agenda important 7.1%

Very different from my culture 7.0%

DIRECT REPORTS’ RECOMMENDATIONS

5.6 Thirty respondents answered the question: “What is the single communication behavior your manager should make more use of to support your performance and productivity?” Their responses fell into seven categories as illustrated in Figure 8 below: Sharing more information, treating direct reports with trust and respect, and valuing their work made up the majority of the responses.

Figure 8. Single Communication Behavior Manager Should Use More to Support Performance and Productivity

What single behavior should your manager use more to enhance your performance and productivity? (N=30)

0 . 0 %

10 .0%

2 0 . 0 %

3 0 . 0 %

4 0 . 0 %

1

Per

cen

t o

f R

esp

on

den

ts

Share more information more frequently 33.3

Treat us with more trust and respect 23.3

Value and acknowledge our work 13.3

Suport our professional development 10

Continue doing what he's already doing 6.7

Listen and accept that others may be right 6.7

Other 6.7

5.7 Twenty-four direct reports answered the question: “What is the single communication behavior your manager should change to enhance your performance and productivity?” Their responses fell into the seven categories shown in Table 7 below. The first category directly parallels the second and third categories in the previous table; respect and acknowledgement are key issues in enhancing staff performance and productivity.

Page 24: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 18

Table 7. Single Communication Behavior Manager Should Change (N=24)

What is the single communication behavior your manager should change to enhance your performance and productivity?

More acknowledgement, recognition, attention 37.5%

Stop being manipulative 16.7%

Talk less, listen more 12.5%

Support my professional development 8.3%

Communicate priorities clearly and tell me when they change 8.3%

Doesn't need to change anything 8.3%

Other 8.3%

5.8 Twenty-five direct reports answered the question: “What is the single communication behavior you should make more use of to support your manager in becoming more effective? The responses fell into five categories as illustrated in Table 8 below. The top three categories were: (i) initiate regular communication, (ii) provide both feedback and information, and (iii) ask for more feedback—all behaviors that are likely to increase the trust and acknowledgement from the managers that many of these direct reports say they seek.

Table 8. Single Communication Behavior Direct Reports Could Use More Often (N=25)

What is the single communication behavior you should make more use of to support your manager in becoming more effective? Initiate regular and open communication 28.0%

Give manager feedback, share information 28.0%

Ask manager for more feedback 20.0%

Discuss priorities, timelines, deliverables 20.0%

Other 4.0%

5.9 Only fifteen direct reports responded to the final question asking “Do you have any other comments about areas where your manager’s behavior needs to be improved?” The answers fell into only three categories as shown in Table 10 below. More than 50 percent of the respondents said there manager was doing a good job and there was little room for improvement. Another 33 percent, however, commented that their manager was not a team player, did not respect individuals on the team, and was more concerned with his own agenda than the Bank’s.

Table 9. Final Comments on How Manager Needs to Improve (N=15)

Other comments about areas where your manager's behavior needs to be improved?

Effective manager, no need for improvement 53.3%

Teamwork requires organization, focus and respect 33.3%

Other 13.3%

Page 25: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 19

6. CONCLUSION AND RECOMMENDATIONS

CONCLUSION

6.1 The Communication as Power Workshop is an intense interactive two-day experience that is perceived as effectively presented and very relevant. The 31 participants rated it highly for relevance, effectiveness, and potential impact (89 percent overall). There were only two specific questions that individually fell below the 85 percent benchmark set by WBIEG: “knowledge of effective decision making” was rated highly by only 54 percent and “influencing skills for a matrix organization” was highly rated by only 65 percent. Since these two items represent much broader concepts than most of the others presented in the Level 1 questionnaire, it may be that 1) the concepts were not understood, or 2) the course does not make the connection of how the communication skills being practiced can be used in conjunction with these higher level concepts. This is an area that the course designers and presenters will want to investigate further in order to make the connections more evident.

6.2 The participants also rated it as very well designed and well presented (92 percent overall). There was one item which individually was rated lower than the 85 percent benchmark: “relevance of the materials used in the case studies (73 percent).” In the open-ended questions it was revealed that at least for one session the case study used was not relevant for the managers at that session. Given that the participants interact with the case studies intensely during their practice sessions, it is important that the cases be ones that the participants relate to easily. Again, the course designers and presenters will want to follow-up on this apparent disconnect. Perhaps information can be collected about participants in advance ‘to match with available cases.

6.3 The Level 1 feedback paints a clear picture: the Communication as Power workshop is highly relevant, effectively taught, and motivates participants to use the new skills and knowledge acquired during the course. What is the picture three to six months later?

6.4 The Level 3 Impact Evaluation indicates that the manager participants learned most of the communication concepts and have been “consciously” applying them with their own staff. Of the three clusters of skills, Listening, Inquiry, and Feedback, the highest ratings for consciously using the skills were for those in the Feedback cluster. The Listening skills were not being used anywhere near as much, with two skills “using effective body language” and “summarizing the content of a dialogue” both falling below the cut-off of 50 percent. Further, the managers reported that they had changed their behavior in many areas. The one they rated themselves the highest on was that of creating an open learning environment (82 percent). They further reported that the changes in their communication behavior had

Page 26: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 20

affected their direct reports’ behavior, reporting improvement in “self confidence, initiative and drive for results, and innovation and risk taking” all rated in the 70’s. The only facilitating factor the managers gave much credit to (>50 percent) was their own sense of ”personal commitment” (79 percent) and only a little more than a third (38 percent) mentioned too much work or differing priorities as impediments or barriers to making the changes.

6.5 The picture drawn three months later by the participants is that the course was an important and enjoyable learning experience and that the skills and attitudes are consciously being applied. They realistically state that it takes reminding and practicing to change one’s behavior. The main recommendation they make three to six months later is that the course should have a built- in “refresher” course or monthly e-mail reminders. Their attitude is very positive about the management program in general and the communication course in particular: “good communication takes constant effort.”

6.6 The 48 direct reports that responded to the Level 3 Impact Evaluation Questionnaire, also gave the managers fairly high marks for their communication behavior. They rated their managers on “Getting Heard, Being Valued, and Empowered to Learn” fairly highly: all at or above the 50 percent level. They also gave high marks for the managers’ communication behavior improving their Being Motivated, Enhancing the Work Environment, and contributing to their improved self-confidence and successful completion of tasks, among others. Interestingly, while they rated these all positively and all at >50 percent, the managers rated these same items much higher (by 10 or 20 percentage points).

6.7 While none of the open-ended questions on the direct reports’ questionnaire yielded ratings of 50 percent or more, the items listed most frequently are informative. For instance, when asked about what single behavior their manager should increase or change to improve their performance, the top answers to both questions were remarkably consistent: “share more, treat us with trust and respect, value and acknowledge our work,” and “more acknowledgement, recognition, and attention, talk less and listen more.” These answers provide validation for the need for this and other communication courses throughout the Bank. Improving these communication behaviors is critical to improving the staffs’ effectiveness and thus the Bank’s productivity.

6.8 Even though this was not a pre and post situation, the managers’ reports of consciously applying their new skills and attitudes are verified by the direct reports, who as a group, gave their managers’ communication behavior credit for improving their self confidence, initiative, risk taking and successful completion of tasks. It is unfortunate that the number of managers matched to direct reports was too small to yield reliable results of a matched-pair analysis.

RECOMMENDATIONS

• Make the course available to more people. The course is highly rated, motivating, and participants continue to practice what they’ve learned. Many participants suggested

Page 27: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 21

that more managers, indeed all managers, should take the course. Consider spreading it out over four half days (over two to four weeks) to give participants a chance to practice in between sessions. This would also allow for two groups to go through the course simultaneously (one morning, one afternoon). This would provide more people who “know the communication rules” to practice with, to change the way the Bank communicates doing everyday business.

• Investigate why some very basic communication skills of listening, inquiry, and feedback are not being consciously adopted – specific skills such as “positive body language”, “starting with a positive statement” and “summarizing.” Perhaps more practice of these skills in the role plays during the workshop would get these skills above the 50 percent level.

• Add a half-day refresher course at three months and maybe six months. Tape the critical information and send it to those who are too busy to attend in person.

• Include a short session (even 15 minutes) comparing and contrasting several different communication styles. This would validate the styles that many Part II nationalities bring with them. Communication is easiest and most effective among people who know each other well and understand the rules under which they communicate. One of the participants mentioned that she now understood her boss’s feedback much better, but that the “American” style of communication that worked well within the Bank probably wouldn’t work very well back in her country. Encourage this kind of reality check.

• Investigate setting up partnerships for both informal coaching and formal mentoring to increase long term success rates. Since the managers reported that fewer than half of them had support ”from their colleagues, managers, or HR managers” in changing their communication behavior after the workshop, this appears to be a critical issue. Establishing coaching or mentoring partners to provide this kind of support could make the difference in whether or not these skills are internalized, as follow-up and practice are critical to that process. Several participants mentioned that they were using the skills in conjunction with a mentoring program. Both structured mentoring and informal coaching could provide the reminding and reinforcement that participants reported needing.

• Do a pre-post design with participants’ and their direct reports using the same questionnaire both a week before and three to six months after the manager completes the workshop. Then you could attribute changes to what was learned in the workshop and trace how the training affects the manage.

• Vary the above design by training the managers and the managees at approximately the same time, but in different sessions. Positive communication might improve exponentially if everyone in a group learns and practices the rules simultaneously.

In summary, the Communication as Power workshop is an effective training course that could have an even more positive impact on the Bank if the training was (i) linked more specifically to the higher level objectives and (ii) if reminders were sent or refresher sessions for the positive communication behaviors were scheduled as part of the initial course.

Page 28: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 22

Page 29: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 23

ANNEXES

Page 30: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 24

Page 31: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 25

ANNEX A

Communication as Power Level 1: Feedback Evaluation Questionnaire

We would like your assessment of the course quality and your recommendations for improvement.

Please complete this questionnaire. Use a pencil, a black or blue pen to darken the oval like this •. If you wish to change an answer, erase cleanly or if using a pen, place an “x” over the first mark and fill in the oval for your preferred answer. This form is designed to be scored by machine. Make no stray marks of any kind. Your cooperation is appreciated.

I. Course Outcomes: Relevance, Effectiveness, Potential Impact

Low High 1. To what degree is the training relevant to improving your

performance as a manager. ¡1 ¡2 ¡ 3 ¡ 4 ¡5 2. To what degree is the training effective in increasing your

knowledge and skills in the following areas:

a. Knowledge of effective communication ¡1 ¡2 ¡ 3 ¡ 4 ¡5 b. Knowledge of effective decision making ¡1 ¡2 ¡ 3 ¡ 4 ¡5

c. Skills for fostering relationships ¡1 ¡2 ¡ 3 ¡ 4 ¡5 d. Influencing skills for a matrix organization ¡1 ¡2 ¡ 3 ¡ 4 ¡5 e. Listening skills ¡1 ¡2 ¡ 3 ¡ 4 ¡5 f. Inquiry skills ¡1 ¡2 ¡ 3 ¡ 4 ¡5 g. Skills for giving feedback ¡1 ¡2 ¡ 3 ¡ 4 ¡5 h. Knowledge and skills of how to gain more control in

one-on-one communication ¡1 ¡2 ¡ 3 ¡ 4 ¡5 3. To what degree will you now use the knowledge or skills you

have acquired. ¡1 ¡2 ¡ 3 ¡ 4 ¡5 4. Of the skills you acquired, which ones would be most significant for your use at work?

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

Page 32: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 26

5. What type of support would you need in using these skills at work?

_____________________________________________________________________________

_____________________________________________________________________________

_____________________________________________________________________________

II. Quality of Course Design and Delivery Poor Excellent

6. How would you rate the quality of the design and delivery of the course? ¡1 ¡2 ¡ 3 ¡ 4 ¡5

a. The plenary presentations ¡1 ¡2 ¡ 3 ¡ 4 ¡5 b. The facilitation of the small group sessions ¡1 ¡2 ¡ 3 ¡ 4 ¡5 c. The relevance of the materials used in the case studies ¡1 ¡2 ¡ 3 ¡ 4 ¡5

d. The depth of coverage of the course content ¡1 ¡2 ¡ 3 ¡ 4 ¡5 e. The opportunity to practice the skills of the training ¡1 ¡2 ¡ 3 ¡ 4 ¡5

Too Slow Adequate Too Fast f. The pacing of the delivery ¡ l ¡ l ¡ l

III. Overall Comments and Recommendations

7. What did you find most useful about the course?

_____________________________________________________________________________

_____________________________________________________________________________

_____________________________________________________________________________

8. What did you find least useful about the course?

_____________________________________________________________________________

_____________________________________________________________________________

_____________________________________________________________________________

9. What should be done differently next time?

_____________________________________________________________________________

_____________________________________________________________________________

_____________________________________________________________________________

10. Other comments and recommendations for the course.

_____________________________________________________________________________

_____________________________________________________________________________

Page 33: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 27

ANNEX B

Communication as Power Impact Evaluation

Form A: Questionnaire for Managers and Course Participants

1. You are: 2. Gender O First time Manager in the World Bank O Male O Manager in a new unit O Female O New externally recruited manager O First time Director O VP O Other: _________________ 3. Grade Level 4. National:

O GG O GH

O Part I O Part II

O GI O GJ O Other: _________________

5. VPU : _______________________

6. Since your participation in this course, to what degree have you consciously used

the following communication skills? Please rate the degree of use on the scale provided. If you have not used the skill, please indicate “Not Used”.

Not

used Low degree

High degree

Listening

Use effective body language ? ? ? ? ? ?

Use eye contact well ? ? ? ? ? ?

Use silences/ pauses constructively ? ? ? ? ? ?

Summarize the content of dialogue ? ? ? ? ? ?

Reflect on and verbalize feelings ? ? ? ? ? ?

Inquiring

Use one question at a time ? ? ? ? ? ?

Use open-ended questions ? ? ? ? ? ? Reinforce answers

Question assumptions ? ? ? ? ? ?

Page 34: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 28

Shape the conversation ? ? ? ? ? ? Feedback

Start with a positive message ? ? ? ? ? ?

Describe problem and consequences (not the person) ? ? ? ? ? ?

Give clear and specific messages ? ? ? ? ? ?

Focus on actionable behavior ? ? ? ? ? ?

Seek commitment ? ? ? ? ? ? 7. As a result of the course, to what degree has your behavior changed in the following

areas? Low

degree High degree

a. Valuing differences and using them to maximize the quality of the work ? ? ? ? ?

b. Increasing tolerance of the differences among staff ? ? ? ? ?

c. Empowering staff ? ? ? ? ?

d. Creating an open learning environment ? ? ? ? ?

e. Inspiring trust ? ? ? ? ?

f. Other effects __________________________________ ? ? ? ? ? 8. To what degree has the change in your behavior led to improvements in the behavior and

performance of your staff in the following areas? Low

degree High degree

a. Self-confidence ? ? ? ? ?

b. Initiative and drive for results ? ? ? ? ?

c. Innovation and risk taking ? ? ? ? ?

d. Successful completion of tasks ? ? ? ? ? e. Other effects ________________________________ ? ? ? ? ?

Page 35: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 29

9. To what degree did each of the following facilitate or enhance your behavior change?

Low degree

High degree

a. Your personal commitment to work effectively through others ? ? ? ? ?

b. Your staff readiness to be empowered ? ? ? ? ?

c. Support from colleagues in the Bank ? ? ? ? ?

d. Support from others who attended the course with you ? ? ? ? ?

e. Support from your Supervising manager ? ? ? ? ?

f. Support from your HR Manager and Officer ? ? ? ? ? g. Follow-up coaching or consultations provided by the training team ? ? ? ? ?

h. Coaching provided via the Individual Coaching Initiative ? ? ? ? ?

i. Peer support provided via the HRSLO Peer learning programs ? ? ? ? ?

j. Mentoring from an assigned mentor ? ? ? ? ? k. Other facilitators (Please describe) ___________________________________________ ? ? ? ? ? 10. To what degree did the following act as barriers to your behavior change?

Low

degree High degree

a. Lack of understanding of how to use the knowledge and skills of the training ? ? ? ? ? b. The heavy work load which limited the time for me to focus on and use knowledge and skills ? ? ? ? ?

c. Other priorities I established with my Manager ? ? ? ? ? d. The knowledge and skills of the training not being appropriate in my situation ? ? ? ? ?

e. Lack of a safe environment for practicing skills ? ? ? ? ? f. Other barriers (Please describe) ___________________________________________ ? ? ? ? ? 11. What are your recommendations for improving The Communications as Power

workshop

12. What can you do to reinforce the learning that you had from this workshop?

Page 36: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 30

13. What other recommendations do you have for improving management development at the Bank?

14. Other comments.

Thank You

Page 37: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 31

ANNEX C

Communication as Power Impact Evaluation

Form B: Questionnaire for Staff 1. O Part I National 2. Grade Level: O GA-GD 3. Your Gender: O Female O Part II National O GE O Male O GF O GG O GH O GI 4. Length of time you have been reporting to the Manager:____________________________ Over the past 3 to 6 months, to what degree do the following describe your manager’s behaviors?

Low High Degree Degree

1 2 3 4 5 Getting Heard - My Manager: Is fully attentive when I speak to him/her. O O O O O Promotes open and candid communications. O O O O O Invites my comments and feedback. O O O O O Summarizes content of conversation to ensure O O O O O understanding. Remembers substance of our last conversation. O O O O O Being Valued - My Manager: Respects my opinions. O O O O O Acknowledges my contributions. O O O O O Respects my views even when we disagree. O O O O O Takes an interest in my professional growth. O O O O O Empowered to Learned – My Manager: Clearly communicates his/her expectations and tasks. O O O O O Provides clear standards of interpersonal behaviors. O O O O O Provides specific guidance/feedback for improvement. O O O O O Recognizes mistakes as learning opportunities. O O O O O Fosters an environment of innovation. O O O O O Clearly communicates priorities O O O O O Being Motivated – My Manager: Uses feedback to motivate me to take actions O O O O O Seeks to achieve mutual agreement O O O O O

Page 38: Impact Evaluation of Communication as Power: Listening ...documents.worldbank.org/curated/en/... · The findings, recommendations, and implications of this impact evaluation were

Impact Evaluation of Communication as Power Page 32

Low High Degree Degree 1 2 3 4 5 Takes time to nurture my commitment. O O O O O Empowers me by removing obstacles. O O O O O Encourages desirable behaviors from me. O O O O O 6. Over the past 3 to 6 months, to what degree has your manager’s behaviors enhanced the following aspects of the work environment of your unit?

a) Level of trust O O O O O b) Level of commitment O O O O O c) The valuing of differences O O O O O d) Tolerance among staff O O O O O e) The feeling of empower among staff O O O O O f) An open learning environment O O O O O g) Other effects: _______________________________ O O O O O

7. Over the past 3 to 6 months, to what degree has your manager’s behavior improved your:

a) Self-confidence O O O O O b) Initiative and drive for results O O O O O c) Innovation and risk taking O O O O O d) Successful completion of tasks O O O O O e) Other effects: _____________________________ O O O O O

What is the single behavior that your manager should make more use of to support your performance and productivity? What is the single communication behavior that your manager should change to enhance your performance and productivity? What is the single behavior that you should make more use of to support your manager in becoming more effective? Do you have any other comments about areas where your manager’s behaviors needs to be improved?