using surveys to support od projects

Using Surveys to Support OD projects

You are free to copy and distribute the work under the following conditions: you must attribute the work to “Mick Verran [email protected]” ; you may not use the work for commercial purposes; if you alter, transform, or build upon this work, you may distribute the resulting work only under the same or similar license to this one.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.

Mick Verran for Boston Facilitators Roundtable

02-07-2014

Handout www.MickVerran.com Password BFR

mailto:[email protected]

http://creativecommons.org/licenses/by-nc-sa/3.0/deed.en_US







http://www.mickverran.com/


Issues in gathering data

Sensitivities about the role of an outside ‘expert’Impact of the intervention itself

Barriers to participation – distribution, trustChallenges in balancing the loudest voices

Availability & accessibility of usable dataWhole or representative population required

Degree of accuracy required

Funding & time available

2 Using Surveys to Support OD Projects Mick Verran

Data-gathering methods


Pros Cons

Observation, Test, Simulation

High face-value.Allows for probing & clarification.Observe hidden dynamics. More accurate than e.g. recall. Comparability.

Skilled, knowledgeable observer.Time-consuming, high $.

Personal Interview

Follow-up to nuance data.Unexpected information.

> Quality < Quantity. Time consuming.Analysis and comparisons harder.

Focus Group Interaction between viewpoints.Buy-in.

Time-consuming.Quantifying data.Skillful facilitator.

(Electronic) Survey

Low $.Reach large or distributed population.Easy for respondents.Easy to nudge completion.Higher response than paper.Easy data analysis & reporting.

Requires internet access.No opportunity for clarifying or probing.(Rare) concerns about anonymity.

Survey tools

Google

Constant Contact

SearchGizmo

Capricity

Qualtronics – QuestionPro – Vertical Response

‘… where I do not speak the language’http://qualtrics.com/ https://www.questionpro.com/

‘Analysis, vendors who specialize in analysis and reporting, current tools and trends, live poling, web-based live poling’www.DiscoverSurveys.com – Bruce Katcher


http://qualtrics.com/

https://www.questionpro.com/

http://www.discoversurveys.com/

‘How others are using surveys’


Download Handout 1

‘Other ways to leverage surveys’


Enter, Scope

Initiate, Analyze

Design, Plan

Implement

Sustain, Exit

M O V E

Motivation

Ownership

Voice Expectation

(Role = moving the process along)

Download Handout 2

‘What makes for a good survey?’

Everyone concerned agrees that the survey asks

an important question that each respondent you

need to hear from will understand, know their

answer to, and be willing and able to answer, to

produce information you can reliably interpret and

use.


Online survey workload

Activity

Agree survey goals, measures, population, design considerations, follow-up actions, scheduleDraft questions, intro & nudge emails, example reportReview with client

Create online survey

Launch

Monitor & nudge

Draft results

Review results with client

Finalize report of results


Additional Workload

-

2 hours

-

< 1 hour

-

< 1 hour

1 hour

-

< 1 hour

Activity Additional Workload Watch-out for ….

Agree survey goals, measures, population, design considerations, follow-up actions, schedule

- • Vague follow-up actions• Push for demographics• Shift patterns

Draft questions, intro & nudge emails, example report 2 hours

Review with client - • ‘Scope creep’

Create online survey < 1 hour

Launch -

Monitor & nudge < 1 hour

Draft results 1 hour

Review results with client - • Heavy-handed editing• Vague follow-up actions

Finalize report of results < 1 hour

Needs of interested parties1. Inform the content of what you’re doing

2. Move the process along

Client• Validation• Efficiency • > Participation• Temperature check

Survey Participants• A voice• Understand what the others involved think.

Consultant• Reach around the client to uncover hidden issues• Platform for success• Attune to the language and issues


Applications – Goals


Download Handout 3

Response rates

Online, ‘out of the blue’ market research = 30% averageOur pre-survey = 27/30 = 90%

What impacts the response rate?• Connected with issues of importance to the respondents.• Promise of report back on results. • Credible commitment of the sponsor to address issues.• Confidence in confidentiality.• Accommodation of e.g. shift schedules.• Easy-to-answer.

– Questions & scales– Goal; 20 questions or 5 minutes (5 closed or 2 open per minute)


Choosing respondentsWho to ask?Stakeholders – Knowledgeable – Opinion Leaders

‘Gathering data from a large sample…’http://www.calculator.net/sample-size-calculator.html

‘Surveying groups for the same project with variants of the survey’Separate surveys – no value in complexities of creating too much conditional/skip logic/branching/piping


http://www.calculator.net/sample-size-calculator.html

http://www.calculator.net/sample-size-calculator.html

Employee EngagementExperiencesWhich ‘drivers of engagement’ do you experience?CommitmentHow committed are you to our new strategy?ConfidenceHow confident are you that we will successfully execute our new strategy?ConcernsWhat concerns do you have about our new strategy?What concerns do you have about our ability to successfully execute our new strategy?


Organization/Team EffectivenessExperienceHow often does the team ……?EvaluationHow well does the team do ……?SatisfactionHow satisfied are you with the way the team ………?Opportunities for improvementImprovements in which of the following would have a significant impact on the team’s effectiveness?


Training Needs AnalysisRelevanceHow often do you use this skill at work? ImportanceHow important is this skill to success in your work?CompetenceHow skilled are you in ……?Development NeedI need to improve my ability to ….Commitment to ImproveHow likely are you to attend ….?


Applications – Measures


Download Handout 3

Survey sections

1. Why doing survey, benefits, how results will be used

2. Time commitment, when required3. Contacts – content AND completion4. Call for Honesty – use full scale, specific examples5. Confidentiality framing6. Demographic data7. Questions8. Open ended comments 9. Thanks


‘Collecting demographic info’

Only those that are vital to the achievement of your objectivesNot so narrow that one can guessOptional space for self-identification if open to follow-up SurveyMonkey tracks IP address; individual tracking requires unique survey

Positioning Demographic questionsAt the end of the survey, when respondents have developed confidence in the survey and it’s objective

OR

Early, straight after the reassurance of anonymity, so don’t feel cheated if asked at the end


So many options!How many times in the last (period of time) have you bought a Coca-Cola?01 – 56 – 12More than 13How much do you enjoy Coca-Cola?I enjoy Coca-ColaI neither enjoy nor do not enjoy Coca-Cola I do not enjoy Coca-Cola

Given a choice, which would you choose?Coca-Cola PepsiI prefer Coca-Cola to Pepsi; Strongly disagreeDisagreeNeither agree nor disagreeAgreeStrongly agree If you were to buy 10 soft drinks, how many of each would you buy?Coca-Cola Pepsi

I have chosen Coca-Cola over Pepsi;AlwaysOftenAbout half and halfRarelyNever I enjoy Coca-Cola;More than I enjoy PepsiAs much as I enjoy PepsiLess than I enjoy PepsiPut these soft drinks in order of your preference; Coca-Cola PepsiMountain DewAre you willing to; Drink Coca-ColaBuy Coca-ColaOffer Coca-Cola to your guestsBuy Coca-Cola for your children Consult to Coca-Cola Inc.Invest in Coca-Cola Inc.Work for Coca-Cola Inc.?


Question types


Download Handout 4

Respondent TendenciesAgree with statements as presented.Respond from general opinion of positives & negatives.

Have own definition of e.g. ‘frequently’ v. ‘occasionally’.Avoid using extreme response categories/poles.“I Tend to Disagree” is more negative than “I Tend to Agree” is positive.

Portray selves or group in a more favorable light when asked about; • Feelings of powerlessness, "kindness" or "benevolence“, honesty,

willingness to contribute.• Ability to do job, stress levels, company loyalty, value of own

contribution, compliance with procedures.


How the question and the scale work togetherHow important are the following to the success of a restaurant?

Very Unimportant

Somewhat Unimportant

Neither Important Nor Unimportant

Somewhat Important Very Important

ServiceParkingCost


How important are the following in attracting families to a restaurant?

Very Unimportant

1 2 3 4Very Important

5

Quality of ServiceEase of ParkingValue for MoneyI take the following into account when choosing a restaurant for a family outing;

I Strongly Disagree I Mostly Disagree I Neither Agree

Nor Disagree I Mostly Agree I Strongly Agree

Quality of ServiceEase of ParkingValue for Money

‘Preventing bias’ – ‘Examples of what not to do’

Reliability – Problem questions & scales


Download Handout 5

Anchored rating scales

Process X produces an acceptable quality output.Strongly AgreeSomewhat AgreeNeither Agree nor DisagreeSomewhat DisagreeStrongly Disagree


Effectively examine events issues, problems and generate solution alternatives.

Strongly Agree

Somewhat Agree

Neither Agree nor Disagree

Somewhat Disagree

Strongly Disagree

Behaviorally-anchoredEffectively examine events issues, problems and generate solution alternatives

Strongly Agree 1 Makes sound decisions on everyday issues and problems.

Somewhat Agree 2

Neither Agree nor Disagree 3 Makes sound decisions on complex functional issues and problems.

Somewhat Disagree 4

Strongly Disagree 5 Makes decisions that take into account broad, system-wide implications

Anchored thru question AND scaleThe output of Process X compared with our competitors is;Far Above AverageSomewhat Above AverageAverageSomewhat Below AverageFar Below Average

Rating Scale Template – ‘gold standard’


Download Handout 6

Favorite questions & scales


Download Handout 7

Questions about scales 1How many points should there be on the scale?• How many degrees of difference can respondents discriminate?• How many responses will give rise to different actions?

Should I include a mid-point?• Use is heavily influenced by “social desirability bias”.• “… a four point scale without a mid-point appears to push more respondents

towards the positive end of the scale.”• (a four point scale) “… can … (force) subjects to indicate their opinion when

they actually have no opinion and are not simply reluctant to reveal it.”• “This study found the opposite … the way in which people will respond to a

balanced Likert type scale without a mid-point is content-specific.”• (To minimize the use of the mid-point) “… either not include it at all or use

scales with many points.”


Questions about scales 2Should I start with the “agree” or “disagree” options?• Some evidence of a small bias towards left-side options.• No evidence of significant bias whether start with positive option or

negative in a survey (different when face-to-face due to “social desirability bias”).

Should I give people the option to NOT respond to a question?• I Don’t Know – I Have No Evidence – I Have No Experience – I Have No

Opinion.• SurveyMonkey: “By allowing respondents to opt out due to non-familiarity

with the question or if it does not apply to their situation, this will help increase the response rate and quality of the collected data.”

• Exclude the question if too many respondents will be unable to respond. • How will high “don’t know” responses impact your interpretation of the

results?• Omit if it’s impossible NOT to have an opinion on or experience of the issue.


Sequencing the questions“Respondents asked to answer questions dealing with fraud and waste in government programs, were subsequently MORE likely to oppose welfare programs than those who were asked the questions in the reverse order.”Meeting Review • Content we covered• Actions you will take • What’s left to be done• (Logistics)


Team Assessment• Goal clarity • Results• Roles &

responsibilities• Goal clarity

• Processes • Roles & responsibilities• Resources • Processes• Results • Resources

Ok, …. one ‘Tip on using SurveyMonkey’

To allow multiple entries from same computer or IP address

1. Create the survey2. Click ‘Collect Responses’3. Select ‘Web Link’ & click ‘Next Step’4. Click ‘Change Settings’5. Under ‘Allow Multiple Responses’ click ‘Yes’6. Click ‘Save Settings’


Deployment schedule (time zones, shift patterns)

You Sponsoring ManagerLaunch -2 days - Heads-up

Day 1 a.m. - Distribute link; 3-day deadline

Day 2 a.m. Send manager x/n - Day 2 p.m. - NudgeDay 3 a.m. Send manager x/n - Day 3 p.m. - Extend deadline by one dayDay 4 a.m. Send manager x/n - Day 4 noon - Count down to deadline

Day 5 a.m.Send manager x/n(Leave open for stragglers)

Thank you

Day 5 p.m. Draft report -


Download Handout 8

Analyzing responses

‘Evaluating data based on … demographics’• SurveyMonkey filters

What to do about missing responses• What response rate & coverage is critical to the

goals?


‘Communicating results to the event sponsor’ ‘Improving the presentation of data’

http://www.edwardtufte.comBoston, March 11–14 $380

http://www.edwardtufte.com/

http://www.edwardtufte.com/

Reporting principles

Separate results & interpretation.

Low response; how representative are the responses you got?

‘Use-by’ date on the report?

Summarize themes in open-ended comments.Respondents’ own words – including typos.

.pptx for the report, .docx for the record.Present all the data at end in a Table.


Means, standard deviations, confidence interval

A = 2.60 : B = 2.95 : 5 point scale : 12 respondents

Very Strongly Agree is not 20% > Mostly Agree

“I can do that for you, but I’m afraid I’ll have to add $nx2 to my bill”.


Presenting the results

Responses

Strongly Agree

Mostly Agree

Mostly Disagree

Strongly Disagree

Q1Q2Q3Q4Q5Q6Q7Q8Q9


Q Mean/4.00

Q1 2.50Q2 2.50Q3 2.50Q4 2.75Q5 2.75Q6 3.00Q7 3.00Q8 3.25Q9 3.25

Number of Responses to each Rating Option

4 Strongly

Agree

3 Mostly Agree

2 Mostly Disagre

e

1 Strongly Disagre

e

0 2 2 02 0 0 21 1 1 10 3 1 01 2 0 10 4 0 02 1 0 11 3 0 03 0 0 1

Format options

All responses• Bar chart

% responses• Pie chart

Changes over time• Line graph• Side-by-side bar chart

Comments• Verbatim• Themes• Word frequency

www.Wordle.net

You can struggle with Excel & CSv or PDFs or …..Manual Templates!


http://www.wordle.net/

Priorities for a new CEOBusiness Development

Creating Bank’s vision for growth

Generating new sources of revenue

Capturing new customers

Differentiating Bank through customer service & marketing

Operations Management

Protecting capital & asset quality

Managing the regulatory process

Restructuring operating costs

Generating capital for investment

Organization Leadership

Developing an effective executive team

Attracting & developing leadership talent

Maintaining a healthy culture

External Relations

Engaging in the political process

Connecting with the local community

Retaining the confidence of the Board


Hospital Housekeeping1 2 3 4

N/CommentI Strongly Disagree I Somewhat Disagree I Somewhat Agree I Strongly Agree

Housekeeping staff adhere to all health and safety standards.

5 11 10 1 1

17% 39% 36% 4% 4%

Housekeeping staff act appropriately toward patients and visitors.

4 1 17 4 2

14% 4% 61% 14% 7%

Housekeeping staff interact collaboratively with medical staff.

5 5 16 0 2

18% 18% 57% 0% 7%

Housekeeping staff respond flexibly to changing needs.

8 14 5 0 1

28% 50% 18% 0% 4%

Housekeeping Supervisors are responsive to Unit needs.

6 12 10 0 0

21% 43% 36% 0% 0%

Housekeeping staff respond in a timely manner to urgent needs.

9 14 5 0 0

32% 50% 18% 0% 0%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

... adhere to allhealth and safety

standards.

... act appropriatelytoward patients

and visitors.

... interactcollaboratively with

medical staff.

... respond flexiblyto changing needs.

... respond in atimely manner to

urgent needs.

No OpinionI Strongly AgreeI Somewhat AgreeI Somewhat DisagreeI Strongly Disagree


Team surveys


Team Effectiveness

Initiative Implementation

Team survey with leader’s response


360 feedback


Repeat 360


January April July

Other

s

4

Other

s

4

Others

4

3 3 3

2 2 2

1 1 1

N R O F N R O F N R O F

January April July

Other

s

4

Other

s

4

Others

4

3 3 3

2 2 2

1 1 1

N R O F N R O F N R O F

1 Making condescending or dismissive statements N = Never R = Rarely O = Often F = Frequently

2 Talking over and/or interrupting people N = Never R = Rarely O = Often F = Frequently

‘How to summarize open-ended data’ 1

Read through once

List the themes

Cut ‘n’ paste sentences to themes

Cluster overlapping themes

Editing decisions; • Every theme? • Every sentence, or representative quotes?• Edit easily-identified sources?• Omit obvious grudges?• Edit same point by same person over and over?• Correct spelling, grammar?


Strengths• Leadership Team members step up & take a leadership role when required.• Leadership Team meetings & conference calls are effective. • Leadership Team members collaborate with & support each other.• Leadership Team members give & receive feedback without becoming

defensive or aggressive.• XX is willing to listen & be influenced, & offers significant support.• Leadership Team members are optimistic about the team's future & ability

to accomplish it's mission.

Noteworthy Limitations• Team goals & responsibilities are not as clear as individual assignments.• Team capabilities and resources are a concern.



www.Wordle.net 46 Using Surveys to Support OD Projects Mick Verran


http://www.wordle.net/

Event reviewsGoalsAssess achievement of objectivesIsolate impact of content – process – presenterAssess commitment to ActionsIdentify need for further support & ‘next agenda’ItemsPre-work relevance & use Content RelevancePresenters: Relevance of examples – Responses to questions –Interactions Clarity of action planOutstanding issuesPersonal commitment to changeOngoing support requirements


Handout www.MickVerran.com Password BFR

[email protected]

978-505-5275

Thank you.

You are free to copy and distribute the work under the following conditions: you must attribute the work to “Mick Verran [email protected]” ; you may not use the work for commercial purposes; if you alter, transform, or build upon this work, you may distribute the resulting work only under the same or similar license to this one.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.

http://www.mickverran.com/











using surveys to support od projects

Documents

od projects mick verrandata

od projects mick verran4how

od projectsyou

good survey

resulting work

easy data analysis reporting

representative population

quantifying data