make or break customer satisfaction

PROPRIETARY AND CONFIDENTIAL, MARITZ COPYRIGHT 2010

Make or Break Customer Satisfaction

Improving Customer Satisfaction Measurement With New Methods

Keith Chrzan

Chief Research Officer, Maritz Research

1


Current practice for customer satisfaction modeling

• “Driver analysis” using linear, compensatory customer satisfaction models (regression, correlation, PLS, SEM)– Each attribute has an importance weight

– Sum of attribute importances times their respective performance scores reflects overall satisfaction

– When done well, we account for the multicollinearity that’s pervasive in customer satisfaction research (e.g. Theil’sinformation-theoretic averaging-over-orderings regression model)

• Entered the marketing and economics fields in the 1950s and 1960s and never left

• Fits well with statistical models

• But how realistic is this model?

2


A visit to <NAME REMOVED> hotel

• Accurate reservation

• Quick check in process

• Nice room, upgraded bath

• Larger TV than most movie theater screens

• Easy internet access

• Excellent workout room

• Tasty room service

• Comfortable bed, neatly folded

3


A visit to <NAME REMOVED> hotel

• Accurate reservation

• Quick check in process

• Nice room, upgraded bath

• Larger TV than most movie theater screens

• Easy internet access

• Excellent workout room

• Tasty room service

• Comfortable bed, neatly folded with dead cockroaches between the sheets

4


Non-compensatory effects

• Sometimes performance on an attribute is so bad that, all by itself, it causes dissatisfaction– It doesn’t matter how well the brand performs on other attributes,

poor performance on just one ruins the entire experience

– This is a “non-compensatory” effect because adding the good effects still can’t overcome the effect of the poor performance on overall satisfaction

• Of course the opposite is also possible – great performance on one attribute outweighs shortcomings on others

• Standard regression models miss these kinds of effects


A non-compensatory model

• Joffre Swait (1997) developed a questionnaire method that allowed analysts to incorporate non-compensatory effects to conjoint modeling– His method created better-fitting models, models that explain

brand choice better

– His models generated additional insights not otherwise available• Which are “must have” attribute levels and for which and how many

people?

• Which are dealbreakers for which and how many people?

– Within three years, however, the advent of HB analysis made thismodel obsolete, for conjoint studies


A non-compensatory customer satisfaction model

• Adapting Swait’s approach to customer satisfaction modeling creates a non-compensatory model featuring rewards and penalties

• The result is Make or Break customer satisfaction


Test case

• Online survey, May 2010

• 599 respondents rating a recent trip to a mobile phone retail store

• Overall satisfaction

• Nine attributes identified as drivers of retailer satisfaction in qualitative research

9


Sample design

• Two cells – Control cell (n=305): Standard customer sat survey

– Test cell (n=294): Penalty/Reward approach


Questionnaire Outline

• First respondents rate their overall satisfaction

• We ask of satisfied respondents – If any attributes were so wonderful that, all by themselves, they

made the experience great

– We use a checklist, to make this easy for respondents

• We ask dissatisfied respondents to check any attributes that were so terrible as to ruin, by themselves, the overall experience

• We ask respondents to rate only the attributes not checked above

• Thus there are no additional keystrokes required from respondents


Questionnaire

Q1. Please indicate how satisfied you were with your mobile phone shopping experience at <INSERT MOBILE PHONE RETAIL STORE> using the scale below. [ ] Completely satisfied [ ] Somewhat satisfied [ ] Neither satisfied nor dissatisfied [ ] Somewhat dissatisfied [ ] Completely dissatisfied

Q2c. ASK IF Q1< 3. Was the performance on any of these aspects so good as, all by itself, to make your overall mobile phone shopping experience satisfactory?

Q2d. ASK IF Q1> 3. Was the performance on any of these aspects so bad as, all by itself, to make your overall mobile phone shopping experience satisfactory?

Aspect Yes (for the good) Yes (for the bad) Store location [ ] [ ]Speed of service [ ] [ ]Friendliness of sales representative [ ] [ ]Professionalism sales representative [ ] [ ]Information the sales representative had for me [ ] [ ]Phone prices [ ] [ ]Network coverage [ ] [ ]Availability of the phone I wanted [ ] [ ]Price of the calling plans [ ] [ ]None of these CANNOT BE CHOSEN WITH ANY OTHER [ ] [ ]

Q2e.

Strongly Agree Agree

Neither agree nor disagree Disagree

Strongly Disagree

The store was conveniently located [ ] [ ] [ ] [ ] [ ]

I was waited on quickly [ ] [ ] [ ] [ ] [ ]

The sales representative was friendly [ ] [ ] [ ] [ ] [ ]

The sales representative was professional [ ] [ ] [ ] [ ] [ ]

The sales representative had the information I needed [ ] [ ] [ ] [ ] [ ]

The phones were reasonably priced [ ] [ ] [ ] [ ] [ ]

The network has adequate coverage [ ] [ ] [ ] [ ] [ ]

The phone I wanted was available [ ] [ ] [ ] [ ] [ ]

The calling plans were reasonably priced [ ] [ ] [ ] [ ] [ ]

Please indicate how much you agree or disagree with each of the following statements about your mobile phone shopping experience at <MOBILE PHONE RETAIL STORE>. SHOW ONLY ATTRIBUTES NOT CHECKED IN 2C OR 2D. RANDOMIZE ORDER.


Results

• We get these results– Basic coefficients (weights) for each attribute

– An additional bonus weight for those people saying each attribute was wonderful and made their experience great

– An additional penalty, a negative weight, that detracts from theoverall rating for those people reporting attributes that ruined their experience

– Patterns of which attributes were particularly wonderful or terrible vary

• Not all respondents get the same attribute weights

• The model accommodates respondent heterogeneity

• This test case study uses regression analysis and shows the statistically significant attribute coefficients

13


Standard regression model

14

Attribute CoefficientRep had info I needed -My phone was available -Price of phone .13Price of plan -Coverage .15Rep friendly .38Rep professionalism -Quick service -


Base of Make or Break model

15

Attribute CoefficientRep had info I needed .18My phone was available .07Price of phone .15Price of plan

Coverage

Rep friendly

Rep professionalism

Quick service


Adding in penalties

16

Attribute Coefficient

% reporting attribute

ruined the overall

experience PenaltyRep had info I needed .18 5 -.40My phone was available .07 5 -.68Price of phone .15Price of plan 4 -.48Coverage 1 -.64Rep friendly

Rep professionalism

Quick service 6 -.96


Topping it off with gains

17

Attribute Coefficient


ruined the overall

experience Penalty

% reporting attribute perfected the overall experience Reward

Rep had info I needed .18 5 -.40My phone was available .07 5 -.68 31 .26Price of phone .15Price of plan 4 -.48Coverage 1 -.64 30 .25Rep friendly 39 .22Rep professionalism 34 .17Quick service 6 -.96


Correcting for multicollinearity

• Using Theil’s model (True Driver Analysis) shows the impact of all the penalties and gains, taking into account shared variance among the scale questions, the penalties and the rewards

18

Scale Questions 42%

Reward 32%

Penalty 26%

-10% -5% 0% 5% 10%

Represenative - hadinfo I needed

Representative -professional

Representative -friendly

Waited on quickly

Phone i wanted wasavailable

Phones werereasonably priced

Calling plans werereasonably priced

Network hasadequate coverage

Conveniently located


Evaluation

• The non-compensatory Make or Break model yields additional insights

• The model incorporates respondent heterogeneity– Different respondents can have different patterns of penalties and

rewards

– In this case, 47 distinct patterns

• The model VASTLY improves prediction– Control cell with standard customer sat questions: R2 = 30%

– Above plus non-compensatory penalties/rewards for ruining/making my experience: R2 = 65%

19


Maybe that was too easy?

• Experiences at mobile phone retailers vary quite a bit (long waits for service, phone availability, etc.)

• How does the model perform when most respondents are happy with their experiences?

• Will a shortage of penalties allow the model enough to work with?

• Let’s try retail banking

20


Case studies 2-6

• Web surveys fielded October 2010

• Control groups doing standard ratings and test cells identifyingnon-compensatory penalty/rewards

• Five surveys of banking satisfaction– Branch satisfaction

– ATM satisfaction

– Call center satisfaction

– Customer service representative satisfaction

– Online banking satisfaction

• Overall satisfaction and 3-11 attributes, depending on study

21


Sample sizes – case studies 2-6

22

Aspect Test SampleControl Sample

Branch 395 377

ATM 367 369

Call center 113 128

CSR 181 180

Online 363 363


Bank branch satisfaction model

23

AttributeStandard

regression Base


ruined the overall

experience Penalty


Wait time in line .15 .10 2 -1.42

Staff courteous 41 .27Speed of completing request

.19 .17

Staff knowledgeable .28 .22

Staff provides accurate answers

.18 .17 1 -.40

R2 .62 .68 .70

• Significant effects for 4/10 attributes

• Model improves with addition of penalties and then again with addition of rewards


ATM satisfaction model

24

AttributeStandard

regression Base


ruined the overall

experience Penalty


Safe and secure .14 .14 71 .11

Ease of transaction .37 .19 2 -.99 42 .36Wait time .13 .13 1 -1.61

R2 .51 .61 .64

• Significant effects for all three attributes

• Model improves with addition of penalties and rewards

• Service failures are uncommon and catastrophic – 1.61 points on a 5 point scale!


Automated call satisfaction model

25

AttributeStandard

regression Base


ruined the overall

experience Penalty


Easy to get live rep 8 -.53

Easy to navigate .31 .13Useful response options

.53 .42 5 -.72

Reasonable hold time

4 -.85 37 .50

R2 .74 .81 .84

• Significant effects for all four attributes

• Model improves with addition of penalties and of rewards

• Service failures are more common

• Captures eight different varieties of customer experience


Phone rep satisfaction model

26

AttributeStandard

regression Base


ruined the overall

experience Penalty


Authority to address your issue

.22 .18 1 -.49

Explains things clearly

2 -.80

Takes responsibility to resolve your issue

.28 .22 4 -.58

Handles call quickly 43 .31

Provides complete answers

.27 .19

R2 .71 .75 .77


• Three significant and injurious penalties

• Improved model fit


Online banking satisfaction model

27

AttributeStandard

regression Base


ruined the overall

experience Penalty


Safe and secure 1 -1.45

Easy to navigate .21 .19Easy to complete tasks

.15 .14 50 .17

Able to conduct desired transactions

.19 .13 1 -1.22

Helps you manage your finances

.11 .11

R2 .57 .61 .62


• Rewards are less good than penalties are bad (but more common)

• Improved model fit


Location of penalty/reward questions

• If before attribute ratings, as in Study 1, an adaptive survey flow can keep the amount of respondent effort (in terms of number of keystrokes) the same as standard customer satisfaction studies

• If after, we require additional work from respondents

• We split respondents, half with penalty/reward questions before and half with after

• Similar models, same R2 either way

• Adaptive set-up doesn’t seem to hurt the resulting data

28


Summary

• In all six cases tested, the Make or Break model significantly outperforms the standard customer satisfaction ratings measurements– Higher R2 – explains more variance on overall satisfaction

– Additional insight about non-linear penalties and boosts for excellent/poor performance

– Additional insight about respondent heterogeneity

• These benefits remain even if we only use the penalties part of the model

29

make or break customer satisfaction

Documents

process nice

rewards service

attributes

penaltyreward

sales representative

calling plans

compensatory

experience