knappen smartid presentation_final
TRANSCRIPT
May 2015
Heather Knappen, MSResearch Analyst
Metrix Matrix Inc.
Founded in 1997.
Located in Upstate NY.
Extensive expertise and knowledge in survey design, administration, analysis and data submission.
National survey work includes B2B, B2C, government, medical, telecommunications and utility projects.
MetrixMatrix.com (888) 865-0065 2
MetrixMatrix.com (888) 865-0065 3
In 2012, Metrix Matrix implemented the SmartID method by which outbound caller ID area codes were matched to target respondents’ area codes.
This SmartID method was expected to improve efficiency and survey completion rates.
In a 2014 poster presentation to AAPOR, preliminary research revealed that projects conducted with SmartID were less likely to result in a completed survey.
MetrixMatrix.com (888) 865-0065 4
Can this method increase response rates within an individual project when respondents receive calls that are associated with local (i.e., familiar) area codes?
And…
Does the use of SmartID reduce cost(s) by improving performance?
Looking more deeply into the before-and-after effects of SmartID…
MetrixMatrix.com (888) 865-0065 5
Utility Authority within the State of Oklahoma
Residential customers were asked about satisfaction with the utility service/pricing
Telephone survey, 38 questions
Wave 1: 9/5/2014-10/7/2014 SmartID was not used Records were attempted up to 3 times
Wave 2: 10/8/2014- 11/5/2014 SmartID was used in the 405 area code only Records with 3 attempts (from Wave 1) were re-called, with no
more than 2 additional attempts New phone records were added
MetrixMatrix.com (888) 865-0065 6
Region Name Area CodeTarget Sample
CompletedSurveys
(Wave 1)
% CompletedSurveys /Target
(Wave 1)
South West 580 350 319 91%
South Central 405, 580 350 255 73%
Central405, 580, 918/539
450 188 42%
North580, 918/549
700 549 78%
TOTAL: 1,850 1,311 71%
MetrixMatrix.com (888) 865-0065 7
20%
36%
45%
60%55% 57%
100%
51%
100%
70%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
City A City B City C City D City E City F City G City H City I City J
Midpoint Sample Assessment: Central Region Completes, by City
After Wave 1, completed surveys for three cities within the Central Region were still less than 50% complete based on their target sample size. All three cities were also within the 405 area code.
MetrixMatrix.com (888) 865-0065 8
(Contact Made)
24%
6%
63%
7%
34%
11%
52%
4%
38%
9%
50%
2%0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Contact Fail Completed Survey Declined Survey Problem Record
Wave 1 Record Status, by Area Code
405
580
918/539
(9%)
Records within the 405 area code were more likely to result in a contact, when compared to records in the remaining area codes.
However, when contacts were made, calls made to the 405 area code were less likely to result in a completed a survey (9%) when compared to the 580 area code (17%) and the 918/539 area code (15%).
% Complete/Contact Made
(17%) (15%)
MetrixMatrix.com (888) 865-0065 9
(Contact Made)
24%
6%
63%
7%
55%
4%
38%
4%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Contact Fail Completed Survey Declined Survey Problem Record
Wave 1 and Wave 2 Final Record Status for SmartIDin the 405 Area Code
Wave 1
Wave 2
After implementing the SmartID method (Wave 2), records were more likely to result in a failed contact.
When a contact was made, SmartID calls were no more likely to generate a completed survey when compared to Wave 1.
% Complete/Contact Made
(10%)(9%)
MetrixMatrix.com (888) 865-0065 10
(Contact Made)
40%
5%
51%
4%
58%
3%
35%
4%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Contact Fail Completed survey Declined Survey Problem Record
Wave 2, Record Type with SmartID
Wave 2: Re-loaded Records
Wave 2: NewRecords
In Wave 2, 15% of the SmartID calls were re-loaded sample records from Wave 1 and 81% were new-purchase records.
Overall, new-purchase records were more likely to result in a failed contact.
Re-loaded sample records were more likely to result in an overall contact.
% Complete/Contact Made
(8%)(9%)
MetrixMatrix.com (888) 865-0065 11
OutcomeMinutes
/CallUnit Cost
Unit Count
Total Project Cost
Wave 1 Dials 0.75 $ 0.19 3,861
Contact Made –Completed Survey
10 $ 2.50 116 $1,606.09
Contact Made –Declined Survey
2 $ 0.50 1,165
Cost per Completed Survey: $13.85
Wave 2 Dials 0.75 $ 0.19 19,825
Contact Made –Completed Survey
10 $ 2.50 264 $5,750.25
Contact Made –Declined Survey
2 $ 0.50 2,647
Cost per Completed Survey: $21.78
MetrixMatrix.com (888) 865-0065 12
The overall cost per completed survey increased by 57.3% from Wave 1 ($13.85) to Wave 2 ($21.78)
Calling effort increased in Wave 2 with more dials needed to complete a survey. Meanwhile, the cost remained nearly equivalent for declined and completed surveys when individuals picked up the phone.
Wave 1 Wave 2
Do
llar
s
Cost per Completed Survey, Wave 1 and Wave 2
Dials Declined Surveys Completed Surveys
MetrixMatrix.com (888) 865-0065 13
Overall, the use of SmartID did not have the anticipated benefits on completion rates and production costs.
Calls made with SmartID were more likely to generate a failed contact.
Records that were re-called in Wave 2 using SmartID were more likely to result in a contact.
When a contact was made, SmartID calls were no more likely to result in a completed survey.
The overall cost per completed survey increased by 57.3% after using the SmartID method (due to increased effort).
However, SmartID may help improve survey coverage…
14MetrixMatrix.com (888) 865-0065
Jim Antonevich, PresidentMetrix Matrix
Chris Horn, Chief Information OfficerMetrix Matrix
Thank you to those who also contributed to this research:
Ann Rogers Lane, Director of OperationsMetrix Matrix
Contact Information:
Heather KnappenResearch AnalystMetrix Matrix [email protected](585) 739-0661
MetrixMatrix.com (888) 865-0065 15