survey tool and internet research program

UCD Survey Tool & Internet Research Program

Tom Callaghan

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

The Survey Tool: WebSurveyor/Vovici

2


Survey of Tool Users -- Summary

- 32 of 99 users from across the company responded.

- Users find WebSurveyor (now Vovici) useful, support

adequate, and would recommend it to a colleague.

- Training and the intranet site could be improved.

- Users indicated they saved $98,000 and 1282 hours

using WebSurveyor.


Users saw WebSurveyor as a useful tool they would recommend to others, but they wanted to see additional training and additions to the intranet site.

Indicate the degree with which you agree with the following statements regarding WebSurveyor.

(Percentages) Strongly

Disagree Disagree Neutral Agree

Strongly

Agree

WebSurveyor is a useful tool. 0.0% 0.0% 0.0% 43.8% 56.3%

Support for WebSurveyor is adequate. 0.0% 3.1% 12.5% 59.4% 25.0%

Training for WebSurveyor is adequate. 3.1% 6.3% 25.0% 46.9% 18.8%

The WebSurveyor intranet site is useful. 0.0% 0.0% 46.9% 31.3% 21.9%

I would recommend WebSurveyor to a colleague at CA. 0.0% 3.1% 3.1% 34.4% 59.4%


There was considerable return on the $36,000 investment. - 32 users saved $98,000 (~ $300,000 for 99 users)

- 32 users saved 1282 hours (~ 4000 hours for 99 users)

- Monetary savings ranged from $0 to $25,000

- A number of users said they found it hard to quantify

due to the lack of a comparable alternative, and

entered $0

- Time savings ranged from 0 to 120 hours.

*Statistics do not include past account owners and those sharing

accounts (~ 60 users), and of course, 67 account owners that did not

respond to the survey.


Internet UCD Survey Program

- Goal: Monitor the usability of site

- Strategy: Use online surveys to segment users

and collect feedback on user tasks.

- Timing: Site section importance survey (1) begun

Oct 5; redesign survey (6) in May


Internet Survey Program

9


Goals

- Impetus: Need for a subjective and “direct from the user”

compliment to the information in the site content ratings, web

metrics, search metrics, support calls, general feedback, and other

business owner customer input channels.

- Primary Goal: Provide a process for gathering end user

experiences regarding the usability of the site web presence in a

cost-efficient manner.

- Secondary Goals:

- Create a survey infrastructure for an Internet UCD Program.

- Generate a pool of profiled users to provide timely answers to

questions about site site design and performance.

- Create a community of site users by providing summary

information and insightful conclusions to survey results.


Strategy

Using a series of 5 (or 6) online surveys to segment users by 18 sections

of the site, then collect feedback on specific user tasks they indicate are

important and that they use frequently.

- Survey 1: Importance of site sections, and join web survey panel

- Intercept survey, respondents volunteer with email address

- Survey 2: Gather users tasks

- Email invite, opt out, track previous responses, more demographic info

- Survey 3: Rate task importance and frequency of use

- Email invite, track previous responses

- Survey 4: Rate usability of site in relation to specific task

- Email invite, track previous responses

- Survey 5: Repeat survey 3 when significant functionality is added to site

- First instance being the redesign

- Survey 6: Repeat survey 4 (1-2 months after launch)


Results: Survey 1

Site Section Average

Importance

Rating

Site Section Average

Importance

Rating

Support 4.1 News 2.9

Products 3.8 About site 2.8

Solutions 3.8 SMB 2.7

Trials and Downloads 3.7 Worldwide 2.6

Search 3.7 World 2.6

Contact 3.6 Events 2.6

Services 3.6 Partners 2.4

Home and Home Office 3.3 Careers 2.3

Education/Training 3.2 Investor Relations 2.1

Detail survey report: http://surveys.com/wswebtop.dll/WSPubReport?esid=1060&subaccountid=1

Survey example: https://surveys.com/wsb.dll/1/-Web-UCD_1_Example.htm

- Joined Internet UCD Survey Program: 1903 (1100 “Not sure”)

- 5000+ responses to date at 27% completion rate of survey

- Rating of site section importance

http://surveys.com/wswebtop.dll/WSPubReport?esid=1060&subaccountid=1


https://surveys.com/wsb.dll/1/CA-Web-UCD_1_Example.htm







Results: Survey 2 - Gather User Tasks

- Users provided top 3 tasks for site sections only if they rated that section

important, or higher, in Survey 1

- Tasks were refined and compiled for each site section. They were shared

with Internet Marketing and section owners for review and additions.

- Example:

- User Profile expanded with additional questions

Detail Survey Report: http://surveys.com/wswebtop.dll/WSPubReport?esid=1098&subaccountid=1 Survey Example: https://surveys.com/wsb.dll/1/-UCD_2-Example.htm

Emails Sent: 770

Usable Completed Surveys: 53



https://surveys.com/wsb.dll/1/-UCD_2-Example.htm







Results: Survey 3 - Rate task importance and frequency of use

- Tasks from Survey 2 ranked by importance and frequency of use to

understand what is important to our users to help us better focus our efforts.

- Tasks were sorted by importance and frequency of use, and a “top-ten” list

of tasks for each of the 18 sections was created to represent that section.

Detail Survey Report: http://surveys.com/wswebtop.dll/WSPubReport?esid=1291&subaccountid=1

Survey Example: https://surveys.com/wsb.dll/1/-UCD_3-BO-Tasks-Participants-EXAMPLE.htm

Emails Sent: 2015

Usable Completed Surveys: 88



https://surveys.com/wsb.dll/1/-UCD_3-BO-Tasks-Participants-EXAMPLE.htm













Results: Survey 4

- Rate usability of site in relation to specific task - Benchmark setting of site task usability

- Over 2500 users were invited to rate the usability of the top site section

tasks, and 258 acceptable responses were received. The percentage of

users that clicked on the survey link versus completing the survey

(49.9%) was nearly twice that of Surveys 1, 2, and 3.

Detail Survey Report: http://surveys.com/wswebtop.dll/WSPubReport?esid=1494&subaccountid=1

Survey Example: https://surveys.com/wsb.dll/1/-UCD_4-Task-Usability_DUMMY.htm

231 users rated the site

usability for Support tasks.

https://surveys.com/wsb.dll/1/CA-UCD_4-Task-Usability_DUMMY.htm









Strategy: Survey 5 In Survey 5, we rank the tasks by importance and frequency again if we feel

these tasks have changed or need to be expanded.


Appendix


Results: Survey 2 Demographic feedback on our users also provide insight.


Survey Invite Email site Business Owner,

You are being invited to complete this survey because as a business owner you have insight into the tasks your

web site visitors need to complete. The results of this survey will be used to improve site design and focus

resources.

The purpose of this survey is to ask you to rate the importance and frequency of use of tasks your users might

complete on site. The tasks were compiled from the responses to another survey, completed by site end

users. While not exhaustive, they represent some of the key tasks by which we can judge the usability of the

site.

You will be asked only to rate the sections for which you are responsible. Also, you will have the opportunity to

add up to two addition tasks for that section. There are eighteen (18) site sections, and a number of tasks in

each. It should take about 15 minutes to complete. Your ratings will refine the list presented in our next user

survey

In a follow up survey, users will rate each of these tasks according to how well the site supports them now. We

will then repeat the survey periodically, most notably after the April redesign.

Please click the following link to access the survey: |LINK1|

We ask that you complete this survey by January 12.

Thank you for your time and input,

33

survey tool and internet research program

Documents