Assessing User Satisfaction for the Digimap Service: Surveys, spread sheets and a different kind of chart.

The 2011 Winter Survey

• Required as part of the JISC services portfolio review

• Assess user satisfaction as part of the review of all their services

• Common questions across all JISC services

• In house design for interface and database• Bristol Online Surveys used in the past

• Detailed review for Geo-Services in 2011-12• Survey made more prominent in a pop-up

• Users made to wait 10 seconds for the pop-up to disappear

• Results were made publicly available online: http://edina.ac.uk/impact/

Considerations before starting a survey

PlanningSurveys are exercises in Public Engagement, be prepared!• Don’t carry them out lightly, have a plan• Survey are an inconvenience, weigh the costs with the benefits• Plan carefully how you are going to carry out the survey• Plan how you are going to give feedback• Plan how you are going to handle complaints

Good and swift feedback can turn someone with a negative view of your service into your best customer; if they have been part of an improvement they will often help promote your services.

“Tell me and I'll forget; show me and I may remember; involve me and I'll understand.” Chinese Proverb

Considerations before starting a survey (2)

TimingSurvey fatigue can dent your response rate and increase

the amount of overly negative responses you get through misrepresentation:

• When possible, avoid times that you know your customers/users are having to fill in lots of other surveys.

• Don’t survey too often / balance with length. • Find out the best time of year to survey your users.

LengthKeep it as short as possible, and use yes/no answers and

options as much as you can to minimise the time users need to fill it in.

Considerations before starting a survey (3)

MisrepresentationThis can be further mitigated by the amount of information you

give respondents:• Tell them why they are being surveyed• Tell them how there responses will be used• If possible, show how previous results have affected the service• Feedback the results of the survey• Feedback how the results have changed the service

The more the respondents understand why they are doing the survey and what impact it is going to have, the more useful the responses they are likely to give.

The Questions – Who are the users?1. Where do you study/work? 2. In which role are you using Digimap?

Student, Postgraduate, Lecturer, Researcher, Information professional

3. Please indicate your subject area:

Automatically answered for registered users

The Questions – How do you find the service?

Q1. On average, how often do you use this service?– Several times a week, Once a week, Once or twice a month, Less often, This is my first use

Q2. How do you find using Digimap? – Very easy, Easy, Neither easy nor difficult, Difficult, Very difficult

Q3. My work would take longer if Digimap were not available. – Strongly agree, Agree, Neither agree nor disagree, Disagree, Strongly disagree

Q3b. [optional] Why do you say that?

The Questions – What else is there?

Q4. Would you use an alternative if Digimap was not available? – Yes, No, Don’t Know

Q4b. If yes, what would you use?

Q5. I would recommend Digimap to others. – Strongly agree, Agree, Neither agree nor disagree, Disagree, Strongly disagree

Q5b. [optional] Please tell us why you would, or would not,

recommend this service:

The Questions – What do you use Digimap for?

Q6a. What do you use Digimap for? Tick all that apply to you:– For Research– For teaching the use of maps, spatial data or GIS – For Post Graduate Thesis – For teaching materials in a general subject – For Undergraduate Thesis – For supporting students use of maps, spatial data or GIS – For Coursework – For personal interest

Q6b. [optional] We’d really like to know more about what you are doing with Digimap, please give as much detail about what you use the service for below:

Can we contact you about using your example as a case study?

The Questions – What do you use the most?

Q7a. Rate the following elements of the service in terms of how important they are to your studies, research or work:

Essential, Important, Useful, Little value, I don’t use it

Viewing Online Maps

Downloading Data

Accessing Streamed Maps

Gazetteer and Postcode Query tools

Support Materials

Q7b. [optional] We’d really like to know how important the service is to you; please share any stories you may have about how Digimap has helped you:

The Questions – What do you want in the future?

Q8a. What would you like Digimap to provide next? Rate the following options in terms of how useful they would be to your studies, research or work:Essential, Important, Useful, Little value, I wouldn’t use it

Mobile Maps

More Data Products

More Data Formats

Advanced Online GIS Tools

More printing options

More mapping support

More GIS support

Q8b. [optional] Please give us any details about the choices you would use from the table above such as which products, formats, options or support you would like to see:

The Results

• 8537 responses received

• The published results are based on the 7000+ responses from returning users (omitting those who indicated this was their first time):

• OS = 4969• Historic = 1462• Geology = 776• Marine = 117

• Results in full: http://edina.ac.uk/impact/

• Thank you to anyone who filled one in!

Doughnut Charts

Mmm... Donuts!

Diverse Users

English Lit, History, Fine Art, Equine Studies, Medicine

Easy To Use?

YES

Saves Effort?

YES!

Would You Recommend Digimap?

YES!

Difficulties Users Experienced

• About 62% of respondents found the service easy or very easy to use… but what about the others?

• These were the main areas of concern:• Waiting for maps• Printing• Complex interfaces• Registration

The SolutionsDigimap is Slow?

• Problem = Bandwidth• Solution = Shrink JavaScript & Compress Map Images

Printing?• Problem = Previews & Legends• Solution = A new interface

Complex Interfaces?• Problem = MasterMap Download & Carto• Solution = New Data Download, Roam+ & USeD

Registration?• Problem = Complicated & the wait• Solution = The next Digimap licence

Priorities for the Future

OS1. Printing

2. Mapping Support

3. Data Formats

Historic1. Printing

2. Mapping Support

3. Data Products

Geology1. Printing

2. Mapping Support

3. GIS Support

Marine1. GIS Tools

2. GIS Support

3. Data Formats

Over 300 people commented in this section

Priorities for the Future (2)

• The Print interface needs to be enhanced.• New functionality: Legends, 2up maps & more formats

• We need to offer downloads in many more formats.

• Pre-generate data & on the fly transformations

• Mapping and GIS Support• The new home page resource centre

• New Datasets• We are continually looking to add to Digimap

• Environment Digimap

Survey Results Summary

• The service used by an incredibly broad range of people from across academia.

• The service is well liked and very well used:

...but...

• There is more that can be done to make the service:• Faster• Easier to use• Meet all it’s users requirements

• We continually need feedback to make sure Digimap continues to meet its users needs.

Lessons Learned from the survey

JISC want us to carry out the survey every year.

Changes we need to make:• No survey at all for first time users• Easier to ignore• Tighten up the wording of certain questions• Timing of the appearance of the pop-up• More thorough testing

Important considerations before making changes:• Surveys need to be comparable to previous years (especially

questions 1-6)• We need a good amount of responses

Beyond the Annual SurveyContinual Feedback Via:

• Helpdesk• No news is good news!

• “Comment on this service” links• A mechanism for people to let us know about something while they

are doing it• Surprising number of glowing reviews from this

• Shorter, one question pop-ups to gauge opinion on certain features etc. are also being considered.

• This has proved successful in the past for users waiting for downloads

• Social media• Popular with certain groups of users• Needs careful monitoring as it is all public

Beyond the Annual Survey (2)

More in-depth user engagement during service development

•Usage statistics are being used to create personae

•Users who fit into the categories are asked to test services in development

•Iterative testing and design phases are used to hone the user experience

•USeD Project: http://used.blogs.edina.ac.uk/