“Do users do what they think they do?” – a comparative study of user perceived and actual information searching behaviour in the National electronic Library of Infection (NeLI)

Anjana Roy 1, 2, Patty Kostkova1, Mike Catchpole 2 & Ewart Carson1

1City ehealth Research Centre and Centre for Health Informatics, City University, Northamption Square, London. UKAnjana.Roy@hpa.org.uk

2 Health Protection Agency61 Colindale AvenueColindale London NW9 5EQ

1. Introduction – importance of evaluation

2. Research questions

3. Methodology

a. Website & navigation structure

b. Study design

4. Results

5. Limitations of study

6. Conclusions & implementations

Outline of presentation

eHealth professionals

Challenging issues & different responsibilities

Demand of regular evaluation

Understanding of underlying issues of user satisfaction

Content

Healthcare websites - importance of evaluation

Medical websites provide

Evidence based information

Informed decision

Ease of navigation

Research Questions:

• Difference between perceived (reported) and actual (observed) searching behavior?

• Which navigation strategy most popular?

• Within each navigation strategy most popular options?

• Did users find answers to their queries?

• Was there an information seeking behavior pattern?

Web site used as test bed – National electronic Library of Infection

• The aim of NeLI is to provide a single information gateway in a portal of evidence based information.

• Information related to investigation, diagnosis treatment and management of infection and communicable diseases.

NeLI Navigation Structure

Snap shot of “Browse” options available on NeLI during study period

Snap shot of “Search” options available on NeLI during study period

NeLI Navigation Structure

On-line questionnaires Weblog report

• IP addresses of respondents were used to track their navigation .•Answers were compared with the weblogs.

Study design

1. Was there any difference between perceived (reported) and actual (observed) searching behavior?

Questionnaire

WeblogBrowse

(%)Search

(%)Combination (%)

Browse 20 7 13

Search0 40 0

Combination 0 7 13

• Questionnaires and weblog answers compared by kappa measurement of agreement. • The mean of the agreement was 0.59 - moderate agreement between answers in the questionnaire and actual behavior recorded on weblog

Cases where the users reported and navigation behaviour did not match

• Users were ‘observed’ used combination of navigational options while reporting only one option.

Possible explanation:

• The most likely explanation is web space disorientation.

Recommendation: Make the navigation access points clearer and simpler to understand

• Users reported using a different navigation technique e.g. they reported using ‘search’ but used ‘Browse option’.

2. How did the users seek their information – searching or browsing?

Questionnaire

Weblog Browse (%) Search(%)

Combination(%) Total (%)

Browse 20 7 13 40

Search 0 40 0 40

Combination0 7 13 20

Total 20 53.3 26.7 100

• According to weblogs both search and browse are equally.• Respondents are more likely to report or recall searching activity rather than browsing.

   Options

Questionnaire (percent)

Web logs (percent)

Browse  

  1. A – Z listing of pages on NeLI 7 13

  2. Top 10 Topics 13 20

  3. Factsheets listed on NeLI 13 13

  4. Guidelines listed on NeLI  13 13

  5. Antimicrobials Resistance 0 0

 6. List of Infectious Disease Society Websites 7 7

Search  

  1. Pull down Menu 67 60

  2. Free Text Search 0 0

3. Within each navigation strategy most popular options?

Overall, most users reported their methods of navigation accurately.

“Browse” - no significant difference in the choice of choice of options available.

“Search” – the ‘Free Text Search’ was not used at all

Questionnaire : All the users reported that they were able to find the answers to their questions.

One exception ……………………….

4. Were the users able to find the answers to the queries?

Web logs: indicated that all respondents did access a page of their interest as reported in their questionnaire.

• Users understand and are able to navigate the website as expected by the developers.

• Information on diverse aspects of infectious diseases available on NeLI.

5. Was there any information seeking behavior pattern typical of certain users?

• 80% of the respondents visited other pages of interest after submitting questionnaire.

• Web logs: 93% of those who visited these other sites used “browse”.

• 27% used “Top 10 Topics”.

Recommendations: Need to identify “Top 25 Topics” in infectious diseases.

Limitations of the study

• External web site when accessed from cannot be tracked using the available web logs.

• Questionnaire has excluded “how users would surf NeLI”.

• Sample size limited.

• According to web logs both search and browse are used equally.

• Respondents are more likely to report or recall searching activity rather than browsing.

• Moderate agreement between answers in the questionnaire and actual behavior recorded on web log.

• Users understand and are able to navigate the website retrieve information on ‘question of interest’.

• When surfing though NeLI their preference was to browse, using “Top 10 Topics.

Conclusions

Navigation access points clearer and simpler to understanding

NeLI now has a list of 30 Infectious Diseases as an independent navigation item.

Implementation of recommendations to actions: