1© disclosure or duplication without consent

prohibited

COMET K2 Competence Center - Initiated by the Federal Ministry of Transport, Innovation & Technology (BMVIT) and theFederal Ministry of Economics & Labour (BMWFI). Funded by FFG, Land Steiermark and Steirische Wirtschaftsförderung (SFG)

Is Enterprise Search useful at all? Lessons Learned from Studying Usage behavior.

I-KNOW 2014Graz, Austria

Dr. Alexander Stocker

2© disclosure or duplication without consent

prohibited

Intro User Satisfaction User Evaluation

Enterprise SearchAgenda

3© disclosure or duplication without consent

prohibited

Enterprise SearchBusiness Case

Findability Survey 2013 http://de.slideshare.net/findwise/enterprise-search-and-findability-survey-2013

4© disclosure or duplication without consent

prohibited

Huge demand from industry Enterprise Search is becoming more and more popular EU-Market for EU Enterprise Saerch vendors is between 100 and 200 Mio. €

Lack of application oriented research Lack of academic case studies Lack of academic best practices Lack of academic user evaluations

Is the topic Enterprise Search in the hand of practicioners, only?

Enterprise SearchIntroduction

5© disclosure or duplication without consent

prohibited

Enterprise search includes any organization with text content in electronic form, search in an organization’s external website, search in the organization’s internal websites (i.e. its intranet), and search in other electronic text held by the organizations in the form of email,

database records, documents on fileshares, etc.

(„Challenges in Enterprise Search“, Hawking, 2004)

Enterprise SearchDefinition

6© disclosure or duplication without consent

prohibited

Enterprise SearchSatisfaction

Mindmetre: Report „Minding the Search Gap“

7© disclosure or duplication without consent

prohibited

Enterprise SearchSatisfaction

Findability Survey 2013 http://de.slideshare.net/findwise/enterprise-search-and-findability-survey-2013

8© disclosure or duplication without consent

prohibited

Web Seach Interlinked Web pages.

Standard formats

Relevance of a web page is (initially) calculated by assessing, how many (good) links point to it

Information on the Web wants to be found. Hence content is optimized for search.

Web search has indexed open content, access rights are not relevant

Searcher is satisfied with a good answer (out of many possible good answers). Search queries are of a more common nature.

…

Search on the Web vs. Search in the Enterprise

Enterprise Search

Heterogeneous content (structured and unstructured)

No links between the documents. Different ways of relevance assessment needed

Users are not motivated to optimize their documents for search

Complex group & roles structures. Access rights first, information access second

In most cases, there is only one document relevant for a user. This has to be „found“ as quick as possible.

..

9© disclosure or duplication without consent

prohibited

Case Study Research organization in the automotive domain, employing about 200 people Enterprise Search Pilot to find information in project-relevant documents on a

fileshare (Microsoft Sharepoint Foundation 2013) Pilot study with 10 participants to identify user-centric aspects of Enterprise Search

implementation

Evaluation Short structured interviews

(tasks, documents and tools, information demand, search-scope and time) Definition of possible & realistic search tasks Participants performed these search tasks afterwards Participants had to think aloud what they did and why they did it Transcription and analysis of interviews Identification of commonalities and differences

Enterprise SearchUser Evaluation

10© disclosure or duplication without consent

prohibited

Enterprise Search(Microsoft Sharepoint Foundation 2013)

11© disclosure or duplication without consent

prohibited

Finding the right keywords is crucial, but very challenging Better keywords lead to more relevant search results Thinking process for finding the best keywords is very challenging Explication of a personal information demand for a search engine is a challenge

Evaluation results Keyword selection and query formulation

12© disclosure or duplication without consent

prohibited

The benefit of metadata for search gets lost, if metadata is not administered in organizational practice Metadata is used in search to filter search results („facets“) Missing or/and false metadata reduces search quality (e.g. the shown author is not the

correct author of a document, but the creator of its template) Users lack motivation to provide correct & useful metadata, as they do not perceive

any benefit

Evaluation results Usage of metadata

13© disclosure or duplication without consent

prohibited

Consistent document content (formulations) would increase perceived usefulness of search The availability of essential terms in documents increases search success

(e.g. authors, location, date, consistent project titles, MOM vs. meeting minutes, ..) Knowledge about document content and writing practices is crucial

(terms, syntax, semantics, …) Common writing practices of essential project-information in documents have to be

found, first, otherwise searchers may always have to guess(“..what terms could be used in the document containing the information I seek..”)

Evaluation results Document Content

14© disclosure or duplication without consent

prohibited

The relevance assessment of a search engine does not necessarily meet the user needs The search engine calculates relevance displayed in the hit list based on the

appearance of query terms within documents The ranking is crucial for perceived usefulness of search. Employees requested

other mechanisms, e.g. ranking in term of up-to-dateness

Evaluation results Relevance

15© disclosure or duplication without consent

prohibited

Enterprise Search has to compete against existing individual information seeking practices (search alternatives). So far, anything had to be „found“ without enterprise search. Employees (may) have

created individual search strategies. Face-2-Face conversations between colleagues are the number one tool for

information acquisition (e.g. the location of a document is gained by asking a colleague who knows it)

Users know the structure of project-folders, they browse instead of search Users take advantage of links to projects, project documents and others via their

desktop or via messenger/skype etc.

Evaluation results Search Alternatives

16© disclosure or duplication without consent

prohibited

Enterprise Search has to immediately provide an added value from a user‘s perspective for daily work. Enterprise Search can require more time to find information compared to browsing

through (known) folder structures Finding the right office-document was challenging, because of the high number of

similar documents in different versions (e.g. V21_meeting_minute_xy) User satisfaction was especially very low, when keywords were not available in

documents

Evaluation results Overall usefulness and satisfaction

17© disclosure or duplication without consent

prohibited

1. Enterprise Search Introduction

2. Enterprise Search Satisfation

3. Enterprise Search Potentials & Pitfalls

Pioneering work to motivate other researches studying user-aspects of enterprise search (adoption)

Call for Research Cases: Enterprises having a successful enterprise search implementation with a very high user satisfaction are warmly welcome

Enterprise SearchConclusion

18© disclosure or duplication without consent

prohibited

Kontakt

Dr. Alexander Stocker

VIRTUAL VEHICLE

alexander.stocker@v2c2.at

www.alexanderstocker.at