Library & Information Science Research 34 (2012) 300–307

Contents lists available at SciVerse ScienceDirect

Library & Information Science Research

Scenarios in information seeking and information retrieval research: Amethodological application and discussion

Jeonghyun KimDepartment of Library and Information Sciences, College of Information, University of North Texas, Denton, TX 76203-5017 USA

E-mail address: jeonghyun.kim@unt.edu.

0740-8188/$ – see front matter © 2012 Elsevier Inc. Alldoi:10.1016/j.lisr.2012.04.002

a b s t r a c t

a r t i c l e i n f o

Available online 4 August 2012

Scenario has been used as a research tool to study information seeking and information retrieval. This reviewof the nature and development of scenario use reveals a variety of applications and a number of practicalmethodological issues. While scenarios have a number of advantages, there are limitations. Considerationof these advantages and challenges leads to recommendations for constructing and presenting scenarios.

© 2012 Elsevier Inc. All rights reserved.

1. Introduction

Imagine you are going to visit your friend in Paris during springbreak, and you want to find information about cultural events, suchas concerts, painting, and photography exhibits, that will take placeduring your stay. You decide to use the Internet as a starting point.Now suppose you believe you have a high risk of breast cancer, asyour mother had it 4 years ago, and your chest is sore to the touch.You go to WebMD (www.webmd.com) to learn about the symptomsof breast cancer.

These search-task scenarios are simulations of real-life searchtasks employed in research conducted with information-retrievalsystems and their users' information behavior. The scenarios depicthypothetical situations and provide some contextual information forsubjects to understand the situation being portrayed; they areintended to reproduce a trust situation and facilitate an explorationof subjects' responses to those hypothetical situations. The use ofsuch a scenario is a type of unobtrusive measure, with the underlyingassumption that subjects will give the same response for this briefrepresentation of a real-life situation as they would give if the situationactually applied to them.

Social sciences research has employed scenarios since the 1950s,including applications in anthropology, sociology, psychology, andcognitive psychology. The literature reveals a large number of differentand, at times, conflicting definitions, characteristics, andmethodologicalideas. Researchers have coined different terms, including vignette, casestory, case simulation, hypothetical scenario, task scenario, and situationscenario, and used them interchangeably to describe the “brief, concretedescriptions of realistic situations” (Kerlinger, 1986, p. 457). All havebeen used to gauge how people might respond in a given situation, toobserve a respondent's behavior, and to elicit a respondent's opinion,belief, or attitude.

rights reserved.

Likewise, the scenario is a popular tool to help library and informationscience (LIS) researchers understand people's cognitive, emotional,and behavioral aspects, and for evaluating information-retrievalsystems and processes. It has been used as a trigger for simulatedinformation needs in information behavior studies, promptinginformation users to reveal their information-seeking strategiesand choice of information sources (Urquhart, 1999). The use of realisticscenarios became popular in information retrieval studies after Borlund(2000) proposed the approach for interactive information retrievalevaluation, or “simulated work task situation.” It has been supposedthat scenarios may trigger more predictable information needs fromthe end user of information retrieval systems, producing evidencein the form of more natural requests. For this reason, the scenariohas been useful in information-seeking studies, and in interactiveinformation-retrieval experiments (Ingwersen & Järvelin, 2005).

2. Nature and history of scenarios

According to the Oxford English Dictionary, the word “scenario”has two general definitions: 1) a sketch or outline of the plot of aplay, ballet, novel, opera, story, and so forth, or a film script with allthe details of scenes, appearances of characters, stage directions,and so on, and 2) a sketch, outline, or description of an imaginedsituation or sequence of events (Scenario, 2012). The first definition,a classical, common English usage, fits best with the humanities (i.e.,folktale scenario, myth scenario, saga scenario, etc.), though it isoften called a story, script, or narrative. On the other hand, the seconddefinition has turned the term “scenario” into a buzzword, as it has amyriad of uses with different meanings in various disciplines. Scenariohas been used as a term for war-game simulations in the military;as a strategic planning tool in business; in the form of simulationsof future situations in crisis management; as an illustration of systemrequirements, analyses, and component interactions in softwareengineering; and as a series of user activities of system use in human–computer interaction. There are varying definitions of scenario, but

301J. Kim / Library & Information Science Research 34 (2012) 300–307

some characteristics are inherent in all the definitions: hypothetical,causally coherent, internally consistent, and/or descriptive. Scenariosare a simulation that describes real or hypothetical situations in orderto prompt people to consider them.

The literature reveals a large number of different, and at timesconflicting, definitions, characteristics, principles, and methodologicalideas about scenario as a research tool. The social science roots ofscenario-based research can be traced to Piaget's (1932) use ofwhat he called “story situations” to investigate moral reasoning inchildren. Piaget offered an important rationale for using scenarios:

While pure observation is the only sure method, it allows for theacquisition of no more than a small number of fragmentary facts…Let us therefore make the best of it and… analyze, not the child'sactual decisions nor even his memory of his actions, but the wayhe evaluates a given piece of conduct… We shall only be able todescribe [it]… bymeans of a story, obviously a very indirectmethod.To ask a child to say what he thinks about actions that are merelytold to him—can this have the least connection with child morality(Piaget, 1932, p. 112–113).

As a common methodological technique, scenario has been usedwithin social science, including applications in anthropology, sociology,and psychology, since the 1950s. Herskovits (1950) introduced themethod of what he termed the “hypothetical situation” in his fieldresearch on the culture of people of West Africa. Anderson andAnderson (1951) devised the “projective technique,” which askedresearch subjects to verbalize their thoughts and feelings by givingthem something with which they could connect. Star (1955) employed“descriptions” of a person with problematic behaviors to assess thegeneral public's recognition of mental illness. Finch (1987) renamedthis approach “vignette,” which allowed participants to respond invariousways to “short stories about hypothetical characters in specifiedcircumstances” (p. 10).

Vignettes have beenused in variousways and for different purposes.For instance, intervieweesmay be presentedwith a scenario and invitedto comment on how it makes them feel, or what they might do in thatsituation. Scenarios have beenpresentedwith a series of predeterminedresponses in a survey questionnaire in order to elicit the respondents'judgment about a given situation. Analysis involved comparingresponses or using a Likert scale, which enabled respondents to rate aparticular response (Alexander & Becker, 1978). Vignettes also havebeen administered in controlled, experimental evaluations of thefactors that affect the respondents' judgments (Rossi & Anderson,1982). The technique has been recognized as a valuable research toolthat “facilitates the expression of knowledge, opinion, beliefs, values,and attitudes” (Johnson, Newton, Jiwa, & Goyder, 2005, p. 326), andfurthers preferences, intentions, reasoning, or intended behaviorabout a specific situation. For this reason, vignettes have been appliedin health and medical research, such as nursing research, medicaleducation, and research on clinical practice (Hughes & Huby, 2002),especially for abstract concepts related to health and illness. In addition,vignettes have been employed as an effective instruction tool topromote discussion and topic exploration, and as an assessment toolto rate student learning (Jeffries & Maeder, 2011).

While using a short story that depicts a hypothetical character orsituation as a stimulus has been popular as a research tool in socialsciences, and more recently in the health fields, the library and infor-mation science field is relatively new to using scenarios. In simulatedexperimental studies of search behaviors and evaluations of retrievalsystems, researchers formerly assigned search tasks for subjects toperform (Haas & Kraft, 1984). Arising from “a need to simulate a real-istic interactive searching task within a laboratory environment”(Beaulieu, Robertson, & Rasmussen, 1996, p. 85), there was a call forthe use of realistic scenarios in information seeking (IS) and informationretrieval (IR). In 1997, Borlund introduced the concept of “simulated

work task situations,” which frames the simulated information-needsituation. Borlund (2003). The scenario format for simulated worktask situations corresponded with the one for vignette: “short storiesabout hypothetical characters in specific circumstances, to whose situa-tion the research participant is invited to respond… moving fromabstract to context-specific” (Finch, 1987, p. 106).

It is evident that scenario is a popular term. It may have the widestapplication in human–computer interaction studies. Conventionalusability evaluation, which is mostly laboratory-based, has tried tosimulate scenarios, defined in this case as “short stories of specificuser situations” (Courage, Redish, &Wixon, 2009, p. 47), for developingrealistic sample tasks for usability testing. In one instance, a scenariowas constructed on the basis of a task analysis of the actual users andtheir work; as a result, the scenario was as representative as possibleof the eventual use of the system (Campbell, 1992). Such a task-basedscenario states only what the user wants to do, however, rather thanserving as a form of stimulus. Nevertheless, this methodology hasproven its value over many years. Task-based scenario is stronglygrounded in carefully prepared descriptions that test users are askedto perform in very controlled conditions; all participants perform thesame tasks under as similar conditions as possible in order to makethe gathering of quantitative data easy and reliable.

Scenarios are often applied as a strategic planning tool in scenarioplanning for a different purpose. Scenario planning is a useful meansof conducting an organization's strategic analysis and planning. Theapplicability of scenario planning in a library organization has beendiscussed in many LIS articles (e.g., Giesecke, 1999; Hannabuss,2001; Wilmore, 2001). Scenarios in scenario planning are plausiblestories about alternative futures, and focus more on what the futuremight hold. These scenarios are then used to review or test a rangeof plans and policy options, or can be used to develop new policies.

Here, the use of scenarios focuses on scenario as defined as a narra-tive, a detailed description that contains real or hypothetical events,incidents, or situations that can be used in research studies to elicitresearch subjects' knowledge, attitudes, or behaviors regarding howthey would behave in the depicted situation. As such, scenario, whichis essentially a short story, for the purpose of imagining a given situationin both vignettes and simulated work task situations, is included. How-ever, scenario as employed in the human–computer interaction area,which describes specific user situations to ask subjects to do specifictasks, and scenario planning, which simulates possible futures, are notincluded.

3. Application of scenarios in IS and IR research

Scenarios frequently have been used as a given stimulus in both ISand IR in the LIS field. To review scenario-based studies in the LISfield, several databases, including Library and Information ScienceAbstract, Web of Science, and ACM Digital Library, were searchedfrom 1990 to 2011. Studies were located using a variety of relatedterms, including “scenario,” “vignette,” “search task,” and “simulatedwork task situation.” The methodology section of each selected paperwas reviewed. Articles were selected if a narrative that described a real-istic situation was given to research subjects to investigate subjects'reaction to the situation—that is, for assessing or predicting behavior,eliciting attitudes, beliefs, and opinions, or elucidating the decision-making process.

Citation and bibliographic searches using those databases indicatethat the use of scenarios has rapidly increased in the study of IS andinteractive IR, as presented in Fig. 1. Although an extensive literaturesearch yielded 125 studies using scenario as a research tool, this maynot represent all studies, as many terms are used to describe scenarios.It was noted that scenarios were used in a variety of ways, such as onsurveys, to supplement interviews, and to collect experimental databoth inside and outside the laboratory.

0

5

10

15

20

25

30

1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011

Fig. 1. Scenario-based research 1994–2011.

302 J. Kim / Library & Information Science Research 34 (2012) 300–307

3.1. Nonexperimental studies

Scenarios have been incorporated in interview questions in qual-itative research. As early as 1994 in their study assessing nurses'information-seeking skills and perception of information sources,Urquhart and Crane (1994) created a series of information problemsin a scenario-format information problem:

“Imagine you are in charge of an acute ward. You have been askedby your manager to investigate the feasibility of primary nursingfor your particular type of ward. As far as you know, primary nurs-ing is not yet practiced in your type of ward. How would youapproach this request for information?”

This problemwas given to the participants at the end of the structuredinterview. Participants then were instructed to write brief notes on thegiven problem.

Interviews incorporating a scenario have been used to promptinformation users to reveal their information-seeking strategies,choice of information sources, and the reasoning underlying theirchoices (e.g., Landry, 2006; Meyers, Fisher, & Marcoux, 2009;Rowley & Urquhart, 2007; Williams & Coles, 2007). This approachwas assumed to be effective “when direct questioning might be lessproductive” (Urquhart, 1999, p. 280). Landry (2006) designed fivework-task scenarios to elicit and capture the information-seekingbehavior of dentists. For instance, a patient education task-associatedscenario was presented as follows:

“You currently provide your patients with educational materialsrelevant to your dental practice. However, you feel that whatyou make available appears out-of-date and are looking to replaceit with something fresher and more pertinent to the direction thatyou are taking your dental practice.”

After the scenario, several open-ended questions were asked todetermine the first source of information preferred by participants,as well as their awareness of information.

Scenarios were also employed as a warm-up exercise in a focusgroup interview to stimulate a purposeful discussion (e.g., Zuccala,2010). In addition, this technique has been used as a complementarytool to anchor survey questions (e.g., Beldad, Jong, & Steehouder,2010; Hunter, Whiddett, Norris, McDonald, & Waldon, 2009;Stansbury & Ludwick, 2009; Stefl-Mabry, 2003). These studies devel-oped a short scenario, followed by a question asking how the respon-dent would feel about the situation, or what they would do in thesituation, ranking their feelings on a scale provided in the form.Stefl-Mabry (2003), in her study on users' satisfaction with a genericinformation source, asked research participants to assume they wereseeking information to use in making an important decision on anissue that was completely new to them. She presented participants

with a set of information-judgment task scenarios, and participantswere asked to score their satisfaction level for six information sourceson a scale of 0 to 100.

3.2. Experimental studies

In 1997, Borlund proposed a novel approach for interactiveinformation-retrieval evaluation that she called “simulated worktask situations,”which was based on the assumption that “informationneeds are dynamic mental constructions, thus simulations of informa-tion needs should hence allow for their dynamic nature to evolve duringexperiments” (Borlund & Ingwersen, 1997, p. 226). This approach waswidely accepted in the interactive information-retrieval field becauseit was driven by the expected benefits of positing the searchers withina realistic context and helping generate natural behaviors. Theysolicited real searchers and presented them with realistic search-taskscenarios that reflected real-life search situations. The research partici-pants were asked to imagine that theywere the person described in thescenario, and to find the information they needed for the describedsituation. Table 1 presents selected experimental studies that employedsearch-task scenarios.

A number of studies using scenarios were conducted to observevarious aspects of information-seeking behavior. Some focused onsearchers' search strategies (e.g., Fu, 2010; Kim, 2009; Thatcher, 2008)or relevance judgment behavior (e.g., Papaeconomou, Zijlema, &Ingwersen, 2008; Tombros, Ruthven, & Jose, 2005). Others looked intointeractive search behavior. This was oftenmeasured by the observablephysical actions logged during search sessions, such as entering searchqueries, viewing search results, reading web pages to assess their rele-vance, saving pages judged as relevant, and so on (Aula, Khan, &Guan, 2010; Gwizdka & Lopatovska, 2009; Hansen & Karlgren, 2005;Kim, 2008; Kules & Shneiderman, 2008; Li & Belkin, 2010; Ruthven,Lalmas, & van Rijsbergen, 2003; Saito et al., 2009). Kim (2009) exploredthe effect of different search tasks on information-seeking strategiesusing three search-task scenarios: factual, interpretive, and exploratorysearch tasks. Such tasks were assumed to have different task attributes,such as information need, task structure, types of information required,number of information required, and so forth. As the focus of Kim'sstudywas on theWeb, topics for the scenarioswere chosen to representthe general interest ofWeb users. Li and Belkin (2010), in their study onthe relationship between work-task and search behavior, created sixtask scenarios corresponding to the six work-task types that werevaried with the values of product and objective task complexity.

A large number of studies were conducted on the evaluation ofinteractive information retrieval (e.g., Balatsoukas & Demian, 2010;Joho, Birbeck, & Jose, 2007; Joho & Jose, 2008; Jose, Furner, &Harper, 1998; Pharo, 2008; Urban & Jose, 2005; White, Jose, &Ruthven, 2003; White, Jose, & Ruthven, 2006; White & Marchionini,2007; White & Ruthven, 2006; Yuan & Belkin, 2010). The scenario

Table 1Selected experimental studies using scenario.

Number; type of searchtask scenario

Development ofscenario

Time limit Number ofparticipants

Datacollectionmethods

Scenario

BalatsoukasandDemian(2010)

6; Fact finding anddecision making

NR NR 12 University facultyand students

QuestionnaireInterview

You need to develop an action plan for the renovation ofhistorical buildings in the area of Leicestershire… Which ofthe past project plans that dealt with the renovation ofhistorical buildings could be considered for reuse in thepresent situation?

Fu (2010) 6; Closed-ended (factfinding or known item)

NR 1 hour for6 searchtasks

22 University staffs InterviewQuestionnaireEye tracking

You heard that in Russia, people drink a lot of vodka. Canyou find out on average how much vodka a Russian drink?

Gwizdka(2010)

12; Fact finding andinformation gathering

Created byotherresearcher

1½ to2 hoursfor 6searchtasks

48 LIS undergradu-ate/ graduatestudents

QuestionnaireSearch logs

As a history buff, you have heard of the quiet revolution, thepeaceful revolution, and the velvet revolution. For askill-testing question to win an iPod, you have been askedhow they differ from the April 19th revolution.

Kim (2009) 3; Factual, interpretive,exploratory

Pilot-tested 10 minper eachtask

30 LIS graduatestudents

QuestionnaireInterviewSearch logs

You plan to visit San Francisco next week. One of yourfriends who has been there suggests that you visit therestaurant that is known as the oldest seafood restaurant intown. You want to know the name of the restaurant.

Kules andShneiderman(2008)

4; Exploratory Reviewed by ajournalismprofessor

2 hoursfor 4searchtasks

24 Journalismundergraduate/graduate students

QuestionnaireInterviewThink-aloudSearch logs

Imagine that you are a reporter for a national newspaper.Due to some recent events, your editor has just asked you togenerate a list of ideas for a series of articles on humansmuggling.

Li and Belkin(2010)

6; Intellectual anddecision/ solution

Revised fromthe real worktasks

15 minper task

24 Undergraduate/graduate studentswith diverse majors

QuestionnaireThink-aloudInterviewSearch logs

You need to take at least three courses next semester, andyour advisor has asked you to check the classes offered inyour program before you make this decision.

Pharo (2008) 3; General, challenging,and one formulated bythe individualparticipant

NR 20 minper eachtask

73University-affiliatedparticipants

QuestionnaireSearch logs

Video games are being played by an ever increasing numberof people of all ages, and the game industry is becoming amajor economic player. You would therefore like to findnontechnical information about how video games haveaffected people's lives, as well as how the games havechanged the entertainment industry.

Ruthven et al.(2003)

6; Topic search Adopted fromInteractive trackof TREC6

15 min foreach task

30 Computer scienceundergraduate/graduate students

QuestionnaireInterviewSearch logs

You and a friend are trying to choose a holiday for later thissummer. One possible destination will mean taking severalferry trips, but you have heard rumors that ferries in thisarea have a poor safety record. You need to book yourholiday soon, but need more information on the dangers offerry travel.

Shah andMarchionini(2010)

2; Exploratory NR 20 minper task

84 Undergraduate/graduate students

QuestionnaireSearch logs

The College Network News Channel wants to do adocumentary on the effects of social networking servicesand software. Your team is responsible for collecting variousrelevant information from the Web.

Tombros et al.(2005)

3; Background, decision,and many items

NR 15/30 minper eachtask

24 Undergraduate/postgraduatestudents andacademic staffs

QuestionnaireThink-aloudSearch logs

You are considering a career as a Web-page designer andhave an interview next week with a company you reallywant to work for. The position will involve designing sitesto allow local companies to sell their products on-line. Yourealize, however, that you know little about who actuallyuses the Internet.

White et al.(2006)

4; Fact-finding, decision,background, and manyitems

NR 10 minper eachtask

24 Undergraduate/postgraduatestudents

QuestionnaireInterviewSearch logs

You have recently inherited a large sum of money left by arecently deceased distant relative. A number of friends haveadvised you that it may be worth investing this money in afinancial instrument, such as a bond or corporate stocks. Atpresent you are unaware of stock market trends and lackthe knowledge required to make a sound judgment on whatto do with this money.

White andMarchionini(2007)

4; Known-item andexploratory

Adopted frompreviousresearch

5/10 minper eachtask

36 Undergraduate/graduate students

QuestionnaireSearch logs

You are doing some research for a term paper you arewriting and need to find the name of the first woman totravel in space and her age at the time of her flight.

Yuan andBelkin(2010)

4; Scanning andsearching

Scenarios wererevised from thereal work tasks

12 minper task

32 Graduatestudents

QuestionnaireSearch logs

As a graduate student, you are asked to write an essay aboutglobal warming for one of your courses. You believe itwould be interesting to discover factors that affect globalwarming, and would like to collect documents that identifydifferent factors.

Xu & Wang(2008)

1; Exploratory NR NR 133 Undergraduatestudents

QuestionnaireSearch logs

Assume you are taking a health education course. You willbe asked to search online for documents about “therelationship between mobile phone radiation and health.”

NR: Not Reported.

303J. Kim / Library & Information Science Research 34 (2012) 300–307

approach was often used when researchers wanted to prove the ef-fectiveness of a particular information retrieval system or applicationon searchers' search performance and relevance judgment. Such pro-totype systems or applications were designed to support interactiveinformation-retrieval and were often compared with another. Many

of those studies employed user activities through automatic loggingfrom the system with subjects' indication of their perceptions orsatisfaction. White and Ruthven (2006) used three different versionsof the search system that varied how users indicated which documentrepresentations were relevant, modified their queries, and made

304 J. Kim / Library & Information Science Research 34 (2012) 300–307

search decisions. In this experiment, subjects were asked to chooseone scenario that was most interesting from the given sixsearch-task scenarios; they were not allowed to choose more thanone task scenario for a particular topic. Pharo (2008) studied whichelements of the documents searchers judge to be the most relevantto searching in XML retrieval systems. His approach was differentfrom other studies, as he used two categories of task scenarios,challenging and general, from which the searchers could select oneof each. Pharo also used an additional scenario formulated by theindividual searcher.

4. Benefits and challenges of scenarios

4.1. Indirect elicitation

Scenarios have been accepted as a valid tool to elicit responsesthat might be more difficult to reveal using a more direct approach(Kerlinger, 1986). They can be regarded as a less-threatening wayof uncovering difficult-to-explore and sensitive topics. In addition,they are often constructed to predict behaviors—that is, to under-stand how individuals might behave in the hypothetical situationdepicted. In a sense, scenarios can be less intrusive compared witha direct observation of behavior. Therefore, the use of scenarioscan serve as a mechanism for avoiding the potential of theHawthorne effect (Gould, 1996), whereby those who are being ob-served alter their usual behavior in order to create a more favorableimpression.

Most studies asked participants to respond to a range of selectedscenarios regardless of whether the participants had disclosed a similarsituation. Williams and Coles (2007) took a less-personal approach;instead of being given scenarios, the interviewees were invited torespond with what they would advise others to do when looking forand using information. Then they were invited to relate the scenarioto a personal experience.With this approach, participantswere allowedto tell their own stories about information-seeking in a nonthreateningway.

4.2. Efficiency

Scenarios appear to offer a feasible alternative to observation,which is the traditional method of obtaining data about behavior(Gould, 1996). This is because scenarios are relatively easy to con-struct and administer; scenarios are proven to generate sound datarelatively fast and at a lower cost than observational studies, which areoften expensive and time-consuming. In addition, they are easy to repeat,and researchers do not have to wait for certain events to occur. Thus,many researchers draw attention to the convenience of scenariosas a tool of collecting extensive amounts of data from large samples.Scenarios can quickly generate considerable amounts of data from alarge participant group because they are often used to answerquantitative-focused research questions, especially when incorpo-rating a survey questionnaire. For example, the survey instrumentemployed in Stansbury and Ludwick's (2009) study was adminis-tered to a total of 450 participants; Beldad et al.'s (2010) surveyattracted 223 respondents.

4.3. Internal validity

Another advantage of using scenarios is that they can improve theinternal validity of a study by standardizing the conditions underwhich the research is carried out. In other words, standardized infor-mation in scenarios enables all participants to respond to the samestimulus, and it allows researchers to control extraneous variables.Scenarios can provide a potential advantage with regard to internalvalidity, as compared to observational studies, where the information

obtained for the overall sample cannot be standardized, as data arecollected from subjects in a range of different situations.

When scenarios are used in conjunction with interviews or surveyquestionnaires, the information contained within the scenario can bedefined and standardized, which enables all participants to respondto the same stimulus and focus on the same issue. Moreover, by hold-ing the stimulus constant over a heterogeneous respondent popula-tion, the survey researcher can gain a degree of uniformity andcontrol over the stimulus situation, approximating what is achievedby researchers using experimental designs.

This also allows replication studies and the relative ease of cross-study comparisons (Weber, 1992). Especially for experimental stud-ies, the use of a standardized and defined scenario allows compari-son of systems (e.g., Joho et al., 2007; Pharo, 2008; White &Ruthven, 2006; White et al., 2003; Xu & Yin, 2008; Yuan & Belkin,2010) or a comparison of the characteristics of searchers workingon the same search tasks (e.g., Gwizdka & Lopatovska, 2009; Saitoet al., 2009). Many evaluation studies employing scenarios havebeen conducted in an experimental environment, where great carewas taken to control the situational variables that can influence auser's search experience or performance. That is, the researcher canmaintain experimental control over the search tasks by placing allparticipants in the same hypothetical situation (Borlund, 2000). Sinceprocesses in information-seeking are complex and multiple, scenariosoffer researchers the opportunity to manage this complexity by isolatinga search-task factor that may influence information-seeking behaviorand information-retrieval performance.

4.4. External validity

As discussed, scenarios can be used to enhance the internal valid-ity of research, particularly when it is utilized in experimentalresearch designs, but with reduced external validity. External valid-ity relates to how much the situation depicted in a scenario genu-inely represents the phenomenon being explored. As portrayedsituations in the scenario are artificial, subjects must imagine whatit would be like for them in that particular situation, and the actualsituation may result in different responses by the individuals in-volved. Thus, there is no guarantee that the responses to a given sce-nario will mirror actual behavior of the respondent in real-lifesituations. Abbott and Sapsford (1993) criticized the artificial natureof scenarios by saying that they do not reflect the full picture. In ad-dition, participants' responses to questions about scenarios may failto reflect the complexity of real life. Even though Borlund's (2000)simulated work-task situation assumed that subjects' informationneeds would evolve in the same dynamic manner as their typicalworking lives, it often “failed to provide a comprehensive pictureof the impacts of contextual factors” (Choi, 2010, p. 2015), or reflectthe complexity of information seeking itself. For instance, usersoften search on more than one task during a single search interac-tion. How can we observe this multitasking aspect with givenscenarios?

This question also raised concern about whether the data collect-ed are comparable to data collected in real-life situations. It was in-teresting to note that Russell and Grimes (2007) comparedparticipants' own self-chosen tasks and assigned tasks in regards totheir behavioral differences; they found that personal search taskshave longer sessions and more browsing through the results sets,while assigned search tasks generate fewer unique queries. Poddarand Ruthven's (2010) study on the emotional impact of searchtasks was another attempt in this direction; they found the estimat-ed difficulty of own search task was lower than assigned searchtasks. When searching on personal search tasks, however, subjectsmade greater use of strategies. Therefore, as Carifio and Lanza(1989) pointed out, generalizations of results from simplified

305J. Kim / Library & Information Science Research 34 (2012) 300–307

scenarios to real situations in natural settings should be made withcaution.

5. Recommendations for using scenarios

5.1. How scenarios should be constructed

Scenarios must appear plausible and real to participants (Barter &Renold, 1999), depicting actual people or real-life situations. Asdiscussed earlier, threats to external validity can be minimized bymak-ing the approximation of reality as close as possible. The general princi-ple for constructing a realistic scenario is to describe specific situationsand frame the scenarios in a setting that is familiar to the participants.To achieve this goal, it is important to consider where the scenarioswere generated; naturalistic study of potential participants' everydayactivity is the appropriate method for scenario generation. While mostresearchers construct scenarios around their actual experience or gen-eral knowledge, only a few number of research works in LIS followedthis naturalistic approach. For instance, Urquhart and Crane (1994)consulted articles in the nursing literature and library users' queries toprovide problems related to nurses' area of practice. Li and Belkin(2010) constructed six scenarios based on actual work tasks collectedthrough semi-structured in-depth interviews with their potentialstudy subjects. Kules and Capra (2008) examined the log data fromthe online public access catalog in order to extract possible topics fortheir exploratory search-task creation. Zhang's (2012) scenarios camefrom Yahoo! Answers (answers.yahoo.com), a question-and-answerWebsite where people post their own questions and answer questionsposted by peer users. If naturalistic study is not available, literaturestudies and pilot studies are usually helpful. Some researchers con-firmed the feasibility of constructed scenarios by consulting with theirpotential study subjects (e.g., Kules & Shneiderman, 2008; Landry,2006). Others recycled previously constructed scenarios that were suc-cessfully employed in earlier studies in order to earn a proven record ofreliability (e.g., Gwizdka, 2010; Gwizdka & Lopatovska, 2009; Ruthvenet al., 2003; White et al., 2006).

Scenarios should be constructed in a clear way and consist of cer-tain basic elements. Before presenting scenarios to subjects, it is al-ways helpful to provide a brief instruction to subjects, encouragingthem to treat the hypothetical situation depicted in the scenario asif it were taking place in their own lives. The story should describethe topic, but also emphasize the context wherein the relevant infor-mation should be used—that is, “the work task from which the needfor information originated” (Balatsoukas & Demian, 2010, p. 461). Italso should include some details necessary to accurately depict thescenario's conditions and events. To minimize the external validityissue, however, the scenario approach leaves room for participantsto “define the situation in their own terms” (Finch, 1987). Barterand Renold (1999) asserted that scenarios should be vague enoughto compel participants to provide additional factors that influencetheir responses. This was reiterated in Borlund's simulated work-task situation, which “ensure[d] a degree of freedom for each individ-ual to react and respond in relation to his or her interpretation of thegiven indicative request for that particular situation” (Borlund &Ingwersen, 1997, p. 232).

Additionally, scenarios should be constructed in an appealing wayfor different types of participants. As discussed, searchers exhibited be-havior differences when searching their own tasks versus given tasksbecause of their level of interest; it was inferred by Bilal (2002) thatpeople are more motivated, challenged, engaged, and successful withtheir own topics rather than imposed or assigned topics. A number ofresearchers have recommended that participants be given searchtasks of interest to them in order to enrich theirmotivation and increasetheir challenges (Hirsh, 1999; Solomon, 1994). Nardi (1992) claimedthat a vivid, compelling description of scenarios is grounded in a coher-ent story that has the kinds of illustrative details that stimulate interest

and understanding; this always has been challenging in the controlledenvironment of searches.

5.2. How scenarios should be presented

Scenario formats are flexible, but the most frequent method ofadministering them is in written form. Consideration should also begiven to the length of time allocated for completion of the writtenexercises in interviews and survey questionnaires. In the case ofexperimental study, the length of time allocated for reading andexploring search task scenarios and completing the search itselfshould be considered.

The stage at which scenarios are introduced in the data collectionprocess is also important. Scenarios can be introduced in the begin-ning of an in-depth interview to break the ice and to frame and focusthe conversation on the research issue. It is common for theinterviewer to read situations aloud and then ask interviewees togive their typical responses to each situation. However, asking inter-viewees to read a short story may break the ice by encouragingsomeone other than the researcher to gain confidence fromspeaking.

The number of scenarios should be carefully selected to reduceparticipants' burden. Kelly (2009) noted that subjects are likely to be-come fatigued if they are in experiments that last over an hour, whichcan influence the results of the study. Generally speaking, the ade-quate number of scenarios depends on the time available and onthe difficulty and complexity of the scenarios. This should be deter-mined by a particular purpose of the study, however, and should beconsistent with the research question.

The order of presentation of scenarios is important to acknowl-edge, because randomizing the order can minimize threats to internalvalidity. Even though it is assumed that the repeated evaluation of alarger number of situations prevents respondents from depicting abiased and artificial attitude (Hechter, Ranger-Moore, Jasso, &Horne, 1999), there would be a learning effect because of repeatedpresentation of the scenarios. In addition, there is some evidencethat the order of scenarios might affect respondents' reaction to thequestion item (Sha & Pan, 2009). Search task scenarios employed inmost LIS research were rotated and counterbalanced among subjectsto reduce any order effect, because scenario itself does function asvariables that should be controlled (Kelly, 2009). This also helpedsustain participants' interests. The survey questionnaire and inter-view are no exceptions. Stefl-Mabry (2003), for example, randomizedthe order of 40 scenarios to minimize the danger that some respon-dents may become task-fatigued.

6. Conclusions

While not a new idea, this review shows that the use of scenariosin LIS has been rather limited. Few empirical studies have addressedrelevant methodological issues, and scenarios have not been widelyshared or published. However, as scenarios are commonly used tocollect empirical data on various aspects of information seeking andretrieval, and most likely will continue to be used in the future, meth-odological improvement is highly needed. Accordingly, researchersshould ensure that the scenario's content is credible, recognizable,and relevant to the research participants, and in line with the re-search question.

To support such effort, it is important to provide a comprehen-sive overview of published research using scenarios and develop alist of recommendations for using scenarios to best understandpeople's cognitive, emotional, and behavioral aspects, and to assessinformation-retrieval systems and processes. This review and cri-tique, which identified exemplary research efforts, will serve as amodel for future research using scenarios. In addition, the recom-mendations presented here will serve as a guide for researchers to

306 J. Kim / Library & Information Science Research 34 (2012) 300–307

consider and determine how to strengthen the research design of ascenario-based study.

Acknowledgments

This paper was based on a presentation at the TEFKO 2010 confer-ence which was held in New Brunswick, NJ on November 7–8, 2010.The author would like to thank Dr. Tefko Saracevic for helping hersee the forest through the trees throughout her academic journey.

References

Abbott, P., & Sapsford, R. (1993). Studying policy and practice: Use of vignettes. NurseResearcher, 1(2), 81–91.

Alexander, C. S., & Becker, H. J. (1978). The use of vignettes in survey research. PublicOpinion Quarterly, 42(1), 93–104.

Anderson, H. H., & Anderson, G. L. (1951). Introduction to projective techniques. EnglewoodCliffs, NJ: Prentice-Hall.

Aula, A., Khan, R., & Guan, Z. (2010). How does search behavior change as search be-comes more difficult? In E. Mynatt (Ed.), Proceedings of the 28th International Con-ference on Human Factors in Computing Systems (CHI'10) (pp. 35–44). New York,NY: ACM (Chair).

Balatsoukas, P., & Demian, P. (2010). Effects of granularity of search results on therelevance judgment behavior of engineers: Building system for retrieval andunderstanding of context. Journal of the American Society for Information Scienceand Technology, 61, 453–467.

Barter, C., & Renold, E. (1999). The use of vignettes in qualitative research. SocialResearch Update, 25 (Retrieved from http://sru.soc.surrey.ac.uk/SRU25.html)

Beaulieu, M., Robertson, S., & Rasmussen, E. (1996). Evaluating interactive systems inTREC. Journal of the American Society for Information Science, 47, 85–94.

Beldad, A., Jong, M., & Steehouder, M. (2010). Reading the least read? Indicators ofusers' intention to consult privacy statements on municipal websites. GovernmentInformation Quarterly, 27, 238–244.

Bilal, D. (2002). Perspectives on children's navigation of the World Wide Web: Doesthe type of search task make a difference? Online Information Review, 26(2),108–117.

Borlund, P. (2000). Experimental components for the evaluation of interactive infor-mation retrieval systems. Journal of Documentation, 56, 71–90.

Borlund, P. (2003). The IIR evaluation model: A framework for evaluation of interactiveinformation retrieval systems. Information Research, 8(3) (Retrieved from http://informationr.net/ir/8-3/paper152.html)

Borlund, P., & Ingwersen, P. (1997). The development of a method for the evaluation ofinteractive information retrieval systems. Journal of Documentation, 53, 225–250.

Campbell, R. L. (1992). Will the real scenario please stand up? ACM SIGCHI Bulletin,24(2), 6–8.

Carifio, J., & Lanza, M. (April). On the need to use control vignettes in social science re-search. Paper presented at the Annual Meeting of the New England Research Organiza-tion, Portsmouth, NH, April 26–28, 1989 Retrieved from http://www.eric.ed.gov/ERICWebPortal/detail?accno=ED319657

Choi, Y. (2010). Effects of contextual factors on image searching on the Web. Journal ofthe American Society for Information Science and Technology, 61, 2011–2028.

Courage, C., Redish, J., & Wixon, D. (2009). Task analysis. In A. Sears, & J. A. Jacko (Eds.),Human–computer interaction: Development process (pp. 33–53). Boca Raton, FL:CRC Press.

Finch, J. (1987). The vignette technique in survey research. Sociology, 21(1), 105–114.Fu, X. (2010). Towards a model of implicit feedback for Web search. Journal of the

American Society for Information Science and Technology, 61, 30–49.Giesecke, J. (1999). Scenario planning and collection development. Journal of Library

Administration, 28(1), 81–92.Gould, D. (1996). Using vignettes to collect data for nursing research studies: How

valid are the findings? Journal of Clinical Nursing, 5(4), 207–212.Gwizdka, J. (2010). Distribution of cognitive load inWeb search. Journal of the American

Society for Information Science and Technology, 61, 2167–2187.Gwizdka, J., & Lopatovska, I. (2009). The role of subjective factors in the information

search process. Journal of the American Society for Information Science and Technol-ogy, 60, 2452–2464.

Haas, D. F., & Kraft, D. H. (1984). Experimental and quasi-experimental designs for re-search in information science. Information Processing and Management, 20,229–237.

Hannabuss, S. (2001). Scenario planning for libraries. Library Management, 22(4/5),168–176.

Hansen, P., & Karlgren, J. (2005). Effects of foreign language and task scenario on rele-vance assessment. Journal of Documentation, 61, 623–639.

Hechter, M., Ranger-Moore, J., Jasso, G., & Horne, C. (1999). Do values matter? An anal-ysis of advance directives for medical treatment. European Sociological Review,15(4), 405–430.

Herskovits, M. J. (1950). The hypothetical situation: A technique of field research.Southwestern Journal of Anthropology, 6(1), 32–40.

Hirsh, S. G. (1999). Children's relevance criteria and information seeking on electronicresources. Journal of the American Society for Information Science, 50, 1265–1283.

Hughes, R., & Huby, M. (2002). The application of vignettes in social and nursing re-search. Journal of Advanced Nursing, 37(4), 382–386.

Hunter, I. M., Whiddett, R. J., Norris, A. C., McDonald, B. W., & Waldon, J. A. (2009). NewZealanders' attitudes towards access to their electronic health records: Preliminaryresults from a national study using vignettes. Health Informatics Journal, 15(3),212–228.

Ingwersen, P., & Järvelin, K. (2005). The turn: Integration of information seeking andretrieval in context. Dordrecht, The Netherlands: Springer.

Jeffries, C., & Maeder, D. W. (2011). Comparing vignette instruction and assessmenttasks to classroom observations and reflections. The Teacher Educator, 46(2),161–175.

Johnson, M., Newton, P., Jiwa, M., & Goyder, E. (2005). Meeting the educational needsof people at risk of diabetes-related amputations: A vignette study with patientsand professionals. Health Expectations, 8(4), 324–333.

Joho, H., Birbeck, R. D., & Jose, J. (2007). An ostensive browsing and searching on theWeb.In B. -L. Doan, J. Jose, & M. Melucci (Eds.), Proceedings of the 2nd International Work-shop on Context-Based Information Retrieval (pp. 81–92). New York, NY: ACM.

Joho, H., & Jose, J. M. (2008). Effectiveness of additional representations for the searchresult presentation on the Web. Information Processing and Management, 44,226–241.

Jose, J., Furner, J., & Harper, D. J. (1998). Spatial querying for image retrieval: Auser-oriented evaluation. In W. B. Croft, A. Moffat, C. J. van Rijsbergen, R.Wilkinson, & J. Zobel (Eds.), Proceedings of the 21st Annual International ACMSIGIR Conference on Research and Development in Information Retrieval (SIGIR' 98)(pp. 232–240). New York, NY: ACM.

Kelly, D. (2009). Methods for evaluating interactive information retrieval systems withusers. Foundations and Trends in Information Retrieval, 3(1/2), 1–224.

Kerlinger, F. N. (1986). Foundations of behavior research. New York, NY: Holt, Rinehartand Winston.

Kim, J. (2008). Task as a context for information seeking: An investigation of Websearching for daily life tasks. Libri, 58(3), 172–181.

Kim, J. (2009). Describing and predicting information seeking behavior on the Web.Journal of the American Society for Information Science and Technology, 60,679–693.

Kules, B., & Capra, R. (2008). Creating exploratory tasks for a faceted search interface.In D. Tunkelang, & R. White (Eds.), Proceedings of the SecondWorkshop on Human–Computer Interaction and Information Retrieval (HCIR' 2008) (pp. 18–21). Re-trieved from http://research.microsoft.com/en-us/um/people/ryenw/hcir2008/doc/hcir08-proceedings.pdf

Kules, B., & Shneiderman, B. (2008). Users can change their web search tactics: Designguidelines for categorized overviews. Information Processing and Management, 44,463–484.

Landry, C. F. (2006). Work roles, tasks, and the information behavior of dentists. Journalof the American Society for Information Science and Technology, 57, 1896–1908.

Li, Y., & Belkin, N. (2010). An exploration of the relationships between work task andinteractive information search behavior. Journal of the American Society for Informa-tion Science and Technology, 61, 1771–1789.

Meyers, E. M., Fisher, K. E., & Marcoux, E. (2009). Making sense of an informationworld: The everyday life information behavior of preteens. The Library Quarterly,79(3), 301–341.

Nardi, B. A. (1992). The use of scenarios in design. SIGCHI Bulletin, 24(4), 13–14.Papaeconomou, C., Zijlema, A. F., & Ingwersen, P. (2008). Searchers' relevance judg-

ments and criteria in evaluating Web pages in a learning style perspective. In P.Borlund (Ed.), Proceedings of the 2nd International Conference on Information Inter-action in Context (IIiX' 08) (pp. 123–130). New York, NY: ACM.

Pharo, N. (2008). The effect of granularity and order in XML element retrieval. Informa-tion Processing and Management, 44, 1732–1740.

Piaget, J. (1932). The moral judgment of the child. New York, NY: The Free Press.Poddar, A., & Ruthven, I. (2010). The emotional impact of search tasks. In N. J. Belkin

(Ed.), Proceedings of the 3rd International Conference on Information Interaction inContext (IIiX' 10) (pp. 35–44). New York, NY: ACM (Chair).

Rossi, P. H., & Anderson, A. B. (1982). The factorial survey research: An introduction. InP. H. Rossi, & S. L. Nock (Eds.), Measuring social judgments: The factorial surveyapproach (pp. 15–67). Beverly Hills, CA: Sage Publications.

Rowley, J., & Urquhart, C. (2007). Understanding student information behavior in rela-tion to electronic information services: Lessons from longitudinal monitoring andevaluation, part 1. Journal of the American Society for Information Science and Tech-nology, 58, 1162–1174.

Russell, D., & Grimes, C. (2007). Assigned tasks are not the same as self-chosen Websearch tasks. In R. H. Sprague (Ed.), Proceedings of the 40th Annual Hawaii Interna-tional Conference on System Sciences (HICSS' 07) (pp. 83–91). Washington, D.C: IEEEComputer Society Press.

Ruthven, I., Lalmas, M., & van Rijsbergen, C. J. (2003). Incorporating user search behav-ior into relevance feedback. Journal of the American Society for Information Scienceand Technology, 54, 529–549.

Saito, H., Terai, H., Egusa, Y., Takaku, M., Miwa, M., & Kando, N. (2009). How task typesand user experiences affect information-seeking behavior on the web: Usingeye-tracking and client-side search. In N. J. Belkin (Ed.), Proceedings of the Work-shop on Understanding the User (UUIR 2009) (SIGIR 2009 Workshop). New York,NY: ACM Retrieved from http://ceur-ws.org/Vol-512/paper05.pdf

Scenario (2012). In Oxford English dictionary online. Retrieved from http://dictionary.oed.com

Sha, M. M., & Pan, Y. (May). The use of vignettes in evaluating multilingual question-naires. Paper presented at the 64th Annual Conference of the American Associationfor Public Opinion Research (AAPOR), Hollywood, FL Retrieved from http://www.rti.org/pubs/aapor09_sha-pres.pdf

Shah, C., & Marchionini, G. (2010). Awareness in collaborative information seeking. Journalof the American Society for Information Science and Technology, 61, 1970–1986.

307J. Kim / Library & Information Science Research 34 (2012) 300–307

Solomon, P. (1994). Children, technology, and instruction: A case study of elementaryschool children using an online public access catalog (OPAC). School Library MediaQuarterly, 23(1), 43–51.

Stansbury, M., & Ludwick, R. (2009). Results of a factorial survey investigating thehealth information seeking behaviors of older adults. In A. Grove (Ed.), Proceedingof the 72nd Annual Meeting of the American Society for Information Science and Tech-nology. Medford, NJ: Information Today, http://dx.doi.org/10.1002/meet.2009.1450460268.

Star, S. A. (November 5). The public's ideas aboutmental illness. Paper presented at the Annu-al Meeting of the National Association for Mental Health, Indianapolis, IN Retrieved fromhttp://www.norc.org/PDFs/publications/StarS_Publics_Ideas_1955.pdf

Stefl-Mabry, J. (2003). A social judgment analysis of information source preference pro-files: An exploratory study to empirically represent media selection patterns. Journalof the American Society for Information Science and Technology, 54, 879–904.

Thatcher, A. (2008). Web search strategies: The influence of web experience and tasktype. Information Processing and Management, 44, 1308–1329.

Tombros, A., Ruthven, I., & Jose, J. M. (2005). How users assess Web pages for informa-tion seeking. Journal of the American Society for Information Science and Technology,56, 327–334.

Urban, J., & Jose, J. M. (2005). Exploring results organisation for image searching. In M. F.Costabile, & F. Paternò (Eds.), Proceedings of the tenth IFIP TC13 International Conferenceon Human–Computer Interaction (INTERACT' 05) (pp. 958–961). Berlin/Heidelberg:Springer (Chairs).

Urquhart, C. J. (1999). Using vignettes to diagnose information seeking strategies: Op-portunities and possible problems for information use studies of health profes-sionals. In T. D. Wilson, & D. K. Allen (Eds.), Proceedings of the 2nd InternationalConference on Research in Information Needs, Seeking and Use in Different Contexts(pp. 277–289). London, UK: Taylor Graham.

Urquhart, C., & Crane, S. (1994). Nurses' information seeking skills and perceptions ofinformation sources: Assessment using vignettes. Journal of Information Science,20(4), 237–246.

Weber, J. (1992). Scenarios in business ethics research: Review, critical assessment,and recommendations. Business Ethics Quarterly, 2(2), 137–160.

White, R. W., Jose, J. M., & Ruthven, I. (2003). A task-oriented study on the influencingeffects of query-biased summarisation in web searching. Information Processingand Management, 39, 707–733.

White, R. W., Jose, J. M., & Ruthven, I. (2006). An implicit feedback approach forinteractive information retrieval. Information Processing and Management, 42,166–190.

White, R. W., & Marchionini, G. (2007). Examining the effectiveness of real-time queryexpansion. Information Processing and Management, 43, 685–704.

White, R. W., & Ruthven, I. (2006). A study of interface support mechanisms for inter-active information retrieval. Journal of the American Society for Information Scienceand Technology, 57, 933–948.

Williams, D., & Coles, L. (2007). Teachers' approaches to finding and using research ev-idence: An information literacy perspective. Educational Research, 49(2), 185–206.

Wilmore, J. (2001). Scenario planning: Creating strategy for uncertain times. Informa-tion Outlook, 5(9), 22–28.

Xu, Y., & Wang, D. (2008). Order effect in relevance judgment. Journal of the AmericanSociety for Information Science and Technology, 59, 1264–1275.

Xu, Y., & Yin, H. (2008). Novelty and topicality in interactive information retrieval. Jour-nal of the American Society for Information Science and Technology, 59, 201–215.

Yuan, X., & Belkin, N. (2010). Evaluating an integrated system supporting multipleinformation-seeking strategies. Journal of the American Society for Information Scienceand Technology, 61, 1987–2010.

Zhang, Y. (2012). The impact of task complexity on people's mental models ofMedlinePlus. Information Processing and Management, 48, 107–119.

Zuccala, A. (2010). Open access and civic scientific information literacy. InformationResearch, 15(1) (Retrieved from http://informationr.net/ir/15-1/paper426.html)

Jeonghyun Kim is an assistant professor at the Department of Library and InformationSciences, College of Information, University of North Texas. She earned her PhD at RutgersUniversity. She has various areas of research interest ranging fromdigital libraries, humancomputer interaction, research methods, evaluation, and human information behavior, toconvergence issues surrounding libraries, archives, and museums.