Rank Xerox Research CentreCambridge Laboratory

61 Regent StreetCambridge CB2 1AB

Tel:+44 1223 341500Fax:+44 1223 341510

The Design of a Human Memory Prosthesis

Mik Lamming, Peter Brown,Kathleen Carter, Margery Eldridge,

Mike Flynn, Gifford Louie,Peter Robinson and Abigail Sellen

Technical Report EPC-1993-119

Copyright © Oxford University Press 1994.

Published in:The Computer Journal,Volume 37 (3), 1994,

pp 153-163.

The Design of a Human Memory ProsthesisMik Lamming‡, Peter Brown*, Kathleen Carter, Margery Eldridge, Mike Flynn,

Gifford Louie, Peter Robinson†, Abigail Sellen°

RANK XEROX CAMBRIDGE EUROPARC61 REGENT STREET, CAMBRIDGE, CB2 1AB

workplace and on technological possibilities fordealing with these problems. These guidelinesdefine this new class of application, provide thebasis for our continuing work in support of theproblems of everyday office life, and offer a newchallenge for computer systems research.

Key words: human memory, officesystems, memory prosthesis, informationretrieval, Personal Digital Assistant (PDA),mobile computing, distributed computing

2. THE PROBLEM

If you observe yourself in action, youwill probably see that the everyday proceduresyou employ to get your work done often involvesome process designed to compensate for yourfallible memory. You organise files to make themeasier to find, you keep lists, you set traps foryourself to remember to do things. Look aroundyou: you design your environment to makeremembering easier - file cabinets, notice boards,alarm clocks. How much of your life is spentmemorising and remembering small things thatare nevertheless important? How much is spentfailing to remember and then finding anotherway to get the job done? How much time isspent organising information to make it easier tofind or to ensure it does not get forgotten? Ourstudies show that these are common concerns:memory problems impose a hidden tax on work.

We believe that a new and distinctiveclass of applications, called “memory prostheses”is needed to tackle this memory tax. We expectthese systems to provide help with a range ofeveryday memory problems, including: findingfiles, papers and notes, in whatever medium theyare expressed, recalling names of people andplaces, procedures and lists, remembering to

* on sabbatical leave from the University of Kent atCanterbury

† on sabbatical leave from the University of CambridgeComputer Laboratory

° also at the MRC Applied Psychology Unit, Cam-bridge

‡ Corresponding author: Mik Lamming.lamming@europarc.xerox.com

Key words: human memory, office systems,memory prosthesis, informationretrieval, Personal DigitalAssistant (PDA), mobilecomputing, distributedcomputing

1. ABSTRACT

Memory is the forgotten problem ofoffice systems research. We believe that a newclass of applications, which we call “memoryprostheses”, are needed when memory problemsarise. We expect these systems to provide helpwith a range of everyday memory problems,including: finding files, papers and notes (inwhatever medium they are expressed), recallingnames of people and places, procedures and lists,remembering to perform tasks. A memoryprosthesis will be sensitive to its environmentand able to record data automatically about itsuser’s activities. These data can later be retrievedto help users remember things they haveforgotten, especially things they did not knowthey would need to remember. This sensitivity tothe environment will also enable the memoryprosthesis to issue context-sensitive reminders ofthings that the user intended to do.

In this paper we present guidelines forthe design of memory prostheses, drawing onstudies that have been carried out on thepsychological basis of memory problems in the

2

perform tasks. The differences between theseproblems mean that there will be many differentapplications but there are still certain featuresthat will be important for all memoryprostheses.

We envisage memory prostheses aspersonal systems, extending the user’s ability torecall things that are specific to his or her life.Such systems were foreshadowed by the Memexenvisaged by Vannevar Bush (1945), a devicethat would collect all of a user’s books, notes,comments, sketches etc. into a personal databasefor future use. A recent system calledMEMOIRS (Lansdale & Edmonds, 1992) treatsa personal filing system as a history of events,and allows retrieval of filed documents on thebasis of memorable characteristics of theseevents. The Memex was described as a desk-sizedelectromechanical device, and MEMOIRS runson a Macintosh II, but current advances intechnology will allow us to fulfil many similarfunctions in a small portable package.

Electronic organisers and more recentvariants such as personal digital assistantspurport to address these problems, but they arelittle more than high technology versions ofconventional diaries and note books, perhapsalso with the capability of an alarm clock. Weare considering something more radical: a devicethat will be sensitive to your environment andwill be able to support you in a much moreproactive manner. This device will help youremember things that you didn’t know you weregoing to need to remember at the time youencountered them and, by being sensitive toyour surroundings, will remind you inappropriate ways of things you had planned todo, whether in a particular context or at aparticular time.

A memory prosthesis should become anunobtrusive companion in everyday life,automatically capturing data that may be of uselater. It will not require data to be explicitlyrecorded or categorised, but on the other hand,it will allow you to make notes or annotate thedata if you wish. Given the large amounts ofdata that are captured, it will providemechanisms that help you to focus on relevant

information and to search flexibly for what youneed. In order to be a useful companion it willbe easy to use, reliable and available everywhere.It will be integrated with a wide variety ofcomputer applications to enable you to use themmore effectively. It will also be respectful ofprivacy, only recording the kinds of things thatyou wish to have recorded and making it clearwhat is being captured at any time.

These requirements have emerged fromwork that has been done in both computerscience and psychology. In this paper we presentthis groundwork and then, from this, developour guidelines for the design of memoryprostheses. First we consider psychologicalstudies into the nature of human memory andpresent the results of two studies into memoryproblems at work. Secondly, we summarise somerecent technological developments that open upnew possibilities for experiments with memoryaids. Thirdly, we describe some initialexperiments with applying technology tomemory problems. Finally we present in detailthe guidelines we have developed for memoryprostheses.

Our interest is in supporting healthypeople in their everyday work. We are notconsidering the problems resulting fromtraumatic brain injury or devising physicalimplants of any form, although our work mayhave implications in these areas.

3. PSYCHOLOGICALBACKGROUND

Understanding the processes underlyinghuman memory is important if we hope todesign effective technological support formemory in the workplace. Although there hasbeen a great deal of psychological research onmemory, little attention has been paid tomemory processes in the workplace. We need tounderstand what kinds of memory problemspeople have at work, how frequently they occur,and how troublesome they are when they occur.

3

Memory Problems at Work

Two studies were carried out todocument the kinds of memory problems peopleexperience at work (Eldridge et al, 1992). Thefirst was a diary study in which people wereasked to note down the memory problems theyexperienced during the course of their dailywork. The second was a questionnaire studybased on the results of the diary study.

The Diary Study

The main purpose of the diary study wasto understand the range of memory problemsthat occur at work. People at two researchorganisations in Cambridge (RX EuroPARC andthe MRC Applied Psychology Unit) were askedto write down their memory problems as theyoccurred. A total of 182 problems werecollected, and these were divided into threebroad categories:

retrospective problems -

problems with recalling past events orinformation acquired in the past;

prospective problems -

problems in remembering to carry outplanned actions; and

action slips -

problems arising from losing one’s trainof thought in the middle of some action.

About half of the problems reportedwere retrospective memory problems, eitherinvolving difficulty in recalling information, orin recalling some aspect of a specific event. Someexamples from the data collected are:

• We spent an awfully long timelooking for the name of someonewhose first name is John, who worksin the EEC —and I can’t rememberfor the life of me what his secondname is. So we’re looking to see if wecan find it.

• I’ve just forgotten the word for“taking a responsibility”.

• Confusion about whether I hadgiven Mary something (somedocuments). Had no recollection ofthe event, let alone what happened.

• Lost expenses claim form I wassupposed to fill in. Hunted high andlow. Then picked up a file I had tostart work on and found it there.

• Forgot how to e-mail John inCalifornia.

About a third of the problems reportedwere prospective memory problems. Someexamples from the data collected are:

• Had to phone my doctor. Hadintended to do this last week butnever remembered. At lastremembered, but number wasengaged. Forgot to try again.

• I forgot that I was supposed to askJohn Smith about a meeting.

The remaining problems reported wereaction slips. These are somewhat different innature from the previous two categories in thatthey are short-term and immediate. Most are theresult of inattention, either due to externallygenerated distracting events, or due to general“absent-mindedness”. Examples from the datacollected are:

• Wrote a letter saying I would enclosea program for a meeting, and thensealed up the envelope, omitting theprogram.

• When I left the building, someoneopened the door for me. I quicklywalked out. The next morning I sawmy “name switch” said “IN”.

• Off to fax room with letter to befaxed—forgot to take the fax number(again!).

Action slips do not involve deliberateattempts to remember, as is the case withprospective memory problems. Thus, it is notlikely that a person would be likely to use amemory aid for these temporary and often quite

4

semantic memory is a person’s knowledge of theworld that is shared by others. Although somememory prosthesis applications may supportprocesses of semantic memory (i.e., memory forfacts), our primary focus is on the processes ofepisodic memory. One part of our goal is toprovide support for remembering past activities,even those activities you think you will neverneed to remember.

There are three characteristics of episodicmemory which are of particular importance.First, the basic unit of information in episodicmemory is an episode. The dictionary defines anepisode as an “event that is distinct and separatealthough part of a larger series”. Thus, as Tulving(1983) points out, one can think of a person’slife as being a series of successive episodes.

Second, although people are notoriouslypoor at remembering the precise date of anepisode (Wagenaar, 1986), they are often verygood at remembering temporal relationshipsbetween episodes (e.g., Loftus & Marburger,1983; Saywitz, Bornstein, & Geiselman, 1992).For example, if you were trying to rememberwhen you posted a letter to a friend, you mightnot be able to remember exactly when you hadposted it, but you might well remember that youposted it just before you went on your summerholiday, and just after you had a phone call fromthat person.

Third, presenting partial informationabout an episode can help people remembermore information about that episode (Brewer,1988; Cohen & Faulkner, 1986; Linton, 1986;Wagenaar, 1986). For example, if you weretrying to remember the name of someone youmet at a meeting last week, you might be morelikely to remember the person’s name if someonere-created the meeting for you by telling youwho else was there, where the meeting tookplace, and so on.

We aim to capitalise on these threecharacteristics of episodic memory in our designof technological support for human memory.For example, consider the problem of trying tofind a recently-filed letter from your managerabout your performance appraisal. The relevant

unpredictable lapses. For this reason, action slipsare not addressed in the general design of thememory prostheses.

Although this diary study was carried outin research organisations, other diary studies ofmemory problems in everyday life (e.g., Crovitz& Daniel, 1984; Terry, 1988) have found similarranges of problems. To our knowledge, no otherdiary studies of memory problems in the work-place have been carried out.

The Questionnaire Study

To better understand the relativefrequencies and severities of the different kindsof memory problems, a memory lapsequestionnaire was designed. The questionnaireconsisted of 21 questions, covering the threebroad categories of problems identified in theDiary Study. The frequency of occurrence wasrated for each question on a 5-point scale,ranging from “daily” to “never”. Thequestionnaire was distributed to approximately300 people at a Rank Xerox engineering site,and 118 completed questionnaires werereturned.

The results of this study showed that themost frequently occurring retrospective memoryproblems were: forgetting a person’s name,forgetting the location of a paper document, andforgetting a word or phrase. As a general class,prospective memory problems (forgettingplanned actions) were also rated among the mostfrequent, as well as being rated as fairly severewhen they occurred.

Retrospective Memory

The first class of memory problemsreported in the Diary Study, retrospectivememory problems, is by far the most thoroughlystudied class in the psychological literature onhuman memory. Many distinctions have beenmade within the area of retrospective memory,one of the most well-known being Tulving’s(1983) distinction between episodic andsemantic memory. Episodic memory (or“autobiographical memory”) is a person’smemory for unique personal experiences, and

5

episode is “filing the letter”, but you do notremember where you filed it. However, you doremember that you filed it before you attended aseminar by someone from the University ofDundee. You also remember that you receivedthe letter just after your appraisal, and thatappraisals occur in August. Now imagine thatyour memory prosthesis has stored informationabout when these other episodes (“attending theseminar” and “having your appraisal”) occurred,thus allowing you to pinpoint when the letterwas filed. Imagine also that your memoryprosthesis has stored information about everydocument you filed for this period of time,along with many pieces of information abouteach document (e.g., who wrote it, what it wasabout, where it was filed, etc.). By telling yourprosthesis what you do remember about theletter, it can now tell you where you filed it.

In summary, then, a memory prosthesisshould provide: information in a form whichallows people to retrieve past episodes;information about many different episodes andthe temporal relationships among them; andinformation about many different characteristicsof episodes. With a memory prosthesis designedfollowing these principles, you may never haveto “remember” the information yourself-thememory prosthesis will remember it for you.

Prospective Memory

The second class of memory problemsreported in the Diary Study, those involvingprospective memory, is an almost unexploredarea of research within the study of humanmemory (see Kvavilashvili, 1992 for a review).However, both the Diary and the QuestionnaireStudies highlight its importance as a class ofproblems which needs technological support.These studies confirm that remembering one’sintentions at some point in the future, withoutthe use of memory aids such as lists,appointment diaries, and alarm clocks, is a verydifficult task. The most difficult part of theremembering task is “remembering toremember” at an appropriate point in time.Once one “knows” that one needs to remembersomething, recalling exactly what that was is notusually a problem.

The Diary Study also illustrates thatthere are different kinds of prospective memoryproblems. Some are “time-based” in that thingshave to be remembered either at some specificpoint in time in the future (such as anappointment or meeting), or within a timewindow (such as remembering to post a birthdaycard). Other problems may be called “event-based” in that the thing to be remembered iscontingent on the occurrence of some event. Forexample, trying to remember to pass on atelephone message to a colleague is contingenton encountering that particular person.Remembering to make a phone call when youget back to your office is a case whereremembering is dependent on entering a specificplace (i.e., the office). Thus people, places, andactivities form the cues that should trigger event-based remembering. Unfortunately, such cuesare often ineffective in everyday life.

A memory prosthesis for prospectivememory needs to be sensitive, therefore, notonly to intentions which must be triggered atparticular times, but also to intentions whichdepend on the occurrence of particular events.Because plans and intentions can occur to usanywhere, at any time, we should be able to setreminders for ourselves when they occur, beforewe forget them.

Implications for the Design ofTechnology

The results of both the Diary andQuestionnaire Studies emphasize the wide rangeof problems that are reported as both frequentand troublesome. One of the main implicationsof these studies for the design of technology isthat different memory problems will require verydifferent kinds of memory aids. Problems occurin a variety of situations, and a range of retrievalcues may be useful depending on the problem. Amemory aid developed to help people rememberproper names may have features which are quitedistinct from one designed to support thelocation of paper documents. Applicationsrequired to support prospective memoryproblems will again be fundamentally differentfrom those required to support retrospectivememory problems. Although these applications

6

need to be tailored to fit the problem, there arenonetheless some requirements for technologicalsupport of memory which are more general, andwhich are outlined in the following sections.

4. TECHNOLOGICAL BASIS

As we have seen, for a memory prosthesisto be useful to us it must know a lot of thingsabout us, such as where we are, what we aredoing, who we are with. This will enable it toprovide us with reminders of things we intendedto do in this context, and, by recording suchinformation, it will later be able to help us torecall details of events that would otherwise beforgotten. The more personal information itknows the more potential it has to be of help tous. We call this kind of device an “IntimateComputer”. Ideally an intimate computer wouldbe one that goes with us everywhere and is ableto sense and record things about ourenvironment that may be of use to us later.Fortunately, computer technology is about toundergo yet another major revolution that opensup exciting possibilities for such devices.

Using cellular radio and infraredtechnology computers are able to communicatewith each other without wires. Until recently,radio transceivers were large and power hungry,so much so that the machines to which thetransceivers were attached were also fairly large.Credit card sized radio transceivers are nowavailable as commercial products (NCR, 1993)and we only have to look a short while into thefuture to see mobile computers that will be smallenough to be worn rather than carried - perhapsresembling a watch or piece of jewellery. Peoplewon’t have to remember to take their computerwith them, they will wear it and take iteverywhere.

Such systems will have severalfundamental capabilities not previously availableon such a wide scale. They will communicate,not only with each other, but with officeequipment, domestic appliances and theplethora of electronic equipment that alreadycontain one or more single-chipmicroprocessors. We are comfortable in thisassumption because inexpensive communication

facilities will inevitably become just anotherstandard part of tomorrow’s single-chipmicroprocessor.

The wireless communication technologyused by these systems will be cellular, perhapsbased upon the new digital cellular telephonestandards. The requirement for low power usageand the need to handle many transceivers in asmall area means that the cells will be small, eachcell being connected by wire or higher-poweredradio to its neighbours. The consequence of thisis simple yet profound. Computers will knowwhere they are. To find out their location theysimply ask the nearest non-mobile object, forexample, the base station connecting the localwireless cell to the building’s conventional wire-net.

So, to summarise the anticipateddevelopments in technology:

• computers will be small enough towear and take everywhere;

• they will be embedded in domesticappliances, office and consumerequipment;

• they will talk to each other usingcellular wireless communications;and

• they will know where they are.

Commercial products

A number of computer and electronicequipment manufacturers have recentlyannounced products that nearly meet thesespecifications. One of the first plausible productson the market is the PenPad from Amstrad inthe UK; but, unfortunately, it has no built-inwireless communications and can only take aRelease 1.0 PCMCIA card (credit-card sizedperipheral), precluding the use of plug-incommunications. The EO 440 and 880 supportradio communications, but use conventionalcellular telephone technology which means thattheir cells are too large for fine-grain location.More interesting are the Casio/Tandy Zoomerand Apple/Sharp Newton MessagePad devices,which support infrared wireless

7

communications, but which are only justbecoming available.

Moreover, these are all significantlybulkier than we would like. We have thereforebecome involved with colleagues in theComputer Science Laboratory of Xerox PARCin the development of suitable portablecomputers as an interim solution.

The PARC Tab

The resulting device is known as thePARC Tab (see Figure 1) developed by Want andothers (Schilit et al., 1993). It is small (100 mmx 75 mm x 25 mm, or 4" x 3" x 1") and light(225 gm, or 8 oz.) and can be worn eitherclipped to the user’s belt or hung from a pocket.It has a bitmapped, touch-sensitive screen with aresolution of 128 x 64 pixels. In addition thereare three buttons and two-way infraredcommunications. The user can interact with theTab by pressing the buttons or by pointing witha finger or stylus on the screen. Power is suppliedby rechargeable batteries that give a full workingday’s use from an overnight charging.

Each Tab has a beacon that is used toindicate its location in much the same way as anOlivetti infrared active badge (Want et al., 1992)and a more sophisticated protocol maintains a

link using infrared, serial line and Ethernetcommunications between each Tab and a homecomputer associated with its owner (Adams et

al., 1993). A suite of librarysoftware presents the Tab to

application programs as asimple bitmap terminal.

We believe that theTab provides a feasiblevehicle for implementing amemory prosthesis, both

by building up theautobiographical data used by

the system, and by providing a userinterface to it.

5. INITIAL EXPERIMENTS

Exploratory work on applications tosupport both retrospective and prospectivememory has been carried out. Our experiencewith these applications, along with thepsychological studies, provides the basis for theguidelines that we develop in the next section.

Systems to Support RetrospectiveMemory

The Activity-based Information Retrieval(AIR) project (Lamming & Newman, 1992) wasthe beginning of many of the ideas related to thememory prosthesis. The goal of the AIR projectwas to improve the ability of the user to retrieveinformation and recall past events in theworkplace. The basic idea was first to collectinformation about activities automatically, andto time-stamp each activity as it occurred; and,second, to translate these data into descriptionsthat correspond to the way users wouldthemselves describe their activities. Thus aperson could use easily remembered informationabout events to help recall other less memorableinformation.

The AIR project led to a number ofpieces of work, summarised here, to test thefeasibility of some of these ideas. Theseexplorations were inspired both by technologicalpossibilities and the psychological findingsdiscussed earlier.

The PARC Tab

8

Pepys

A paper by Newman et al (1991)describes an early project to use automaticallytime-stamped data about peoples’ movementsaround our laboratory. The Olivetti infraredactive badge network (Want et al., 1992) atEuroPARC senses the locations of badges aroundthe building and these data can be gathered togive a log of the locations of people wearingbadges. The raw badge data are very detailed, forexample walking from an office to the commonroom to meet a visitor will generate events at awhole series of sensors along the way.

In the Pepys project techniques weredeveloped for extracting significant episodesfrom the raw badge data. These included periodsspent mostly alone, gatherings of two or moreusers (e.g. meetings) and travel by users betweenlocations. Part of a typical diary generated in thisway can be seen in Figure 2. Discounting thoseepisodes in which errors occurred due to peoplenot wearing badges, at least 90 percent ofepisodes are correctly identified. This has beengood enough to encourage people to make use of

Pepys for such things as reminding them of theneed to follow-up activities from the previousday. Visitors to EuroPARC have used Pepys tohelp in writing trip reports.

On the other hand, the episodesrecognised by Pepys are purely location basedand often lack sufficient detail to make themdistinct. For example, “In Office [2hr 20m]”does not give much information about whatactivities were actually being done in the office.Several other experiments were carried out withother recording methods in order to providefurther detail to the Pepys diary.

Video Diary

The Video Diary provides a way ofaugmenting the Pepys diary with videosnapshots (Eldridge et al, 1991). The videonetwork around the building (Buxton andMoran, 1990), directed by the active badgenetwork, enables the recording station to switchto the camera nearest a particular person as theymove around the building. Any significantchange in the image being received triggers the

14:14 In office [50 mins]

15:04 In and out of event in Nathan’s office; with W. Nathan, R. Hatton [45 mins]

15:50 In office [10 mins]

16:00 In Conference room [4 mins]

16:05 Attended part of event in Commons; with B. Andrews, M. Morton, R. Hatton [7mins]

16:13 Mostly in office [44 mins]

16:57 Attended event in Wright’s office; with P. Wright [7 mins]

17:04 Looked in on event in Morton’s office; with I. David, M. Morton [1 min]

17:05 Mostly in office [2h 3m]

17:05 In office [5 mins]

17:11 In event in office; with P. Wright, I. David [1h 2 mins]

18:13 In office [36 mins]

18:50 Meeting in office; with W. Nathan [13 mins]

19:03 In office [5 mins]

19:09 In 2nd floor rear area [2 mins]

19:11 Last seen

A Typical Pepys Diary

9

recording of a frame. The study by Eldridge et al(1991) tested the ability of the video diary to aidrecall. Considerably more activities were recalledwhen users viewed their video diaries than whena Pepys diary was used, especially when a monthor more had elapsed since the original activities.

NoTime

Although Pepys can inform its user thata meeting occurred, it can provide little usefuldata on the content. Hence other methods areneeded to capture this information. NoTime(Lamming, 1991) is a system that electronicallycaptures the hand-written notes normally madeby a person attending a meeting. The electronicnotepad links each note to the appropriate partof an audio and video recording that is made ofthe meeting. Each stroke is given a time-stampand this is used as the link to the recordings.Later, when the user sees “In conference roomwith Mik and Peter 1hr 45m” she can go back tothese notes and find out more about the contentof the meeting. Because each stroke in the notesis time-stamped she can listen to selected partsof the meeting relating to points in the notes, forexample if the notes are incomplete or unclear.Other projects have successfully used theprototype to gather and analyse annotated videodata.

Monitoring Paperwork

Long periods spent alone in the officealso provide little detail for Pepys, and the VideoDiary presents only a view of the office, and nodetail of paper documents. A system thatrecognises and tracks paper documents couldcontribute details of paper-based activitiesduring these periods.

A prototype system, called Marcel(Newman & Wellner, 1992), is able to recogniseactivities involving paperwork by using a videocamera mounted over the user’s desk. Images ofdocuments are digitised, processed and thencompared with a database of known documents.Recognition is sufficiently fast and reliable tomake this a plausible approach for logging thearrival and departure of documents on the desktop.

Monitoring Workstation Activities

Monitoring workstation activities isanother way to provide further useful details ofactivities in the office. For example, recordingwhich electronic files were touched could aid inthe retrieval of documents relating to particularactivities. Similarly, recording sequences ofworkstation commands could help the user recallcomplicated procedures performed earlier.

Experiments have been carried out togather various kinds of workstation activity data,such as what the user types, which files are openand the creation and destruction of processes onthe machine. There were various technicaldifficulties to be overcome, but a large quantityof potentially useful information was gathered.The major problem is in processing the raw datainto higher level episodes that match the user’srecollections. The records were very detailed andextensive, and although it is possible for a skilleduser to recognise what they were doing, it takesconsiderable effort. We anticipate more successby directly instrumenting applications togenerate more meaningful data.

A System to Support ProspectiveMemory

Prospective memory either involvesdelaying an action until some future pre-specified time or until some pre-specified event,such as entering a room, meeting a person, orperforming an activity. This suggested areminding system which allows reminders to beset not only on the basis of time, but also on thebasis of place, people, and activity. Sinceintentions can spring to mind any time, anyplace, another requirement of the system is thatit be portable, allowing users to set reminderswherever they are.

A prospective memory experimentalprototype was built to explore ways of providingreminders about intentions (Sellen, Lamming, &Louie, 1992). This system used the existingKhronika system (Lovstrand, 1991) inconjunction with active badges. The system waslimited in that it relied on the audio-videonetwork (Buxton and Moran, 1990) to play

10

audio reminders, required that reminders be setfrom a workstation, and could not effectivelydeliver the content of a reminder (i.e., what theuser wanted to be reminded about). However,this prototype was useful in showing howreminders based on time, place and people couldbe implemented.

The system is now being implementedon PARC Tabs. Tabs are ideal for this kind ofapplication in that audio reminders can begenerated by the Tab itself; reminders can be setany time and in any place within range of Tabsensors; and the contents of the reminder can bedisplayed on the Tab screen as not much screenarea is required for a typical reminding message.

Future plans include incorporating“smart” reminders so that the delivery ofreminder cues will be sensitive to certain aspectsof the environment. For example, one does notwant a series of bleeps going off during aseminar, and so in this situation audio cuesmight be replaced by visual ones. Notificationmight also be sensitive to whether the user is onthe telephone or is in a meeting. Another featureof this system is that it might monitor the userto see whether planned actions are actuallycarried out. For example, after playing areminder to attend a meeting, the system couldmonitor the movements of the user to seewhether that person went to the meeting. If not,the system could replay the reminder. Both thebasic functionality of the reminding system, aswell as the utility of features such as thosedescribed above, will be tested and refined bystudies of use within an actual workplace.

6. THE DESIGN OF A MEMORYPROSTHESIS

Memory problems are a serious challengein office automation. Our workplace studieshave revealed a range of problems, particularlyretrospective memory problems concerned withrecall for past events and prospective memoryproblems concerned with remembering to carryout intended actions. Psychological studies haveshown the value of partial information aboutpast events in aiding recall of further details.

Information about the location, the peoplepresent and the kinds of activities they wereinvolved in are of special value. In the case ofprospective memory, actions are often intendedto be carried out in a particular location or whenmeeting a particular person but frequently theperson forgets the action when they actuallyencounter that context. These kinds of problemshave led us to define a new class of applications,called “memory prostheses”, that providesupport when these problems arise. The natureof human memory problems makes memoryprostheses distinctly different from any currentoffice information system.

The central feature of these applicationsis the ability to sense aspects of their user’senvironment and make records which can thenbe used later to enable the user to recall thingsthey did not know they were going to need toremember. This sensing of the environment alsoenables the issuing of context-sensitivereminders, an important feature of support forprospective memory. The various experimentsreported above demonstrated the feasibility andutility of making records of a person’senvironment and activities. However, analysingand presenting the data in a form that makessense to users and does not overwhelm themwith a mass of detail was found to be achallenge. On the other hand, the data relate toa person’s own experiences and thus provide aframework for interpretation and triggeringrecall of what they need to know, even if thatinformation is not present in the data.

We are now in a position to present ourcurrent design guidelines for memory prosthesesand to lay out the challenges that these presentfor computer systems research. The first fourguidelines are specific to the functionalityprovided by a memory prosthesis:

a: Sensing of the environment

b: Automatic data capture

c: Manual data capture, note taking,annotation

d: Focussing on relevant informationfor retrieval

11

The remaining guidelines are normalrequirements for any interactive system but havea particular interpretation in the context of amemory prosthesis:

e: Easy to use

f: Available where needed

g: Integrated with other applications

h: Reliable and fail-safe

i: Respectful of privacy

In what follows we elaborate on each ofthese guidelines.

The Guidelines in Detail

a: Sensing of the environment

A memory prosthesis must be able tosense its external environment if it is to be ofhelp to its user. Experiments with Pepys and thevideo diary have shown that the moreinformation a system can gather about the user’sactivities then the more help it can be wheninformation needs to be recalled later. Thissensing of the environment is also important inproviding support for prospective memory.Many intentions are related to particular peopleor places, such as remembering to hand over amessage to someone when you next meet themor remembering to phone someone when youreturn to your office. A memory prosthesis thatis sensitive to where you are and who you arewith can provide reminders at an appropriatetime. This sensitivity to the context also enablesreminders to be delivered in a mannerappropriate to current activities. For example, avisual rather than audio reminder would beissued in the middle of a seminar.

A wide range of aspects of theenvironment can be sensed. We have alreadyexperimented with information from an audio-video network, from an active badge network(yielding people’s locations) and from softwarethat recognises documents from video images ofdesks. Many other things can also be sensed,such as whether a door is open or closed orwhether someone is using the telephone or someother electronic device.

b: Automatic data capture

A memory prosthesis is intended to haverecords of information that will be of help in anunspecified future situation when something isforgotten. The problem with many currentinformation systems is the need to recognise thatinformation is going to be important and shouldbe stored. The user also needs to decide how tocategorise what is recorded so that it can beretrieved later. It is often not possible torecognise what will be needed in future or inwhat way it will need to be retrieved, and as aresult the information may be irretrievably lost.A memory prosthesis that relied on explicitrecording and on-the-spot categorisation wouldbe less useful. Hence it should automaticallycapture as much data as possible, with nointervention from the user. Minimal structureshould be imposed at the time of recording sothat a variety of categorisations can be appliedlater.

Automatic capture should potentiallycover the whole range of what can be sensed bythe memory prosthesis. In addition to theimmediate aspects of a person’s activities that wehave already mentioned, there are many otherthings that can provide powerful cues forremembering, e.g. the weather and the newsheadlines. Services exist to provide such data on-line and so these too can be captured into therecords. Instrumented software is anotherpotential source of useful data.

Certain aspects of support forprospective memory might also be handledautomatically. For example, centrally-organisedevents could be entered through an eventsdatabase such as Khronika (Lovstrand, 1991).The user can set up daemons that detectparticular kinds of events in the central databaseand then carry out some action, which couldinclude entering it into their personal records.

c: Manual data capture, note taking,annotation

There will be situations in which theuser wishes to make some explicit record, such asa note or sketch. This should be simple to do

12

and the record should then become part of theon-going collection of information about theuser’s activities. The device should also supportthe annotation of existing records, whetherautomatically or manually generated. The kindof functionality provided by NoTime, describedearlier, is a step towards this. It demonstratesnote-taking at the time of recording, and similartechnology could be envisaged for annotation ofvideo records after the event. A fully functionalmemory prosthesis would allow annotation ofany kind of record.

The system for supporting prospectivememory is one important aspect of a memoryprosthesis that cannot rely entirely onautomatically recorded data. The user will needto enter many future events manually and willalso want to set up various kinds of remindersassociated with future events. Such reminderscan be seen as a special kind of annotationassociated with the event.

d: Focussing on relevant information forretrieval

The automatic recording will generatelarge quantities of data to be searched. Amemory prosthesis should provide simple waysfor the user to constrain the search to relevantparts of the archive.

Different memory problems requiredifferent strategies for solving them and somemory prostheses should provide for a range ofstrategies. The user may be able to rememberenough to specify accurate retrieval of an item,or at the other extreme they may only remembera little and therefore will want to browse aroundguided by what cues they do recall. The usershould be able to switch freely betweenconstrained pattern-matching and free-rangebrowsing.

This is a case of a guideline where theparticular nature of the data being searched orbrowsed simplifies the requirements. Gettinglost is a serious and well-known problem whenpeople browse large quantities of data (Nielsen,1990). An advantage of the memory prosthesis isthat the data is all related to events in a person’s

life. Although the user may have forgotten manyof the details, their memories, along with thenatural temporal organisation of the data, willhelp to orient them (Loftus and Marburger,1983; Saywitz, Bornstein and Geiselman, 1992).This should make it much easier for them tonavigate than would be the case with completelynew and unfamiliar data.

The kinds of cues remembered will varyand so the search could be started with any of anumber of kinds of information. For example,we may remember that we last had the lostdocument when we were with a particularperson in a certain room and so wish to startfrom person and location information. On theother hand, we may remember that it was on theday when we had a very hot and sticky officemeeting, and so we start from the office calendarand the weather records.

The information we wish to retrieve willvary, too. If we have lost a document, we maywish to follow our movements from the time welast remember having it, to see if we are thenreminded of the place we might have left it. Butif we simply remained in our office for a longperiod, then snap shots from a video diary orinformation from a document recogniserwatching our desk would be more helpful.

This approach to retrieval provides a newapproach to the management of electronicdocuments in general. Currently suchdocuments must be explicitly named and storedin an appropriate location. The name andlocation must then be remembered in order toretrieve the document. This is particularly adisincentive to producing small electronic notesabout miscellaneous things that arise as theoverhead involved in ever finding them again istoo great. A memory prosthesis should associatea rich variety of retrieval cues with any electronicnote such that we can still retrieve it even if wenever got around to naming or categorising it.

e: Easy to use

Quantifying “ease of use” is hard. But fora memory prosthesis we know that it must beeasier to use than other means of recalling

13

something that has been forgotten. This meansthat it must have a user interface that readilyreveals how it should be used and how thesystem records, stores and retrieves information.

A memory prosthesis has to capture avariety of different types of information andallow it to be browsed using a single userinterface, whatever the type of data. The systemshould also convey the status of the informationthat is presented to the user, ranging from itstechnical reliability (e.g. was the systemunavailable for a time because of maintenancework), through conveying in-built limitations(e.g. moving out of range of the wirelesscommunication system) to user-imposed privacyconstraints.

For the users to feel in control of theirpersonal data it is important that they can seeclearly what information is being revealed toothers and how they can control this. An unclearand complex system makes it unlikely that userswill be able to take this control.

The information retrieved should bepresented in a human-recognisable form. Forexample, in the Pepys experiments raw locationdata were processed into chunks such as“meetings” that approximated more closely tothe user’s experience. Such processing will alwaysbe an approximation, but it makes the data moremanageable and recognisable.

f: Available where needed

A traditional diary is available whereverthe person carries it and a memory prosthesisshould be similarly available for both captureand retrieval, provided it is within range of thenetwork. Events of interest happen in both the“physical” and “electronic” worlds and so thisdevice should record activities in both.Obviously there will be limits to this, both inphysical extent and in the kinds of datarecorded. The system must make it apparentwhere these limits are so that the user knowswhen and what it can be relied on to record.Similarly when information is being retrieved, itshould be clear what the limits of theinformation are.

One way to achieve ubiquitousavailability is with a small, portable device suchas the PARC Tab, combined with a network.Ideally the device should be sufficiently smalland unobtrusive that it can be carried at alltimes. Another form of ubiquity is to have non-portable devices available in any location wherethey might be needed. In fact, portable and non-portable devices can play complementary rolesin an integrated system. Small devices are idealin situations such as meetings where aworkstation would be obtrusive andinappropriate. Brief notes could fairly easily bemade on the small screen. On the other hand,the small screen is highly constraining, forinstance if you wished to browse quantities ofprevious notes. In such a situation it may bemore appropriate to find a nearby workstationwith a large screen. Memory prosthesisapplications should move gracefully betweendevices in such a mixed environment.

g: Integrated with other applications

We expect a memory prosthesis tofrequently be used during the use of othercomputer applications, for example to retrievemail addresses, find files, refer to notes and soon. It should provide an integrated set offacilities that can be accessed by the designers ofother applications so that memory prosthesisfunctionality can be made available whereever itis needed. It could be thought of in a similar wayto file-browser “widgets” in that it will provide aconsistent way to browse through the memoryprosthesis records, whatever those records willeventually be used for.

Another essential form of integration isthe collecting of records from other applications,such as the creation of files in a documentprocessing system or the sending of electronicmail to particular people. All such data could beuseful as cues for remembering.

A successful memory prosthesis willintegrate seamlessly into the user’s normaleveryday activities and be available to providehelp when needed.

14

h: Reliable and fail-safe

If a user is to depend on this system forremembering things, it should be very reliableand fail in safe ways. It should always be obviouswhen it is not working correctly and it should beclear to the user what strategies should beadopted to deal with this.

An analogy with the telephone networkmight be helpful here. If your own telephone isnot working then the next thing you try to do isuse a neighbour’s telephone to report theproblem and request that it be fixed. If all yourneighbours’ telephones are not working then itmust be a much larger, central problem and sothere is nothing more you can do. An interestingproblem with current telephones is that anonworking telephone is not apparent until youtry to make an outgoing call. You might happilysit waiting for a call on a nonworking telephonewithout knowing this. We would wish a memoryprosthesis to make it much more obvious that itis not working correctly.

System failures should also be apparentin the records when they are browsed later. Forexample, it should be possible to see whetheryour absence from the record was because youwere away that day, or because your active badgewas not working or because the whole networkhad failed.

i: Respectful of privacy

People treat traditional paper diaries aspersonal and usually confidential. The memoryprosthesis should similarly allow its user tomaintain his or her desired level of privacy. Animportant underlying requirement here is thatthe memory prosthesis convey a clear andcoherent model of its behaviour to its user. Theuser should be presented with no difficulties inunderstanding what is being recorded, what isbeing revealed to others and how all this can becontrolled. Different people may have differentboundaries between what they consider accept-able behaviour and what they consider to beintrusion. In addition, the same person mayhave different boundaries in different contexts.Hence the system must be easily tailorable at anytime.

Interesting questions arise about therecording and reuse of publicly availableinformation about a person. A record of mylocation at a particular time will appear in therecords of other people who meet me. Theremay also be an audio recording of ourconversation which also becomes part of bothmy records and those of the other person. It isunclear what are appropriate policies, and nodoubt these will vary between people and workorganisations. What we can say is that a memoryprosthesis should allow for a variety of policiesand a high level of personal control. Furtherissues arise about how and whether a user isinformed of the preferences of another person, asrevealing these could itself be considered anintrusion.

10. CONCLUSION

We have defined a new class ofapplications to support human memory that wecall memory prostheses. The guidelines we havedeveloped for the design of these applicationsderive from psychological studies of memory inthe work place as well as from technologicalconsiderations. A person’s memory for details ofpast events can be triggered by providing partialinformation. It has been demonstrated thatsensing and recording a variety of informationabout a person’s activities, such as location andother people present, can be a useful aid toremembering. We have also shown the feasibilityof providing people with context-sensitivereminders to carry out intended actions. Thisclass of applications will be sensitive to theirenvironment, able to record what they sense andable to present the recorded data in acomprehensible and useful form. Many of theguidelines are similar to those for more generalinformation capture and retrieval but theparticular nature of human memory problemsprovides its own difficulties and opportunities.

The guidelines for memory prosthesesthat we have developed in this paper pose manyinteresting research questions. We are nowworking on a prototype model which tries tomeet some of the challenges of data capture andretrieval by a user carrying a portable device.

15

There are also interesting (and sometimesdisturbing) issues of privacy-issues nevercontemplated when current privacy and dataprotection legislation was conceived-and we areaddressing these too.

11. REFERENCES

Adams, N., Gold, R., Schilit, W.N., Tso, M.M.and Want, R. (1993) An infrarednetwork for mobile computers. Toappear in Proceedings of USENIXSymposium on Mobile Location-independent Computing, Cambridge,Massachusetts, August 1993.

Brewer, W. (1988) A qualitative analysis of therecalls of randomly sampledautobiographical events. In Gruneberg,M.M., Morris, P.E. and Sykes, R.N.(eds), Practical aspects of memory:Current research and issues, Vol 1:Memory in Everyday life (pp. 371-376).John Wiley and Sons, Chichester.

Bush, V. (1945) As we may think. AtlanticMonthly, 176(1):101-108, July 1945.

Buxton, W. and Moran, T. (1990) EuroPARC’sintegrated interactive intermedia facility(iiif ): Early experiences. In Proceedingsof the IFIP WG8.4 Conference onMulti-User Interfaces and Applications,Herakleion, Crete, September 1990.

Cohen, G. and Faulkner, D. (1986) Memory forproper names: Age differences inretrieval. British Journal ofDevelopmental Psychology, 4, 187-197.

Crovitz, H. F., & Daniel, W. F. (1984)Measurements of everyday memory:toward the prevention of forgetting.Bulletin of the Psychonomic Society, 22,413-414.

Eldridge, M., Lamming, M.G. and Flynn, M.(1991) Does a Video Diary Help Recall?In Monk, A., Diaper, D.and Harrison,M.D. (eds), People and Computers VII.Cambridge University Press, Cambridge,England.

Eldridge, M., Sellen, A. and Bekerian, D. (1992)Memory Problems at Work: TheirRange, Frequency and Severity.EuroPARC Technical Report EPC-92-129.

Kvavilashvili, L.(1992). Rememberingintentions: A critical review of existingexperimental paradigms. AppliedCognitive Psychology, Vol. 6, 507-524.

Lamming, M.G. (1991) NoTime - A Tool forNotetakers. EuroPARC Technical ReportEPC-91-135.

Lamming, M.G. and Newman, W.M. (1992)Activity-based Information Retrieval:Technology in Support of PersonalMemory. In Vogt, F.H. (ed.), PersonalComputers and Intelligent Systems.Proceedings of Information Processing92, Vol III. Elsevier Science Publishers(North-Holland).

Lansdale, M. and Edmonds, E. (1992) Usingmemory for events in the design ofpersonal filing systems. InternationalJournal of Man-Machine Studies, 36,97-126.

Linton, M. (1986). Ways of searching and thecontent of memory. In Rubin, D.C.(ed.), Autobiographical memory (pp. 50-67). Cambridge University Press, NewYork.

Loftus, E.F. and Marburger, W. (1983) Since theeruption of Mount St. Helens, hasanyone beaten you up? Improving theaccuracy of retrospective reports withlandmark events. Memory andCognition, 11, 114-120.

Lovstrand, L. (1991) Being Selectively Awarewith the Khronika System. In Bannon,L., Robinson, M. and Schmidt, K. (eds),Proceedings of the second EuropeanConference on Computer-SupportedCooperative Work, Kluwer AcademicPublishers, Amsterdam.

16

NCR (1993) NCR WaveLAN/PCMCIA.Product announcement, 14 May 1993.

Newman, W.M., Eldridge, M., and Lamming,M.G. (1991) PEPYS: Generatingautobiographies by automatic tracking.In Bannon, L., Robinson, M. andSchmidt, K. (eds), Proceedings of thesecond European Conference onComputer-Supported CooperativeWork, Kluwer Academic Publishers,Amsterdam.

Newman, W.M. and Wellner, P. (1992) A desksupporting interaction with paperdocuments. In Proceedings of CHI’92,Conference on Human Factors inComputer Systems, ACM, New York.

Nielsen, J. (1990) Hypertext and Hypermedia.Academic Press, San Diego, Ca. (seeChapter 8, Navigating large informationspaces)

Saywitz, K., Bornstein, G. and Geiselman, E.(1992). Effects of cognitive interviewingand practice on children’s recallperformance. Journal of AppliedPsychology, 77(5), 3-15.

Schilit, W.N., Adams, N., Gold, R., Tso, M.M.and Want, R. (1993) The PARC Tabmobile computing system. Xerox PARCTechnical Report.

Sellen, A.J., Lamming, M.G., & Louie, G.(1992) Technology in support of humanmemory. Paper presented at the ScienceFestival sponsored by the BritishAssociation for the Advancement ofScience, Southampton, U.K., 26 August1992.

Terry, W. S. (1988) Everyday forgetting: Datafrom a diary study. PsychologicalReports, 62, 299-303.

Tulving, E. (1983) Elements of EpisodicMemory. Oxford University Press.

Wagenaar, W.A. (1986) My Memory: A study ofautobiographical memory over six years.Cognitive Psychology, 18, 225-252.

Want, R., Hopper, A., Falcao, V. and Gibbons,J.J. (1992) The Active Badge LocationSystem. ACM Transactions onInformation Systems, 10(1), 91-102.