cognitive support: extending human knowledge and processing capacities

HUMAN-COMPUTER INTERACTION, 1998, Volume 13, pp. 73-106 Copyright 8 1998, Lawrence Erlbaum Associates, Inc.

Cognitive Support: Extending Human Knowledge and Processing Capacities

Mark A. Neerincx TNO Human Factors Research Institute

H. Paul de Greef Unirrersity of Amsterdam

ABSTRACT

The idea of aiding as cognitive support is to offer the user the knowledge he or she is missing. Recently, we developed a design method for aiding that is based on explicit requirements of the human problem solver. This proved to be able to supplement a lack of human knowledge in a statistical analysis task. In this article we extend the aiding concept to time-pressured tasks and we investigate whether aiding can supplement lack of knowledge and capacity under tasks with high mental loading, such as dealing with irregularities in process control. We developed a simulator of the workplace of a railway traffic controller with an aiding function for dealing with irregularities (e.g., a switch getting out of order). Application of the design method proved to be possible for this task. We then conducted an experiment to study effects of the aiding on task performance, mental effort, and learning under low and high task load

Mark A. Neerincx is a ps chologist with an interest in the development of B user interfaces that provi e cognitive support; he is a Researcher in the Human-Machine Interface oup of the TNO Human Factors Research Insti- tute in Soesterberg, The Ne tr erlands. H. Paul de Greef is a ps chologist with an interest in cooperative problem solving and agent mo d eling; he is a Researcher in the Department of Social Science Informatics at the University of Amsterdam.

74 NEERINCX AND DE GREEF

CONTENTS

1. INTRODUCTION 2. AIDING IN PROCESS CONTROL

2.1. Requirements 2.2. A Design Method

3. THE DESIGN OF RAILWAY TRAF3FIC C O ~ R O L (RTC) AIDING 3.1. Step I: Designing the Plain Interface

Generate a Design of the Plain Interface Test the Design of the Plain Interface

3.2. Step 11: User Analysis 3.3. Step 111: Designing the Aiding Function

Generate the Design of the Aiding Function Test the Design of the Aidin Function

E v A L u m o N OF TBE R T ~ A I D I ~ G 4.1. Method

Participants Variables Design Task Procedure

4.2. Results Training Performance Mental Effort Knowledge

4.3. Interpretation of Results 5. DISCUSSION

5.1. The Method for Design of Aiding 5.2. The Effect of Aiding

Aiding, Task Load, and Mental Effort Aiding "Real* Traffic ControIlers

conditions. Users of the simulator dealt better and faster with irregularities when the computer provided aiding. The higher the task load was, the larger this beneficial effect was. For theory about human-computer interaction, this research points to possible positive effects of aiding on performance and learning as a consequence of reducing cognitive demands.

In classical task automation, the roles of the human and the software system have been limited mainly to either a master or a slave role. From a technology perspective, knowledge-based systems have been developed in which the software system performs the primary task and the users merely

COGNITIVE SUPPORT 75

have the secondary task of providing the data for the system's problem solving (e.g, MYCIN; Buchanan & Shortliffe, 1984). From a user perspective, usability principles for software have been developed according to which the users should be in full control of the software system as if they were using a tool (e.g., Norman & Draper, 1986).

In modern information technology, new opportunities arise for arrang- ing joint human-computer task performance: Instead of a master-slave task execution, we can design a task as cooperative problem solving. An important advantage is that the human's task is not taken over by the software system, so that the human remains in the loop of the task execution. Recently, cooperative software functions have been designed that aim to improve human information processing (e.g., Fischer, Nakak- oji, Ostwald, Stahl, & Sumner, 1993; Jones & Mitchell, 1994; Silverman, 1991; Terveen, 1995). Terveen (1995) distinguished between the human emulation approach and the human complementary approach. The first aims at mimicking human discourse functions using artificial intelligence (AI) and multimedia technology. The second assumes a fundamental difference between the abilities of human and computer, but even in this human-complementary approach, A1 techniques seem to be more central than problems and difficulties of the user. The cognitive requirements of the tasks to be performed by the user are often hardly addressed (cf. CoopGroup, 1996). Empirical research on cooperative problem solving is scarce (e.g., Silverman, 1992) and it is rather unclear which support function is most efficient for certain purposes (cf. Layton et al., 1994).

In practice, the effect of such a function may be detrimental: There is a trade-off between the potential benefits of the support (i.e., providing knowledge or off-loading the human) and the overhead of actually utilizing it (Adelman, Cohen, Bresnick, Chinnis, & Laskey, 1993; Kirlik, 1993; Neerincx & de Greef, 1993). The human not only lacks knowledge, but he or she also has a limited capacity for processing. The task load can-possibly temporarily-be too high to do additional work for utilizing cognitive support functions. Further, the human may overlook the benefits of the support and be unwilling to accept the costs, that is, do this extra work (Carroll & Rosson, 1987).

We have sought to develop a form of support-called aiding~entailing an optimal cost-benefit trade-off (de Greef & Neerincx, 1995). Aiding centers around supplementing human lack of knowledge and the basic idea is simple: The system provides the user with the knowledge he or she is missing. Such cognitive support may improve the performance of the human and of the overall system (e.g., faster task execution, fewer errors) and can be an adjunct or alternative for training or personnel selection. However, it is far from obvious how to design aiding so that cognitive overhead is minimized and benefits of supplementing missing knowledge are maximized.


Important sources of overhead are extensive dialogues to provide the system with data. These can be avoided by integration of cognitive support in the primary work environment-the base system-from which the support function can obtain data without bothering the user, and in which the system can act without recourse to the user as action implementer. Therefore, we view aiding as an add-on to a base system that in itself is worthwhile enough to use. Based on current theories of human information processing, we derived general requirements for such aiding functions. In short, the system will take the initiative to present the missing knowledge in a context-specific, procedural format using an easy-to-use interface. Subsequently, we developed a design method for user interfaces that provide aiding and satisfy these requirements. This method was applied to design aiding for a statistical analysis program and an experiment showed a beneficial effect of aiding both on performance and learning (de Greef & Neerincx, 1995; Neerincx & de Greef, 1993).

In this article we investigate whether aiding can be designed to extend human knowledge and capacities. To test the generality of the design method and to convey conditions under which aiding may be beneficial, the research centers around a process-control task: railway traffic control. In process control the needs for aiding seem to be quite different from a self-paced task like statistical analysis. In the railway study, the method will be applied to design aiding and an experiment will be conducted in which effects of aiding are studied under different conditions of task load to investigate whether aiding can compensate for deficient knowledge and capacities. Task load can be manipulated in railway traffic control simulations rather directly and realistically by changes in traffic density.

Section 2 discusses aiding for process-control tasks, such as railway traffic control, and recapitulates the design method for aiding hnctions briefly. Section 3 presents the design of a base-system interface plus aiding function for railway traffic control, Section 4 contains an evaluation of this function, and Section 5 discusses the results of the evaluation.

2. AIDING IN PROCESS CONTROL

Process-control tasks are environmentally paced and the environment to be controlled has tended to become more dense and hectic (e.g., more trains, more planes, larger plants). Also, due to modem information technology, the tasks tend to become more abstract. Task demands can run high as a consequence of disturbances and unexpected events. Task performance of human operators may be seriously diminished as a consequence of high task load apparent in abnormal situations (Gopher & Donchin, 1986; Rasmussen, 1986; Reason, 1990; Wickens, 1992). Wier- wille (1988) noticed that research in this domain has for a long time focused on workload evaluation techniques, and proposed to shift the


emphasis of research to preventing workload-related problems. Morris and Rouse (1985) noticed that enabling the operators to deal with unfamil- iar situations may not be achieved solely through training, and that more attention should be devoted to providing operators with assistance in using appropriate knowledge during abnormal conditions. We investigate the possibility of cognitive support for a specific process-control task: railway traffic control (RTC), specifically targeting high-load conditions.

Traffic controllers are responsible for the safe transportation of passengers and goods on a portion of the railway network. They are working at a control post and, among other things, have to set the signals and switches according to the timetable. Neerincx and Griffioen (1996) applied a cognitive task load analysis to the RTC task of the Netherlands Railways. This task has been changing over recent years, posing new demands on the processing capacities of the traffic controllers. In general, the task has been changing from the control of signals and switches to the control and adjustment of train routes. In the current situation, momentary task load rises rapidly when irregularities appear (e.g., a train engineer arriving too late, a cow on a track, or theft of cables) because problems have to be solved and extra actions have to be undertaken in addition to the setting of train routes. Due to the high task load, performance may be nonoptimal and, therefore, Neerincx and Griffioen (1996) advised the Netherlands Railways to design cognitive support for dealing with irregularities. In this article, we investigate whether the aiding concept, as developed by de Greef and Neerincx (1995), can be fruitfully applied to such a task.

2.1. Requirements

Initially, de Greef and Neerincx (1995) developed their aiding concept to compensate for lack of human knowledge in such a manner that the task demands for communicating and processing of the knowledge offered by the aiding function are minimal. They derived the following three general requirements for such a function.

First, the aiding facility should take the initiative to provide the right information at the right time; that is, at the point, or just before, it is needed. Consequently, there is no extra task of consulting a help facility and no extra knowledge needed for this task (i.e., knowing when help may be profitable and where to find it).

Second, the information should consist of context-specific, procedural task knowledge; it should be minimal, but comprise complete routines (i.e., complete and operational directions for how to solve the problem). Aiding provides procedural knowledge, because such knowledge reflects efficient expert task performance (Rasmussen, 1986). What is being provided is minimal, but complete: When needed, the aiding provides a complete solution to the subproblem. Just providing a hint may not lead


to successful task performance or might invoke too much effort in complex and possibly hectic situations.

Third, the user-aiding communication should be a well-integrated part of the user-system dialogue with a minimal interface. The mapping of users' goals on the system is most direct for a minimal interface, because it provides precisely the functionality the users need and no more. Thus, a minimal interface is easy to learn and use (Carroll, 1984).

Neerincx and de Greef (1993) showed that an aiding function that satisfies these three requirements can compensate for missing knowledge of users in statistical analysis. Statistical analysis is a self-paced task, because subtasks can be performed one by one and the tasks do not-in principle-have to be performed under time pressure. The tasks may be difficult, but can be spread out in time so that task load remains manageable. In process control, on the other hand, task load is determined more by events in the environment, and may rise high when handling disturbances. The question is whether an aiding function, originally intended to supplement human knowledge, can be of any benefit in work situations where limited capacity (i.e., overloading) is the most prevalent problem, rather than insufficient knowledge as it was in statistics.

Nevertheless, we hypothesize that even in demanding process-paced tasks, such as RTC, users may benefit from aiding, because the presentation of procedural task knowledge can invoke less demanding cognitive processing. We expect that the second requirement carries specific benefits for process-control tasks because it involves off-loading. Task demands depend on the type of knowledge available, and context-specific procedural knowledge is associated with lower task demands than reasoning or problem solving using general knowledge (e.g., first principles).

Rasmussen (1986) and Reason (1990) distinguished two levels of human problem solving: rule based and knowledge based. The knowledge-based level comes into play in novel situations for which actions must be planned. Based on a mental model, the person sets local goals, initiates actions to achieve them, observes the extent to which the actions are successful, and, if needed, poses new subgoals to minimize the discrepancy between the present state and the desired state. A switch out of order may, for example, require that new destinations for trains be planned. The new plan may invoke extra delays, so that it has to be adjusted. As task experience-and possibly training-grows, more and more problem solving takes place at the rule-based level. The rule-based level represents problem-solving actions in which the solutions are governed by stored rules (productions) of the type "if state then action." If, for example, a particular switch has been out of order regularly, then the traffic controller will have developed a plan or procedure to deal with this situation. Task performance at this level is less demanding than performance at the knowledge-based level: Actions can be accomplished without time-expen-


sive, knowledge-based problem solving (Rasmussen, 1984). However, process operators hardly ever have procedural knowledge available about how to deal with rare situations, such as a switch out of order. In these situations, even when operating instructions have been issued by the system designer, system operators often tend to formulate the solution procedures directly by a trial-and-error operation rather than spending the effort reading a manual, at least initially (Rasmussen, 1986).

Automatic presentation of context-specific, procedural task knowledge might propagate mainly rule-based behavior (instead of knowledge-based behavior) by conveying rule-based shortcuts to solve the problem so that elaborate knowledge-based search actions are minimized (Rasmussen, 1986). Applying such compiled knowledge requires less capacities than problem solving using general conceptual knowledge. Consequently, aiding might be effective especially in situations of high task load.

The first and third requirement do not carry any benefit in off-loading the process operator; they aim to minimize the cost of using the support. In overloading situations, they are even more pressing, as to achieve a positive trade-off with the off-loading effect of offering context-specific procedural knowledge.

In summary, in process control, compared to self-paced tasks, benefits of aiding may differ. We hypothesize that aiding might be helpful in managing high task load; that is, aiding might extend the knowledge and capacities of process operators, such as railway traffic controllers.

2.2. A Design Method

The previous section presented three general requirements for aiding functions. Given these, the question becomes this: By what method can one design applications that satisfy these requirements the best? De Greef and Neerincx (1995) provided a comprehensive method for efficient development of aiding functions. The design method integrates principles of human-computer interaction into current methods for software engineering. Thus, most elements of the method already exist, but are arranged in a new way and used for new purposes. The method extends existing model-based software engineering techniques to address the highly inter- active cooperative involvement of both the human and the technology. The method allows human factors specialists to intervene at an early stage in the development process, when decisions are made that influence the usability and learnability of software. This section presents a brief summary; the next section shows its application to RTC in more detail.

Model-Based Design. Similar to current methods in software en@- neering (Rumbaugh, Blaha, Premerlani, Eddy, & Lorensen, 1991; Your- don, 1989) and knowledge acquisition (Schreiber, Wielinga, & Breuker,

8 0 NEERINCX AND DE GREEF

1993), we view design as a modeling activity where an abstract model of the prospective artifact is constructed and successfully transformed into more detailed models, with the code of the final implementation as the final and most detailed model. Large models are made manageable by distinguishing submodels or perspectives-functional, data, behavioral, and social-as part of the overall model (Curtis, Kellner, & Over, 1992). The transfer of more abstract models to less abstract models can be supported or even partly automated (e.g., Kim & Foley, 1993; Johnson, Wilson, Markopoulos, & Pycock, 1993).

Although much is known about human task performance and about user interface design, the eventual outcome cannot be predicted completely and errors can be made along the way. Therefore, the design method follows a generate-and-test approach. First, based on current knowledge about users' task performance, a prototype is designed. Then, the prototype is user tested in performance studies, and-based on the test-the design model can be improved, starting a new generate-and-test cycle. This generate-and-test cycle, using a model-based approach to system development, is used in Step I and Step I11 of the design method for aiding functions (see following).

D d g ~ P r o d w e . To develop effective, minimal support (see the second requirement), it is necessary to establish accurately which knowledge prospective users are lacking. An efficient and accurate method is to let users of different abilities try criterion tasks or scenarios using a prototype of the base system. This motivates the global structuring of the design method as a sequence of three steps. Step I is the design of the user interface for the base system. Testing in Step I is done with experts to make sure the base system is easy to learn and use by users who do not lack knowledge. Step I1 uses the base system to see whether prospective users have problems in performing criterion tasks. Step I11 is the design of the aiding function. This involves the acquisition of the knowledge for task parts with which users have difficulties. Here the base system can be used again to acquire the knowledge from experts. Finally, a prototype of the aiding function is added to the base system and the effect of aiding can be tested with the prospective users.

Figure 1 shows the three steps in more detail. The first step consists of the generation and test of a design of the plain interface. This is a simple, minimal user interface for the base system. It aims to be optimal with regard to ease of learning and it is designed for minimal system experience. To accomplish Step I, the model-based design approach is used to generate design models and to implement a mock-up prototype. To complete Step I, the prototype is tested (e.g., by having experts solve task scenarios using the prototype) and improved for those parts that proved to be nonoptimal.

COGNITIVE SUPPORT

Figure I. The method for the deaign of aiding.

I. Designing the plain interface for the base system a. Generate a design of the plain interface b. Test whether the plain interface is satisfactory for experts

11. User analysis Test users' performance with the Plain Interface and identify the subtasks

for which users need aiding 111. Designing the aiding function

a. Generate the design of the aiding function Generate an expert model (procedural task knowledge) Generate a model of the joint execution process

b. Test the design of the aiding function

In Step I1 of the design method, user analysis, the users' performance with the plain interface is investigated. Among users, the level of expertise may vary considerably. The mock-up implementation of the plain interface and the task scenarios are used again, but now to provide a context to assess the availability of task knowledge and capacity of users. This user analysis provides information about the problems and difficulties users may have and helps to identify in which parts of the task they appear to lack knowledge and capacities. It may of course turn out that there is no need for aiding or other cognitive support.

Step I11 of the design method is concerned with design of the aiding function for the task parts in which users lack knowledge, capacities, or both. For this, the generate-and-test framework is applied again. An analysis model has to be generated that comprises the expert task knowledge to be presented by the aiding function. To acquire this task knowledge from experts, the mock-up prototype and the task scenarios can be used again. The analysis model must also contain a process model for the social perspective; that is, a specification of the system in the process of joint task execution. The analysis model is transformed to a mock-up implementation. The aiding function can then be tested with task scenarios tried by users.

Modeling and Testng in Process Control. Compared to the first application of the design method for statistical software, there are two new issues. In contrast to statistics, the process-control tasks have to be performed real time in a dynamic and sometimes hectic environment. A process-control task can be interrupted by events and subtasks with different priorities may have to be executed in parallel. The design models must be able to capture these characteristics.

Besides the demands on the generation of design models, process-control tasks pose higher demands on the test of the aiding function compared to self-paced tasks like statistical analysis. Task execution takes place, and thus task scenarios have to be simulated in real time. Further, task load is an extra ingredient of the evaluation of aiding effects.

NEERINCX AND DE GREEF

In summary, process-control tasks pose higher demands on the generation of design models and the test of the design than self-paced sequential tasks. Still, we hypothesize that the design method, including a generate- and-test approach, can be applied to RTC effectively.

3. THE DESIGN OF RAILWAY TRAFFIC CONTROL (RTC) AIDING

In this section we apply the design method for aiding in order to develop a prototype interface for RTC. Our primary interest is not to build a system for the Netherlands Railways, but to investigate whether an aiding facility can be designed and be of help for certain problems in process control. For such an investigation it is important to capture the crucial aspects of a process-control task, design aiding for this task, and test it in a controlled experiment (in which the environment comprises important aspects of the "realn work environment). Software for prototyping and simulation is available with which realistic mock-ups can be implemented, so that laboratory experiments can be conducted relatively inexpensively. These experiments can provide empirical data about the success of computer support in different conditions. Expensive field evaluations should be carried out only after demonstrating superior performance in such an experiment (Johannsen, Levis, & Stassen, 1992; cf. Sanders, 1991).

Currently, the Netherlands Railways uses several systems for traffic control. Recently, a new traffic control system was designed for the Neth- erlands Railways that is in operation for a small part of the railway network. This system is here regarded as a base system that can be extended with an aiding function. However, the user interface does not comply with the method's requirements (i.e., plain, and easy to use and learn). As the user interface of the aiding function must be well integrated with that of the base system, we employed Step I of the method in a kind of reverse engineering of the existing system, to abstract to an analysis model, from which we designed our own plain user interface prototype. We also developed a simulator of railway traffic scenarios as part of the prototype for realistic real-time simulation of task scenarios and to enable an evaluation of this interface.

With respect to the second step, user analysis, we had prior knowledge regarding general problems of human process operators and also about specific performance problems in RTC. In a previous study, Neerincx and Griffioen (1996) applied a cognitive task load analysis to the RTC task. The controller's subtask to carry out the work plan (i.e., the timetable) under normal conditions is not difficult, but to handle rare, unpredictable events is a difficult subtask with a relatively high task load for which the traffic controllers have little procedural knowledge available. Thus, dealing with irregularities seems to be a good subtask on which to investigate

COGNITIVE SUPPORT 8 3

the possibilities of support. In the third step, we designed an aiding function for this subtask that should be able to compensate for deficient human knowledge and capacities. In the following we explain the design of RTC aiding following the three steps of the design method.

3.1. Step I: Designing the Plain Interface

In the first step of the design method, the generate-and-test procedure is followed to develop a plain user interface that is easy to use and learn.

Generate a Design of the Plain Interface

This method for the design of aiding uses four perspectives in system analysis: functional, data, behavioral, and social (Curtis et al., 1992). The first three perspectives are also present in the object-oriented modeling method of Rumbaugh et al. (1991) and the structured analysis of Yourdon (1989). These two methods use similar modeling constructs and support the functional, data, and behavioral perspectives on system design. In the structured analysis approach, the functional perspective dominates, whereas the object-oriented approach regards the object (i.e., data) perspective as most important. In the following we explain the use of the four perspectives in the design of the plain interface.

Functional and Social Perspective. We investigate the situation in which a railway traffic controller is responsible for the subtasks Carry Out Work Plan and Deal With Irregularities. Based on the cognitive task analysis of Neerincx and Griffioen (1996), a functional model of the RTC task was generated. The social perspective is obtained by allocating each function to the user or to the system (Figure 2).

The system has a specific role in the subtask Carry Out Work Plan. On the basis of the process information and the work plan the traffic controller decides to plan a train route. The system checks whether the desired settings are possible and-if they are-the system then sets the elements concerned in the correct position (i.e., change the setting of points and signals on the railway yard). A standard route can only be set for tracks without trains for which no routes have been set already (see Figure 2).

The subtask Deal With Irregularities is performed by the user. On the basis of the process information and the work plan the traffic controller may make adjustments to the work plan and inform passengers where necessary. The tasks Carry Out Work Plan and Deal With Irregularities are discussed in more detail later.

Data and Behavioral Perspective. Data to be communicated between user and system cross the user-system partition in Figure 2. These data consist of process information, work plan, personnel information, and


Ftgrrre 2. A functional model of the RTC task with a user-system partition to represent the roles of user and system.

deal with

information for personnel

. . . . . . . . . . . . state of state of state of information elements

PROCESS traffic

objects

I I data

0 task or process

route plan. To model these data, an object-oriented approach was used (Rumbaugh et al., 1991). The resulting data model provides a complete definition of the information that is exchanged between the functions in Figure 2. The data model for the information crossing the user-system partition in Figure 2 serves as an abstract user-interface specification. To add the behavioral perspective, it has to be established whether the user or the system takes the initiative to transfer the data. Following current practice, the user initiates information for personnel, the system for presenting process information and for presenting the work plan, and the user again initiates changing the work plan and making a route plan.

Test the Design of the Plain Interface

To test the design, it has to be transformed to a mock-up implementation (i.e., a prototype) that provides the simulated work environment in which users can perform tasks. In process-control tasks, the prototype needs to include a simulator for the process to be controlled.

A graphical user interface was designed and implemented in XPCE- prolog, a high-level object-oriented environment for direct-manipulation interfaces, which also provides facilities for process simulation (Wiele- maker & Anjewierden, 1992). Here we present the user interface, the two subtasks in RTC, and the simulation of railway traffic in the human performance studies.

COGNITIVE SUPPORT 8 5

Figure 3 shows the prototype of the plain interface for the railway yard of Amersfoort (the same railway yard is used by Netherlands Railways for the training of railway traffic controllers). The upper window presents the yard. In Amersfoort, trains go to and come in from different directions: Amsterdam, Utrecht, Zwolle, and Apeldoorn. The railway yard has 18 track segments guarded with signals that appear as small rectangles on the screen, four platforms (next to Tracks 6, 7, 8, and 9), and one level crossing. Trains are represented as rectangles somewhat larger than the signal rectangles. In Figure 3, Train 85 is on Track 15 of the railway yard. The time is presented in the clock at the top of the window.

In correspondence to "realn railway traffic controllers, the user of the prototype has to perform two subtasks: Carry Out Work Plan and Deal With Irregulari- ties. The second subtask has a higher priority, is more difficult, and consists of several actions.

Carry Out Work Plan. The work plan is a timetable on paper prescrib- ing at which time a route has to be set (see Figure 4). A user sets routes for trains by clicking with the mouse on the starting track and then clicking on the terminal track. The system sets the signals and switches at the correct color and correct position, respectively. The route is displayed as a bold line at the railway yard. Trains will move to their destination if the required route is set. In Figure 3, Train 85 is moving from Track 15 to Track 6.

The setting of a route is not possible if a part of it is locked; for example, when another train is already on that route or when the control of a switch in the route is disabled. In such situations, the mouse clicking has no effect.

Deal With Irregularities. Next to carrying out the work plan, a user has to Deal With Irregularities such as a switch getting out of order. In the left bottom area of the screen a message may appear to inform the user about an irregularity (Figure 3). Such a message is always accompanied with a beep. Dealing with messages has a high priority: The user has to interrupt his or her normal setting of routes for this. With the buttons in the window at the bottom right of the screen (see Figure 3) the following actions can be taken to deal with the messages.

The user can put a specific diversion into memory so this route can be easily set for several trains. The user can send four messages to a train: wait for delayed train, give permission to pass red signal, remove train from timetable, and sprinkle sand on iced track. The user can send three messages to the maintenance personnel: give permission for repair, redraw permission for repair, and give a repair order.

Figure3. The screen of the railway traffic controller (i.e., the plain interface of the base system). The bottom left area is used to display messages and in the aiding version this area is also used to present context-specific procedural knowledge.

TRACK SEGMENT LEVEL


Figure 4 Part of a timetable.

OK Train Setting Time Time of N D / P Type of Route From Track To Track

85 0O:OO 00:03 Arrival 15 7 88 00:Ol 00:02 Departure 9 18

. . . . . . . . . . . . . . . . . . . . .

Note. OK = to annotate that a route has been set; Train = train number; Setting Time = time before which the route should be set; Time of A/D/P = planned time of arrival, departure, or passing; Type of Route = arriving, departing, or thrbugh train; From Track = starting track of the route; To Track = terminal track of the route.

0 The user can decide that the barriers (gates) not only close when trains pass, but also when trains are supposed to stop near the level crossing (for safety reasons). The user can mark a track as being slippery or with an out-of-order signal. The user can disable the setting of a track or switch.

Simulator. A simulation of train traffic and irregularities has to be developed to test the design with users performing the RTC task. The simulation does not need to incorporate all aspects of the job, but it should tap the crucial aspects (Ackerman & Kanfer, 1993). We developed and implemented scenarios incorporating those aspects of the RTC task that were shown to be important in the cognitive task analysis of Neerincx and Griffioen (1996). In the scenarios, the train movements are specified with respect to the following variables:

The trains that will appear on the railway yard. The time and position that a train appears on the railway yard. The direction that a train moves. The time at which a train will start moving from a platform.

Within a scenario, irregularities can also be specified. The user is informed about an irregularity via a message. An irregularity can be added to a scenario by specifying:

The type of irregularity (e.g., a switch gets out of order). The messages that will appear on the screen. The time a message appears.

This section discussed the implementation of a prototype interface for RTC and a simulator of railway traffic. According to an expert of the Netherlands Railways, this prototype and the scenarios matched to the RTC task well. Expert traffic controllers, who have the required knowl-

NEERINCX AND DE GREEF

edge and do not run into capacity problems, can perform the central task elements with the plain interface easily. In section 4, the plain interface is evaluated in a controlled experiment.

3.2. Step 11: User Analysis

In the user analysis we investigate to what extent prospective users of the system are capable of doing their tasks with the system. However, we already had prior knowledge about needs and difficulties in process control in general and in RTC in particular. In a sense, user analysis using a working version of the plain interface (i.e., the existing system) had already been performed by Neerincx and Griffioen (1996). They decom- posed the RTC task into knowledge-, rule-, and skill-based actions according to the framework of Rasmussen (1986; see section 2.1) and subsequently observed task performance in three control posts of the Netherlands Railways to assess the cognitive task load. Among other things, task load proved to be too high for Deal With Irregularities in busy control posts. Railway traffic controllers had no procedural knowledge to do this task with relatively little effort and, using the plain interface of the Nether- lands Railways' system, showed a need for computer support when task load was high (i.e., when the traffic controllers did not have enough cognitive capacity). In other words, Neerincx and Griffioen (1996) identi- fied a need for an aiding function that supplements human knowledge and capacity deficiencies for the task of dealing with irregularities.

3.3. Step 111: Designing the Aiding Function

In correspondence with the first step, the third step of the design method follows a generate-and-test approach.

Generate the Design of the Aiding Function

MOM Based on the cognitive task load analysis of Neerincx & Griffioen (1996), existing documents, and interviews with experts, an expert model was formulated for a part of Deal With Irregdacities. It should be noted that the expert traffic controllers could not directly provide complete procedural knowledge about this subtask, but-using additional sources of information-a complete, formalized procedural expert model could be constructed for some of the irregularities. This model of the controller's task prescribes the actions to take when one of these irregularities occurs. Figure 5 shows the actions that should be taken when switch X goes out of order. The expert model was written in Prolog (somewhat adjusted so that it could be used directly in the XPCE-simulation). In this program, the irregularities or events were embedded in a main loop.

COGNITIVE SUPPORT

Fipre 5. Part of the RTC expert model.

IF switch X has got out-of-order THEN (I) disable settingof switch X (2) NEXT make a diversion around switch X (I) NEXT order maintenance personnel to npair switch X

Fgrre 6. The joint execution behavior with the aiding interface.

USER SYSTEM

event

context-specific 4 deal with

rules for event

irregularities rules for event

1 1 0 task, function or process

1 flow of control

data flow

Model ofJoint Execution. After constructing the expert model for Deal With Irregularities, the operation of the aiding function has to be specified. Figure 6 shows a model of the joint execution process and the role of the aiding function for the task Deal With Irregularities. Notice that this model represents the roles of user and system in the execution process for one event. During one execution process a new, parallel process can be started.

Test the Design of the Aiding Function

To test the design of the aiding function, it must be integrated into the prototype of the plain interface. Because we reengineered the base system as a simulator in a modern, object-oriented, graphical user interface environment, implementing and integrating the prototype of the aiding function proved to be rather straightforward. Irregularities appear in the simulation as messages in a separate window. Figure 5 presented the part of the expert model-a three-step procedure-to deal with a switch that goes out of order. The aiding interface presents this procedure at the moment the message arrives that switch X is out of order. Furthermore, it presents the corresponding action button to minimize the search for the


Figure Z Example of a mewage and the corresponding aiding. Thir information is displayed in the bottom left area of Figure 3.

BECAUSE it Is a serlous (low) delecf -> Make a dlverslon amund back 12

1 Select track segments Wh middle-mouse

2 Put the selected seuments tn memory

II Inform maintenance personnel to repalr switch 12 --

AIDING

needed buttons. Figure 7 shows the message in the top and the corresponding aiding at the bottom. In the aiding interface, these two are presented in the bottom left part of the screen shown in Figure 3 (the interface without aiding presents only the message). The context-specific help for RTC can be summarized as follows.

A message is always accompanied with the corresponding help window. Only relevant information appears. For example, only if the switch is not disabled yet is the user informed that the switch should be disabled. The object classes of the expert model are instantiated. For example, if Switch 12 is out of order, the user is informed that Switch 12 should be disabled.

0 The help window displays the reason for an action if possible. For example, if Train 43 is delayed, the user gets the following advice: Because Train 40 is connected with Train 43, order Train 40 to wait for Train 43.

The next section shows the evaluation of the aiding function in a controlled experiment.

COGNITIVE SUPPORT

4. EVALUATION OF THE RTC AIDING

The preceding section described a prototype interface that provides context-specific help for RTC. In an experiment, detailed here, we test whether this kind of support can really be of help and, if it helps, under which conditions it is effective. Of special interest is the effect of aiding under different conditions of task load. For the RTC task, this can be investigated very well: The task consists of an easy subtask, Carry Out Work Plan, and a difficult subtask, Deal With Irregularities. Aiding can be provided for the second subtask and, at the same time, task load can be manipulated on the first subtask by varying the number of trains for which routes have to be set. Two hypotheses were tested in the experiment.

First, we hypothesize that aiding has a positive effect on the performance of the subtask Deal With Irregularities. Users will perform more of the required actions because they are told what to do. Further, they will need less time to perform these actions because they do not have to construct a procedure of actions themselves to solve the irregularity problem and hardly have to search for the corresponding buttons in the display.

Second, we hypothesize that aiding has a positive effect on the performance of the subtask Carry Out Work Plan. Although the aiding centers on dealing with irregularities, the subtask with the highest priority, its effect on carrying out the work plan is also important. Aiding may enhance the performance of important subtasks, but can in turn lead to inferior performance of less critical, yet still important subtasks (Adelman et al., 1993). Whether an irregularity is dealt with correctly or not, the aiding must not hinder the setting of routes. We expect that the aiding has a positive effect on Carry Out Work Plan, because it brings about an efficient strategy for dealing with irregularities, leaving more time and capacity available for the rou- tine work.

4.1. Method

Participants

Forty psychology students participated in the experiment for class credit. Students could choose among a number of experiments, so partici- pation in this specific experiment was more or less voluntary.

Variables

The independent variables are aiding (aiding or no aiding) and task load level (low or high). Task load is manipulated by means of the number of routes the participant has to set in the simulations. Three dependent variables are measured: performance, mental effort, and knowledge.


P&@name. In the experiment a distinction is made between the performances on the subtasks Carry Out Work Plan and Deal With Irregularities. For Carry Out Work Plan, a measure of performance on the setting of routes must be obtained. We need a proportion measure in order to compare the performance in the low and the high load conditions with, respectively, few and many routes to be set. To this end, we could measure the proportion of routes set correctly in a scenario. However, this does not tell us much as, in practice, railway traffic is almost always routed to the correct destination so that this performance measure is at ceiling level. Performance differences are shown in the mean delay of a route setting that is a more realistic measure.

The mean delay of a route setting is calculated from three variables:

1. t-set= time a route is set correctly. 2. Q l a n = the setting time of the timetable (see Figure 4). 3. t e n d = time the scenario stops and routes cannot be set anymore.

The delay of a route if it is set correctly, is:

delay = t- set - t- plan, if t-set > t- plan, else 0

The mean delay of n routes set correctly is:

delayi mean-delay = x=, (2)

In the experiment participants have only limited time to set a number of routes; this time may be too short to set all the routes correctly. For routes not set or routes set with an incorrect destination, it has to be estimated at what time-or delay-a route is set that "deliversn the train at the correct destination. For this, the mean delay of the routes set correctly is added to the delay at the end of the scenario. Thus, the estimated delay for routes not set correctly is:

est-delay = ( t - end - t- plan) + mean-delay (3)

The estimated mean delay of a route setting for m routes set correctly and n routes not set correctly is:

(r delay, + zfl est- delay, (4) i=1 j=1

Delay- per- Route- Setting = m + n


This is the performance measure used for the subtask Carry Out Work Plan. To assess the performance on the subtask Deal With Irregularities, two variables

are measured. The first one is the number of correct actions, which are, according to the expert model, the actions that have to be performed to deal with the irregularities adequately. The second variable is the mean time spent for a single action.

Mental Effort. Rating instruments are among the most sensitive, most transferable, and least intrusive techniques for workload estimation (Wier- wille & Eggemeier, 1993). In this experiment, mental effort is assessed with a one-dimensional rating scale developed in the Netherlands: the Subjec- tive Mental Effort (SME) questionnaire (Zijlstra, 1990, 1993; Zijlstra & Meijman, 1989). This self-rating questionnaire consists of a single question and a rating scale with labels. The minimum score is 0 (no mental effort) and the maximum is 150. Participants estimate their perceived effort immediately after each scenario.

Knowledge. To measure the knowledge of the participants two ques- tionnaires are used. The first one assesses general knowledge. The following is an example question: What is the normal destination of trains departing from Platform 6? The number of correct answers is the measure for general knowledge.

The second questionnaire centers on knowledge about how to deal with the irregularities occurring in the scenarios, such as what to do when a switch goes out of order. The questionnaire consists of messages in the same format as they appear in the simulation. The participant has to describe on paper what a railway traffic controller must do to deal with the messages. The number of correct answers on the second questionnaire is the measure for knowledge about irregularities.

Design

The experiment consisted of a completely randomized between-subject design with four conditions. There were 10 students assigned to each condition.

Task

Each of the participants was tested on six scenarios. In each scenario they had to set routes for trains according to a timetable that was available on paper. The participants in the low load condition received scenarios with fewer trains than the participants in the high load condition. On top of this, participants had to deal with one or two messages about irregularities in each scenario. All participants received the same messages.


Procedure

The procedure consisted of two parts, a training phase and a test phase, consisting of the following elements:

I Tramng instruchon of 20 min trammg scenario Subjechve Mental Effort @esho~aJre general knowledge queshonnme

2 Test 6 scenmos mth loggng of user achons after each scenario: the Subjechve Mental Effort Queshonnare after all scenarios - general knowledge queshonnare (same as m the trammg) - queshonnare about how to ha1 Nth Irreguhnt~tr

The training objectives were to bring about a fast and error-free communication with the user interface, to bring about a good task performance for the subtask Carry Out Work Plan, and to get the participants acquainted with the functions of the interface. First, the participants received written instructions that provided information about the RTC task. They were instructed how to use the system to set a number of routes according to a timetable, how to make diversions and put them in memory, and how to use all the other buttons of the interface. The experimenter observed participants' performance to ensure that they worked according to the instructions and, when necessary, to point out aspects of the written instruction that had been overlooked. After the instruction, participants went through a training scenario in which eight routes had to be set for trains according to a timetable. In the training scenario no message appeared. The training phase ended with the general knowledge questionnaire.

In the test phase, participants had to perform the RTC task as already described for six scenarios. User actions (i.e., button clicks and strings typed) were logged in a file with the corresponding times. After each scenario the SME questionnaire was filled in. At the end of the test, the participants answered the general knowledge questionnaire and the questionnaire about dealing with irregularities.

4.2. Results

We applied an analysis of variance (ANOVA) to the individual data collected for the dependent variables. This section presents only those results that proved to be significant in this statistical analysis.

Training

In the training scenario the students had to set eight routes. Almost all students succeeded in setting these routes correctly (99010 of the routes

COGNITIVE SUPPORT

Figure 8. Performance on Carry Out Work Plan.

Delay per Route Setting /

with aiding

low load high load

were set correctly). The mental effort score on the training scenario, which was the same for all students, varied enormously among students (range = 3-73). With respect to the general knowledge questionnaire, no significant differences were found among the four groups.

Performance

First, we present the results on Carry Out Work Plan. Overall, the students set 92% of the routes correctly; in other words, performance was at ceiling level as expected and no differences between conditions appeared. To analyze the performance on this subtask in more detail, the mean delay of the setting of a route was estimated according to the formula in section 4.1. Students working in the high task load condition set a route with a mean delay of 48.5 sec, whereas the mean delay in the low task load condition was 25.7 sec, q l , 36) = 9.5, p < .O1 (see Figure 8). The interaction effect between aiding and task load is significant, q l , 36) = 4.2, P < .05: The delay is large, especially in the high-load-and-aiding condition.

Second, we present the results on Deal With Irregularities. Figure 9 shows the total number of correctly executed actions that have to be performed in order to deal with the irregularities adequately. The maximum score is 65 actions. With aiding an average of 59.3 actions were executed, whereas without aiding an average of 29.7 actions were executed, q l , 36) = 230.3,


Figure 9. Performance on Deal With Irregularities.

aiding

0 - - - - - - - - - - - - - - - _ _ _ _ - -0 without aiding I

low load high load

p < .001. The interaction effect between aiding and task load is significant, fll, 36) = 4.2, p < .05: The beneficial effect of aiding is larger when task load is higher.

For each message, the time that the message appeared was subtracted from the time that the last of the corresponding actions was executed. The sum of these time periods is the total time students spent for the execution of the actions. As expected, the number of actions correlated with the total time spent (r = 59). With aiding, students executed more actions and, consequently, spent more time accomplishing these actions (694 sec with aiding and 543.9 sec without aiding), fll, 36) = 7.8, p < .01. For each student the mean time spent for a single action was estimated by dividing the total time spent by the number of actions (Figure 10). With aiding the mean time per action is 11.7 sec, whereas it is 18.9 sec without aiding, F(1, 36) = 517, p < .001.

Mental Effort

The scores on the SME questionnaire varied widely among participants. The mean score of participants varied between 2 and 88. The experiment had a between-subject design and, as can be expected with such a large variance, no effects appeared of aiding and task load. There was, however, one significant effect: Participants estimated the mental


Figure 7a Time spent per action in seconds for the subtask Deal With Irregularities.

Time per - - - - - a

_ _ _ - - * - _ _ - - - - - D--

without aiding

____I with aiding

low load high load

effort for the training scenario lower (M= 34) than the other scenarios in which they had to deal with an irregularity (M= 39), t = -2.34, p < .05.

The SME score on the training scenario had a high correlation with the mean SME score on the six scenarios of the experiment ( r = .74). The variance among participants on the SME scores might be the result of a different base rate and range of the subjective scores. The score on the training scenario can be considered such a base rate. Calibrating the SME scores to this base rate and compensating for range differences did not produce other results than the raw SME scores. Summarizing, an irregularity raised the mental effort, but effects of the between-subject variables, aiding and task load, on mental effort could not be found.

Knowledge

Participants filled in the same general knowledge questionnaire after the training (pretest) and at the end of the experiment (posttest). Further, they filled in the irregularities questionnaire at the end of the experiment. The results on these knowledge tests correlated with each other. For general knowledge the correlation between pre- and posttest was .65; the correlation between the pretest for general knowledge and the irregularities questionnaire was .45; the correlation between the posttest for general knowledge and the irregularities questionnaire was .51.

General Knowkt&e. The maximum score a participant can acquire on the general knowledge questionnaire is 33. No differences appeared between the groups on the pre- and posttest. The scores on the pretest with a mean of 10.2 were less than the scores on the posttest with a mean of 15, t = -6.66, p < .001.


Figure 77. The ditrerences between the pre- and posttest b r general knowledge.

General 8 Knowledge

I I I

low load high load

Figure 11 shows the differences between the scores on the general knowledge questionnaire before the test and after the test phase. An ANOVA on these differences showed no significant main effects; however, the interaction effect is significant, F(1, 36) = 4.99, P < .05. For high load, learning seems to be best with aiding, and for low load, learning seems to be best without aiding.

The score on the pretest of general knowledge correlated nemvely with the mean delay of a route setting (r = -.36) and did not correlate with the measures for dealing with irregularities. The correlations of the score on the posttest of general knowledge with the other variables show the same pattern. Unexpectedly, having more general knowledge seemed not to be helpful for establishing the correct actions when an irregularity occurred.

Knowledge Abut tbties. The maximum score a participant can acquire on the irregularities questionnaire is 23. The mean score was 12.9; no differences appeared among the groups. The score on the irregularities questionnaire correlated slightly with the number of irregularity actions executed correctly ( r = .32).

4.3 Interpretation of Reaults

The RTC task consists of the subtasks Carry Out Work Plan and Deal With Irregularities. Our first hypothesis was that aiding has a positive effect on the performance of the second subtask, which is more difficult and has a higher priority than the first. With aiding, users of the railway simulator performed 91% of the actions prescribed by the expert model correctly, whereas without aiding, users performed only 460h of these actions. Furthermore, users spent 38% less time accomplishing a single action with aiding. This is


important because dealing quickly with irregularities promotes safety and minimizes delays. Thus, the first hypothesis can be accepted.

Our second hypothesis was that aiding has a positive effect on the performance of the subtask Carry Out Work Plan. Although the aiding only provides knowledge for dealing with irregularities, it also is proven to influence performance on this subtask. At first sight, carrying out the work plan seems to be hindered by the aiding: The delay of route settings is large especially in the high-load-and-aiding condition. However, this result is not the consequence of a direct interference between the use of the extra help window and the setting of routes. With aiding, more actions are executed to deal with the irregularities, so that more time is spent on this subtask and less time is left to set routes. The more time spent on these actions, the larger the mean delay of a route setting was (r = .42). The setting of a large number of routes is not possible within the time left and, therefore, the delay of route settings is large in the high-load-and-aiding condition. The important finding is that with aiding, less time is spent on a single action so that more time is available for carrying out the work plan in case an irregularity is dealt with adequately. Thus, in correspondence with the second hypothesis, the aiding for Deal With Irregularities has a positive effect on the subtask Carry Out Work Plan, leaving more time, and not a negative effect as the mean delay of a route setting suggests.

The experiment conveys two interesting interaction effects of task load and aiding. First, the beneficial effect of aiding on dealing with irregularities is larger when task load is high. Second, the acquisition of general knowledge is only better with aiding when task load is high. The discussion elaborates on this finding; here suffice to say that the aiding is especially beneficial when task load is high.

No effect of aiding or task load on mental effort appeared. The SME questionnaire is a standard instrument to assess effort. Contrary to other research with this instrument (Zijlstra & Meijman, 1989), in this experiment there was such a large intersubject variance that no differences between the experimental groups could be discovered even when the baseline measure was factored out.

Taken together, the context-specific, procedural support proves to be very beneficial for users of the railway simulator, who just received 30 min of training. This result is not as trivial as it may seem. Recent empirical evaluations of help systems show possible harmful effects of support (de Jong, de Hoog, & de Vries, 1993; Kirlik, 1993; Layton et al., 1994; Neerincx & de Greef, 1993). Support may also have unforeseen negative side effects; for example, adding a rule-generation capability to an interface for aircraft identification enhances the number of correct identifica- tions, but in turn increases the time spent for the examination of a single aircraft (Adelman et al., 1993). This experiment did not convey such an undesirable effect for the aiding function.


5. DISCUSSION

Classical user-centered design with graphical user interfaces solves many difficulties in human-computer interaction, but, with the user in the master role, it cannot prevent or remediate performance problems due to lack of domain knowledge or lack of capacity on behalf of the user. Aiding functions aim to complement human lack of knowledge, but because of human capacity limitations, the effect of aiding depends on a trade-off between potential benefits and the effort to make use of the aiding (Section 1). Based on user psychology, three requirements for aiding have been formulated (Section 2.1) and a method has been devised for the efficient development of aiding functions that meet these requirements (Section 2.2). This method was applied to develop an aiding interface with a simulator for railway traffic (Section 3). This prototype was used in an experiment to study the effect of aiding under various loading conditions on criteria for performance, learning, and mental effort (Section 4). We first discuss the application of the method and then the aiding effects.

5.1. The Method for Design of Mdiq

The first general conclusion of this article is that our method for the design of aiding was successfully appled to a process-control task, namely RTC. The design method is based on explicit requirements for cognitive support and for aiding in particular. The method integrates and extends software engineering, user psychology, and experimental techniques. It explicates what needs to be done to develop aiding and how this can be achieved efficiently, The RTC task sets higher demands on the modeling and experimental evaluation of the design method than the first application domain of this method-statistics. By embedding the expert model in a main loop and by developing a real-time simulator for railway traffic scenarios, these demands could be satisfied.

The complete method with three steps-design of base system, user analysis, and design of aiding-provides a comprehensive procedure to develop a new system from scratch. When prior knowledge is available, such as in the RTC application, the method can be easily configured to utilize the available information.

MoBel-Baaed A#t#roctcB. The method provides a systematic approach for user interface design of a base system and aiding function. The model- based approach decomposes the design process in analysis, design, implementation, and testing. The use of modeling languages and the decomposition of models in functional, data, behavioral, and social perspectives helps to manage complexity and facilitates the design of user interfaces that are minimal and consistent.


The functional perspective of the analysis model comprises a task decomposition that fits well with the cognitive task load analysis of Neer- incx and Griffioen (1996). This task analysis centers on human task performance and the set of tasks allocated to one person, whereas the design method centers on the human-system cooperation aspects by modeling the communication (data transfer) and the expert task performance (behavioral perspective) more precisely. To model the data, an object-oriented approach proved to be useful.

EmpricalApproach. Design is often an iterative process in which unforeseen problems are detected in a first version or prototype and the problems are subsequently removed in the design of the next version. To optimize this process, we developed a systematic generate-and-test procedure for the design of cognitive support. Theories about human task performance are used to generate a first design that should fit the task performance of future users. Current theories cannot predict human task performance completely, and therefore the design has to be tested for possible shortcomings and to acquire users' needs for support. Compared to desktop analyses, such as cognitive walk-throughs and heuristic evaluations, empirical testing seems to provide a better and more complete assessment of severe user problems (Desurvire, Kondziela, & Atwood, 1992; Jeffries, Miller, Wharton, & Uyeda, 1991; Karat, Campbell, & Fiegel, 1992; Nielsen & Phillips, 1993). Current prototyping and user-testing tools support empirical assessments. In our research, the controlled experiment provided knowledge about the potential benefits and costs of the aiding interface. The prototype can be adjusted relatively easily in a next itera- tion to study other new design options.

5.2. The Effect of Aiding

The second general conclusion of this article is that the proposed type of aiding can supplement human knowledge and capacities. Our experiment centered on aiding that provides task knowledge about how to deal with the current irregularity. A pop-up window tells or reminds the user what should be done. This type of context-specific support proves to be able to compensate for knowledge deficiencies of traffic controllers and to bring about a relatively fast task execution.

Aiding, Task Load, and Mental Mort

The experiment showed significant interaction effects of aiding and task load on the performance of Deal With Irregularities and on the acquisition of general knowledge. Neerincx and Griffioen (1996) maintained that procedural guidance can reduce task load, because the users do not have to


analyze the problem or plan a procedure to solve it, but only have to execute the presented plan or procedure. The results of the experiment correspond to this view: Aiding proved to be most effective for the tasks that were constrained most by human processing capacities (i.e., under overload). With aiding, a single (sub)task seems to require fewer capacities and, therefore, more capacities are available for other (sub)tasks and learning (cf. Paas, 1993; Sweller, 1988).

In this experiment, it proved difficult to measure mental effort. There was an enormous variance among the scores of participants on the SME rating scale and the mental effort score seemed to be influenced by factors other than the actual levels of loading experienced (cf. Wierwille & Egge- meier, 1993). Every scenario ended in a relatively quiet process after which the questionnaire had to be filled in. Users may have forgotten the busy moments or may have found it difficult to provide a single estimate for the task consisting of several different parts. Further, the subtask Deal With Irregularities is perhaps not the best task on which to investigate effects of aiding on mental effort. When they are not supported, traffic controllers may simply not realize that some goals should be reached and-ignoring these goals-the task may thus appear simple. Their invested effort may be low without aiding as a consequence of the execution of fewer actions than necessary to deal with the irregularity. In sum, the experiment does not provide a decisive answer to the question of whether mental effort is or can be an important factor in the success of aiding. This is an important issue for future research.

Aiding "Real* Traffic Controllers

In the laboratory experiment of Ackerman (1992), students worked with a simulator for air traffic control to discover determinants of learning success for complex tasks. The results of this experiment generalized to the learning of air traffic controllers in practice (Ackerman & Kanfer, 1993). How far do the results of our experiment generalize to reality?

ExpWe. An important gap between the experiment and reality is the difference between the user populations. Railway traffic controllers are well trained and they have much experience with the railway network they control, whereas the students were trained for only 30 min. An experiment with railway traffic controllers can determine whether they behave funda- mentally differently from the students. Unfortunately, field investigations often face considerable limitations with respect to many variables, such as limited access to job applicants, insufficient numbers of applicants, and limited testing time (Ackerman & Kanfer, 1993).

Lenior's (1993) research suggests that the population gap is relatively small for the focus of aiding, the nonroutine subtask Deal With Irregularities, and


that provision of context-specific task knowledge can have similar effects on experts' and on students' performance of nonroutine tasks. His research conveyed correspondences between the cognitive processes of railway traffic controllers in complex tasks and the problem solving of test participants in experimental tasks. In correspondence with Reason's (1990) notion that the performance of experts begins to approximate that of novices in novel situations, expert controllers and students proved to apply similar problem-solving strategies in such situations.

Task. A second gap between the experiment and reality is the difference between the simulation task and the RTC task. The simulation does not encompass all aspects of the control task; communication with other personnel is for example only possible in a very limited way (cf. Lenior, 1993). However, the simulation seems to tap the crucial aspects of the job (Ackerman & Kanfer, 1993). In the cognitive task analysis of Neerincx and Griffioen (1996), which studied "realn RTC task performance, Carry Out Work Plan and Deal With Irregularities proved to be two important subtasks of the RTC task. Their analysis showed that the first task hardly is a problem for railway traffic controllers. In correspondence with this finding, the students in the simulation experiment were very capable of setting the routes of trains according to the work plan. Further, according to the cognitive task analysis, task load proves to increase substantially when an irregularity occurs. The only effect of mental effort in the simulation experiment was in accordance with this finding: The average effort score with the training scenario in which no irregularity appeared was lower than the effort scores with the scenarios in which irregularities appeared. Thus, the simulation task seems to correspond to two general characteristics of the real RTC task: "expert" performance for Carry Out Work Plan and high task load for Deal With Irregularities.

In sum, the results of our experiment suggest that aiding will be effective in the current practice of RTC. However, the experiment provides no final proof of the benefits of aiding for this. A decisive experiment requires extensive collaboration of experienced railway traffic controllers.

NOTES

Acknowledgments. We are very grateful to Rene Vink for his contribution to the experiment and are indebted to the Netherlands Railways for providing information about the RTC task. In particular, we thank Edwin Grifioen for giving his "practice-basedn view on the simulator, user interface, and aiding function during the design process. Thanks are also due to the anonymous reviewers who made helpful comments on previous versions of this article.

Support. The Netherlands Organization for Scientific Research (NWO) is gratefully acknowledged for funding this project. This research was mainly conducted


while Mark Neerincx was supported by a grant (575-59-044) from the Foundation for Behavioral Sciences of this organization, awarded to Dr. G. Mulder and Dr. H. P. de Greef. Dr. R. Visser is gratefully acknowledged for being responsible for the original proposal.

Authrs' Awsrt Aklrmb~. Mark A. Neerincx, TNO Human Factors Research Institute, P.O. Box 23, 3769 ZG Soesterberg, The Netherlands. E-mail: neer- [email protected]. H. Paul de Greef, University of Amsterdam, Department of Social Science Informatics, Roetersstraat 15, 1018 WB Amsterdam, The Netherlands. E-mail: [email protected].

HCZ EdttarSal Rscorci. First manuscript received March 29, 1995. Revisions received January 29,1996, andJanuary 27,1997. Accepted by Ruven Brooks. Final manuscript received May 29, 1997. - Editor

REFERENCES

Ackerman, P. L. (1992). Predicting individual differences in complex skill acquisition: Dynamics of ability determinants. Journal ofAMlied Psychobogy, 77,598-614.

Ackerman, P. L., & Kanfer, R. (1993). Integrating laboratory and field study for improving selection: Develo ment of a battery for predicting air traffic controller success. Joumnl of A ~ I ~ ~ B P ~ C ~ I ~ ~ , 78,4113-432.

Adelman, L., Cohen, M. S., Bresnick, T. A., Chinnis, J. 0 . J., & Laskey, K. B. (1993). Real-time expert system interfaces, cognitive processes, and task performance: An empirical assessment. Human Factors, 35,243-261.

Buchanan, B. G., & Shortliffe, E. H. (1984). Rubbased expert sys&m: Tlu Mycln experiments of the Stanford Heuristic Rogyamming Roject. Reading, MA: Addison- Wesley.

Carroll, J. M. (1984).Minimalist design for active users. Human-Computer Interac- tion, Interact'84,219-225. Amsterdam: North-Holland.

Carroll, J. M., & Rosson, M. B. (1987). Paradox of the active user. In J. M. Carroll (Ed.), Interfacing thought: Cognitive aspecls of human-computer interaction (pp. 80-1 11). Cambridge, MA: MIT Press.

CoopGroup. (Ed.). (1996). F'roceedings ofthe Second International Conftrence on the Design of Cooperative Systems. Juan-les-Pins, France: INRIA.

Curtis, B., Kellner, M., & Over, J. (1992). Process modeling. Communicatiom of the ACM, 35, 75-89.

De Greef, H. P., & Neerincx, M. A. (1995). Cognitive support: Designing aiding to supplement human knowledge. International. Journal of Human-Computer Studies, 42,531-571.

De Jong, T., de Hoog, R., & de Vries, F. (1993). Coping with complex environ- ments: The effects of providing overviews and a transparent interface on learning with a computer simulation. International Journal of Man-Madine Studies, 39, 621-639.

Desurvire, H. W., Kondziela, J. M., & Atwood, M. E. (1992). What is gained and lost when using evaluation methods other than empirical testing. People and Computers VII: Proceedings of the H C W Conference, 89-102. New York: Carn- bridge University Press.

Fischer, G., Nakakoji, K., Ostwald, J., Stahl, G., & Sumner, T. (1993). Embedded computer-based critics in the contexts of design. Roceedings of InterCHI'93, 157-164. New York: ACM.


Gopher, D., & Donchin, E. (1986). Workload: An examination of the concept. In K. Boff, L. Kaufman, & J. Thomas (Eds.), Handbook of perception and human petformame: Vol. II. Cognitiueprocesses andpe@rmame (PP. 41- 1-41-49). New York: Wiley.

Jeffries, R., Miller, J., Wharton, C., & Uyeda, K. (1991). User interface evaluation in the real world: A comparison of four techniques. CHI'97 Confirence Proceed- ings, 119-124. New York: ACM.

Johannsen, G., Levis, A. H., & Stassen, H. G. (1992). Theoretical problems in man-machine systems and their experimental validation. Symposium on Analysis, Design and Evaluation of Man-Machine System, 1-11. Hague, The Netherlands: International Federation on Automatic Control.

Johnson, P., Wilson, S., Markopoulos, P., & Pycock, J. (1993). ADEPT-Advanced environment for prototyping with task models. Proceedings of InterCHZ'93, 56. New York: ACM.

Jones, P. M., & Mitchell, C. M. (1994). Model-based communicative acts: Hu- man-computer collaboration in supervisory control. ZnternationalJournal of Hu- man-Computer Studies, 47, 527-551.

Karat, C. M., Campbell, R., & Fiegel, T. (1992). Comparison of empirical testing and walkthrough methods in user interface evaluation. CHI'92 Confirence Pro- ceedings, 397-404. New York: ACM.

Kim, W. C., & Foley, J. D. (1993). Providing high-level control and expert assistance in the user interface presentation design. Proceedings of InterCHI'93, 430-437. New York: ACM.

Kirlik, A. (1993). Modeling strategic behavior in human-automation interaction: Why an aid can (and should) go unused. Human Factors, 35,221-242.

Layton, C., Smith, P. J., & McCoy, C. (1994). Design of a cooperative problem- solving system for en-route flight planning: An empirical evaluation. Human Factors, 36, 94- 1 19.

Lenior, D. (1993). Analyses of cognitive processes in train traffic control. Ergonom- ics, 36, 1361-1368. Morris, N. M., & Rouse, W. B. (1985). The effects of type of knowledge upon human problem solving in a process control task. IEEE Trans- actions on Systems, Man and Cybernetics, 75, 698-707.

Neerincx, M. A., & de Greef, H. P. (1993). How to aid non-experts. Proceedings of InterCHI'93, 165-171. New York: ACM.

Neerincx, M. A., & Griffioen, E. (1996). Cognitive task analysis: Harmonizing tasks to human capacities. Ergonomics, 39,543-56 1.

Nielsen, J., & Phillips, V. L. (1993). Estimating the relative usability of two interfaces: Heuristic, formal, and empirical methods compared. Proceedings of InterCHI'93,214-221. New York: ACM.

Norman, D. A., & Draper, S. W. (1986). User-centered system design: New perspectives on human-computer interaction. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.

Paas, F. (1993). Instructional control of cognitive load in the training of complex cognitive tasks. Unpublished doctoral thesis, University of Twente, Enschede, The Netherlands.

Rasmussen, J. (1984). Strategies for state identification and diagnosis in supervisory control tasks, and design of computer-based support systems. In W. B. Rouse (Ed.), Advances in man-machine system research (Vol. 1, pp. 139-193). Greenwich, CT: JAI.


Rasmussen, J. (1986). Information processing and human-machine interaction: An approach to cognitive engineering. Amsterdam: Elsevier.

Reason, J. (1990). Human error. Cambridge, England: Cambridge University Press. Rumbaugh, J., Blaha, M., Premerlani, W., Eddy, F., & Lorensen, W. (1991).

Object-oriented modeling and design. Englewood Cliffs, NJ: Prentice Hall. Sanders, A. F. (1991). Simulation as a tool in the measurement of human perform-

ance. Ergonomics, 34,995-1025. Schreiber, A. T., Wielin a, B. J., & Breuker, J. A. (Eds.). (1993). KADS: Aprincipled B approach to knowledge- aced system d e v e ~ e n t . London: Academic. Silverman, B. G. (1991). Expert critics: Operationalizing the judgement/decision-

making literature as a theory of "bugs" and repair strategies. Knowh&e Acquisi- tion, 3, 175-214.

Silverman, B. G. (1992). Human-computer collaboration. Human-Computer Inter- action, 7, 165-196.

Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 72, 257-285.

Terveen, L. G. (1995). An overview of human-computer collaboration. Knowledge- Based System, 8, 67-8 1.

Wickens, C. D. (1992). Engineeringpsychology and human performance (2nd ed.). New York: HarperCollins.

Wielemaker, J., & Anjew~erden, A. (1992). PCE-4: Functional overview (Tech. Rep.). Amsterdam: Sociaal-Wetenschappelijke Informatics, University of Amsterdam.

Wierwille, W. W. (1988). Important remaining issues in mental workload estimation. In P. A. Hancock & N. Meshkati (Eds.), Human mental workioad (pp. 315-327). Amsterdam: Elsevier.

Wierwille, W. W., & Eggemeier, F. T. (1993). Recommendations for mental workload measurement in a test and evaluation environment. Human Factors, 35, 263-281.

Yourdon, E. (1989). Modern structured analysis. Englewood Cliffs, NJ: Prentice Hall. Zijlstra, F. R. H. (1990). Mental efficiency in computer work. In P. Richter & W.

Hacker (Eds.), Mental work and automation (pp. 139-144). Dresden, Germany: Technical University Dresden.

Zijlstra, F. R. H. (1993). Efjiciency in work behaviour: A design approach for modern tools. Unpublished doctoral thesis, Delft University, The Netherlands.

Zijlstra, F. R. H., & Meijman, T. (1989). Het meten van mentale inspanning met behulp van een subjectieve methode [Measuring mental effort with a subjective method]. In T. Meijman (Ed.), Mentale Belasting en Werkstress (pp. 139-144). Assen, The Netherlands: Van Gorcum.

cognitive support: extending human knowledge and processing capacities

Documents