temporal aspects of interaction in shared virtual worlds

Temporal aspects of interaction in shared virtual worlds

David Englanda,*, Philip Grayb

aSchool of Computing and Mathematical Sciences, Liverpool John Moores University, Byrom Street, LiverpoolL3 3AF, UK

bDepartment of Computing Science, University of Glasgow, Glasgow G12 8QQ, UK

Received 1 September 1997; received in revised form 16 February 1998; accepted 18 May 1998

Abstract

In this paper we examine the problems of usability and related temporal issues in shared virtualenvironments. Shared virtual environments involve physically distributed users interacting with eachother and with distributed objects via complex highly graphical user interfaces. These factors canresult in interaction which suffers from serious and unpredictable delays in system response times.Careful interaction design can alleviate the problems resulting from such delays. We look at severalareas of shared virtual world design, including object interaction, avatars and scene rendering,investigating ways of dealing with communicating information, preserving world coherence andproviding users with effective real-time interaction. Our medium of analysis is the Extended UserAction Notation (XUAN), a variant of the User Action Notation that enables us to express explicitlyand clearly the temporal features of our examples. Finally, we discuss the implications of ourtemporal analysis for further research and development of shared virtual worlds, and the implicationsfor the further development and use of XUAN.q 1998 Elsevier Science B.V. All rights reserved

Keywords:Temporal aspects; Usability; Shared virtual worlds; Virtual reality; Formal specification;Temporal specification; UAN; XUAN

1. Introduction

Unpredictable system response times and complex interleaving of tasks in user inter-action with computers are the source of a number of interaction breakdowns. Users maynot know how to react when delays occur and may not be able to determine the effects oftheir actions. We first encountered such problems in our own research when investigating

0953-5438/98/$ - see front matterq 1998 – Elsevier Science B.V. All rights reservedPII: S0953-5438(98)00033-2

Interacting with Computers 11 (1998) 87–105

* Corresponding author. Tel.: +44-151-2312271; e-mail: [email protected]

novel painting programs on parallel machines [6]. It became clear that such problemswould soon become more general. Problems of response time and interleaving are cur-rently exacerbated by a number of developments, including:

(1) the Internet and client-server computing, where users can face long and unpredict-able delays in system response;

(2) languages like Java that enable the development of multi-threaded applicationssupporting complex interleaving of system and user actions;

(3) Virtual memory, once the preserve of time-sharing mini-computers, but which isnow commonly found on desktop machines. As virtual memory is used on such smallersystems, it can result in lengthy and unpredictable response times.

In this paper we will focus on yet another such development: shared virtual worlds.These applications are a particularly useful test-bed for investigating temporally inducedinteraction problems. First, such worlds exhibit the problems of the three developmentsmentioned above. Users interact with remote users and objects and may thus be faced withnetworking delays. Interactions between users and objects may be multi-threaded as eachacts as an autonomous agent in the system. Virtual worlds usually have large memoryrequirements. Second, shared virtual worlds introduce additional interaction design chal-lenges because

(1) users interact with local and remote objects and with remote users in collaborativetasks; and

(2) interaction is usually via manipulation of 3D representations of the virtual world andits constituent elements, causing difficulties of rendering scenes at a sufficiently highframe rate.

There are a number of ways of representing the temporal aspects of interleaved inter-action with significant real-time aspects, including time-lines, temporal logics, and Petrinets [8]. There is evidence that temporal logic and related formalisms are superior toothers for the identification of temporal interaction problems [12]. In this paper, however,we are concerned less with problem identification than in examining the ways in whichinteraction features (sequencing mechanisms, forms of feedback) might be exploited toreduce known problems. For such an examination, the notation should enable suchfeatures to be salient in the specification. The User Action Notation (UAN) is a semi-formal task description notation designed to highlight just such action–feedbackrelationships. Therefore, for our analyses of virtual world interaction we have chosen touse XUAN, a variant of UAN extended to deal with complex interleaving and feedbackdelays.

2. Shared virtual worlds

Shared virtual worlds allow remote co-workers (or players) to meet and interact in thesame virtual, three-dimensional (3D) environment. There are many examples of suchworlds; DIVE [4], Community Place [18], Active Worlds [19]. Many of them share thesame features, for example, the user is embodied as an avatar which is a 3-dimensionalobject which they can pilot around the world. In addition users can interact with each otherby text chat or speech. Avatars may also have limited animation controlled by the user to

88 D. England and P. Gray/Interacting with Computers 11 (1998) 87–105

add expressions such as frowning or smiling. The users can also use their avatars tointeract with the world and its objects.

In such environments an avatar also signals to other users that a person is available forinteraction. The reactions of the avatar (running away, smiling, frowning, jumping) are part offeedback mechanism of interaction. In other words, avatars are anaffordancemechanism invirtual worlds, or in CSCW terms (Computer-Supported Co-operative Work) one can say thatan avatar is part of the awareness mechanism of the shared environment. Awareness is a keyconcept in co-operative systems. In its simplest form it can just be a mechanism to notify usersof the presence of others. More advanced awareness mechanisms also show various states ofother users, whether they are active or inactive, what particular task they are involved with,their emotional state, whether they can hear and be heard. In general, an awareness mechanismis any technique that allows users to co-operate, co-ordinate and, perhaps, compete, with theirfellows. In any case, avatars are a key feature for interaction with others in the world.

Avatars are also used to control awareness in shared environments by affecting thespace surrounding a user’s avatar. For example, in DIVE and Community Place, there isthe aura mechanism [2]. An aura is a volume in space around an avatar which, when otherobjects enter, they become aware of its owner. The aura maybe be controlled by the userand set at different volumes for different media, for example, speech and vision. So a usermight be able to see someone’s avatar before he or she could speak to them. CommunityPlace also uses the aura to control which group of people one can communicate with. Sothe user might, for example, be limited to communicating to the first eight avatars thatenter his or her aura. An aura is just one example of the general technique of spatialpartitioning of virtual worlds. This technique is used to limit the potential flood informa-tion (visual, aural, etc.) which may face the user.

Thus, we can see that an avatar is a multi-functional object which: (1) gives the user alocation in space; (2) provides a virtual presence to others; (3) supports action and feed-back of interactions with objects; (4) supports communication with others; (5) is part of anawareness mechanism.

In a shared virtual world the information supporting the above functions has to becommunicated over a network. One can quickly imagine that if there are delays in networkcommunication one or more of these functions may break down. For example, I, as a user,may not be able to work out if another person is available for communication. Delays inthe network may mean that I am not receiving feedback from the other user’s avatar. Inaddition a complex 3D scene may be slow to render on the user’s screen. To furthercomplicate matters, network delays may be unpredictable. Thus, I may at one point seea burst of activity, followed by a further period of apparent inaction. It is these types ofproblems which motivate us to analyse shared virtual worlds from a temporal perspective.

3. Temporal requirements for shared virtual worlds

The Living Worlds [13] proposal was formulated as a way of introducing multi-uservirtual worlds to the VRML community. Amongst its requirements are that the virtualworld should: (1) communicate object state and behaviour; (2) preserve scene integrity;(3) allow users to drive avatars in real time.

89D. England and P. Gray/Interacting with Computers 11 (1998) 87–105

In our first requirement we wish to ensure that the state of avatars and interactive objects inour world is communicated to all users. This is vital to ensuring that individual users are awareof each other. Thus when two or more users wish to co-operate and use an object, they see eachother and the object in the same state. In our second requirement we want the world to appearthe same to all users as the result of some user or system action. So if a new object is created inthe scenery, all users around the network see that new object. Lastly we want to ensure that asusers move around the world they are able to do so smoothly and without unnecessary delays.In Section 5 we shall see how we can specify and analyse these requirements in XUAN.

In trying to support these requirements we are constrained by two factors, the under-lying network technology and the speed of the graphics system.

Most virtual environments differ in the underlying technologies used to support sharingand interaction between users and objects. In Ref. [3] we showed how users could beaffected by the different networking and object-locking strategies. Two common strategiesto networking the users in virtual worlds are to use either the client–server approach or touse multi-casting.

3.1. Client–server worlds

Client–server approaches are generally more common as they are simpler to implementand install. One server process (with, potentially, multiple threads of execution) providesthe central point of communication for the users’ client applications. However, client–server approaches have one main weakness, namely, that the server can become a bottle-neck as network traffic grows. So, as more users join a virtual world, the server has to domore work in transmitting information to the clients. To the users this appears as growingand less predictable delays in the movements of other avatars and the reactions of sharedobjects to their input. In order to cut down on network traffic the servers and clients maychoose to filter out some information.

If we are sharing interactive objects via a central server then we might also need to lockthose objects. That is, if one user begins to interact with an object then other users are preventedfrom operating on the same object until the first user has finished with it. Locking is a way ofpreserving scene integrity so that all users see a consistent world, and not a world in a constantstate of flux. For example, two users may try to change the colour of a shared object. The firstuser to get a lock on the object will be able to change its colour. The second user will not beable to change its colour until the lock is released. Unfortunately the server is the only means oftransmitting information of the presence of locks so if the server is busy delayed locks may leadto some unexpected behaviour. To the users this may appear as unexpected state changes(colour, position, size and so on) in the objects they are attempting to interact with.

3.2. Multi-casting worlds

An alternative approach is multi-casting where there is no central server. Instead parti-cipating applications listen and send information to a multi-cast address and only those inthe same multi-cast group can join. Rather than having client applications we now refer topeer or participating applications. However, multi-casting is not supported by every computerplatform or operating system. A further problem with multi-casting is that the delivery of the


packets, of information transmitted over the network, cannot be guaranteed, neither can theirorder of arrival. It is left to the recipient application to deal with missing or out-of-orderinformation. Now, it happens that in shared virtual worlds we do not need to transmit everypiece of information. The recipient may chose to collect information and re-order it or simplyignore it. If information is missing or ignored it may be filled in by the recipient usingalgorithms appropriate to the type of data. For example, in a busy virtual world, avatarmovement might only be transmitted every few seconds. Other users’ client programsmight then fill in the gaps in information to give the appearance of smooth movement.

Multi-casting does not prevent the locking problems described above but it does avoidthe server bottleneck. Some systems avoid locking altogether and rely on social protocols,that is, users have to negotiate and co-ordinate their interactions with shared objects.

As we can see, both approaches have their drawbacks and we shall see how these impacton users later on. In some virtual worlds a dual approach is used, where a central server isused to hold static information (the world, its objects and avatar shapes) and multi-casting isused to communicate more rapidly changing information, such as avatar position changes.

3.3. Screen updating

Another important resource factor in virtual worlds is the speed at which the screen canbe updated. Users of virtual worlds are initially disappointed because of the lack of coloursand the cartoon-like nature of the scenes. This lack of scene complexity is necessary to ensurethat when the user moves their avatar the world scene can be quickly updated around them.That is, the frame rate is adequate to give the appearance of smooth motion. A slow updatewould lead to a jerky or unusable display. However providing simple virtual worlds wouldseverely limit the domain of applications of VR. Rather than keeping the whole world simplethere are techniques which allow the users current view to be simplified. For example, Levelof Detail Nodes in VRML [1] allow the author of the world to select alternative views of anobject or group of objects depending on the users distance from the object. Thus, a flamemaybe be rendered as a texture at a long distance but close up it maybe represented by a groupof animated objects. Of course this technique relies on the author choosing suitable alter-natives. Another technique which can be automated is that of using visual acuity. In thistechnique the volume of an object is calculated and only objects of a certain volume arerendering, again depending on their distance from the viewer. Thus, large objects would bevisible at a distance but smaller objects would only be visible at close range. The main aim ofthese techniques is to provide authors with ways of striking a balance between the need topresent information and the need to reduce the complexity of the virtual world.

In summary, the temporal aspects of interaction in a virtual world are affected by theunderlying technologies. In Section 5 we will present further scenarios of shared worldinteraction, but firstly we will present our notation for describing those scenarios.

4. XUAN

XUAN (eXtended User Action Notation) is a variant of the User Action Notation(UAN) [10] developed as part of the Temporal Aspects of Usability (TAU) project [7]


to support investigation of temporal issues in interaction. The features described belowwere added both to extend the range of interactive behaviours that can be described and asthe basis of an executable task description language. The executable version of XUAN isthe subject of a further report [14] but will not be discussed further here.

One of the main problems with the User Action Notation is that the temporal relation-ships within and across tasks are not all stated explicitly, but are represented at least in partby the ordering of tasks, feedback and system operations within the UAN’s tabular taskdiagrams. Hence, some important temporal relationships are represented only implicitly.Furthermore, those temporal relationships which can be expressed explicitly in UAN focusprimarily on user actions, thus neglecting the temporal aspects of task categories notconsidered to be user actions.

XUAN attempts to resolve the above limitations by partitioning a task descriptioninto separate sections that specify task details such as local and global variables, pre-and post-conditions, constituent sub-tasks and, most importantly perhaps,a componentthat describes the complete set of temporal relationships among the specified sub-tasks. It is in this explicit partitioning of task aspects that XUAN improves uponUAN, particularly with respect to temporal relationships both within and betweentasks.

4.1. Tasks and actions

In the User Action Notation, tasks consist of a set of temporally related user actions. Thetemporal ordering among elements in the action set is specified in the UANaction lan-guage. For example, if task T contains the action set {A1,A2}, the relationship of strictsequence would be expressed by:

A1 A2 (usually shown on separate lines)

Order independent execution of the set (i.e., all must be executed, but in any order) isshown with the operator ‘&’:

A1 & A2

A full description of the action language can be found in Ref. [11] (definitions of the maintemporal relationships are given in Ref. [9]). XUAN retains all of these relationships,usually varying only in concrete syntax:

UAN XUAN Note

Sequence A1 A2 A1, A2Order independence A1 & A2 sameOptionality A1lA2 sameInterruptibility A1 → A2 A1 → A2 also: A2← A1Concurrent A1kA2 sameInterleavability A1⇔ A2 A1 , l . A2Iteration A* or A þ same also: while (condition) AConditionality condition: A if condition then AWaiting A1 (t , n) A2 various alternatives (see Section 4.3)


User actions are either primitive actions, typically manipulations of physical inputdevices (pressing a key, moving a mouse), or tasks:

〈user action〉 :: ¼ 〈primitive user action〉 l 〈task〉Additionally, and optionally, a UAN action specification may be annotated with informa-tion about system feedback (perceivable changes in system state), non-perceivablechanges to user interface state and application-significant operations. Syntactically, aUAN specification places its user actions in a vertically organised list, with annotationsin columns to the right. Thus, consider a specification of clicking a typical screen buttonwidget (Fig. 1).

This approach, although possessing the virtues of simplicity and readability, is overlyrestrictive in a number of important ways. In particular, it makes it difficult to specifycomplex temporal relationships between user actions and system activity (the annotationdevice assumes that each feedback element immediately follows the action to which it isattached). Further examples of the difficulties forced by these restrictions are reported inRef. [5].

In XUAN, we chose to solve this problem by expanding the language, roughlysimilar in approach to Shneiderman’s multi-party grammar [17], so that systemactions are fully-fledged constituents of a task. That is, tasks are viewed as collabora-tive enterprises involving one or more people (‘users’) and one or more computersystems.

Thus, an XUAN task is, like a UAN task, a set of the actions comprising the task,separated out into columns (or types) for some communicative purpose. The UANtabular layout is used as one way of writing a XUAN description (see below for analternative method). The left/right–top/down ordering of the table elements may beviewed as apartial representation of temporal ordering on actions and the columnarorganisation as modelling the task agents. Fig. 2 is a tabular XUAN description of thesame task as above.

One consequence of treating user and system function similarly in the task description isthat the somewhat ad hoc nature of system annotations of user actions is replaced by asymmetrical treatment of each task agent. Both perform visible actions and (hypothesised)invisible internal actions. These internal actions for the system have already been

Fig. 1. UAN task description.


considered (dialogue state, application action); those for the user are similar (changes toSTM, LTM, planning and other cognitive actions).

4.2. Temporal constraints

The UAN is primarily concerned with expressing temporal relationships of sequenceamong the actions forming a task. The UAN tabular display is a syntactic device forshowing strict sequence simply and effectively. Actions and their annotations are readfrom left to right and from top to bottom. However, certain interactive sequences demandthat the ordering imposed by the tabular format be relaxed.

Consider specifying a two-keypress sequence with type-ahead (i.e., the second keypressmay precede the feedback from the first keypress). If we follow strict left/right–top/bottomsequencing, then type-ahead would not be possible, since k2 must occurafter echo k1.What is wanted, however, arepartial orderings of the actions:

k2 follows k1

echo k1 follows k1

echo k2 follows k2

This set of temporal relationships can be satisfied by an interaction in which type-aheadmay occur; that is, they may be true even though

k2 follows echo k1

is false.To deal with this problem, the temporal ordering imposed by the vertical and horizontal

axes is limited in scope by row or column. That is, the left to right order of items in a rowand the top to bottom order of items in a column imply temporal sequence. However, thetemporal ordering of items which do not share a column or row cannot be inferred. Thus,

Fig. 2. XUAN task description.


in the following table, it cannot be inferred that ‘press key2’ will follow ‘echo key1’.

To specify a variant of this interaction without type-ahead (prohibited type-ahead), it isnecessary to add an annotation to the table which expresses the additional constraint that‘press key2’ follows ‘echo key1’.

In order to accommodate these features systematically in XUAN, we treat each asser-tion about the temporal relationships among actions as a temporal constraint partiallydefinitive of the task. That is, a task may be defined as a tuple of (1) a set of agent actionsand (2) a set of temporal constraints on that action set:

task ¼ (agent actions, temporal constraints)

The set of constraints may also be represented as a list since each constraint is self-contained. However, part of the value of UAN, and XUAN, lies in the representationalpower of the tabular layout. Thus, XUAN uses tabular layout as the preferred representa-tion, with the use of annotations for expressing additional temporal constraints whennecessary.

4.3. Handling action duration

In dealing with time-critical tasks, it is often necessary to express temporal constraintsbased on the actual duration of actions. XUAN includes several functions for this purpose,including the following.

4.3.1. Time functions

start(a:ACTION), stop(a:ACTION)

These primitive functions return a value indicating the start and stop times of a parti-cular action. These two primitives can be built upon to derive more specific temporalrelationships (see below).

duration(a:ACTION)

Returns a value which is the length of a in seconds. duration() is defined in terms ofstart() and stop() as:

duration(a) ¼ stop(a) ¹ start(a)


4.3.2. Time comparison relations

, , , ¼ , ¼ , . ¼ , .

Comparison operators to evaluate temporal relationships, such asstart(a1) ,start(a2) to assess whether the absolute start time of action a1 was less than (earlier)than the absolute start time of action a2.

4.3.3. Additional control structures

if (condition) then a: ACTION

UAN has a special operator for conditionality. In order to improve the readability oftemporal constraints, we have found it helpful to introduce a more conventional condi-tional structure.

5. Temporal issues in shared virtual worlds

We have set the scene for virtual worlds and provided a notation for examining temporalissues. We now wish to bring the two together by exploring some key critical issues oftemporal behaviour. Most of the examples come from our experiences with the sharedvirtual worlds discussed in Sections 2 and 3 above, in particular our development of aVirtual MerseyWorld using Sony’s Community Place. We will not examine every aspector go into full detail with each example. Instead we hope to provide sufficient illustrations,firstly to show the main temporal problems of virtual world interaction and, secondly, toshow the value of XUAN.

5.1. Object interaction

Our first requirement was that the shared virtual world be able to communicate object stateand behaviour. In our first example we will consider a lift, or elevator, in a virtual world.Initially let us imagine that this is like a real-life lift which can take a person between floors ofa virtual building. The lift has floor number buttons to indicate where the user wishes to go.A tabular XUAN description of the floor selection buttons would be as follows:


The tabular form only gives a partial ordering. We could add further constraints toensure that the operations were in sequence. However we might not care if the button wasunlit before or after the door began to open, i.e.

push_button( N), light_button( N), move_lift_to( N),(unlight_button(N) & open_door(N)) , exit_lift

Now, this specification will work in a single-user lift or a lift which is in a multi-userworld where actions on the lift button are not shared. In the latter case every user wouldhave their own local copy of the lift and be unaware of the state of other lifts.

Now consider a shared lift available to all users of a shared virtual world. This time thefirst user selected the button and it will not be available to other users until the lift hasreached the chosen floor.

This time we might be more strict about the unlighting of the button and the opening ofthe door and all specific actions are in strict sequence with no concurrency. Users in thissystem have to wait on actions of shared objects. How that waiting operation is imple-mented depends on the underlying communications protocol. It could be a single lockimplemented in a central server. In this case the time taken to obtain and release a lockmay be long or unpredictable on a busy server. In order to introduce fairness in such asystem we might use time-outs to decide that a lock can expire if it is not used within acertain period. Here we can specify that if the duration of thewait_button operation islonger than some time interval, the lock is released and other users can use that button.

if duration(wait_button( N)) . 5

then end_wait_button( N)

Our lift is now acting more like a computer entity than a real-life elevator. Now we haveproduced a lift which meets the requirements set out in Section 3. We have communicatedthe state of the lift button and preserved the world integrity so all users should be seeingthe same world.

We can take this further and introduce the mischievous lift, which is more in the spirit ofan abstract virtual world. This lift allows multiple users to push the floor choice button buteach push increases a time delay between the last push and the starting of the lift.


The delay is only reset when the lift reaches its destination. Of course with sufficientimpatient button presses the lift may never reach its destination! Nowpush_button andlight_button are concurrent withadd_delay anddelay_lift , i.e.

push_button( N), light_button( N) & (add_delay1( N), delay_-lift),

move_lift_to( N), (unlight_button(N ) & open_door( N)),exit_lift.

5.2. Awareness of avatars

As we discussed earlier, avatars are the embodiment of users in a virtual world and arethe only cue that other users have of each others presence. Not only can they indicatepresence, they can also convey other information about the state of the user. Thus, it isimportant the avatars are introduced into the world in a timely manner. In some client–server systems it has been observed that the name of a user is listed as present but theiravatar is not yet visible. This usually happens when the server is reaching its capacity limitfor dealing with new connections from users. The client user’s program may also haveproblems rendering the 3D images of a large number of avatars.

In this scenario a new avatar is attempting to join the world but there is a variable delaybetween the join operation and the avatar being available to other users. Rather than try tomake the new avatar immediately available we are going to highlight the possibility ofdelay. Such feedback might take the form of a pre-loaded shimmering shape as used insome popular science fiction television series.


Hereadd_avatar adds the new user to the underlying data store of the world and thisis then broadcast to the other participating applications. The actions ofshow_world andadd_avatar are concurrent, which are then followed bybroadcast_new_avatarandshow_simmer .

join_world, accept_join, show_world & add_avatar,broadcast_new_avatar & (show_shimmer, show_avatar).

Again we may need to check the duration of some of these operations and add time-outswhere appropriate.

A similar problem can occur when an avatar moves around in a busy world or is updatedin some other way (e.g. by changing the style of avatar). The problem here is that if we tryto broadcast every update of every avatar position we will quickly swamp the network.Then the updates of all avatars will be delayed and we will get unexpected updates andjumps in avatar behaviour. Users will then be confused about the location and availabilityof their co-users. There are several possible solutions to this problem and we will examinethem in the following scenarios.

One solution is to examine the update behaviour of avatars and provide interpolatorsthat mimic this behaviour at the client applications. This is similar to the dead-reckoningtechnique used in military simulations like DIS [15]. We know, for example, how a tank isgoing to move and can predict its path locally. We can occasionally check our predictionswith the tank’s client and update them accordingly. Tanks are simpler objects to deal withthan avatars so we present a tank example first.

Task: add tank

Feedback Internal operationsAdd_tank

Show_tank1 Interpolate_tankMove_tank

Show_tank2 Check_tank_positionUpdate_tank

Show_tank1


We can add a temporal constraint to make sure the position is checked after some timeperiod N. Here we have chosen to check the difference between the initial drawing of thetank and its re-drawing after a local update by an interpolator;

add_tank, (show_tank1, (interpolate_tank, move_tank,

show_tank2)*, check_tank_position, update_tank)*

if (start(show_tank2)–stop(show_tank1) . N) then

check_tank_position

Now this technique works for predictable objects like tanks but avatars driven byusers are less predictable. Instead we have to consider some dynamic filtering ofupdate information. For example, if there are several rapid changes in position in ashort time we may choose to only send those updates with a longer time intervalbetween them. This technique is commonly used in distributed window systems suchas the X Window System. So in the following example we are going to use timeintervals between updates as a deciding factor on whether to broadcast them. We mightcompute on the value of time interval depending on the quality of service on thenetwork. However, deciding the quality of service is beyond our the scope of our currentdiscussion.

Task: move avatar

Articulatory actions Feedback Internal operationsMove to new position Update_avatar

tmp ¼ start(update_avatar)Broadcast_avatar

Redraw_avatar

Now when we move the avatar its position is updated and the time of that updaterecorded. We then redraw the avatar and broadcast its new position

move_avatar, update_avatar, (redraw_avatar & broadcast_-avatar)if (start(update_avatar)–tmp . N) then

broadcast_avatar

Note we only broadcast the new position if the time interval between one update and thenext is greater than our factorN.

Our discussion about the joining and movement of avatars also applies to any other statechanges in avatars. We need to examine what factors affect the broadcast and receipt of thosechanges to ensure that we are maintain any awareness requirements for our virtual world.

5.3. Avatar interaction

In Section 5.2 we saw one initial problem with avatars, namely joining the world. Afurther common problem for users of virtual worlds is knowing whether their fellow


cybernauts are alive or dead. An avatar may be visible in the world but not reacting toothers for several reasons; (1) its user may be busy with another task; (2) the server may betoo busy to allow interaction with that avatar; (3) the client may have disconnected but theavatar is still in the server database.

Shared objects may also be in these states. There is usually no indication to other usersas to which of these states the avatar is in. This example covers our second requirement;ensuring consistency in the users’ views of the shared world. Some systems use aheart-beatsignal to show that an avatar is still alive. This is mainly used for consistency of thevirtual world database but we will show here how it might be used to give feedback tousers as to the state of other avatars. This signal is sent out periodically so as not to add toogreatly to network traffic.

In this scenario clients receive the heart-beat signal and use it with recent updateinformation to decide on the state of the avatar. In addition time-outs are used to cleanup deceased avatars.

Task: monitor avatar

Feedback Internal operationsLast_hearbeat ¼ now()(current_heartbeat ¼ check_heartbeat()

Draw_avatar or draw_sleeping_avatar orremove_avatar

Last_heartbeat ¼ current_heartbeat())*

Here we check the heartbeat of the avatar which returns one of three possible states:alive if we receive the heartbeat within a certain interval of the last update;asleepif aheartbeat is received but no recent updates; anddeadwhen another time interval elapsesbetween heartbeats, i.e.

if (current_heartbeat–last_update . N) then

draw_sleeping_avatar

else draw_avatar

if (current_heartbeat–last_heartbeat) . M)

remove_avatar

5.4. Rendering issues

In our final example we will see how we might use temporal measures to controlthe rendering of the world scene. Our third requirement in Section 3 outlined the needfor real-time graphical response from the system. To some extend the techniques wehave already mentioned will cut down on scene complexity, for example, by reducingthe need for avatar updates and by removing dead avatars and unwanted shared objects.Rather than concentrating on any one technique we will assume there is a rendering


complexity function (be it based on level of detail or acuity) and this will be controlledby the current frame-rate. Thus we will automatically adjust the scene renderingcomplexity.

Task: control rendering

Feedback Internal operations

Check_frame_rateAdjust_scene_complexity

Change_rendering_complexity

Say we always want to maintain a frame-rate of 12 frames per second for our world. Ifwe find we are falling below this value we might choose to lower the scene complexity.For example, if we were using acuity measures we might decide that only larger volumeobjects would be visible at some range, i.e.

check_frame_rate, adjust_scene_complexity, change_-rendering_complexity

if (check_frame_rate , 12) then

adjust_scene_complexity(acuity_volume ¼ 12)

Thus the system attempts to maintain a smooth visual image regardless of the actualcomplexity of the scene.

6. Implications

The implications for our work fall under four main categories: (1) the design of virtualworlds; (2) research and development of the underlying technologies; (3) the evaluation ofvirtual worlds; and (4) the further development of XUAN and other specificationnotations.

6.1. Design of virtual worlds

The designers of shared virtual worlds need to be aware of the current technical limita-tions of their medium. This means making better use than is exhibited in many VRMLworlds of, for example, level of detail control and world partitioning. By using thesetechniques scene complexity can be reduced, giving smoother interaction for users.These techniques can also make more efficient use of the available networking bandwidth(although designers can in general do little about networking delays).

Unfortunately, most authoring tools for virtual worlds do not provide tools to controllevel of detail. To introduce such control, authors must edit model descriptions by hand.This is clearly an unsatisfactory approach and better tool support is needed.


6.2. Implications for the supporting technologies

Communications infrastructures can be improved by providing faster networks; ofcourse, it is likely that many virtual world users will remain on slower parts of the network.Therefore we need to examine ways of making better use of the available networking.Improved communications protocols could be developed which address quality of servicein shared virtual worlds. That is, as the number of shared users grows, the amount ofinformation actually transmitted could be kept relatively constant. We then rely on theuser’s client doing some of the work to fill in or indicate any missing information. Suchclient level developments imply a need for further research into browsers and other enduser tools.

There is also a need for research into the area of graphics for shared virtual worlds. Twomain issues need further investigation: reducing the amount of graphical information thatneeds to be held and presented on a user’s system and reducing the amount of graphicalinformation that needs to be transmitted over a network. In the first case we need moreadvanced techniques for level of detail control so that only those parts of the world that areimmediately needed by the user are part of the rendering process. In the second case weneed to ensure that changes to the virtual world can be transmitted efficiently. For exam-ple, virtual world descriptions could be transmitted as complex, but compact fractaldescriptions rather than as full three-dimensional models.

6.3. Evaluating shared virtual worlds

One of our initial goals with XUAN was to provide a means of communication betweensoftware engineers and human factors specialists. The development team would specifytemporal requirements for an interaction scenario. Engineers could formulate this as anXUAN specification which would be executable. The human factors specialist couldthen run the XUAN-based prototype with the end users to evaluate the temporal require-ments. Whether the specification is executed or not, we still believe XUAN provides a goodmeans of documentation and discussion of temporal issues which can form the basis ofevaluation.

However, the evaluation of shared virtual worlds is complicated by their social nature. Itis the goal of shared virtual worlds to support virtual communities and evaluating thesuccess of the community can be very difficult and may outweigh individual interactionconcerns. We can be sure, however, that if a system fails to meet basic temporal interactionrequirements it will also fail to supportany community activity.

6.4. Implication for user interaction notations

We have found XUAN useful in describing situations for which it was not envisaged.This gives us some confidence that the set of features and principles we initiallyprovided cover a wide range of interaction issues. XUAN tries to describe interactionin a neutral way without regard to the deeper state of the user or the system. We believethis neutrality is important in enabling designers to focus upon interaction issueswithout becoming overwhelmed by other systems issues. This is especially important in


the area of temporal specification where, at the systems design level, there can be manyconflicting requirements and issues. The question is not whether XUAN is the best tem-poral notation but whether it covers the right issues.

XUAN has served as a suitable medium for these early, somewhat tentative investiga-tions. The basic UAN notation is low cost and XUAN’s additional temporal constraints areadded as a way of overcoming the problems encountered in the more simple specification.We envisage a number of developments to improve XUAN for dealing with advancedmulti-user multimedia interaction. For example, XUAN currently has no notion of a groupor members of a group. The concepts of groups in CSCW is generally difficult. Groups canbe dynamic, and efforts to enforce grouping via system mechanisms can lessen theirflexibility. What is required in XUAN is the treatment of users (and groups) as objectswhich carry out methods.

In our earlier work we provided a simulator for XUAN, written in Ada. This allowedhypotheses to be formulated in XUAN and then tested experimentally on the simulator.Ada was chosen at that time for its built-in support of tasking. This naturally sup-ported the temporal relations of the XUAN. Java also has support for tasks via itsThread class. In addition many researchers and developers (e.g. Ref. [16]) are usingJava to construct shared virtual world systems. Thus an XUAN simulator built inJava may be a useful step in supporting experiments with the temporal aspects ofshared interaction.

7. Conclusions

We have looked at some of the key temporal issues surrounding virtual worlds andexamined some of them more formally using the XUAN notation. Using such a notationallows us to concentrate on some of those issues which may need special attention in thedesign, evaluation and implementation of virtual worlds, namely, networking and render-ing. We saw that we either needed to cut down extraneous information or provide time-outs so that networking resources were not monopolised.

This investigation, identifying problems and looking at possible fixes, is only a first step.What is needed now is further work to build on these results. For example, are thesetemporally related problems the main ones? How costly are they in terms of theusability of shared virtual worlds? We also hope the examples we have used can beexpanded to influence the development of systems and proposals for shared virtualworlds.

Acknowledgements

Our work on XUAN was initially funded bythe Joint (UK) Councils Initiative in HCI/Cognitive Science, grant number g9201233. The work on virtual environments was carriedout at GMD, Sankt Augustin, Germany funded by a European Commission,HumanCapital and Mobility fellowship. We also thank the referees for their helpful commentson an earlier version of the paper.


References

[1] A. Ames, D. Nadeau, J.L. Moreland, The VRML 2.0 Sourcebook, 2nd ed., John Wiley, New York,1997.

[2] S.D. Benford, L.E. Fahle`n, A spatial model of interaction in large virtual environments, in: Proceedings ofthe Third European Conference on CSCW (ECSCW’93), Milano, Italy, Kluwer, Dordrecht, 1993.

[3] W. Broll D. England, Bringing worlds together: adding multi-user support to VRML, in: Proceedings ofVRML 95 Symposium, New York, ACM, 1995.

[4] C. Carlsson, O. Hagsand, DIVE—A platform for multi-user virtual environments, Computer Graphics 17(6) (1993) 663–669.

[5] J. Coutaz, G. Faconti, F. Paterno, L. Nigay, D. Salber, MATIS: a UAN description and lesson learned,AMODEUS Project Report WP14, 1993.

[6] D. England, T. Cockshott, Painting with wet paint on a Transputer-Based system, in: Durani et al., (Eds.),Proceedings of Transputer Applications 3, IOS Press, Amsterdam 1991.

[7] P. Gray, D. England, S. McGowan XUAN: Enhancing the UAN to capture temporal relationshipsamong actions, in: Proceedings of BCS HCI ’94, Cambridge University Press, Cambridge, September1994.

[8] P.D. Gray, C.W. Johnson, Requirements for the next generation of user interface specification languages, in:Proc. DSV-IS 1995, 1995, pp. 113–133.

[9] H.R. Hartson, P.D. Gray, Temporal aspects of tasks in the user action notation, Human Computer Interaction7 (92) (1992) 1–45.

[10] H.R. Hartson, A.C. Siochi, D. Hix, The UAN: a user-oriented representation for direct manipulationinterface designs, ACM Trans. Information Systems 8 (3) (1990) 181–203.

[11] D. Hix, H.R. Hartson, Developing User Interfaces, Wiley, New York, 1993.[12] C. Johnson, The evaluation of user interface notations, in: Proc DSV-IS 1996, 1996, pp. 188–206.[13] Living Worlds, Making VRML 2.0 Applications Interpersonal and Interoperable, http://www.livingworlds.-

com/, accessed September 1997.[14] S. McGowan, P.D. Gray, D. England, XUAN: An Executable Task Description Lanaguge, Dept of

Computing Science, University of Glasgow, 1994.[15] NPSNET Research Group, http://www npsnet.cs.nps.navy.mil/npsnet/, accessed September 1997.[16] B. Roehl, J. Couch, C. Reed-Ballreich, T. Rohaly, G. Brown, Late Night VRML 2.0 with Java, Ziff Davis

Press, 1997.[17] Shneidermann. Multiparty grammers and related features for designing interactive systems IEEE Transac-

tions in Systems. Man and Cybernetics 12 (2) (1992) 148–154.[18] Sony, Virtual Society on the Web, http://sonypic.com/vs/index.html, accessed September 1997.[19] Worlds Inc, Active Worlds, http://www.activeworlds.com/, accessed September 1997.


temporal aspects of interaction in shared virtual worlds

Documents