Which way to move?Incremental recognition of trajectories with context information

Maaike Veltmaats0628972

November 3, 2010

M. J. T. Veltmaat m.j.t.veltmaat@student.ru.nlBachelor thesis in Artificial Intelligence

Supervisor Dr. L. G. VuurpijlDonders Institute for Cognition and BehaviourRadboud University Nijmegen

Second Supervisor Dr. I.G. Sprinkhuizen-KuyperDonders Institute for Cognition and BehaviourRadboud University Nijmegen

Contents

1 Introduction 31.1 Brain computer interfacing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.1.1 Two-dimensional cursor movement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.2 Context information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.3 Incremental recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.3.1 Fluid Sketches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.4 Organization of thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2 Approach 7

3 Modeling user-driven noisy trajectories 93.1 Basic user model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.2 Basic user model with momentum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.2.1 Results basic user model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.3 Extended user model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.3.1 Estimation of number of steps for performance model . . . . . . . . . . . . . . . . . . 143.3.2 Results extended user model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

4 Trajectory matching and system-guided trajectory generation 184.1 Incremental trajectory matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184.2 System-guided trajectory generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194.3 Results of system guided trajectory generation . . . . . . . . . . . . . . . . . . . . . . . . . . 20

5 Discussion and future research 245.1 User driven versus system guided generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245.2 Influence of the performance model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255.3 Future research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265.4 Usage of results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

References 28

2

Abstract

Brain-computer interfaces for 2D trajectory movement still lack sufficient bandwidth and classificationperformance. Learning and sustaining the required mental tasks is hard and the signal to noise ratiois very low. Given a lexicon of 2D trajectories and various trajectory coordinate sequences distortedunder various conditions, this research explores how knowledge about the possible target paths can beemployed to incrementally predict the most probable intended direction. We found that both the numberof errors and the trajectory length significantly decrease when the generation of trajectories is guided bythe system. The results may be used for, e.g., robot planning, mouse control, and the generation of 2Dtrajectories in general.

1 Introduction

In today’s society the use of computers is becoming more and more important. Traditionally, the communi-cation between humans and computers is with a mouse and keyboard, but over the past decades more andmore new input devices were developed to control a device. This enables many new interaction methodslike, gaze tracking, speech based communication, handwriting recognition, or 3D gesture recognition. Butwhat if people are not able to control one of the above mentioned devices, for example because they arecompletely paralyzed?

Recent developments in our understanding of neurophysiological phenomena which can be measured andprocessed through signal processing and pattern recognition, have enabled a challenging new paradigm calledBrain Computer Interfaces, or BCI’s. The downside of using a BCI system is that the signals measured by adevice are noisy and distorted since they have to travel through the skull, making it difficult to decode thebrain signals and control a device fast and with high accuracy. Next to that, it also requires a large amount oftraining to use a BCI. [Wolpaw and McFarland, 2004] showed that it is possible to control a two-dimensionalsignal with a non-invasive BCI to move a cursor to a target after extensive training, though improvementsin speed and accuracy could still be made.

[Galan et al., 2008] present an intelligent wheelchair with a shared control system that uses environmentalinformation to take over control if the user does not provide the correct signals to stay at a safe distancefrom walls. The environment information, or context information, helps the user to reach the goal safely.Could context information also help if we want to predict the target someone wants to reach while movinga cursor on a screen, as in [Wolpaw and McFarland, 2004]? Or maybe even help to recognize a letter whileit is still being written?

The aim of this project is to explore how context information can be used to improve the robustness of noisy2D control signals, such as EEG signals as used in a BCI, and if it is possible to predict the target a 2Dcontrol signal is trying to reach. If this is possible, it would be a great improvement for locked-in patientswho can only communicate with the outside world through BCI. If the target a user wants to reach, or theletter someone attempts to write while controlling a BCI, would be recognized before the target is reachedor the letter is fully drawn, this would speed up communication.

Before continuing with our approach we will provide more information about and examples of BCI’s, contextinformation, and incremental recognition.

1.1 Brain computer interfacing

A brain-computer interface (BCI) system is a system that picks up signals from the brain and translatesthese signals into commands to control a device, for instance a wheelchair ([Galan et al., 2008]) or a computerprogram. BCI systems tend to use only brain signals to control a device and no muscle tension. In that wayit is possible for people who are completely paralyzed, so called locked-in patients, to communicate with theoutside world.

3

Brain computer interfaces can be classified in two different types, invasive and non-invasive. Invasive BCI’shave the electrodes placed inside the brain, and non-invasive BCI’s measure brain activity outside theskull. This can be done with several imaging methods, such as functional magnetic resonance imaging(fMRI), magnetoencephalogram (MEG), electrocorticogram (ECoG), but most commonly used is the elec-troencephalogram (EEG). The signals the system has to work with can be synchronous and asynchronous,which means that the devices are controlled by a signal that either starts at a specific and known momentin time (synchronous), or can start at any moment in time, whenever the user wants to (asynchronous).

The intelligent wheelchair created by [Galan et al., 2008], is an asynchronous non-invasive BCI, so it iscontrolled by a signal instantiated when the user chooses it, recorded outside the brain. In this case EEGis used to measure the signal. Subjects were instructed to control both a real wheelchair and a simulatedone by producing brain waves. The BCI system then estimated the probabilities for the different mentalcommands and combined them with environmental information obtained from the sensors on the wheelchair.The shared control system uses the environmental information to take over control in case the EEG accuracyis low. This way, it can prevent the user from steering into a wall or another obstacle when he or she doesnot generate the appropriate commands to stay away from them.

One downside of using a non-invasive BCI, is that there can be a lot of artifacts in the signal and most ofthe time the actual neurophysiological component of the signal is very small. Artifacts are the componentsin the recorded signal that are not caused by neurophysiological activity, such as muscle activity caused byeye blinking, or line noise. The artifacts have to be removed from the signal, or there has to be a smart wayto work around the artifacts or let the system handle the noise caused by artifacts.

Another downside of working with a BCI, is that it requires a large amount of training to use it. The userhas to perform a certain task, for example thinking of left foot movement for moving up, and thinking ofright hand movement to move down, to create the brain waves the system picks up to actually move up ordown. In addition to this, different users have different EEG features which can be used to recognize thesignals, so these have to be found too.

1.1.1 Two-dimensional cursor movement

Another example of a BCI can be found in [Wolpaw and McFarland, 2004]. They attempted to control atwo-dimensional movement signal by a non-invasive BCI, which until then, was widely assumed only to beprovided by invasive BCI’s. Four subjects participated in the study, two of whom had a spinal cord injury.The users with spinal cord injury performed better in the task; they were faster (1,9 and 1,9 seconds versus3,9 and 3,3 seconds) and more accurate (89% and 98% versus 70% and 78% of trials).

The task the users had to fulfill consisted of reaching a target presented on a screen within 10 seconds afterthe cursor appeared on the screen. The cursor was controlled by the user, using the brain waves the userproduced. A trial began when one of the eight targets appeared on the screen. Then the cursor appearedin the middle of the screen and moved according to the brainwaves produced by the user. If the target wasreached within 10 seconds after the cursor appeared, the target flashed. Otherwise the cursor and targetdisappeared and a new trial was initiated. The averaged trajectories for each user are shown in Figure 1.

The horizontal movement of the cursor is controlled by mu-rhythmic (8-12 Hz), and the vertical move-ment is controlled by the beta-rhythmic (18-26 Hz). Both horizontal and vertical movement are controlledsimultaneously and the results of the research show that they do not interfere with each other.

During the trials, both the user and the system learn to work together; the user learns to produce the correctbrain waves to control the cursor movement, and the system learns to focus on the amplitudes within thementioned frequency bands which the user controls best.

In this thesis we will pursue a similar task as described in [Wolpaw and McFarland, 2004]. By modelingnoisy 2D trajectories, BCI control will be emulated. Furthermore, as outlined in the next section, we willemploy context information to estimate the target positions in an early stage.

4

Figure 1: Averaged cursor trajectories for each user to each target, as found in[Wolpaw and McFarland, 2004].

1.2 Context information

One way to make it easier for users to control a device, is to provide context information to the system, asthe shared control system of [Galan et al., 2008] does. Another example of the use of context information isthe Dasher speller by [Ward et al., 2000].

The Dasher speller is an alternative text input device by [Ward et al., 2000], which represents letters inrectangles, on the right side of the screen, and uses a language model to change the height of these rectangles,making it easier to select a letter which is more likely to be chosen according to the language model. AlthoughDasher is slower than a conventional keyboard (34 words per minute, compared to 40-60 words per minute),it can be used on small devices.

According to Fitts’s law [Fitts and Michael, 1967], the movement time for hitting a target depends on a fewempirically determined constants, the distance to the target, and the size of the target. The most commonform is: MT = a+ b log2(d/s+ 1), with a and b the empirically determined constants: d the distance to thetarget and s the size of the target. So, when the distance is kept practically equal, the movement time willdecrease when the target is larger. The language model is used to represent the more likely continuations ofthe string in larger rectangles, thus making it easier to select these rectangles.

1.3 Incremental recognition

Another method to help users with controlling a device is incremental recognition, where information is addedto the recognition process step by step. [Zhao, 1993] presents concepts and techniques to use incrementalgesture recognition in diagram parsing in graphical editors. Diagram editing allows the user to control acomputer using hand-sketches, but it is hard to build editors for this kind of applications. [Zhao, 1993]designed an integrated gesture recognizer, with two components; one for recognizing the hand-sketchedsymbols, and one for the interpretation of these symbols.

The system of [Zhao, 1993] consists of two parts: the low level recognizer (LLR) and the high level recognizer

5

Figure 2: Incremental recognition of a multi-stroke rectangle (see Figure 3 in [Zhao, 1993])

Figure 3: Steps while user is drawing a box-shape. When a shape is recognized, the line color changes tored. (see Figure 1 [Tandler and Prante, 2001])

(HLR). The LLR transforms a sequence of points, the input to the system, into graphical symbols. Eachstroke is recognized after it has been drawn and then stored in the database until a valid symbol is recognized.An example is shown in Figure 2; from (a) to (b), both strokes are merged into an ”L”-form. From thenon, the ”L”-form is stored in the database. When stroke 3 in (c) is added, this is morphed to a ”U”-form.The user sees immediately how the strokes are combined, and can draw additional strokes to complete theshape. The same concept is implemented by [Tandler and Prante, 2001]; they provide feedback to the userby changing the line color after a shape is recognized (Figure 3).

The HLR is responsible for transforming the graphical symbols recognized by the LLR into editing commands.The LLR and HLR are tightly integrated, allowing the underlying diagram syntax to improve the gesturerecognition. Besides the improved recognition, the incremental recognition also makes the gestural interfacemore direct.

1.3.1 Fluid Sketches

An application for the incremental recognition of hand-drawn shapes is provided by [Arvo and Novins, 2000],and is called Fluid Sketches. Fluid sketches is able to recognize simple shapes, such as circles and boxes,by using specific fits for the different shapes. It tightly couples the gesture recognition with rapid morphingto provide feedback to the user as soon as each pen stroke is drawn. In this way, the user can immediatelyadjust the gesture, when the result does not match the intended gesture. This last part is achieved byweighting the recent part of the stroke heavier than the rest of the stroke.

As mentioned before, the recognition of the strokes is done by specific fits for the shapes. In the case of circlesthis is linear least squares; in which an over-constrained system of equations containing the given points ofa stroke can be solved by solving the corresponding normal equations of the form Ax = b. In the case ofweighted linear squares, a weighting matrix can be applied and the normal equations become ATWTWAx =ATWTWb. The best-fit circle will then be centered at point (c1, c2) with radius r =

√c3 + c21 + c22.

For boxes, the specific fit is a relaxation process, starting with the smallest box that contains the wholestroke. With each iteration the shape is updated by applying spring forces on the box; each point pulls the

6

nearest face or vertex of the box towards it and with gradient descent, the net force on all faces will be zero,which results in the optimal box.

While the user is drawing, the systems keeps adjusting the drawn shape to the most likely shape. Due to thisincremental recognition, feedback can be provided to the user immediately, by morphing the drawn shapein to the most likely shape. When the resulting shape is not as intended, the user can adjust the shapeimmediately.

The above examples show that incremental recognition can help to increase input speed, and provide theuser with feedback, while drawing, which makes it easier and faster to input the intended symbol.

1.4 Organization of thesis

The research presented in the current thesis is an exploration of how context information in the form ofknowledge about the possible target paths can help to incrementally predict the most probable target. Ourapproach will be discussed in Section 2.

The first part of the exploration is the trajectory generation which consists of different versions of a usermodel, that will be discussed in Section 3. We will evaluate this model by looking at the number of errors,which is the number of trials in which the target is not reached within a finite or pre-specified number ofsteps. We will also look at the time needed to reach the target, represented as the number of steps.

The second part of the exploration is the trajectory matching. In this part we will incrementally match thetrajectories generated in the first step to the target trajectories. This will result in the number of correctlyrecognized targets for every 10% of the trajectory, and will be discussed in Section 4.1.

In the last part of our exploration trajectory generation and trajectory matching will come together in thedynamic generation of a 2D control signal. This will be discussed in section 4.2.

We will discuss our results and conclude our research in Section ??.

2 Approach

If the BCI system of [Wolpaw and McFarland, 2004] would have had context information, or used incrementalrecognition, this could have decreased the time needed to reach a target. The context information in thiscase, could consist of the optimal paths from starting point to target, and while interpreting the controlsignal from the BCI, a context-driven system would try to estimate the likelihood for the different paths,given the current signal. By estimating the likelihood for the different target paths every time a new pointis added to the signal, incremental recognition can be used to move the cursor towards the target.

In the mean time, the system can provide feedback to the user, by drawing the more likely target path,e.g., in a bigger pen width, or by changing the color of the current line, when the targets are representedin different colors. Another possibility is to change the size of the targets, as in [Ward et al., 2000], or tolet the cursor move faster towards one of the targets, when it is more likely that that target is the intendedtarget, like a magnet (the target) attracts iron (the cursor). Since we are going to model users and we donot have to provide feedback to an actual user, we will not specifically discuss this subject.

Context information can help with the control of a device, making control faster and easier, and incrementalrecognition can decrease the amount of input needed to recognize the intended drawn symbol. Can thecombination of the two help when a user wants to reach a target on a screen with a cursor, or help torecognize a letter while he or she is still writing?

We explored how context information and incremental recognition can be used to improve the robustness ofnoisy 2D control signals for the generation and prediction of 2D trajectories. The question is how knowl-edge about the possible target paths can be used to incrementally predict the intended path, and, moreimportantly, does this matter?

7

Figure 4: The lexicon of 2D gestures with the corresponding targets

8

Figure 5: Noise range

To do this, we will used context information represented in the form of a 2D gesture repertoire, or lexicon,of known 2D trajectories. The lexicon consisted of 8 different trajectories, all reaching to another target,and is shown in figure 4. The targets were the same as in [Wolpaw and McFarland, 2004]. The trajectoriesstarted in the middle of the screen, and the optimal trajectory from the starting point to the target formeda straight line from the starting point to the middle of the target. A trial consisted of a target, and a wayto get to the target: the noisy 2D control. This signal is noisy due to artifacts, as explained in the BCIsection. Personal communication with Jason Farquhar lead to the understanding that most noise in BCIcontrol signals is Gaussian distributed, so we incorporated Gaussian noise in our user model.

First we will discuss the generation of trajectories with our user model.

3 Modeling user-driven noisy trajectories

The user model generates a sequence of 2D control commands that are new points on the trajectory betweenstarting point and target point. In our exploration we will simulate a user using different parameters, thenoise range and the stroke length. This is the basic user model, which will be discussed first. Then we willdiscuss the basic user model, but with the use of momentum. And at last we will present an extension ofthe basic user model.

3.1 Basic user model

The basic user model gets two parameters: the noise range, which is the amount of distortion on the signal,and the stroke length, which is the size of the step that is traveled each time. Generally, the basic user modeldoes a step in a certain direction.

Stroke length The stroke length is the distance that is traveled in each step, and does not change duringthe trajectory generation. Once the model determined the direction, the new point is a stroke length awayfrom the current point, in the generated direction. In our exploration we will use stroke lengths of 10, 20,30, 40, and 50 pixels. In Figure 5 the stroke length is the radius r.

9

Noise range The next parameter is the noise range η. This is the angle in degrees and represents therange in which the Gaussian noise varies. The noise range will be multiplied with a Gaussian (N(0, 1)) value,and this will be added to the ideal angle. The distorted angle that is added to the ideal angle has a meanof 0, which means that on average the new point will be on the straight line to the target. Multiplying theGaussian value with the noise range results in a standard deviation of the noise range, which means that atmost 68,3% of the generated values differ at most with an angle of plus or minus the noise range from theideal angle.

Figure 5 visualizes the situation with the noise range. The start and target points are drawn in black, thestroke length r is drawn in red, and the noise range η is drawn in gray. The mean of the Gaussian distributionis the black line from starting point through desired point to the target, and the area in which 68,3% of theGaussian values will be, is between the gray lines.

In our exploration we will use noise ranges of 15, 30, 45, 60, 75, and 90 degrees. Since the noise range canbe added or distracted from the ideal angle (due to the Gaussian value being positive or negative), 68,3% ofthe generated values for a noise range of 90 degrees will be in an area of 180 degrees.

The new point is generated by first computing the angle between the current point and the target point. The“noisy” new direction is computed by adding the multiplication of the random Gaussian value with the noiserange to the current angle. Finally, the length of the corresponding displacement vector (cos(angle), sin(angle))is normalized to stroke length. The algorithm for the generation of a new point is given in Algorithm 3.1.

Algorithm 1 Generation of new pointnext Pointcompute angle(current point, target point)compute new anglenew_angle = angle + (noise range * Gaussian random)

compute displacement vectordx = cos(new_angle)dy = sin(new_angle)

normalize to stroke lengthdx = strokelength/sqrt(dx*dx+dy*dy)*dxdy = strokelength/sqrt(dx*dx+dy*dy)*dy

compute new pointx = x_start + dxy = y_start + dy

3.2 Basic user model with momentum

A common technique in 2D trajectory generation is the use of momentum, since it models real physicalmovements in a realistic manner. It works as a sort of filter, to average out sudden movement changes. Wewill use the principle of “momentum” over the last three subvectors to compute a new point (xi, yi). First,Algorithm 1 is used to determine the intended new point P ′i = (xi, yi). Using momentum, the new pointis computed by 1/3 ∗ (Pi−2 + Pi−1 + P ′i ). Again, the length of the corresponding displacement vector isnormalized to stroke length. A visualization of the computation of the new point using momentum is givenin Figure 6.

For an example of the trajectories resulting for different parameter settings please consider Figure 7. As canbe observed, there is a clear relation between the noise range and the length of the resulting trajectories,as well as between the relation between the smoothness of the trajectory and the use of momentum. Thetrajectories that were created using momentum (bottom row Figure 7), look much smoother and don’t makesudden movement changes as the trajectories that were created without using momentum (top row Figure

10

Figure 6: Each new coordinate is computed using “momentum”. The gray point is the intended new pointP ′i , and the black point is the resulting point, averaged over the last three subvectors.

(a) (b) (c)

(d) (e) (f)

Figure 7: Different trajectories for various noise ranges. The top row shows trajectories without usingmomentum, a constant stroke length of 20 (pixels) and increasing noise range. Noise ranges are respectively15 (a), 60 (b), and 90 (c). The bottom row shows trajectories generated with the same settings, but usingan average momentum over the last three subvectors.

11

7). For all experiments described below, we will use a momentum computed over three points.

3.2.1 Results basic user model

The results for the basic user model are presented in Figure 11 and Figure 12. A trajectory is considered anerror, if the target is not reached within a certain number of steps; an example of this is shown in Figure 8.

Figure 8: Example of an erroneous trial. The generated trajectory has to reach the green target. Due to thelimited number of steps, the generation is cut off, and the target is not reached.

To determine such a threshold for a given combination of stroke length and noise range, we ran the basicuser model 1000 times for each of the 8 targets, limited to maximally 1000 steps, and computed the meanand standard deviation of the number of steps. The threshold θc for each combination c of noise range andstroke length was subsequently set as θc = µc + 2 ∗ σc. The 1000 steps that were initially allowed weresufficient to reach all the targets.

It can be observed that the number of errors increases with both the stroke length and the noise range,for low values of the noise range and stroke length, there are no errors; so the generated trajectory alwaysreaches the correct target within the allowed number of steps. From a noise range of 75 or higher, errorsoccur even with a stroke length of 10; with a stroke length of 20 errors also occur with a noise range of 60.A noise range of 15 only leads to errors with a stroke length of 50.

The total number of errors for the basic user model, over all conditions, is 2903. This is 0.012%, so themodel had a very good performance in reaching the targets.

3.3 Extended user model

It is well known that user performance in human-computer interaction varies under influence of, e.g., mentalload, fatigue, or emotional factors [Dix et al., 2004]. This holds in particular when using an unreliablecontrol device like a BCI. We model human performance during one trial (i.e., the generation of a particulartrajectory) as an oscillatory process:

12

P (t) = a · sin(t · ω + φ) + b with (1)φ : the phase difference starting from t=0ω : the driving frequency of movement

Our assumption is that user performance is not constant, but rather increases and decreases in a repetitivemanner. Now, to estimate the driving frequency ω, the total number of the oscillations in Equation 1 andthe total duration, measured in number of steps to complete a trajectory, are needed.

In our explorations, we would like to model different kinds of users (see Figure 10):

• users with a constant performance P (t) = c (user 0 in Figure 9)

• users performing increasingly better along the trial (users A and C in Figure 9)

• users performing increasingly worse along the trial (users B and D in Figure 9)

• users who’s performance oscillate during the trial (users E and F in Figure 9)

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0 0.2 0.4 0.6 0.8 1

perf

orm

ance

fraction of trajectory

Performance for the different user groups

u0uAuBuCuDuEuF

Figure 9: Performance of different users. Time ranges from 0 to 1, indicating the fraction of trajectory lengthtraveled. Furthermore, {ω, φ, a, b} are respectively for User A: {1/4, 0, 1,−1/2}, User B: {1/4, 1, 1, 1/2}, UserC: {1/4, 0, 1/2, 0}, User D: {−1, 0, 1/2, 0}, User E: {1, 0, 1/2, 0}, User F: {1, 1, 1/2, 0}.

As depicted in Figure 9, we will compare six extended users for different parameter settings. The performancecan be between -0.5 and 0.5. The basic user model has a constant performance of 0.

The performance influences the noise range in the basic user model. The new noise range is computed as:

nr(t) = (1− P (t)) ∗ nr(0) withnr(t) : noise range at time tP (t) : performance at time t

13

So the noise range decreases when the performance increases. A user with a performance of 0.5, will havehalf less noise range compared to a user with a performance of 0, who has a full noise range, or a user witha performance of -0.5, who will have a noise range 1.5 times as high.

In our model, user A starts with a negative performance, and gradually performs better over time. User Bstarts with a high performance and gradually loses performance. User C starts with a performance of 0, butgradually improves over time; while user D also starts with a performance of 0 but gradually performs worse.Users E and F both have a positive and negative performance; user E starts with increasing performanceand later decreases, to increase again in the end. User F starts with decreasing performance, to increasenext and decrease again in the end.

3.3.1 Estimation of number of steps for performance model

Metric To estimate the duration of a trajectory, for each of the noise ranges n ∈ {15°, 30°, 45°, 60°, 90°}degrees and for each of the stroke lengths n ∈ {10, 20, 30, 40, 50}, 1000 random trajectories to each targetwere generated with the basic user model (with momentum). Figure 10 depicts the average number of stepsrequired to reach a target.

0

20

40

60

80

100

120

140

160

180

0 10 20 30 40 50 60 70 80 90

avg

#ste

ps

noise range

number of steps per stroke length and noise range

sl10sl20sl30sl40sl50

Figure 10: Average number of steps for different stroke lengths and noise ranges. The number of steps toreach a target increases exponentially with increasing noise ranges. This effect becomes more profound forlarger stroke lengths.

From the results depicted in Figure 10, it must be concluded that it is hard to pinpoint a fixed number ofsteps for a given combination of stroke length and noise range. In our performance models, we use the meanplus two times the standard deviation of the number of steps corresponding to each combination as the fulltrajectory length.

3.3.2 Results extended user model

We will discuss the results of the extended user model for each user. Again we will look at the number oferrors and the trajectory length measured as number of steps needed to reach the target. The number ofsteps for each user in each condition is shown in Figure 11 and the trajectory length for each user in eachcondition is shown in Figure 12. As can be observed, the performance of the different artificial users differsconsiderably. For example, a userA generates a fraction

14

0

100

200

300

400

500

600

700

10 20 30 40 50 60 70 80 90

#err

ors

noise range

#errors for different strokelengths and noise ranges (usr0)

sl10sl20sl30sl40sl50

0

200

400

600

800

1000

1200

1400

1600

1800

2000

10 20 30 40 50 60 70 80 90

#err

ors

noise range

#errors for different strokelengths and noise ranges (usrA)

sl10sl20sl30sl40sl50

0

0.5

1

1.5

2

10 20 30 40 50 60 70 80 90

#err

ors

noise range

#errors for different strokelengths and noise ranges (usrB)

sl10sl20sl30sl40sl50

0

50

100

150

200

250

300

350

10 20 30 40 50 60 70 80 90

#err

ors

noise range

#errors for different strokelengths and noise ranges (usrC)

sl10sl20sl30sl40sl50

0

50

100

150

200

250

300

10 20 30 40 50 60 70 80 90

#err

ors

noise range

#errors for different strokelengths and noise ranges (usrD)

sl10sl20sl30sl40sl50

0

20

40

60

80

100

120

140

160

10 20 30 40 50 60 70 80 90

#err

ors

noise range

#errors for different strokelengths and noise ranges (usrE)

sl10sl20sl30sl40sl50

0

50

100

150

200

250

300

10 20 30 40 50 60 70 80 90

#err

ors

noise range

#errors for different strokelengths and noise ranges (usrF)

sl10sl20sl30sl40sl50

Figure 11: Number of errors for all users in the user driven condition.15

0

20

40

60

80

100

120

140

10 20 30 40 50 60 70 80 90

#ste

ps

noise range

#steps for different strokelengths and noise ranges (usr0)

sl10sl20sl30sl40sl50

10

20

30

40

50

60

70

80

90

10 20 30 40 50 60 70 80 90

#ste

ps

noise range

#steps for different strokelengths and noise ranges (usrA)

sl10sl20sl30sl40sl50

0

20

40

60

80

100

120

140

10 20 30 40 50 60 70 80 90

#ste

ps

noise range

#steps for different strokelengths and noise ranges (usrB)

sl10sl20sl30sl40sl50

0

10

20

30

40

50

60

70

10 20 30 40 50 60 70 80 90

#ste

ps

noise range

#steps for different strokelengths and noise ranges (usrC)

sl10sl20sl30sl40sl50

0

20

40

60

80

100

120

140

160

180

200

10 20 30 40 50 60 70 80 90

#ste

ps

noise range

#steps for different strokelengths and noise ranges (usrD)

sl10sl20sl30sl40sl50

0

10

20

30

40

50

60

70

80

90

100

10 20 30 40 50 60 70 80 90

#ste

ps

noise range

#steps for different strokelengths and noise ranges (usrE)

sl10sl20sl30sl40sl50

0

20

40

60

80

100

120

140

10 20 30 40 50 60 70 80 90

#ste

ps

noise range

#steps for different strokelengths and noise ranges (usrF)

sl10sl20sl30sl40sl50

Figure 12: Trajectory length for all users in the user driven condition.16

User A The number of errors is depicted on the left, and the mean trajectory length is depicted on theright.

When the noise range increases, the number of errors first increases and then decreases again for each strokelength. For stroke lengths 10, 20, 30, and 40, the number of errors increases until the noise range is 60, andthen decreases again. For a stroke length of 50, the number of errors decreases from a noise range of 45. Itis also shown that a higher stroke length gives more errors. User A has the most errors of all users (a totalof 22309) which is still only 0.093%.

The mean trajectory length increases when the noise range increases and when the stroke length decreases,the trajectory length increases.

This seems reasonable, since user A starts with the lowest performance (-0.5), to later increase. But whenthe first points are generated, there will be a lot of noise on them, so they will deviate a lot from the points onthe perfect trajectories. The performance increases later on, but due to the high deviation in the beginningof the trajectories, and the momentum it is hard to move to the correct target. This results in a lot of errors.

User B For User B there are only a few errors for a noise range of 30, with a stroke length of 40 (1 error),a noise range of 45 with a stroke length of 50 (1 error), and a noise range of 60 with a stroke length of 50 (2errors). The other configurations showed no errors. User B has the least errors of all users, 4 in total.

Again the trajectory length increases with the noise range, and increases with the stroke length, althoughthe trajectories are longer for user B than for user A (139.7 for user B with noise range 90 and stroke length10, compared to 84,9 for user A with the same configurations).

User B starts with a performance of 0.5, so the trajectories will be very close to the perfect trajectories in thebeginning. This can explain the low amount of errors; the trajectories start out right, and will be on theirway to the correct target from the beginning on. Later on, the performance decreases, and the generatedpoints will deviate more. The big trajectory length will be caused at the end of the trajectory; where a highnoise causes largely distorted coordinates, with a low chance of already being close enough to the target.Therefore new points need to be generated, until one point is close enough to the target.

User C Again the number of errors first increases and then decreases, as for user A. For all stroke lengths,the number of errors decreases for noise ranges higher than 60. User C has the second most number of errors,after user A, a total of 1855, though this is still only 0.008%.

Again the trajectory length increases with noise range and decreases with stroke length; but the trajectoriesof user C are shorter than those for any of the other users.

User C starts with no performance, and gradually increases over time. This leads to a small amount oferrors, compared to user 0, and a short trajectory length, since the performance is always positive.

User D As with the other users, the number of errors first increases and then decreases when the noiserange is bigger than 75. User D has a total number of errors of 1065, which is 0.004% of the generatedtrajectories.

For user D the trajectory length also increases with noise range and decreases with stroke length. User Dgenerates the longest trajectories of all users.

Since user D starts with no performance and gradually decreases, and therefore most of the time has anegative performance, it seems reasonable that user D has the longest trajectories. It is remarkable thatuser D has less errors than user 0, while the performance of user D is worse.

User E For user E the number of errors is constant for a stroke length of 10, namely 0. For a stroke lengthof 20 or 40 the number of errors increases over the noise ranges, and for a stroke length of 30 or 50 the

17

number of errors decreases again for a noise range greater than 75. User E has the second least number oferrors of all users; 816 in total (0.003%).

As for all other users the trajectory length increases with the noise range and decreases with the strokelength.

The performance of user E oscillates during the trial; first it increases to a maximum of 0.5, then decreasesuntil -0.5, and later increases again to 0. The positive performance in the beginning leads to a low amountof errors, because the first steps are in the right direction.

User F The number of errors again increases for noise ranges until 75 and then decreases again, exceptfor a stroke length of 30, for which the number of errors stays equal. User F has a total number of errorsof 1413 (0.006%), which is more than users B, D, and E, but less than user 0, without performance model,user A, and user C.

As for all other users the trajectory length increases with the noise range and decreases with the strokelength.

The performance for user F also oscillates during the trial, but in the opposite way of user E, so theperformance first decreases to -0.5, than increases to 0.5 and again decreases to 0. This leads to moreerrors than compared to user E, because the first part of the trajectory is generated with more noise, whichgenerates trajectories in the wrong directory, but still less errors than user 0. The trajectories are still shorterthan those generated by user 0. As with the smaller number of errors, despite of the negative performancein the beginning there is a positive performance later on in the trajectory, compared to no performance atall. Both users E and F have a smaller trajectory length then user 0, where E has shorter trajectories thanuser F.

As can be seen in Figure 11 the number of errors drops again for a higher noise range. A possible explanationis related to the addition of the noisy angle to the ideal angle. Because of the multiplication with the randomGaussian value, the noise range can become so high (closer to 360 degrees); that the new angle actually iscloser to the optimal angle. This is more likely for higher noise ranges.

Another explanation can be found in the performance model, since the performance model influences thenoise range. Combined with the previous explanation this can make the noise ranges for user A, who startswith a low performance, thus increased noise range, even closer to 360 degrees. Which can result in morepoints generated in the optimal direction and therefore leading to less errors.

4 Trajectory matching and system-guided trajectory generation

4.1 Incremental trajectory matching

The trajectory matching uses target trajectories that are known to the supervisor system. Now, giventhis representation of context information, for each newly generated user coordinate, the system will matchthe trajectory produced by the user to the set of known target trajectories. Basically, this amounts toperforming a trajectory matching (or classification) task. We will use a dynamic time warping matchingalgorithm (DTW) [Vuori, 2002, Niels et al., 2006] to perform this matching task. DTW is a method tocompute the similarity between two sequences of observations (coordinate points). For each point of a targettrajectory, the corresponding closest point of the user trajectory will be searched. The average Euclideandistance between each pair of such coordinate pairs is used as the DTW distance. Matching will return thetarget trajectory with the smallest DTW distance to the user trajectory.

To determine the length of the target trajectories to which a user trajectory is matched, the matchingalgorithm draws an imaginary circle with the starting point in the middle and through the current point,such that the Euclidean distance between the starting point and the current point is the radius of the circle.

18

The corresponding target is compared to actual target, and the result is stored, either a FA (incorrect target)or OK (correct target). This is done for every 10% of the trial trajectory length. Combining these resultsfor all the trial trajectories, 8000 in total, we can get a probability plot, as shown in Figure 13.

0.4

0.5

0.6

0.7

0.8

0.9

1

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Pro

babili

ty

Fraction of trajectory traveled

Probability of correctly matching target, userA, sl=40

nr=15nr=30nr=45nr=60nr=75nr=90

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Pro

babili

ty

Fraction of trajectory traveled

Probability of correctly matching target, userB, sl=30

nr=15nr=30nr=45nr=60nr=75nr=90

Figure 13: Example plots of the performance of an artificial user, for different stroke lengths and noiseranges.

These results will be used as statistics in the system-guided trajectory generation. We will use the probabilityof correctly matching a target for a certain trajectory length to determine the influence of the supervisor inthe system-guided condition. How this is done, will be explained in the next section.

4.2 System-guided trajectory generation

The system-guided trajectory generation still uses the same targets as the user-driven trajectory generation,but instead of knowing all the points beforehand, the points will be generated one by one. The system-guidedtrajectory generation consists of a user model, as in user-driven trajectory generation, and a system thatsupervises the generation of points. The user model ”knows” the target and generates points to reach thistarget, according to different settings. The supervisor matches the path generated by the user to the possibletarget paths and, using the statistics generated in the trajectory matching as context information, changesthe points generated by the user model. In this way we get a trajectory path that is the result of the jointcooperation between the user model and the supervisor and which is generated using context informationabout the possible paths.

The user model generates a new point, and passes this to the supervisor. The supervisor matches the alreadygenerated trajectory to the target trajectories, using dynamic time warping, and this results in a most likelytarget trajectory, with corresponding target. The direction of the last point to the new point generatedfrom the user model, du, and the direction of the last point to the most likely target, ds are then combined,weighting them with the fractions of correct recognitions from the off-line system. Again we need the totaltrajectory length estimate to estimate the current percentage of the trajectory length, to get the correctstatistics.

With usr the direction generated by the user model, sys the direction to the most likely target according tothe system, and ft the fraction of correctly predicted targets, the new direction d′ will be computed as

d′ = (1− ft) ∗ usr + ft ∗ sys (2)

In the trajectory matching, when more of the trajectory is known, more trajectories will be recognizedcorrectly, and the fraction of correctly recognized targets will be bigger. In the system-guided trajectory

19

Figure 14: Two examples of trajectories generated with the first version of the supervisor. In both cases,the target trajectory is marked green. Initially, the user takes a wrong direction (direction vectors markedmagenta), causing the supervisor (marked grey) to opt for the wrong target. Eventually, newly generatedpoints by the user are trying to get away from the recognized (wrong) target, but the supervisor preventsthis, since this target is recognized as correct target. Note that at the end of the target, the user desperatelytries to escape from the supervision of the system.

generation this will lead to more influence of the supervisor, when there are more trajectory points generated.In this way a magnetic target is implemented, as the coordinates will be pulled towards the target.

We will stop generating points when the number of steps exceeds the maximum number of steps. Themaximum number of steps allowed is determined by looking at the average number of steps the userswithout performance need to reach the target for each combination of settings. This mean is added to thecorresponding standard deviation multiplied with two.

After the first runs with the dynamic system, it showed that some targets were not reached, because thecoordinates were pulled too much to the wrong target. Because the supervisor had too much influence, theuser could not get away from this target. An example is given in figure 14. Because of this, we decided toreduce the influence the supervisor has on the new generated points.

To dampen down the strong effect of supervision (which dramatically goes wrong in certain cases), we willfurther compute the new supervised direction as

d′ = (1− f2) ∗ usr + f2 ∗ sys (3)

4.3 Results of system guided trajectory generation

A visualization of the number of errors for the system guided trajectory generation is given in Figure 15 forall users. The trajectory length can be found in Figure 16, again for all users.

User 0 As in the user driven condition, the errors occur again when the noise range is greater than 75,but this time the stroke length has to be at least 20 for errors to occur. The total number of errors is 794(0.003% of the trajectories), compared to 2903 (0.012%) in the user driven generation.

20

0

50

100

150

200

250

300

10 20 30 40 50 60 70 80 90

#err

ors

noise range

#errors for different strokelengths and noise ranges (sys0)

sl10sl20sl30sl40sl50

0

100

200

300

400

500

600

10 20 30 40 50 60 70 80 90

#err

ors

noise range

#errors for different strokelengths and noise ranges (sysA)

sl10sl20sl30sl40sl50

-1

-0.5

0

0.5

1

10 20 30 40 50 60 70 80 90

#err

ors

noise range

#errors for different strokelengths and noise ranges (sysB)

sl10sl20sl30sl40sl50

0

2

4

6

8

10

12

14

16

10 20 30 40 50 60 70 80 90

#err

ors

noise range

#errors for different strokelengths and noise ranges (sysC)

sl10sl20sl30sl40sl50

0

10

20

30

40

50

60

10 20 30 40 50 60 70 80 90

#err

ors

noise range

#errors for different strokelengths and noise ranges (sysD)

sl10sl20sl30sl40sl50

0

5

10

15

20

25

30

35

40

10 20 30 40 50 60 70 80 90

#err

ors

noise range

#errors for different strokelengths and noise ranges (sysE)

sl10sl20sl30sl40sl50

0

10

20

30

40

50

60

10 20 30 40 50 60 70 80 90

#err

ors

noise range

#errors for different strokelengths and noise ranges (sysF)

sl10sl20sl30sl40sl50

Figure 15: Number of errors for all users in the system guided condition.21

0

10

20

30

40

50

60

70

80

10 20 30 40 50 60 70 80 90

#ste

ps

noise range

#steps for different strokelengths and noise ranges (sys0)

sl10sl20sl30sl40sl50

0

10

20

30

40

50

60

70

10 20 30 40 50 60 70 80 90

#ste

ps

noise range

#steps for different strokelengths and noise ranges (sysA)

sl10sl20sl30sl40sl50

0

10

20

30

40

50

60

70

80

90

100

10 20 30 40 50 60 70 80 90

#ste

ps

noise range

#steps for different strokelengths and noise ranges (sysB)

sl10sl20sl30sl40sl50

5

10

15

20

25

30

35

40

45

50

10 20 30 40 50 60 70 80 90

#ste

ps

noise range

#steps for different strokelengths and noise ranges (sysC)

sl10sl20sl30sl40sl50

0

20

40

60

80

100

120

140

10 20 30 40 50 60 70 80 90

#ste

ps

noise range

#steps for different strokelengths and noise ranges (sysD)

sl10sl20sl30sl40sl50

0

10

20

30

40

50

60

70

10 20 30 40 50 60 70 80 90

#ste

ps

noise range

#steps for different strokelengths and noise ranges (sysE)

sl10sl20sl30sl40sl50

0

10

20

30

40

50

60

70

80

10 20 30 40 50 60 70 80 90

#ste

ps

noise range

#steps for different strokelengths and noise ranges (sysF)

sl10sl20sl30sl40sl50

Figure 16: Trajectory length for all users in the user driven condition.22

Again the trajectory length increases with the noise range and decreases with the stroke length. Thetrajectory length is smaller compared to the trajectory length for user A in the user driven generation. Fora stroke length of 10 and a noise range of 90, the system guided generated trajectories have a mean of 71.7steps, while the user driven generated trajectories have a mean of 136.3 steps for the same configuration.

User A Again there is a peak in the number of errors for a noise range of 60. After that the number of errorsdecreases. The total number of errors is 5060 (0.021%); which is a lot less than the 22309 errors (0.093%) inthe user driven generation. The large amount of errors can again be explained by the negative performancein the beginning, which steers the trajectory in the wrong direction. Combined with the momentum, it isquite hard to get to the right target within the allowed number of steps.

Compared to the trajectory length in the user driven system there is a decrease noticeable; for a noise rangeof 90 and a stroke length of 10, the mean of the trajectories in the user driven generation is 84.9 steps, andthe mean of the trajectories in the system guided generation is 62.8 steps.

User B As can be seen, there are no errors for user B in the system guided generation of trajectories; inthe user driven condition there were 4 errors.

Again the trajectories generated by the system guided generation are shorter than the user driven generatedtrajectories. A stroke length of 10 and a noise range of 90 result in a mean trajectory length of 139.7 stepsfor the user driven generation and 90.4 steps for the system guided generation of trajectories.

The high performance in the beginning of the trajectory, ensures that the trajectory is generated in theright direction, which decreases the number of errors. The large number of steps can be explained by thelow performance in the end of the trajectory; which makes the points move around the target a lot, withoutreaching it.

User C Errors only occur for a stroke length of at least 30 and a noise range of at least 30. The totalnumber of errors is 76 (0.000%), compared to 1855 (0.001%) for User C in the user driven generation.

Again the trajectories are shorter when generated with system guidance than when generated by the user.A stroke length of 10 with a noise range of 90 gives 49.3 steps for the system guidance and 69.0 steps for theuser driven generation.

The fact that the performance starts at 0 and keeps increasing until 0.5 results in a small amount of errors,and short trajectories. Combined with system guidance, this leads to even less errors, and the shortesttrajectories.

User D Again the number of errors decreases for a noise range bigger than 75 and no error occur for astroke length of 10 or 20. The total number of errors is now 210 (0.001%), compared to 1065 (0.004%) inthe user driven generation.

The trajectory length for a stroke length of 10 and a noise range of 90 is now 136.3 steps for the systemguided generation compared to 183.2 steps for user driven generation; so again the number of steps decreases.

Again user D produces the largest trajectories, which still seems reasonable because of the negative perfor-mance. It is remarkable though, that user D has less errors than user 0 (210 for user D and 794 for user 0),since the performance of D is smaller than that of user 0.

User E Errors occur for stroke lengths of 20 or greater and noise ranges of 60 or higher. The total numberof errors is now 134 (0.001%), compared to 816 (0.003%) for user E in the user driven generation.

The trajectory length has a mean of 65.9 steps for the system guided generation, and 99.7 steps for the userdriven generation with a noise range of 90 and a stroke length of 10.

23

The positive performance during the first part of the trajectory leads to a small number of errors, and sincethe first steps are in the right direction, leads to a shorter trajectory.

User F Errors occur for a stroke length of 20, with a noise range of 90, and from noise ranges combinedwith stroke lengths of respectively 60 and higher, and 30 and higher. The total number of errors is 227(0.001%), while there were 1413(0.006%) errors in the user driven generation for user F.

The trajectory length also decreased; the mean of the user driven trajectories was 123.6 steps, and the meanof the system guided trajectories was 78.4 steps.

User F has less errors than user 0, but more then user E. The same holds for the number of steps. Again thiscan be explained by the presence of a positive performance, though at the end of the trajectory, comparedto no performance at all (user 0).

5 Discussion and future research

We simulated users with a different performance and let them generate trajectories towards targets with andwithout system guidance.

For all the users, both in the user driven and system guided condition, we can see that the trajectory length innumber of steps increases with the noise range and decreases with the stroke length. This seems reasonable;when there is more noise it will take more steps to reach the target. When you have a larger stroke length,you can move more in one step, so it will take more steps to reach the target when you have a small strokelength.

5.1 User driven versus system guided generation

An overview of the number of errors for the different users in the two different systems, is shown in Table17 and an overview of the length of the trajectories for a stroke length of 10 and a noise range of 90 is givenin Table 18.

user user driven system guided0 2903 794A 22309 5060B 4 0C 1855 76D 1065 210E 816 134F 1413 227

Figure 17: Total number of errors for the differentusers for both the user driven and system guided gen-eration. The number of errors in the system guidedgeneration are clearly smaller than the number of er-rors in the user driven generation.

user user driven system guided0 136.277 71.734A 84.859 62.756B 139.732 90.410C 68.990 49.267D 183.191 136.334E 99.733 65.910F 123.613 78.439

Figure 18: Trajectory length represented as the meannumber of steps for a user and configuration needed toreach the target. Configuration has a stroke length of10, and a noise range of 90.

The total number of errors is much higher for the user driven generated trajectories than the number oferrors in the system guided trajectories. All results are strongly significant, which (given the populationsize) does not come as a surprise. Table 19 shows the percentage of errors for each user combined for allthe settings. As you can see, both the user driven and system guided generation of trajectories performsvery well. So the use of context information as in the system guided generation of trajectories, reduces thenumber of errors.

24

user user driven system guided0 0.012% 0.003%A 0.093% 0.021%B 0.000% 0.000%C 0.008% 0.000%D 0.004% 0.001%E 0.003% 0.001%F 0.006% 0.001%

Figure 19: Percentage of errors for the different users in both the user driven and system guided setting.

The total length of the trajectory, measured in number of steps, is much lower for the system guidedtrajectories than for the user driven trajectories. Again all results are strongly significant. The use ofcontext information helps to reduce the number of steps needed to reach the target. This is useful since itspeeds up communication because less input is needed to reach the target.

5.2 Influence of the performance model

Our different users have a different performance over time, which is already shown in figure 9.

User 0 has no performance at all, and thus serves as our baseline to compare the other users to.

User A starts with a negative performance of -0.5, and increases to 0.5 during the trial. This resulted in thelargest amount of errors; the trajectories started off in the wrong direction and that makes it very hard toreach the target within the allowed number of steps. In the cases where the trajectories were in the rightdirection, the higher performance at the end of the trajectory ensures that the target is reached in less steps.The number of steps needed to reach the target, is therefore smaller.

User B starts with a high performance of 0.5 which decreases to -0.5. Because of the high performance inthe beginning, the first steps of the trajectory are in the right direction, which leads to a small amount oferrors. However, later on there is a negative performance, which makes it hard to exactly reach the target.Therefore there are a lot of steps needed to finally reach the target, which results in a big trajectory length.

User C starts with no performance and increases to a performance of 0.5 at the end of the trial. The higherperformance led to less errors and smaller trajectories than user 0. In the system guided condition, user Cgenerated trajectories with the smallest amount of errors, that were also the shortest.

User D constantly has a performance of 0 or lower. This led to the largest trajectories that were generated.Surprisingly enough, the number of errors is smaller than that of user 0, while the performance for user Dis worse.

User E has an oscillating performance during the trial, where the performance first increases and later ondecreases and increases again. Both the number of errors and the trajectory length are smaller than thoseof user 0.

User F also has an oscillating performance during the trial, but opposite to user E the performance firstdecreases, to later increase and decrease again. User F has less errors and shorter trajectories than user 0,which can be explained by the presence of some performance, although later on in the trajectory generation,compared to no performance at all.

So we can see an effect of the performance model; the different users generate trajectories of different length,and produce a different number of errors, that can be explained by looking at their performance shift overtime.

25

Figure 20: Via points

5.3 Future research

This research is an exploration on the use of context information and incremental recognition for the predic-tion of targets, and it showed that context information reduces both the trajectory length and the number oferrors. We obtained very nice results, which can be made even better by looking at the following suggestions.

Via points The first thing is the use of a via-point, which can be very useful in the domain of roboticpath planning. A via point is a point on the screen through which the user goes before going to the target,figure 20 shows an example. In stead of going to the target immediately the user first goes to the first viapoint, and then to the second via point, and after that to the target. In the domain of robotics, this can bebecause the robot has to pick up food or has to recharge before moving to a target.

It is also possible to implement obstacles, in this way. Instead of moving to a specific point, the generatedpoints get pushed away from a specific point, for example a wall, or an abyss or steep hill.

Error potentials Another possibility, for the use of system-guided trajectory generation is the use ofthe error-signals produced by the brain. The error signals are produces when the user makes a mistake orwhen the BCI system does not behave as intended. This can be applied in the system-guided generation oftrajectories to detect if the system has recognized the wrong target, where the user can not get away fromas in Figure 14. [Dal Seno et al., 2010] showed encouraging results for the use of error potentials in a BCI.

Other incremental matching algorithms It is also possible to vary the incremental matching algorithm.A few examples are the use of a magnetic field, only matching the trajectory tail instead of the entiretrajectory, or looking at the overall direction the user is moving in. It is also possible to temporarily cancelout the direction the supervisor generates, when the user keeps on generating commands in the oppositedirection, possibly combined with the error potentials.

What is an error We decided to call a trajectory an error when it exceeds a maximum number of steps.This maximum number of steps is derived from trial sessions for user 0; it is the sum of the mean numberof steps needed to reach the target and two times the standard deviation. It could be interesting to look into this matter, to see whether there are other possibilities to estimate the maximum number of steps.

26

5.4 Usage of results

The results obtained by this exploration can be used in various applications. As mentioned before we coulduse (sensor) information about the environment in robotic trajectory planning. When applied in a BCI wecould reduce the amount of input that is needed to, which is useful for both the BCI and the user. Anapplication of context information in handwriting could help to reduce the amount of input needed, andperhaps recognize a letter while it is being written.

5.5 Conclusion

Although there are still some interesting things to look into, we can still conclude that the use of contextinformation in the generation of 2D control signals significantly helps to reduce both the number of errorsand the trajectory length. Context information can help to decide which way to move.

27

References

[Arvo and Novins, 2000] Arvo, J. and Novins, K. (2000). Fluid sketches: Continuous recognition and mor-phing of simple hand-drawn shapes. In Proceedings of the 13th Annual ACM Symposium on User InterfaceSoftware and Technlogy, pages 73–80.

[Dal Seno et al., 2010] Dal Seno, B., Matteucci, M., and Mainardi, L. (2010). Online detection of p300 anderror potentials in a bci speller. Comput Intell Neurosci, page 307254.

[Dix et al., 2004] Dix, A., Finlay, J., Abowd, G. D., and Beale, R. (2004). Human Computer Interaction.Pearson/Prentice-Hall, Harlow, England, 3rd ed. edition.

[Fitts and Michael, 1967] Fitts, P. M. and Michael, I. P. (1967). Human performance.

[Galan et al., 2008] Galan, F., Nuttin, M., Lew, E., Ferrez, P., Vanacker, G., Philips, J., and del R. Millan,J. (2008). A brain-actuated wheelchair: Asynchronous and non-invasive brain-computer interfaces forcontinuous control of robots. Clinical Neurophysiology, 119(9):2159 – 2169.

[Niels et al., 2006] Niels, R., Vuurpijl, L., and Schomaker, L. (2006). Automatic allograph matching inforensic writer identification. International Journal of Pattern Recognition and Artificial Intelligence,(21).

[Tandler and Prante, 2001] Tandler, P. and Prante, T. (2001). Using incremental gesture recognition toprovide immediate feedback while drawing pen gestures. In 14th Annual ACM Symposium on UserInterface Software and Technology (UIST 2001, pages 11–14.

[Vuori, 2002] Vuori, V. (2002). Adaptive Methods for On-Line Recognition of Isolated Handwritten Charac-ters. PhD thesis, Finnish Academies of Technology.

[Ward et al., 2000] Ward, D., Blackwell, A., and MacKay, D. (2000). Dasher—a data entry interface usingcontinuous gestures and language models. In Proc. ACM UIST.

[Wolpaw and McFarland, 2004] Wolpaw, J. R. and McFarland, D. J. (2004). Control of a two-dimensionalmovement signal by a noninvasive brain-computer interface in humans. Proceedings of the NationalAcademy of Sciences of the United States of America, 101(51):17849–17854.

[Zhao, 1993] Zhao, R. (1993). Incremental recognition in gesture-based and syntax-directed diagram editors.In Proceedings of the ACM Conference on Human Factors in Computing Systems (InterCHI’93, pages 95–100. Addison Wesley.

28