End-User Robot Programming Using Mixed Reality

Samir Yitzhak Gadre1, Eric Rosen1, Gary Chien1, Elizabeth Phillips1,2, Stefanie Tellex1, George Konidaris1

Abstract— Mixed Reality (MR) is a promising interface forrobot programming because it can project an immersive 3Dvisualization of a robot’s intended movement onto the realworld. MR can also support hand gestures, which provide anintuitive way for users to construct and modify robot motions.We present a Mixed Reality Head-Mounted Display (MR-HMD) interface that enables end-users to easily create andedit robot motions using waypoints. We describe a user studywhere 20 participants were asked to program a robot armusing 2D and MR interfaces to perform two pick-and-placetasks. In the primitive task, participants created typical pick-and-place programs. In the adapted task, participants adaptedtheir primitive programs to address a more complex pick-and-place scenario, which included obstacles and conditionalreasoning. Compared to the 2D interface, a higher number ofusers were able to complete both tasks in significantly less time,and reported experiencing lower cognitive workload, higherusability, and higher naturalness with the MR-HMD interface.

I. INTRODUCTION

For robots to become widely used, humans must be ableto program their actions. For example, consider the task ofbinning items. A roboticist might accomplish this task byspecifying a series of waypoints in computer code for therobot to visit one by one. If the action must be modified, theroboticist would explicitly modify the waypoints specifiedin the code. This method is widely popular, but will notwork for end-users. The abstraction of breaking down actionsinto a series of waypoints could be communicated, butrequiring the use of programming languages to specify thosewaypoints is beyond their scope. Therefore, we need analternate method of interfacing with the waypoint actionsystem.

Visual programming is one such methodology. A broadterm, visual programming refers to a class of interfaceswhere one interacts with a visual representation of ob-jects, variables, classes, etc. [1]. Visual programming hasbeen a popular tool for programming computers by non-programmers in visual art [2], audio production [3], andeducation [4] because it allows users to focus on algorithmicthinking, rather than the syntax needed to express their intent.

In the world of robotics, RViz and the Interactive Markerspackage [5] allow for the creation of visual programming in-terfaces that control and visualize a robot. Robot researchershave also investigated how effective visual programmingis by creating and evaluating their own frameworks [6].Using our binning task from before, it would be possibleto create and modify waypoints visually in a keyboard andmouse interface. However, users would not see the waypointsoverlayed on the real robot environment.

1Computer Science, Brown University2Behavioral Sciences and Leadership, United States Air Force Academy

(a) A screenshot from the MRperspective of a user program-ming a robot motion. Users canspecify green waypoints.

(b) After creating the waypoints,users can visualize the robot armmotion that is planned throughthe waypoints.

Fig. 1: Our mixed reality interface.

We propose an end-user mixed reality-based visual pro-gramming framework for creating and modifying waypointsto create complex, multistep robot actions (Fig. 1). Userscan specify and group waypoints together to create primitivemotions, and adapt these waypoints to perform similar tasks.Our interface allows users to visualize the entire motion therobot plans to perform through the waypoints and have therobot execute it. We use a commercially available MixedReality Head-Mounted Display (MR-HMD), the MicrosoftHoloLens [7].

The MR-HMD at first seems to pose only advantages overother visual programming interfaces by combining the robotworkspace and the GUI space for the end-user. However,there are limitations to the technology that do not makeit obviously preferable to a 2D interface. For example, theHoloLens has a limited field of view, so it relies on theuser to move around to get a full view of the MR scene.Furthermore, imperfect hand tracking via computer visionmakes selection and dragging gestures less reliable thanmouse clicks, especially for novice HoloLens users.

We therefore conducted a user study with 20 participantsand compared the effectiveness of our MR interface toa 2D visual programming interface for two similar pick-and-place tasks. In the first task, participants programmedprimitive robot motions to pick up a block and place it ona platform. In the second task, participants adapted theirprimitive robot programs to sequentially pick-and-place twocubes on different platforms. Our results show that comparedto the 2D interface, a higher number of users were able tocomplete both tasks in significantly less time. Users reportedexperiencing lower cognitive workload, higher usability andnaturalness with the MR-HMD interface.

II. RELATED WORK

The traditional way to program a robot is to write code.ROS [8] is an extremely powerful middleware environmentfor roboticists. ROS includes packages to allow programmersto use languages like C++ and Python to interface with robothardware. However, leveraging the expertise of end-usersthat lack software engineering skills would help make robotsmore widely accessible.

ROS also includes many graphical user interfaces (GUIs),such as RViz [5], for visualization. RViz can display robotsensor data on a 2D screen and can be connected toMoveIt! [9] motion planners to enable users to programrobot movement via keyboard and mouse. 2D interfaces havebeen shown to be useful for robot programming, but havetheir own shortcomings with regards to immersiveness andintuitiveness. They force users to interpret 3D information ona 2D platform and use control interfaces that do not matchhow users interact with the world.

Alexandrova et al. [6] created a 2D visual programminglanguage, RoboFlow, to enable end-users to easily programmobile manipulators to perform general tasks in open-environments. Action editing was important to resolve errors[6].

Alexandrova et al. [10] developed a framework that en-ables users to provide a demonstration of a task and thenuse an intuitive monitor and mouse interface to edit demon-strations for adaption to new tasks. Elliott et al. [11] extendedthis work to allow for the grouping of several poses relativeto landmarks or objects in the scene. Having an intuitiveinterface for non-experts to edit example motions for robotsis especially important in Learning from Demonstration(LfD), where collecting many examples needed for learningis not always feasible. In such cases, a more preferablemethod may be to have the human perform one generaldemonstration of the task, and then adapt parts of theirprogrammed action to new environments. Both Alexandrovaet al. [10] and Elliott et al. [11] used a 2D monitor interfaceto adapt robot programs.

Language is a well-studied modality for programmingrobots because it is one of the main ways humans commu-nicate with each other. Language has been shown to be anefficient way to parameterize a task through descriptions ofkey items and locations in the environment [12]. Forbes etal. [13] developed a natural language interface to allow usersto program robot motions via LfD. However, using languageto program robots has its limitations. For example, Forbes etal. [13] studied basic object manipulation tasks that requiredrobot action classes that could easily be identified throughnatural language. However, tasks that rely on actions that areharder to describe can make language a noisy and difficultinterface to robots.

Recent techniques experiment with virtual reality (VR)teleoperation interfaces. Whitney et al. [14] compared a VRinterface against a 2D monitor interface for teleoperating arobot to perform a cup-stacking task. Whitney et al. [14]found that users had a significantly easier time using the

VR headset, because they could navigate and interact withthe scene by moving their head and hands naturally. Thiscontrasts with keyboard and mouse actions that a typical2D interface provides. Like the VR headset, the MR-HMDallows users to navigate and interact with the perceivedenvironment using natural actions. However, an additionalbenefit of the MR-HMD is that it allows the user to also seethe real world.

Rosen et al. [15] created an open-source ROS package,ROS Reality, which enables robots to convey intent. Thepackage allows the robot to display its intended path as aholographic time-lapse trail to a user wearing a MR-HMD.Rosen et al. [15] conducted a user study to compare thespeed and task completion rate of novice participants usinga HoloLens and a 2D monitor interface to determine if aproposed robot trajectory would collide with the environ-ment. They found that the MR-HMD increased accuracyand lowered task completion time. While Rosen et al. [15]showed the promise of using a MR-HMD for visualizingrobot motion to non-experts, they did not address howeffective MR-HMD is for programming these motions. TheHoloLens’s limited hand-gesture tracking capabilities posethe possible issue that MR-HMD may not be an effectiveinterface for creating these robot actions.

Walker et al. [16] investigated different MR signallingmechanisms for a drone to communicate with a human.They found quantitative task efficiency was significantlyhigher when using MR signals than when using physically-embodied signals.

Fang et al. [17] created an MR interface to program andvisualize robot trajectories. However, they do so using a 2Dinterface. Ni et al. [18] evaluated the effectiveness of anaugmented reality interface against traditional robot weldingprogramming methodologies. Virtual representations of therobot trajectory were visualized over a video feed of the realrobot, enabling novice users to use the augmented realityinterface to program new welding paths for the robot to actout. Ni et al. [18] found that the augmented reality interfaceallowed users to program robot actions more quickly andintuitively, especially for users without prior computer-aideddesign knowledge. However, the 2D interfaces of both Fanget al. [17] and Ni et al. [18] force users to look at a screenand not strictly at the robot workspace, which poses safetyissues when collaborating with a robot in close-quarters.On the other hand, MR-HMDs allow users to both providevisualizations over the workspace and safely interact withthe 3D GUI components using hand gestures.

III. APPROACH

We designed an MR-based interface that allows users tospecify waypoints in 3D trajectories. We wanted to test ifthe notion of adaptation [10], [11] was made more powerfulwhen end-users could edit robot motion in the real world.We therefore created a baseline 2D GUI and a MR-HMD3D GUI. Our contribution is evaluating the effectiveness ofusing a MR-HMD interface for programming and editingrobot motion.

The following subsections address important features ofour approach. We discuss waypoints, groups, visualization,and execution. At the lowest level, users program the robotby creating waypoints through which the robot must move.These waypoints can be collected into groups, which can bevisualized in MR and executed in real life.

The waypoint (Fig. 1a) is the fundamental unit of pro-gramming in a robot motion. A user adds a waypoint tothe tail of a sequence of end-effector poses (3D positionsand quaternions) that the robot arm must move through. Awaypoint is represented as a hologram of the end-effector,which can be open or closed. For example, if an arbitrarywaypoint has state closed, then a command is sent to thegripper to close after it has converged on the waypoint pose.waypoints can be adjusted and deleted by the user.

A group (Fig. 1a) comprises a sequence of waypointsthat together form an action. Hence, every waypoint isassociated with a group. Groups are split into primitive andadapted groups. Primitive groups are taken to be fundamentalbuilding-blocks such as a generic pick-and-place action.After the primitive motions are created, they can be altered—by moving, adding, and deleting individual waypoints—tosolve related or more complex tasks. These altered primitivesare adapted groups.

Given a group, or a sequence of groups, it is possible tovisualize the path that the robot arm will take from waypointto waypoint (Fig. 1b). Visualization allows the robot toconvey how it will move through the specified waypointsin the form of a holographic time-lapse trail. Visualizationis key in the editing process as it allows the user to furtherchange an adapted group if they are not satisfied with thetrajectory.

Once a user is satisfied with the adapted groups, they canexecute it to make the robot arm move in the real world asper the visualization. We use MoveIt! to plan from waypointto waypoint.

IV. SYSTEM

Implementation was split into front-end Unity code for theHoloLens MR-HMD and back-end ROS code for MoveIt!and the Baxter. The 2D monitor baseline used the same back-end as the 3D interface. The front-ends were identical, exceptthat the 2D monitor interface displayed graphics on a screeninstead of holograms. Additionally, the 2D monitor interfacehad a rendering of a point cloud of the workspace (Fig. 3a).

Unity code supported both GUIs. This code made requeststo the back-end as waypoints were altered, updated the robotarm movement trails upon receiving motion plans from theback-end, and sent execution requests to the back-end. Weused ROS Reality [19] to visualize the robot model and 3Dsensor data for the monitor interface.

The hologram of the Baxter was created by parsing andrendering a Unified Robot Description Format (URDF). Therendering of Baxter was overlaid on the real robot in acalibration step.

There were two slotted columns attached to the hologramof the robot in the form of a manager menu (Fig. 3b). The

(a) The primitive task. The user had to program the robot to1) pick up the cube, then bring it over the wall and 2) placeit on the platform.

(b) The adapted task. The user had to program the robot to 1)pick up the cube from a new location, 2) place it on the sameplatform, 3) pick up a second cube, and 4) place on a secondplatform, all while avoiding walls.

Fig. 2: The primitive and adapted tasks used in our experi-ments.

right column contained primitive groups and the left columncontained adapted groups. A user started by creating aprimitive. They could add waypoints or start a new primitivegroup by clicking the “Add Gripper” and “New Group”buttons, respectively. After creating primitives, the user coulddrag groups to the left column where they became adaptedgroups.

The user could add and delete waypoints to alter theprimitive so that it was well suited for an adapted task.The left column also acted as a run queue that executed thegroup actions from top to bottom. By clicking the “Visualize”button, a user was shown a movement trail that the orderof the adapted groups implied. By clicking the “Execute”button, the user made the robot execute the desired motion.

The front-end used a websocket client, ROS# [20] to sendmessages to the back-end. The client connected to ROSbridge [21], which linked the front-end to the Baxter ROSnetwork. The client code marshaled data and sent it to thebridge. The bridge published these messages to the desiredROS topics. If the location, state, or existence of a waypointwas changed, a message was sent to update the back-end’sknowledge of the waypoint. The front-end also sent twoother types of messages, one to visualize a motion and oneto execute a trajectory. Both of these requests hit MoveIt!code. We used MoveIt! to create a Cartesian plan throughthe waypoints of the various groups. These plans were sentto the front-end via the Bridge Server. If the end-user wantedto visualize the path, then the motion plan was displayed asa movement trail on the front-end. If the end-user wished toexecute the motion plan, MoveIt! communicated with Baxterto actuate its joints.

(a) Motion through waypointsvisualized in our 2D interface.The manager menu is not visi-ble.

(b) A picture of the man-ager menu included in boththe 2D monitor and MR inter-face. Users can add waypoints,create primitive and adaptedgroups, and visualize and ex-ecute motions.

Fig. 3: (a) Our 2D visual programming interface and (b)manager menu for 2D and MR.

V. EXPERIMENT

The aim of our evaluation was to test the hypothesis thatthe MR interface would be faster and easier to use than the2D baseline interface. We conducted a user study askingnon-experts to program a 7 degree of freedom (7-dof) robotarm to perform a primitive and adapted pick-and-place taskusing our MR and 2D interfaces. Twenty users—15 maleand 5 female—participated in the study. We measured usertask completion times, system usability, subjective cognitiveworkload, and perceived naturalness for both interfaces.

A. Task

Participants completed a primitive and adapted task (Fig.2). In the primitive task (Fig. 2a), users were instructed toprogram the robot to pick up a cube, bring it over a wall,and place it on a platform. In the adapted task (Fig. 2b), theparticipant was asked to program the robot arm to pick upthe first block from a new location, move it over the wall,and place it on the same platform. As part of the adaptedtask, the participant also had to pick up a second block, takeit over a wall twice the height of the first, and place it on asecond platform.

B. Interfaces

We compared two interfaces:• Monitor Interface: Participants used a monitor, keyboard,

and mouse to program and edit robot motions (Fig. 3).Using this interface, a user navigated the virtual sceneand oriented the camera view in 3D spacing using acombination of right-mouse clicks, drags, and keyboardbutton presses. To interact with the waypoint programminginterface, users were able to use the left-mouse button.This 2D monitor interface framework is similar to that ofAlexandrova et al. [10] and Elliott et al. [11].

• Mixed Reality Interface: Participants used a HoloLensto program and edit robot motions. Users could translateand rotate their perspective by simply moving their headas they normally would. To interact with the programminginterfaces, users perform tapping and dragging hand ges-tures on the GUI components shown in Fig. 1.

C. Study Design

The study used a within-subjects design where all par-ticipants performed both experimental programming tasksusing both interfaces. Participants were randomly assignedto the order in which they used each interface to completethe programming tasks. Furthermore, experimenters coun-terbalanced the order of the two interfaces across studyparticipants. For each interface, participants performed firstthe primitive task, then the adapted task.

D. Experimental Procedure

Participants began the study by reviewing the informedconsent information. After they agreed to participate, theexperimenters explained the two tasks that the users were toprogram. In addition, the experimenters explained the variouscomponents of the programming interfaces as described inSection III. After the users understood the task and howthe programming interfaces worked, they were randomlyselected to first complete tasks on either the 2D monitoror MR interface. For each interface, users were taught thecontrols before completing the primitive task. Then, theymoved on to the adapted task. In the adapted task, userscould access the successful motions they had programmedin the primitive task and edit them to achieve the new goal.Before each task, experimenters gave a 3-2-1 countdown, atwhich point users were timed for programming the necessarymotions up to the point of having the robot execute thefinal motion. For both the primitive and adapted task, userswere given up to five attempts, where an attempt would beconsidered a failure if a) the robot failed to sequentially placethe cubes on the correct platforms or b) if the robot arm raninto a wall while executing a motion. The time of their firstsuccess was recorded. After completing both the primitiveand adapted tasks, users filled out all usability, workload,and naturalness surveys for that interface, and then movedon to the other interface condition.

E. Dependent Measures

Our objective dependent variables were the task successrate and the task completion times for both the primitive andadapted tasks.

If users were unable to create a successful robot motionto complete the primitive task in up to 5 tries in a giveninterface, then that user did not attempt the adapted taskin that interface. Thus, we report task success rate in theprimitive task for each interface as the number of userswho were able to have at least one successfully programmedrobot motion divided by the total number of participants.Furthermore, we report task completion rate in the adaptedtask for each interface as the number of users who were

able to successfully program the robot motion divided bythe number of people who had successfully programmed arobot motion in the primitive task. The subjective dependentmeasures included the NASA Task Load Index (NASA-TLX)[22], the System Usability Scale (SUS) [23], and a surveyintended to measure user perceptions of the naturalness andusefulness of each interface.

NASA-TLX: the NASA-TLX is a subjective measure ofperceived cognitive workload. Participants provide ratingsof their perceived workload while completing a task onsix sub-scales: mental demand, physical demand, temporaldemand, effort, frustration, and performance. Five of thesub-scales are rated from 0 (very low demand) to 100 (veryhigh demand) and the performance sub-scale is rated from 0(perfect performance) to 100 (failure). For this experiment,the weighted measure of paired comparisons among thesub-scales was not included. See Moroney et al. [24] fora discussion. The subjective cognitive workload score wascalculated by taking the average of the six sub-scale scores.

SUS: the System Usability Scale assesses overall systemusability by asking participants to rate ten statements ona 7-point Likert-type scale from “Strongly Disagree” to“Strongly Agree”. The statements cover aspects like systemcomplexity, consistency, and cumbersomeness among others.The SUS scores are converted to a scale of 0 (low usability)to 100 (high usability). See Sauro [25] for more detail onSUS scoring.

Perceived Naturalness Survey: the Perceived Natural-ness Survey was derived to measure how natural usingthe interface felt to each participant. The naturalness of aninterface has been discussed as a necessary component ofbuilding good user interfaces [26], [27] and can lead topositive outcomes like better learning in computer mediatedenvironments [28], but little work has been done on howto directly measure how “natural” an interface feels to auser. Prior work has suggested that components of interfacenaturalness include natural mapping—the ability of a systemto map its control to changes in the mediated environmentin a natural and predictable manner [26], as well as control,maneuverability, direct connections, and salience of inputand feedback, to name a few. This survey was a first attemptin measuring those concepts of naturalness. The surveyincluded items like, “It was easy to understand how changesin the interface would result in changes in the real-world,”“Using the interface, I felt like I had full control over therobot,” “The interface felt predictable,” and “The interfacefelt natural,” among others. Participants responded to thesestatements using a 7-point Likert-type scale that ranged from“Strongly Agree” to “Strongly Disagree.” Scores on thismeasure were calculated by taking users average responseacross the 11 items.

F. Hypotheses

Overall, we expected users to perform better (lower taskcompletion times, higher task completion rate, lower work-load, higher usability and preference) on the MR interface

than the 2D monitor interface in both the primitive andadapted tasks.

H1: Users will quantitatively perform better on the MRinterface than the 2D monitor interface for completing theprimitive task. Better performance is categorized as a) lowertask completion time and b) higher task completion rate.

H2: Users will quantitatively perform better on the MRinterface than the 2D monitor interface for completing theadapted task. Better performance is categorized as a) lowertask completion time and b) higher task completion rate.

H3: Users will report higher subjective impressions of theMR interface than the 2D monitor interface for completingthe primitive task. Better subjective performance is catego-rized as a) lower reported workload, b) higher usability, andc) higher perceived naturalness of the interface.

G. Results

1) Primitive and Adapted Task Completion: Two pairedsample t-tests were conducted to evaluate differences in taskcompletion times between the 2D monitor and MR interfacesfor both the primitive and adapted tasks (Fig. 4a). There wasa statistically significant difference in mean task completiontimes (in seconds) between the 2D interface (M=189.44,SD=105.33) and the MR interface (M=132.50, SD=92.40)on the primitive task, t(17)=2.77, p=0.013, Cohen’s d=0.57.There was also a significant difference in mean task com-pletion times between the 2D (M=263.98, SD =123.88) andMR (M=145.28, SD=108.43) interface for the adapted task,t(17)=5.656, p <0.001, Cohen’s d=1.02. Additionally, allparticipants (100%) were able to complete the primitiveand adapted tasks using the MR interface. On the otherhand, fewer (N=18) participants were able to complete theprimitive task using the 2D interface. Since these participantswere not able to complete the primitive task, they wereunable to adapt that plan in the adapted task. Thus, theydid not complete either. Using the MR interface, participantscompleted both the primitive and adapted tasks faster than inthe 2D interface and with higher task completion rate. Thus,Hypotheses 1 and 2 were supported.

2) Workload: Statistically significant differences werealso found between subjective ratings of mental workloadbetween the 2D and MR interfaces (Fig. 4b), t(19)=4.07,p =0.001, Cohen’s d=0.93. Participants reported significantlylower cognitive workload using the MR interface (M=31.54,SD=15.69) than the 2D interface (M=46.13, SD=15.59).

3) Usability: Participants also reported significantlyhigher usability scores for the MR interface (M=76.75,SD=15.88) as compared to the 2D interface (M=47.75,SD=21.73), t(19)=7.73, p <0.001, Cohen’s d=1.52 (Fig. 4c).

4) Naturalness of the interface: Finally, participants alsoreported that interacting with the MR interface (M=5.8,SD=0.67) felt significantly more natural than using the 2Dinterface (M=3.82, SD=1.37), t(19)=8.72, p <0.001, Cohen’sd=1.83 (Fig. 4d). Taken together, the results of the subjectivemeasures of workload, usability, and perceived naturalnesssupport Hypothesis 3.

Primitive Task Adapted Task0

50

100

150

200

250

300 Mean Task Completion Time (s)

2D

MR

(a) Task completion times foreach interface/task.

Interface0

10

20

30

40

50 NASA-TLX Mean Scores

2D

MR

(b) NASA-TLX scores formonitor and MR

Interface0

10

20

30

40

50

60

70

80

90 SUS Mean Scores

2D

MR

(c) A graph of the results from theSUS Survey.

Interface0

1

2

3

4

5

6 Naturalness Mean Scores

2D

MR

(d) A graph of the results fromour Naturalness Survey.

Fig. 4: A figure of selected quantitative and qualitative results. Error bars show standard error.

5) Order effects: Additionally analyses were conductedto test whether the order in which participants interactedwith each interface biased the results. There were a fewresults affected by order. First, if participants interacted withthe 2D interface after interacting with the MR interface,they rated the 2D interface significantly lower on usabilitythan participants who interacted with 2D interface first. Thisresult may suggest that participants were disappointed orfrustrated by the 2D interface after having first interactedwith the MR system, mean difference=23.31. t(18)=3.17,p=0.008, Cohen’s d=1.46. A look at the frustration subscaleof the NASA-TLX revealed that participants did indicatesignificantly higher frustration scores for the 2D interface ascompared to the MR interface, t(19)=4.13, p=0.001, Cohen’sd=1.27, mean difference=31.5. Similar results were found forthe perceived naturalness measure. If participants interactedwith the 2D interface after interacting with the MR interfacethey rated the 2D interface significantly lower on how naturalthe interface felt than participants who interacted with theinterfaces in the reverse order, t(18)=4.29, p=0.002, Cohen’sd=2.00, mean difference=1.98. The inverse was also truefor perceive naturalness of the MR interface. If participantsinteracted with the MR interface second, then they ratedthe MR interface significantly higher on the measure ofnaturalness of the interface than participants who interactedwith the MR interface first, t(18)=2.45, p=0.033, Cohen’sd=1.14, mean difference=0.69.

VI. DISCUSSION

Overall our results strongly support the use of MR inter-faces for novice users to program and adapt robot motions.In our user study, participants were significantly faster andbetter able to complete the programming tasks using theMR interface than the 2D keyboard and mouse interface.In addition, participants reported lower levels of cognitiveworkload (e.g., frustration, effort, mental, temporal, andphysical demand) in the MR interface. Users reported that theMR interface felt more usable and natural when completingthe programming tasks. This seemed to stem from the tediousnature of using mouse and keyboard to reposition the cameraview and manipulate waypoints. The 2D monitor also didnot seem ideal for interpreting the 3D workspace making itharder to understand the true pose of the specified waypoints.

Using MR, users could simply move their heads to readjusttheir point of view.

Our findings on the order of completion may also re-veal the subjective appeal of using the MR interface. Ifparticipants used the 2D interface after interacting with theMR interface, they rated the 2D interface significantly lowerin usability and naturalness. It was also the case that ifparticipants used the MR interface after having been exposedto the 2D, they rated the MR interface significantly higher.

This interface could be adapted for future research andindustry applications. We see particular applications to LfD,where waypoints are specified to constitute a demonstration.This approach could be used to program personal and indus-trial robots. In a household setting, a user might programa sweeping primitive for a robot outfitted with a broomattachment. In a factory line setting, a worker might programa soldering robot to place molten metal at a precise locationon a chip.

VII. CONCLUSION

We introduced a new MR-based interface for end-users toprogram and adapt trajectories. We evaluated the effective-ness of using our MR system for programming and editingrobot motions in a primitive and adapted pick-and-place taskagainst a 2D monitor and keyboard equivalent. Our studyshowed that users were able to more quickly and accuratelyprogram and edit robot motions using MR than the 2Dbaseline. In addition, users found that our MR interfacerequired less work, was easier to use, and was more naturalto use in comparison to the baseline. This study shows thelarge promise MR has for making robot programming moreaccessible to all end-users.

REFERENCES

[1] B. A. Myers, “Visual programming, programming by example, andprogram visualization: A taxonomy,” in SIGCHI Conference on Hu-man Factors in Computing Systems (CHI), 1986, pp. 59–66.

[2] “Blender,” Blender Foundation, http://www.blender.org, [Accessed:2018].

[3] “Max software tools for media,” Cycling ’74,https://cycling74.com/products/max, [Accessed: 2018].

[4] M. Resnick, J. Maloney, A. Monroy-Hernandez, N. Rusk, E. East-mond, K. Brennan, A. Millner, E. Rosenbaum, J. Silver, B. Silverman,and Y. Kafai, “Scratch: Programming for all,” in Communications ofthe ACM, vol. 52, no. 11, 2009, pp. 60–67.

[5] D. Gossow, A. Leeper, D. Hershberger, and M. Ciocarlie, “Interactivemarkers: 3-d user interfaces for ros applications [ros topics],” in IEEERobotics & Automation Magazine, vol. 18, no. 4, 2011, pp. 14–15.

[6] S. Alexandrova, Z. Tatlock, and M. Cakmak, “Roboflow: A flow-basedvisual programming language for mobile manipulation tasks,” in IEEEInternational Conference on Robotics and Automation (ICRA), 2015,pp. 5537–5544.

[7] “Hololens,” Microsoft, https://www.microsoft.com/en-us/hololens,[Accessed 2018].

[8] M. Quigley, B. Gerkey, K. Conley, J. Faust, T. Foote, J. Leibs,E. Berger, R. Wheeler, and A. Ng, “Ros: an open-source robot oper-ating system.” in Open-source software workshop of the InternationalConference on Robotics and Automation (ICRA), 2009.

[9] “MoveIt!” Willow Garage, 2007, https://moveit.ros.org, [Accessed:2018].

[10] S. Alexandrova, M. Cakmak, K. Hsiao, and L. Takayama, “Robotprogramming by demonstration with interactive action visualizations.”in Robotics: Science and Systems (RSS), 2014.

[11] S. Elliott, R. Toris, and M. Cakmak, “Efficient programming of manip-ulation tasks by demonstration and adaptation,” in IEEE InternationalSymposium on Robot and Human Interactive Communication (RO-MAN), 2017.

[12] T. Kollar, S. Tellex, M. Walter, A. Huang, A. Bachrach, S. Hemachan-dra, E. Brunskill, A. Banerjee, D. Roy, S. Teller, and N. Roy, “Gener-alized grounding graphs: A probabilistic framework for understandinggrounded language,” in Journal of Artificial Intelligence Research(JAIR), 2013.

[13] M. Forbes, R. Rao, L. Zettlemoyer, and M. Cakmak, “Robot program-ming by demonstration with situated spatial language understanding,”in IEEE International Conference on Robotics and Automation (ICRA),2015, pp. 2014–2020.

[14] D. Whitney, E. Rosen, E. Phillips, G. Konidaris, and S. Tellex,“Comparing robot grasping teleoperation across desktop and virtualreality with ros reality,” in International Symposium on RoboticsResearch (ISRR), 2017.

[15] E. Rosen, D. Whitney, E. Phillips, G. Chien, J. Tompkin, G. Konidaris,and S. Tellex, “Communicating robot arm motion intent throughmixed reality head-mounted displays,” in International Symposium OnRobotics Research (ISRR), 2017.

[16] M. Walker, H. Hedayati, J. Lee, and D. Szafir, “Communicating robotmotion intent with augmented reality,” in ACM/IEEE InternationalConference on Human-Robot Interaction (HRI), 2018, pp. 316–324.

[17] H. Fang, S. Ong, and A. Nee, “Interactive robot trajectory planningand simulation using augmented reality,” in Robotics and Computer-Integrated Manufacturing, vol. 28, no. 2, 2012, pp. 227–237.

[18] D. Ni, A. Yew, S. Ong, and A. Nee, “Haptic and visual augmentedreality interface for programming welding robots,” in Advances inManufacturing, vol. 5, no. 3, 2017, pp. 191–198.

[19] D. Whitney, E. Rosen, D. Ullman, E. Phillips, and S. Tellex, “Rosreality: A virtual reality framework using consumer-grade hardwarefor ros-enabled robots,” in IEEE/RSJ International Conference onIntelligent Robots and Systems (IROS), 2018.

[20] “ROS#,” Siemens, https://github.com/siemens/ros-sharp, [Accessed:2018].

[21] C. Crick, G. Jay, S. Osentoski, and O. Jenkins, “Ros and rosbridge:Roboticists out of the loop,” in ACM/IEEE International Conferenceon Human-Robot Interaction (HRI), 2012, pp. 493–494.

[22] S. Hart and L. Staveland, “Development of nasa-tlx (task load in-dex): Results of empirical and theoretical research,” in Advances inPsychology, vol. 52, 1988, pp. 139–183.

[23] J. Brooke, “Sus - a quick and dirty usability scale,” in UsabilityEvaluation in Industry, 1996, pp. 189–194.

[24] W. Moroney, D. Biers, F. Eggemeier, and J. Mitchell, “A comparisonof two scoring procedures with the nasa task load index in a simulatedflight task,” in IEEE National Aerospace and Electronics Conference(NAECON), 1992, pp. 734–740.

[25] J. Sauro, “Measuring usability with the system usability scale (sus),”https://measuringu.com/sus/ [Accessed: 2018].

[26] P. Skalski, R. Lange, and R. Tamborini, “Mapping the way to fun:The effect of video game interfaces on presence and enjoyment,” inInternational Workshop on Presence, 2006, pp. 63–64.

[27] S. Ramm, J. Giacomin, D. Robertson, and A. Malizia, “A firstapproach to understanding and measuring naturalness in driver-car in-teraction,” in International Conference on Automotive User Interfacesand Interactive Vehicular Applications, 2014, pp. 1–10.

[28] S. Bailey, “Getting the upper hand: Natural gesture interfaces improveinstructional efficiency on a conceptual computer lesson,” Ph.D. dis-sertation, University of Central Florida, 2017.