performance evaluation of underwater platforms in the context of space exploration

ARTICLE IN PRESS

Planetary and Space Science 58 (2010) 706–716

Contents lists available at ScienceDirect

Planetary and Space Science

0032-06

doi:10.1

� Corr

E-m

journal homepage: www.elsevier.com/locate/pss

Performance evaluation of underwater platforms in the contextof space exploration

A.L. Forrest a,�, B.E. Laval a, D.S.S. Lim b, D.R. Williams c, A.C. Trembanis d, M.M. Marinova e,R. Shepard f, A.L. Brady g, G.F. Slater g, M.L. Gernhardt h, C.P. McKay b

a University of British Columbia, Department of Civil Engineering, 6250 Applied Science Lane, Room 2010, Vancouver, BC, Canada V6T 1Z4b NASA Ames Research Centre, Space Science & Astrobiology Division, Building 245, Mail Stop 245-3 Room 283, Moffett Field, CA 94035-1000, USAc McMaster Center for Medical Robotics, McMaster University, St. Joseph’s Healthcare, Hamilton, 50 Charlton Avenue East, Hamilton, ON, Canada L8N 4A6d Department of Geological Sciences, College of Marine and Earth Studies, University of Delaware, 109 Penny Hall, Newark, DE 19716, USAe Division of Geological and Planetary Sciences, California Institute of Technology, MC 150-21, Pasadena, CA 91125, USAf Department of Geology, University of California, One Shields Avenue Davis, CA 95616, USAg School of Geography and Earth Sciences, McMaster University, 1280 Main Street West, Hamilton, ON, Canada L8S 4K1h NASA Lyndon B. Johnson Space Center, 1601 NASA Parkway, Houston, TX 77058, USA

a r t i c l e i n f o

Article history:

Received 18 February 2009

Received in revised form

28 July 2009

Accepted 17 August 2009Available online 31 August 2009

Keywords:

Autonomous underwater vehicle

Human–robotic interface

Performance metrics

Exploration metrics

Remotely operated vehicle

DeepWorker

33/$ - see front matter Published by Elsevier

016/j.pss.2009.08.007

esponding author. Tel.: +1604 822 2915; fax:

ail address: [email protected] (A.L. Forrest).

a b s t r a c t

Robotic platforms are essential for future human planetary and lunar exploration as they can operate in

more extreme environments with a greater endurance than human explorers. In this era of space

exploration, a terrestrial analog that can be used for development of the coordination between manned

and robotic vehicles will optimize the scientific return of future missions while concurrently

minimizing the downtime of both human explorers and robotic platforms. This work presents the

use of underwater exploratory robots – autonomous underwater vehicles (AUV), remotely operated

vehicles (ROV), and manned submersibles – as analogues for mixed human–robot exploration of space.

Subaqueous settings present diverse challenges for navigation, operation and recovery that require the

development of an exploration model of a similar complexity as required for space exploration. To

capitalize on the strengths of both robotic and human explorers this work presents lessons learnt with

respect to the fields of human–robotic interface (HRI) and operator training. These are then used in the

development of mission evaluation tools: (1) a task efficiency index (TEI), (2) performance metrics, and

(3) exploration metrics. Although these independent evaluations were useful for specific missions,

further refinement will be required to fully evaluate the strengths and capabilities of multiple platforms

in a human–robotic exploration campaign in order to take advantage of unforeseen science

opportunities in remote settings.

Published by Elsevier Ltd.

1. Introduction

An integral part of establishing a human presence across thesolar system is the development coordinated human–robotexploration:

A partnership between humans and robots is essential to thesuccess of such ventures. Robotic spacecraft are our scouts andproxies, venturing first into hostile environments to gathercritical intelligence that makes human exploration feasible.—The Global Exploration Strategy: The Framework for Coordination

(2008)

Ltd.

+1604 822 6901.

Human and robotic space exploration programs have tradi-tionally been independent; however, the complexity of futurehuman missions will require an unprecedented use of automationand robotics (Mishkin et al., 2006). Technology specialist divisionsof space agencies have traditionally conducted robotic develop-ment; a paradigm shift must occur to couple engineering andscience lessons in mission design to ensure mission success. Thisneeds to be done to maximize the scientific return of a mission bycapitalizing on the strengths and abilities of each survey platform(autonomous, remotely operated, and manned). Robotic explora-tion is driven by balancing the collection of scientific data inremote, and often extreme, environments while concurrentlydeveloping robotic innovation in terms of endurance, versatilityand autonomy. As robots will never be completely self-sufficient,situations will continue to arise where robots fail and humansneed to intervene, particularly cases where autonomy failsor unexpected contingencies develop. Essential for future

www.elsevier.com/locate/pss

dx.doi.org/10.1016/j.pss.2009.08.007

mailto:[email protected]

ARTICLE IN PRESS

Table 1List of exploration metrics used to determine quantify overall success of a mission.

Metric Descriptor Definition

1 Limited Data provides limited scientific

value

2 Adequate Data reaffirms existing

hypotheses and facts

3 Significant Data elucidates existing

hypotheses in new areas or detail

4 Exceptional Data resolves a major scientific

question or highly significant

hypothesis

5 Discovery Data introduces a novel idea or

hypothesis

A.L. Forrest et al. / Planetary and Space Science 58 (2010) 706–716 707

human–robotic space exploration is: (1) the evaluation of eachplatform at a given task and (2) optimizing coordinated platformperformance tasked with a common scientific objective.

Several field robotic studies have been conducted in order tosimulate surface applications in arid conditions (Bualat et al.,2007; Fong et al., 2008a) although it has also been proposed thatunderwater settings could serve as ideal analogue work environ-ments for mixed human–robot exploration in space (Bellinghamand Rajan, 2007; Forrest et al., 2008b). Unlike surface applications,subaqueous settings present similar challenges as those found inspace exploration: the lethal environment requires life support forhuman exploration; limited visible light transmission preventingdirect human sensory input; and limited radio frequency (RF) andmost electromagnetic (EM) band transmission restricts mostavailable modes of underwater communication. There are also anumber of differences that result in unique challenges in bothexploration domains: (1) mass constraints encountered in spaceare less important underwater; (2) tethering of vehicles (e.g., ROVin subsea applications) are not common in space; (3) platformdesign has to adjust for high-pressure, corrosive environmentsunderwater as compared to the vacuum of space; (4) underwatercurrents generate high perturbations as compared to spaceenvironments. The key to improving the fidelity of underwatervehicles as analogues for space exploration is examining thedifferences and similarities between the two different modes ofexploration.

Evaluation of mission success of multiple platforms, the focusof this work, is impossible without understanding the human–robotic interface (HRI), human operator training and how both ofthese relate to real-time and post-processing of data. Futureunderwater and space exploration will require the development ofa common HRI for varied exploration platform operations inunknown environments. HRI development occurs naturally whennew problems, both in lab and field settings, are encountered andsurmounted in order to maximize the scientific return of amission. New autonomous platforms are working towardsdynamic sequencing of high-level science goals onboard whichleads to: missions being able to take advantage of unforeseenscience opportunities and increasing the human data collectingproductivity rather than dealing with low-level activity planning.Lessons learned in analogue environments will contribute tothe overall success of planetary missions, in particular ongoingsemi-autonomous rover exploration of Mars and future humanlunar exploration.

Benefits of using underwater environments are also gained byselecting sites that are relevant science analogue environments ashas been demonstrated with the Deep Phreatic Thermal Explorer(DEPTHX) AUV project (Krajick, 2007), the TROV project under theRoss sea ice in McMurdo Sound, Antarctica (Hine et al., 1994), orthe TROV project in Mono Lake, CA (Stoker et al., 1996). Lessonslearned exploring the surface waters of Earth can then be appliedto further exploration of our solar system and potentially thesurface waters of other lunar bodies (e.g., Europa or Enceladus).The importance of fieldwork activities in analogue environmentsis that they push forward the fundamental scientific under-standing of planetary bodies throughout the solar system, butalso provide unique opportunities to study how real-worldexploration programs, including the decision-making tree ofresource allocation, are conceived and carried out with multipleplatforms.

This research uses UBC-Gavia (an autonomous underwatervehicle, AUV), Outland Scuttle (a remotely operated vehicle, ROV)and DeepWorker (a manned submersible) as analogues forautonomous, remotely operated and manned exploration plat-forms for extra-terrestrial settings (e.g., Moon, Mars, etc.),respectively. The primary campaign objective was the evaluation

of integrated human–robotic operations in the context of spaceexploration through the use of (1) a task efficiency index (TEI), (2)performance metrics (measuring engineering success), and (3)exploration metrics (quantifying the scientific mission success).Specific lessons learned, in the domains of HRI and operatortraining, are presented in the context of how they contributed tomission evaluation.

2. Site description and methodology

The purpose of each evaluation method was to quantify howdifferent platforms behave in each of the field site contexts. Everymethod would ideally be applied at every field deployment;however, as this work represents the composite of severalworking groups through three different campaigns spanning2006–2008, evaluation techniques differed. The developed task

efficiency index (TEI) subdivided a given objective (e.g., travel towaypoint) into a series of timed tasks allowing an across platformcomparison to be conducted. Other performance metrics (e.g.,operator involvement, operator risk, energy consumed, etc.) couldalso have been selected as a basis of comparison. Performance

metrics, sometimes termed functional primitives (Howard andRodriguez, 2003), were mission specific and provided a good basisof comparison between various missions using the same platform(e.g., number of waypoints achieved, trackline covered, etc.). Foreach selected metric, a normalized ranking from 0 to 10 wasdeveloped in order that the different terms could be statisticallycombined. Beyond the engineering success of a given mission (theperformance metrics), the exploration metrics were an importanttool generic enough to rank the overall scientific return of amission. These metrics were designed to be broad enough in scopeto be useful and applicable to any exploration mission, yet focusedenough to make a quality analysis of the scientific merit possible.A generic scale of 1–5 was selected as demonstrated in Table 1.This scale was to be measured at three different points; (1)mission planning, (2) post-flight debriefing, and (3) post dataanalysis. The post-flight debriefing would allow an individual pilotto record their impressions while the post data analysis (2 yearslater) would allow the data to be processed by both theexploration team and the scientific community.

The primary deployment site for both the AUV and mannedsubmersible field campaigns was Pavilion Lake, a node of theCanadian Space Agency’s (CSA) Canadian Analogue ResearchNetwork (CARN) (Osinski et al., 2006). UBC-Gavia deploymentswere conducted during summer and winter field campaigns of2007 and 2008 as part of ongoing analogue studies on organo-sedimentary structures, known as microbialites, found at this site(Laval et al., 2000) while DeepWorker missions were conductedonly during the 2008 summer field campaign. Pavilion Lake is a

ARTICLE IN PRESS

Fig. 1. Pavilion Lake bathymetric map with BC location inset. Contours represent

20 m intervals with maximum depth of 61 m in the Central Basin where the bulk of

the DeepWorker and AUV missions were run.

Fig. 2. Bathymetric map of waters surrounding Bonaire, N.A. Stars indicate

locations where AUVs were deployed in 2008 and illustrate the spatial coverage

possible with small, man-deployable vehicles. Contours represent 200 m intervals

starting at 300 m nearest land.

A.L. Forrest et al. / Planetary and Space Science 58 (2010) 706–716708

5 km2 temperate lake located in central British Columbia, Canadain a limestone valley known as Marble Canyon (see Fig. 1). Asshown in Fig. 1, there are three main basins (North, Central, andSouth) orientated along the longitudinal axis of the lakeinterconnected by sills approximately 6–10 m deep. While thewinter AUV missions were conducted exclusively in the CentralBasin at the widest and deepest point of the lake (maximumrecorded depth of 61 m), the summertime AUV and DeepWorkercampaigns were conducted in all three basins. The centralscientific objective of the AUV deployments is the examinationof how mass transport, associated with thermally drivenconvective processes, influences the distribution of microbialitesin the lake (Forrest et al., 2008a). There were two central scientificgoals for the DeepWorker missions: (1) map the microbialitedistribution by collecting high-definition video imagery alongseveral contours of each basin and (2) collect samples atapproximately 55–60 m depth (impractical for SCUBA retrieval)in the Central basin. The majority of performance and explorationmetrics were developed during the various field missions. BothAUV and manned submersible campaigns also complement theongoing limnological (Lim et al., 2009) and isotopic investigations(Brady et al., 2009) of Pavilion and other nearby lakes. The abilityto combine scientific analogue and exploration objectives withmultiple platforms at Pavilion Lake is an unprecedentedopportunity in exploration science.

In addition to the Pavilion Lake deployments, in January 2008,a National Oceanic and Atmospheric Administration (NOAA)Ocean Explorer (OE) AUV campaign was conducted on the islandof Bonaire. Located in the southern Caribbean near Curacao andAruba, the island of Bonaire is part of the Netherlands Antilles andis part of a World Heritage site where one of the most pristinecoral reef environments in the Caribbean is preserved (see Fig. 2;Steneck and McClanahan, 2004). The overall scientific objectives

of deploying autonomous robotic platforms in the coastal watersof Bonaire included: (1) producing detailed maps of the coralsystems from the backreef (depthso5 m) down to the twilightzone and beyond 100–200 m deep; (2) characterizing thechemical and physical oceanography of the pristine ecosystem;and (3) examining patterns of biodiversity of the reef system. Tomeet these goals, three AUVs were deployed in coordinated andstand-alone missions at 12 different locations around the island(see Fig. 2). Evaluation of these missions focused on the efficiencyof multi-vehicle coordination in a real-world application.

In addition to the AUV and manned submersible campaigns,Outland Scuttle ROV missions were conducted during the NASAExtreme Environment Mission Operations (NEEMO) 9, 18-daycampaign in 2006 based at the National Oceanic and AtmosphericAdministration (NOAA) ‘Aquarius’ underwater ocean laboratorylocated near Key Largo, Florida. Although the main focus of thiswork was the evaluation of medical diagnostic and therapeutictechnologies in extreme environments, analogue activities wereconducted with an ROV to examine the efficiency of robotic versushuman tasking. Located in approximately 20 m of water, this sitewas ideal for simulating the use of remotely operated vehicles(ROV) as substitutes for human extra-vehicular activity (EVA)from the habitat. Scuttle was either controlled directly by the crewwithin the habitat or in the water or through a direct Internetconnection to the Exploration Planning and Operations Center(ExPOC) remotely located at the Johnson Space Center in Houston.Both modes of control used the commercial computer interfacerather than virtual reality telepresence used in earlier ROVmissions (Hine et al., 1994). Scuttle operations were comparedwith crewmember retrieval of simulated lunar specimens throughthe development and use of a mission-specific TEI (Williams et al.,2006).

3. Description of exploration platforms

Three different underwater exploration platforms were used asanalogues for space exploration; autonomous, remotely operated,and manned. As each platform carries different scientific payloads,

ARTICLE IN PRESS

Fig. 3. (Color) Photos of each of the three AUVs used for the coordinated missions in Bonaire, N.A.: (a) UBC-Gavia in inverted flight beneath the ice; (b) Gavia class AUV

during initial deployment in the Bonaire; and (c) Fetch class vehicle being readied deployment.

Fig. 4. (Color) Photo of diver passing off equipment to the Outland Scuttle ROV

during the NEEMO 9 specimen retrieval missions in 2006 (note that the wheels

allow for dual mobility modes).


the analysis of vehicle performance, relative to the specificscientific mission goals, is necessary to determine the relativestrengths and capabilities. The autonomous robotic platform usedwas UBC-Gavia, a Gavia class AUV manufactured by Hafmynd, ehfand operated by the Environmental Fluid Mechanics group at theUniversity of British Columbia, Canada. Two other AUVs wereused in the Bonaire deployments; another Gavia class vehicle(operated by Hafmynd, ehf) and a Fetch! class AUV (manufacturedby Sias Patterson, Inc. and operated by the Autonomous SystemsLaboratory at the Virginia Institute of Marine Science) (see Fig. 3).

Due to the integrated modularity of the Gavia class AUV,different sections can be interchanged allowing an elevateddegree of versatility to match changing exploration and scientificneeds (e.g., a swath bathymetry sonar unit replaced a currentprofiling unit during one set of the under-ice trials; Forrest et al.,2007). Autonomous robotic platforms, underwater or otherwise,are limited by their energy storage capability, and, in contrast tospace robotics, underwater exploration is currently unable toefficiently use solar power to mitigate power needs. Optimizationof underwater survey techniques to address specific missionobjectives (i.e., survey pattern, platform speed, etc.), same as otherexploration modes, reduces power cycling and improves thechances of increased scientific return (e.g., Sherwood et al., 2001).Due to communication limitations or complete blackouts under-water, AUVs have to be autonomous for extended periods of timein a fashion analogous to time-lags associated with extra-terrestrial robotic exploration (e.g., Mars Rovers). Surface naviga-tion of UBC-Gavia is conducted with the use of a global positioningsystem (GPS, SBAS/WAAS enabled); dead reckoning when sub-merged; or acoustic long baseline (LBL) positioning whensubmerged and multiple LBL transponders are present. With thislatter system the AUV triangulates itself via communications withtwo acoustic transponders fixed at pre-determined locations inthe water column and a two-way acoustic modem that is locatedonboard the AUV. Although the LBL system is a much moreaccurate navigational system in that error does not increase withtime submerged, there is a trade-off in functional range andoperational logistics with the added requirement of the morecomplex acoustic transponder network requiring a more specia-lized operator skill-set.

The remotely operated vehicle (ROV) used for the NEEMO 9missions was Scuttle, a Model 1000 ROV produced by OutlandTechnology Imaging Systems. This type of vehicle is controlled

and powered through an umbilical cord connected to surfacesupport or, in this case, the Aquarius habitat. In addition tosupplying power to the vehicle, this cord also allows the live feedfrom the camera back to the operator who controls the vehicle inreal-time. Analogous to early EVA in Gemini IV where theastronaut managed the umbilical, human explorers were requiredto manage the vehicle tether to test the possibility of using anumbilical to connect crew to life support on a rover andunderstand the unique set of challenges for ground-basedexploration (e.g., umbilical snags on rocks). Unlike most ROVs,Scuttle was equipped with wheels in order to be commanded ineither driving or flying modes (see Fig. 4). Scuttle was alsoequipped with an arm used for object retrieval, which wasessential for the performed specimen retrieval analogue activitiesand for the possibility of using the ROV to hand-off tools from theROV to crew outside the habitat. Typical depth and compassoverlays were included in the streamed video feed which wereessential for vehicle navigation in situations where field of viewcan be completely obscured. In underwater environments,

ARTICLE IN PRESS

Fig. 5. (Color) Photo of DeepWorker vehicle being deployed with UBC-Gavia; twin

lights on the port side manipulator camera provide scale for the video imagery.


disturbed bottom sediment can obscure the camera comparableto dust storms in terrestrial based applications.

The manned submersible platform used was a DeepWorker,designed and operated by Nuytco Research, Inc. With no tether, asingle pilot operates this vehicle for up to 6 hours at a timeindependent of surface support. The vehicle is designed for thepilot to be sitting upright in order to maximize the field of visionthrough a hemispherical glass dome. This design minimizesoperator reliance on the built-in sensors and HRI. The vehiclealso has a hydraulic manipulator, with a 2.27 m reach, equippedduring these trials with lights and a continuous high-definitionvideo recording system (see Fig. 5). The manipulator enabled thepilot to adaptively modify the mission to sample if a target ofopportunity (TOP) was observed, as specified on pre-determinedcriteria (defined in subsequent sections). Similar to the ROV,3-axis control and an auto hover/heading system is providedthrough two main and two angled thrusters. Due to the vehiclesize, these thrusters are considerably larger than those of the ROVand, as a result, will create significantly more turbulence. Vehiclecontrol, relative to the ROV, is more complex due to the variablebuoyancy of the platform.

Surface communications with DeepWorker are provided viaVHF, but communications at depth are restricted to a low-frequency (27 kHz), analog, acoustic-communication system as aresult of the transmission physics underwater. Surface supportused an acoustic positioning system (Ultra Short Baseline—USBL)to regularly update the position of the vehicle that is thenreported verbally to the vehicle pilots using the analog acousticcommunications system in order for flight corrections to be made.This is a similar approach as to how astronauts use ground-basedtechnicians and scientists to evaluate interesting findings andaffect repairs beyond their expertise during mission deployment(White and Averner, 2001).

4. Results and discussion

As each of the vehicles had a different interface, efforts weremade to quantify the success of a given mission in order to makean evaluation, in the context of exploration, of the strengths ofeach platform. Efforts were made to develop the evaluation toolsin the planning stages although much of the development wasconducted in the field, particularly as each campaign involveda different flight crew who individually developed their ownperformance metrics. In this work, a series of functionalprimitives are defined, the Technology Impact Score developed

by Howard and Rodriguez (2003) was not used in order that theevaluations remain general. It is also noted that, althoughcommon metrics can generally be subdivided into 5 broad taskcategories; navigation, perception, management, manipulation,and social (Steinfeld et al., 2006), the majority of developedmetrics fell mainly into the first two categories. Future work isdeveloping a common evaluation scheme based on the threemodels presented here; a task efficiency index, performancemetrics, and exploration metrics.

4.1. Autonomous underwater vehicle campaigns

One of the strengths of an AUV platform, relative to mannedexploration, is the lower deployment cost. As such, AUV missionswere generally rerun until the planned objectives were reached,skewing the results of a performance evaluation as compared tothe other platforms. AUV mission success was measured in termsof the percentage of planned waypoints reached withoutconducting repeat missions. In the context of exploration, AUVmissions demonstrate specific lessons that should be applied tospace exploration: the need for modularity and flexibility ofautonomous platforms; logistical planning for coordinated mis-sions between multiple platforms; and the use of autonomousplatforms to survey areas tedious and/or dangerous for humanexploration.

4.1.1. Modularity and flexibility

Ice cover requires the vehicle returning directly to thedeployment hole for recovery, particularly if a mission abortsprematurely. Many different scenarios may cause an autonomousplatform to abort a mission or a given mission task including theinability to reach a waypoint in a given time period, anobstruction to the planned trajectory, or the inability to completea given task. For example, the Mars Exploration Rover (MER)Opportunity will abort taking drill samples if too much resistanceis encountered at the drill tip (Weisbin et al., 1999). In relatively‘safe’ conditions (e.g., in open atmosphere) this is a relativelyminor risk as a new command sequence can be uploaded;however, in the case of non-hovering AUVs operating under ice,this presents a significant challenge when the normal failuremode (simply stopping to allow the positive buoyancy of thevehicle to raise the AUV to be retrieved at the surface) isinappropriate, although possible (Doble et al., 2009), and couldresult in AUV loss beneath the ice surface. The adopted strategy,although not as sophisticated as other terrestrial models(Tompkins et al., 2006), was to introduce a hierarchy of normaland fallback operations in the mission planning in case an errorwas encountered. During the 10-days of testing, an average of 7missions were run daily to measure water column properties.Using overall mission success rate as a Performance Metric, theunder-ice missions ranked 85% with the successful initiation ofthe fallback mission for the remaining 15% of the missions(�1 mission per day). Although successfully initiated, the AUVwas unable to return to the point of recovery due to accumulatedposition error (�10–50 m of target) as well as vehicle inertia oncethe thrusters had stopped.

Selecting the overall mission success as the performancemetric demonstrates how one metric can skew the results. FutureAUV missions should consider a more developed list (e.g., metricsselected for DeepWorker mission analyses). Trends in missionplanning of all autonomous platforms are using concepts ofdynamic in-situ retasking methodologies and adaptive samplingstrategies. The successful use of a fallback mission, demonstratedin this work through the modification of the software andfirmware (e.g., the HRI) of the vehicle, is the first stage of

ARTICLE IN PRESS

Fig. 6. (Color) Cooper’s Barge, �30 m long vessel located in approximately 40 m of water off the coast of Bonaire, N.A., as imaged with: (a) Geoswath SONAR data and (b)

Sidescan SONAR data from the Gavia AUV (white areas represent non-overlapping datasets).


accomplishing autonomous adaptation except, instead of thevehicle going into a fallback mission resulting from an abortcommand, the mission would be triggered by a measured scalar orvector. This strategy has already been tested with AUVs searchingfor freshwater plumes, harmful algal blooms or hydrothermalvents in the Earth’s oceans (Bellingham and Rajan, 2007). A futureobjective of artificial intelligence (AI) techniques for planetaryexploration is for the robots to be able to deal with similarlyunstructured environments as the world’s oceans while at thesame time being able to create a hierarchical system of theirmission goals (Weisbin et al., 1999). With the implementation ofincreasingly complex models, the careful selection of missionevaluation tools will be critical in order to prevent skewingmission success in terms of engineering constraints.

4.1.2. Coordinated missions

Conducting coordinated missions between three AUVs (seeFig. 3) with a common scientific goal of exploring an unknownenvironment was the focus of a study conducted in Bonaire, N.A.in January 2008. Mission success was similarly evaluatedusing performance metrics. As with under-ice deployments(Section 4.1.1), this has applicability between different roboticplatforms (e.g., terrestrial and aerial vehicles) (Fong et al., 2008b)but also multiple vehicles of the same style (e.g., all water-basedvehicles) (Paley et al., 2008). This study demonstrates that thecoordination of multiple vehicles was possible even if thesevehicles were developed by different manufacturers or carrieddifferent scientific payloads. Although missions where all threevehicles were run concurrently were repeated to achieve a givenmission objective (e.g., survey the bay) the initial success rate wasgenerally quite low (o10%). One of the largest restrictions to thejoint AUV missions was the fact that the HRI differed for eachvehicle resulting in a significant degree of post-processing of thedata making it difficult to quickly visualize the missions. Deployedindividually, mission success was much higher (490%) althoughit is then the datasets that are coordinated rather than themissions themselves. One of the major challenges encounteredwas coordinating the surface support with multiple deployedvehicles. For applications where astronauts would need to controlmultiple robotic platforms (e.g., lunar, ground-based exploration)detailed field operation protocols would need to be developed inorder to minimize mission failure.

Initial missions used the three vehicles to explore non-evolvingareas (on the time scale of this study) of the reef systems around

Bonaire to identify targets of scientific interest. These includedboth natural features (e.g., coral reefs and submarine terraces) andanthropogenic debris (i.e., shipwrecks) (see Fig. 6) that either thevehicles could return to for follow-on exploration purposes or forSCUBA divers to conduct detailed transect surveys. One of the keylessons learned was that rendering the coordinated datasets into acommon geo-referenced HRI is necessary for the successfuldeployment of multiple platforms. This limitation points to theneed for unified, scalable and exchangeable mission visualizationsystems for planning, playback and real-time operationalawareness. Such systems have been developed in both military(AUV Workbench) (Weekley et al., 2004), commercial(FledermausTM) (Grasmueck et al., 2006), academic (VEVI)(Piguet et al., 1995) and open source (GEOZui3D) (Arsenault etal., 2004) arenas yet adoption by AUV manufacturers andoperators, as likewise demonstrated by space explorationpractitioners, has been relatively slow and piecemeal.

Coordinated missions were then planned within the varioussurvey areas with either combinations of the two of the vehiclesor, all three vehicles, concurrently. One-day trials were conductedat three different sites on the island; Nukove (the northernmostsite); the Bonaire Marine Reserve (in the center); and Pink Beach(the southernmost site) (see Fig. 2). An example of the style ofmission planned for each of these sites would be for one AUV to becollecting sidescan or multibeam swath bathymetry sonar (e.g.,Geoswath) data (�15 m altitude) while the other vehicles wouldbe measuring the physical properties of the water column orcollecting still images of the benthic region (see Fig. 7). Althoughthe success rate was relatively low, as mentioned above, theposition and altitude of both vehicles was known, and geo-referenced images could then be used to ground-truth targetspreviously identified in the sonar imagery. These techniques werethen used to improve the success of follow-on missions with eachof the individual vehicles on subsequent days.

4.1.3. Autonomous robotic surveys

Using robots to conduct surveys in extreme terrestrialenvironments is directly analogous to the way humans will useautonomous robots for space and planetary exploration. Subse-quent to an initial robotic survey, a choice would be made todeploy manned mobile platforms or to continue with autonomousexploration to investigate the sites of interest. This style ofexploration planning was used during the summer 2008 PavilionLake field campaign. Steep walls (�15–201 slope) characterize

ARTICLE IN PRESS

Fig. 7. Coordinated missions during deployment at Bonaire Marine Reserve on January 18, 2008 of three different vehicles: (1) UBC-Gavia (dashed)—collecting physical

scalar data on the surface waters; (2) Gavia class AUV (solid)—conducting a sidescan and multibeam swath bathymetry survey; and (3) Fetch (not shown)—conducting

vertical profiles of dissolved gases in the water. Bathymetry shown generated from the combined dataset of the three vehicles where the contours represent 20 m intervals.


most of Pavilion Lake’s bathymetry with a flat bottom coveringmost of the lake at approximately 55 m average depth. During themission-planning stage for the DeepWorker missions, the flightcrew had selected two mission objectives; (1) to collect videofootage at specific contours around each basin in the lake and (2)to collect microbialite samples from targets too deep to samplewith SCUBA. It was decided that the exploration of the flat-bottomregion (the benthic plain) was not efficient to conduct using themanned vehicles, as the region was too large and had a lowprobability of significant scientific return. This stage of theplanning was directed by a boat-based multibeam survey ofthe entire site analogous to the way in which orbital data directsthe decision-making tree in mission planning for planetarymissions. The team decided to conduct a sidescan sonar and stillimagery survey of the benthic plain with UBC-Gavia. Thedeprioritization of the manned survey of this region resultedfrom the fact that manned exploration of microbialite-rich regionswas more beneficial towards understanding the driving scientificquestions. This is a result of the fact that the human element isbetter equipped for hypothesis generation of the environmentalfactors affecting the microbialite distribution.

These bottom-following AUV missions had an overall missionsuccess rate of approximate 65%, reduced from success rate of thewater column, under-ice missions (Section 4.1.1). This was aresult of the mission being aborted more often as a result ofinability of the vehicle to track the variable topography. Asmissions were easily rerun, the entire bottom of the benthic plainin the central basin was surveyed over a 2-day period. Althoughcomparable (time-wise) with a DeepWorker vehicle survey, thesemissions are considerably less expensive. Although no significantscientific discoveries were made, in terms of the explorationmetrics used below (see Section 4.3.3), charting unknownterritory (e.g., for military, oil and gas exploration, and spaceexploration) provides necessary information for planning future

missions and exploration science and mission evaluation is anessential part of this.

4.2. Remotely operated vehicle campaign

One of the strengths of using Outland ROV Scuttle during theNEEMO 9 missions was that it could be operated both locally bythe astronauts in the Aquarius habitat or remotely by ExPOC inHouston directly through a cabled connection to the habitat.During the campaign, the crewmembers developed a taskefficiency index (using time as the performance metric) toevaluate the performance of the vehicle at performing varioustasks. Lessons learned are also useful in the evaluation ofintegrated human–robotic operations related to lunar explorationobjectives. This is a result of the fact that operations require notjust the use of a technical operator (either local or remote) butalso require additional crewmembers for umbilical managementduring deployment.

4.2.1. Specialized skill sets

Human–robotic exploration requires that crewmembers have abasic familiarity with both hardware and software for successfuloperation and troubleshooting (Fong et al., 2008b). While theemphasis of much of this work is on mission evaluation, one of thelessons learned from this phase of the work was the importance ofoperator training. Missions will require technical training in eachof the tools to be used and to quickly process the data in as nearreal-time as possible (e.g., NEEMO 9 crewmembers were trainedin the NASA mission management tools). Without adequatetraining in these tools (e.g., Onboard Short Term Plan Viewer—

OSTPV), crewmembers would be at a disadvantage to trouble-shoot problems and potentially unable to configure workingsolutions in pressure situations (Williams et al., 2006).

ARTICLE IN PRESS


Amongst the IT skill-set required by human operators isknowledge of operating systems and network communications.This proved to be invaluable in the NEEMO 9 missions as well asall the AUV operations where network failures between compu-ters, both onboard the vehicle and operator used, were common-place. Operators of both platforms would need to quickly evaluatespecific missions in order to design successful subsequentmissions. This points to the need to have appropriate expertisespecific to each component technology both on-site as well asproviding off-site access to further resources. For example, duringremote operation of Scuttle by ExPOC, technical ROV supportwas required, network administrators needed to providemultiple points of firewall access and end users altered and ransoftware that led to successful operation of the platform. Localoperational problems with the vehicle included serial portfailures, software conflicts and network configuration issues(Williams et al., 2006).

Many of these issues could have been avoided had clear,technical operating protocols been established before vehicledeployment. In order for underwater robotic platforms to beuseful analogues for space exploration (i.e., all three platformspresented in this work) ‘flight rules’ need to be determined for thehuman operators. In the case of Scuttle, some examples wouldinclude minimum power restrictions, operator certificationrequirements and network troubleshooting techniques. The issueof operator certification was particular relevant for ExPOC controlof the ROV as appropriate safety protocols needed to be followedfor the crewmember in the water who managed the umbilical.These flight rules need to be evaluated after each campaign fortheir applicability to future exploration platforms. Optimizingthese rules minimizes the consumption of resources, minimizesdowntime, and maximizes the efficiency of the work crew.

4.2.2. Specimen retrieval missions

Simulated lunar samples were placed in the immediatelyenvirons of Aquarius and the efficiency of searching for, andretrieving, these samples during an EVA or using the ROV wastested. In order to quantify the success of these missions, a taskefficiency index (TEI) was developed specific to these missions(Williams et al., 2006). Initial development of the TEI was to breakdown the sequence of activities into a series of timed tasks

Fig. 8. Contour-following mission tracks of Deepworker 2000 m

(e.g., time from preparation of hardware to water entry) specific tothe assigned task. This is similar to the ‘performance parameters’selected by Howard and Rodriguez (2003) and is a useful conceptfor future cross-platform analysis for task allocation. Using thistime-based TEI, it was generally found that the EVA retrievalproved more efficient than the ROV operations, either locally orremotely, as a result of the ROV operator having insufficientsituational awareness within the operating range of the vehicle.The TEI was also used to evaluate ROV versus human search andtagging of a simulated spacecraft delivering cargo to a lunarhabitat. Although lunar missions will generally be able to trackthe ship, it is generally thought that robots can be used if thevehicle is beyond the operating range of the human operators. TheExPOC provided approximate range and bearing to the habitatcrewmembers with directions on how to conduct the searchdirectly or via remote robotic control underscoring the value ofcontrolling lunar rovers from a habitat.

4.3. Manned vehicle campaign

DeepWorker submersibles were deployed in Pavilion Lake inthe summer of 2008 with the goals (as stated in Section 2) ofmapping microbialite distribution through video imagery at setcontours and collecting microbialite samples from the deepbenthic region that is not feasible for SCUBA sampling (seeFig. 8 for example mission paths). As these goals were mutuallyexclusive, it was necessary to have mission-specific evaluationtools (i.e., metrics). For each mission type, the DeepWorker pilotsgenerated a set of functional primitives to evaluate thecompletion of mission objectives similar to those developed byRodriguez and Weisbin (2003). In addition, a set of explorationmetrics was created to evaluate the exploration success of a givenmission.

4.3.1. Contour following missions

Functional primitives selected for the contour followingmissions reflected not just the ability to follow a given depthcontour but also how they fit into the scientific objectives of theproject. This campaign only examined the DeepWorker missionsbut it is hoped that future missions will look at using mixedplatforms concurrently (e.g., AUVs and manned vehicles). As

issions in the: (a) North; (b) Central; and (c) South Basins.

ARTICLE IN PRESS


mentioned in the methodology (Section 2), primitives were eachgiven a ranking on a scale of 10. The five developed performancemetrics developed for contour following included:

(1)
Morphology identification and description. This primitive dealswith how well the pilots, previously briefed on morphologicaltypes in the lake, were able to describe and identify theobserved microbialite morphology during time of flight.
(2)
Description of significant transitions. Abrupt transitions inmicrobialite distribution are potential indicators of changedenvironmental conditions and provide clues to examineheterogeneous growth conditions across the lake.
(3)
Quantify percent cover of the sediment surface. In the same waytransitions are revealing of growth conditions, percent coverof microbialites on the lake floor are likewise revealing.
(4)
Ability to maintain flight plan. The ability to execute theprescribed flight plan was an evaluation of the performance ofthe piloting abilities as well as the mission-planning abilitiesof the flight crew.
(5)
Percent completion of prescribed contour. Evaluated through theanalysis of how many of the prescribed waypoints in the flightplan were reached during the course of the mission.
The average score of all the primitives was determined eventhough a single primitive could significantly lower the results even ifthe overall success of the mission was high (see Table 3 for overallresults). For example, it was found that the ability to maintain flight

plan metric often resulted in a low overall mission rank, as the flightplans would often not be completed. Interestingly, this metric wouldbe low as a result of either engineering faults or something of highscientific interest was discovered. The average combined ranking ofthe five metrics was 7.6 (71.6) with the lowest combined scoresgenerally resulting from one of three reasons: communication issueswith surface support, minimal microbialite presence (or morphologyvariations), and high terrain complexity. When pilots were unable toestablish surface communications, they could not navigate theprescribed route due to lack of positioning information in thevehicle. Those flights that traversed regions with minimal micro-bialite cover (e.g., areas of extensive sediment or vegetation), wouldreceive a low ranking in the first two metrics, reducing the overallflight’s score even if the contour was followed closely (PerformanceMetric 5). Conversely, terrain with high complexity would alsoreceive a low ranking, as the pilot would be unable to achieve all ofthe waypoints in the allotted time (Performance Metric 4). Thelesson learned for exploration science was that the selection of theprimitives is key in the evaluation of the mission performance andthat combining the primitive scores is fundamental. If the success ofa mission were determined by only using one of these parameters(e.g., length of contour traveled) it would not be an informativeevaluation of the mission performance. It was also concluded that, ofthe five selected primitives, the percent completion of prescribed

contour was the most easily applicable to autonomous roboticplatforms. Future performance metrics will need to be developed forcross-platform analyses that are quantifiable for all platforms.

4.3.2. Sample retrieval missions

Of the five functional primitives selected for the sampleretrieval missions, only one, the ability to maintain flight plan,was the same as the contour following missions. The rest weremission specific for sample retrieval and, similar to the previoussection, are each ranked out of 10:

(1)
Sample identification. This primitive was an evaluation ofwhether the sample resembled what the pilots thought theyhad picked up in terms of size, shape, etc.
(2)
Sampling efficiency. Quantifying how efficiently a sample couldbe retrieved, revealed differences between pilots but alsosample and underlying substrate quality.
(3)
Target localization. Targets were specified in the flight plan(i.e., microbialite at a given depth or of a given morphology)but selection of the specific target was left to the pilot.
(4)
Sample site disturbance. For all flights, minimizing sitedisturbance was a high priority and, as such, this primitivewas evaluated on a reverse scale with 10 representing theleast disturbance and 0 being the most in order for the metricsto be combined in a meaningful way.
The average combined ranking was 6.7 (71.7); a lower valuethan the ranking received for the contour following missions(although not statistically different). While pilots were highlysuccessful at retrieving samples, it was generally discovered thatthe surrounding sediment to the sampling site suspended for briefperiods of time obscuring the view. As such, the sample site

disturbance ranking would be quite low relative to the otherselected primitives. It was also noted that target localization wouldoften be left void, as the specified targets would not be observed(i.e., not reaching the flight plan criteria). In each of these cases,the pilots would select new representative targets. In calculatingthe overall ranking, this primitive was not included rather thanreceiving a value of 0 and skewing results. When evaluatingmissions it is necessary to develop a strategy for the variationsthat humans will introduce into the flight plan and how to preventthat from creating negative bias in the ranking of the mission. Thishas applicability in comparing different missions using the sameplatform, as in this case, or in multiple platforms trying tocomplete the same task, as the case of the ROV. The optimizationfor robotic versus human task allocation is essential for futureground-based operations in space in order to minimize downtimeand operational risk (Fong et al., 2008b).

4.3.3. Exploration success

The goal of creating a list of exploration metrics beyond theperformance metrics was to be able to evaluate the explorationsuccess of a given mission (i.e., the scientific return). Too often,when using robotic platforms, success is quantified in terms ofvehicle performance (e.g., runtime, distance traveled, etc.) ratherthan whether the mission furthered fundamental science. Duringthe mission-planning stage, the DeepWorker flight crew woulduse this as a tool to collectively decide the ranking of the givenmission allowing for a prioritization of missions if it was decidedthat two missions were mutually exclusive. As an example, twosample contour following missions (taken from June 26, 2008) arepresented here; both given exploration rankings of 3 (see Table 1)in the mission-planning stage. Pilots during both missions notednumerous observations that supported several of the scientifichypotheses about Pavilion Lake (e.g., degree of sediment cover,relation of slope angle to microbialite distribution, etc.). All ofthese observations furthered understanding of the fundamentalquestions relating to microbialite formation and distribution,however, due to the limited range of a single flight no hypothesescould be firmly resolved. As such, both pilots ranked theirmissions with an exploration ranking of 3. After analysis of thecollected video, a ranking of 3 was similarly given suggesting thatthe pilots are generally aware if they have made a fundamentaldiscovery. It was also discovered during data processing that itwas difficult to rank the missions with anything other than a valueof 2 or 3 as the collected data at least partly supported at least oneof several hypotheses existing about microbialite growth. Forsimilar sites, where significant a priori knowledge is available, this

ARTICLE IN PRESS


evaluation should be refined by coupling the exploration metrics(Table 1) with data quality metrics (see Table 2).

An advantage to having a ‘human-in-the-loop’ with mannedexploration platforms is that humans have as yet unmatchedabilities to perceive and react when presented with anomalies orunforeseen events in order to maximize the scientific return of amission. With the DeepWorker missions in Pavilion Lake, it wasdecided by the pilot crew to identify targets of opportunity (TOPs)that, if observed, would call for the flight plan to be altered; (1)groundwater flow, (2) datable structures, (3) microbial mats, and(4) unusual morphologies. These were selected to fit into theoverall scientific objective of the site (exploring mechanisms ofmicrobialite growth) rather than the specific mission typescovered in the previous two subsections. For each TOP, aprescribed protocol, including a position lock, sensor allocation,sampling protocol and time on-site, was pre-determined in orderto maximize the scientific return without limiting the rest of themission. All four TOPs were encountered during the DeepWorkermissions that were then used to plan subsequent missions in the2008 and 2009 deployments. It was also noted early on that a fifthTOP was being observed by several of the pilots. They werenoticing a ‘cloud’ in the thruster wake that did not appear to bestirred sediment and instead was possibly calcium carbonatecrystals (the material of the microbialites) floccing out of the

Table 2List of data quality metrics proposed to be coupled with exploration metrics.

Metric Descriptor Definition

1 Limited Video and navigation did not support scientific

observations and other relevant data was of limited use.

Dive must be re-flown.

2 Adequate Quantitative data adequate for general documentation of

findings. Provides useful context and enables efficient

return. Data is marginally publishable.

3 Significant Quantitative data adequate to support specific

documentation of scientific findings and yielding

publishable results.

4 Exceptional High quality video, navigation, and other quantitative

data that supports and enhances scientific merit.

Table 3Overall performance metrics for both sampling and contour missions.

Mission Type Performance metrics score (mission s

I II

20080623A Contour 9 8

20080623B Contour 8 4

20080624A Contour 9 8

20080624B Contour 8 5

20080625D Contour 7 6

20080626Aa Sample 2.5 8

20080627B Contour 6 6

20080627A Contour 9 9

20080627Ca Sample 6.7 4

20080627D Sample 10 8

20080628C Sample 10 8

20080628D Sample 10 10

20080629Aa Sample 10 10

20080629B Sample 7.5 7

20080630Aa Contour 8 5

20080630D Contour 8.5 8.5

20080702A Sample 10 10

20080702Ca Sample 10 9

a Those cells that are left blank (shaded) were because the DeepWorker pilots did no

the average results, these were ignored rather than being weighted with a zero in orde

water column as a result of the stirring energy. For this, and otherunforeseen discoveries that will be made in space exploration,sampling protocols will need to be flexible and adaptable.Essential to this will be the quick processing of mission data inorder that both the flight crew and scientific community can makeassessments of the TOP (Table 3).

5. Conclusions

Whether it is the exploration of Earth’s oceans or the solarsystem, specific questions are being asked that require increas-ingly complex tasks to be completed by robots in either delayed,or completely absent, communications with base command onEarth. In both cases, the use of autonomous, remotely operated,and manned exploration platforms is becoming increasinglymandatory to improve knowledge of unknown and often extremeenvironments. In increasingly complex human–robotic explora-tion campaigns, a partial list of elements that are beingcontinuously refined include operator training, human–roboticinterface (HRI), data visualization and management, missionplanning, coordinated missions, etc. Specific lessons were demon-strated in each of the AUV, ROV and manned submersiblecampaigns presented in this work; however, it is contended thatone of the critical elements to maximize the scientific return of anexploration campaign is the development of mission evaluationtools that measure not just the engineering but also the exploration

success of a mixed human-robotic campaign.Three independent methods were explored in this work; a task

efficiency index (TEI), performance metrics and explorationmetrics. Each of these were found to be useful for specificcampaigns (e.g., the TEI for EVA versus ROV sampling) or specificplatforms (e.g., comparing AUV or DeepWorker deployedindependently) although it was generally found that the tendencywas to use performance metrics in order to evaluate missionsuccess, as these are the most intuitively quantifiable. Futureapplication of these evaluation tools would be to combine themtogether in a single exploration campaign with multipleplatforms. In addition, the TEI should be further refined to useother elements, apart from time, to calculate the efficiency (e.g.,

pecific) Average result

III IV V

7 8 9 8.6

5 10 8 4.6

8 9 8 9.0

6 7 8 7.0

5 8 8 6.8

4 5 4.9

4 7 3 6.0

7 7 7 8.4

7 3 5.2

8 8 4 7.6

5 5 2 6.0

6 8 2 7.2

8 9 9.3

7 7 3 6.3

3 0 4.0

9 9 9 8.7

8 10 2 8.0

9 7 8.8

t complete the evaluation logs during post-mission debriefing. In the calculation of

r to not skew the results.

ARTICLE IN PRESS


power consumption). Future application should also include a riskassessment be built into this evaluation in order to maximizesafety for human explorers.

Although each of the deployment sites has advantages,Pavilion Lake is a unique opportunity to combine science andexploration objectives in order to develop an operational model ofmultiple platform coordination. Methods developed from chor-eographing the increasingly specialized robots associated withthese projects (in terms of data collection, data management,logistics, HRI, etc.) would serve as a complementary model ashumans reach towards manned flight to Mars and exploration ofextra-terrestrial liquid bodies as potentially found on Europa orEnceladus. Concurrent to the development of the operationalmodel would be the development of general mission evaluationtools that can be applied specifically to this site and then generallyto future exploration sites.

Acknowledgements

The Pavilion Lake Research Project and UBC-Gavia operations atPavilion Lake were supported by the Canadian Space AgencyCanadian Analogue Research Network (CARN) program, theNational Aeronautics and Space Administration (NASA) Astrobiol-ogy program, the National Geographic Society Committee forResearch Exploration (CRE), and the Canadian National Scienceand Engineering Research Council (NSERC) Discovery program.Funding for the purchase of UBC-Gavia was provided by theCanadian Foundation for Innovation, the British Columbia Knowl-edge Development Fund, and the University of British Columbia.Support for the AUV operations in Bonaire was provided by theNational Oceanic and Atmospheric Administration’s Office ofOcean Exploration. Alexander Forrest was supported by a CanadaNSERC Postgraduate Scholarship-Doctorate (PGS-D) scholarship.Operations at Pavilion Lake would not be possible without thesupport of the Ts’Kw’aylaxw First Nation, the Pavilion LakeCommunity, Mickey and Linda Macri, Donnie Reid, Dale Anderson,Harry Bohm, and BC Parks. We also acknowledge the technicalsupport of Martin Doble from the University of Cambridge, UKwho worked hard to ensure that our under-ice endeavors weresuccessful. Tom Hiller and James Baxter of GeoAcoustics providedtechnical support and data processing of the Geoswath data. Thismanuscript represents Pavilion Lake Research Project publication#09-01.

References

Arsenault, R., Ware, C., Plumlee, M., Martin, S., Whitcomb, L.L., Wiley, D., Gross, T.,Bilgili, A., 2004. A System for Visualizing Time-Varying Oceanographic 3DData. In: Proceedings of Oceans 2004, IEEE Press.

Bellingham, J.G., Rajan, K., 2007. Robotics in remote and hostile environments.Science 318, 1098–1102.

Bualat, M., Edwards, L., Fong, T., Broxton, M., Flueckiger, L., Lee, S.Y., Park, E., To, V.,Utz, H., Verma, V., Kunz, C., MacMahon, M., 2007. Autonomous roboticinspection for lunar surface operations. Field and Service Robotics. (42).

Brady, A., Slater, G., Laval, B.E., Lim, D.S.S., 2009. Constraining carbon sources andgrowth rates of freshwater microbialites in Pavilion Lake using 14C analysis.Geobiology 7, 1–12.

Doble, M.J., Wadhams, P., Forrest, A.L., Laval, B.E., 2009. Through-ice AUVdeployment: operational and technical experience from two seasons of Arcticfieldwork. Cold Regions Science and Technology 56, 90–97.

Fong, T., Allan, M., Bouyssounouse, X., et al., 2008a. Robotic site survey atHaughton Crater. 9th International Symposium on Artificial Intelligence,

Robotics and Automation in Space (iSAIRAS). February 26–29, 2008, LosAngeles, CA, USA.

Fong, T., Bualat, M., Deans, M., et al., 2008b. Field testing of utility robots for lunarsurface operations. Space 2008 Conference and Exhibit, September 9–11, 2008,San Diego, CA, USA.

Forrest, A.L., Bohm, H., Laval, B.E., Magnusson, E., Yeo, R., Doble, M.J., 2007.Investigation of under-ice thermal structure: small AUV deployment inPavilion Lake, BC, Canada. IEEE Oceans, Vancouver, BC, Canada.

Forrest, A.L., Laval, B.E., Pieters, R., Lim, D.S.S., 2008a. Convectively driventransport in temperate lakes. Limnology and Oceanography 53 (5, part 2),2321–2332.

Forrest, A.L., Laval, B.E., Lim, D.S.S., Trembanis, A., 2008b. Underwater platforms asanalogues for space exploration. In: 6th Canadian Space Exploration Work-shop. December 1–3, 2008, Montreal, PQ, Canada.

Grasmueck, M., Gregor, P., David, E., Viggiano, A., Correa, T., Rathwell, G., Luo, J.,2006. Autonomous underwater vehicle (AUV) mapping reveals coral mounddistribution, morphology, and oceanography in deep water of the Straits ofFlorida. Geophysical Research Letters 33, L23616.

Hine III, B.P., Stoker, C., Sims, M., Rasmussen, D., Hontalas, P., Fong, T.W.,Steele, J., Barch, D., Andersen, D., Miles, E., and Nygren, E., 1994. TheApplication of Telepresence and Virtual Reality to Subsea Exploration. The2nd Workshop on Mobile Robots for Subsea Environments, Proceedings ofROV’94, May, 1994.

Howard, A., Rodriguez, G., 2003. Validating mission relevance of autonomytechnologies through increased science return. 20th International Conferenceon Machine Learning, August 21–24, 2003, Washington, DC, USA.

Krajick, K., 2007. Robot seeks new life – and new funding – in the abyss of Zacaton.Science 19 (315), 322–324.

Laval, B., Cady, S., Pollack, J., McKay, C., Bird, J., Grotzinger, J., Ford, D., Bohm, H.,2000. Modern freshwater microbialite analogues for ancient dendritic reefstructures. Nature 407, 626–629.

Lim, D.S.S., Laval, B.E., Antoniades, D., Forrest, A.L., Pike, W., Pieters, R., Saffari, M.,Reid, D., Slater, G., McKay, C.P., 2009. Hydrological characterization of thepavilion lake region, British Columbia, Canada. Fundamental and AppliedLimnology 173, 329–351.

Mishkin, A., Lee, Y., Korth, D., LeBlanc, T., 2006. Human–robotic missions to theMoon and Mars: operations design implications. IEEEAC #1400. (6).

Osinski, G.R., L�eveill�e, R., Berinstain, A., Lebeuf, M., Bamsey, M., 2006. Terrestrialanalogues to Mars and the Moon: Canada’s role. Geoscience Canada 33 (4),175–188.

Paley, D.A., Zhang, F., Leonard, N.E., 2008. Cooperative control for ocean sampling:the glider coordinated control system. IEEE Transactions on Control SystemTechnology 16 (4), 735–744.

Piguet, L., Fong, T.W, Hine, B., Hontalas, P., Nygren, E., 1995. VEVI: A Virtual RealityTool for Robotic Planetary Exploration. Proceedings of Virtual Reality World,February, 1995.

Rodriguez, G., Weisbin, C.R., 2003. A new method to evaluate human–robot systemperformance. Autonomous Robots 14, 165–178.

Sherwood, R., Mishkin, A., Estlin, T., Chien, S., Backes, P., Norris, J., Cooper, B.,Maxwell, S., Rabideau, G., 2001. Autonomously generating operationssequences for a mars rover using AI-based planning. In IEEE/RSJ InternationalConference on Intelligent Robots and Systems.

Steneck, R.S., McClanahan, T., 2004. A report on the status of the coral reefs ofBonaire with advice on the establishment of fish protection areas. PewCharitable Trust Report, 64 pp.

Steinfeld, A., Fong, T.W., Kaber, D., Lewis, M., Scholtz, J., Schultz, A., Goodrich, M.,2006. Common Metrics for Human–Robot Interaction. 2006 Human–RobotInteraction Conference, March, 2006.

Stoker, C.R., Barch, D., Farmer, J., Flagg, M., Healey, T., Tengdin, T., Thomas, H.,Schwehr, K., Stakes, D., 1996. Exploration of Mono Lake With an ROV: aprototype experiment for the MAPS AUV program. In: IEEE Symposiumon Autonomous Underwater Vehicle Technology, June 3–6, Monterrey, CA,1996.

Tompkins, P., Stentz, A., Wettergreen, D., 2006. Mission-level path planning andre-planning for rover exploration. Robotics and Autonomous Systems 54 (2),174–183.

Weekley, J., Brutzman, D., Healey, A.J., Davis, D., Lee, D., 2004. AUV Workbench:Integrated 3D for Interoperable Mission Rehearsal, Reality, and Replay. In:2004 Mine Countermeasures & Demining Conference.

Weisbin, R.C., Rodriguez, G., Schenker, S.P., Das, H., Hayati, S., Baumgartner, T.E.,1999. Autonomous rover technology for Mars sample return. ArtificialIntelligence, ESTEC, Noordwijk, The Netherlands.

Williams, D., Stott, N., Buckley, J., Garan, R., Broderick, T., Hein, R., 2006. NEEMO 9Crew Report, 90 pp.

White, R.J., Averner, M., 2001. Humans in space. Nature 409, 1115–1118.

performance evaluation of underwater platforms in the context of space exploration

Documents