marine geospatial ecology tools open source geoprocessing for marine ecology

Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat HalpinDuke University Marine Geospatial Ecology Lab4-Mar-2009

Talk outlineOverview of MGETQuick tour of the MGET tool collectionExample application: habitat modeling

What is MGET?A collection of geoprocessing tools for marine ecologyOceanographic data management and analysisHabitat modeling, connectivity modeling, statisticsHighly modular; designed to be used in many scenariosEmphasis on batch processing and interoperabilityFree, open source softwareWritten in Python, R, MATLAB, C#, and C++Minimum requirements: Win XP, Python 2.4 ArcGIS 9.1 or later currently needed for many toolsArcGIS and Windows are only non-free requirementsAlso useful for terrestrial problems!

Set goals & prioritiesMake & implement EBM decisionsMonitor &assessDevelop conceptual modelsAnalyze data & develop models and scenarios Collect physical, biological, and socioeconomic data

Set goals & prioritiesMake & implement EBM decisionsMonitor &assessDevelop conceptual modelsAnalyze data & develop models and scenarios Collect physical, biological, and socioeconomic dataMonitoring & assessment toolsProject management toolsData collection and management toolsConceptual modeling toolsData processing toolsModeling toolsModel development toolsWatershed modelsDispersal and habitat modelsMarine ecosystem modelsSocial science modelsSector-specific decision support toolsConservation and restoration site selectionCoastal zone management toolsFisheries management toolsHazard assessment and resiliency planning toolsLand use planning toolsStakeholder communication & engagement toolsScenario visualization tools

Set goals & prioritiesMake & implement EBM decisionsMonitor &assessDevelop conceptual modelsAnalyze data & develop models and scenarios Collect physical, biological, and socioeconomic dataMonitoring & assessment toolsProject management toolsConceptual modeling toolsSector-specific decision support toolsConservation and restoration site selectionCoastal zone management toolsFisheries management toolsHazard assessment and resiliency planning toolsLand use planning toolsStakeholder communication & engagement toolsScenario visualization toolsData collection and management toolsData processing toolsModeling toolsModel development toolsWatershed modelsDispersal and habitat modelsMarine ecosystem modelsSocial science modelsData collection and management toolsData processing toolsModeling toolsModel development toolsWatershed modelsDispersal and habitat modelsMarine ecosystem modelsSocial science modelsMGET

MGETs software architectureMGET tools are really just Python functions, e.g.:MGET exposes them to several types of external callers:def MyTool(input1, input2, input3, output1)

IMyInterface

IMyInterface

IMyInterface

IMyInterface

IMyInterface

IMyInterface

IMyInterface

IMyInterface

IMyInterface

IMyInterface

IMyInterface

IUnknown

ObjectName

pythoncom2x.dll

IDispatch

IMyTool

MyToolCOM class

MyTool.py

MGET interface in ArcGISThe MGET toolbox appears in the ArcToolbox window

MGET interface in ArcGISDrill into the toolbox to find the toolsDouble-click tools to execute directly, or drag to geoprocessing models to create a workflow

IntegrationThe Python functions can invoke C++, MATLAB, R, ArcGIS, and COM classes.

IMyInterface

IMyInterface

IMyInterface

IMyInterface

IMyInterface

IMyInterface

IMyInterface

IMyInterface

IMyInterface

IMyInterface

IMyInterface

IUnknown

ObjectName

R interpreter

MyTool.m

MyTool.r

MyTool.py

MGET utilizes a lot of other softwareInterpreters / RuntimesPythonMATLAB Component RuntimeRR PackagesgamMASSmgcvrgdalROCRApplicationsArcGISNOAA CoastWatch UtilitiesPython Packagesdocutilshttplib2lxmlnetcdf4numpyosgeopydappyparsingpyprojpywin32rpysetuptoolsC LibrariesGDAL/OGRgziphdflibxmllibxsltnetcdfproj4zlibAll but one of these (pywin32) are installed automatically

Quick tour of the tools

Analyzing larval connectivityCoral reef ID and % cover mapsOcean currents dataTool downloads data for the region and dates you specifyLarval density time series rastersEdge list feature class representing dispersal networkOriginal research by Eric A. Treml

Converting data

Batch processingCopy one raster at a time

Batch processingCopy rasters that you list in a table

Batch processingCopy rasters from a directory tree

Tools for specific productsDownloads sea surface height data from http://opendap.aviso.oceanobs.com/thredds

Identifying SST frontsCayula and Cornillion (1992) edge detection algorithmFrequencyTemperatureOptimal break 27.0 CStrong cohesion front presentStep 1: Histogram analysisStep 2: Spatial cohesion testWeak cohesion no frontBimodalExample outputMexicoArcGIS model

Identifying geostrophic eddiesNegative W at eddy coreSSH anomalyAvailable in MGET 0.8Example output

Mapping species biodiversity

Invoking R from ArcGIS

Example application: habitat modelingPresence/absence observations Sampled environmental dataMultivariate statistical modelProbability of occurrence predicted from environmental covariatesBinary classificationWarning: Habitat modeling is complicated! This simplified example is meant to briefly illustrate tools. Consult the literature for best practices!

Focal species: Stenella frontalisPhoto: Garth MixMap: OBIS-SEAMAPCommon name:Atlantic Spotted DolphinDistribution:Tropical and warm temperate AtlanticStudy area: Eastern U.S.

Species observation dataThe Ocean Biogeographic Information System (OBIS) is a global database of marine species observations.The OBIS-SEAMAP system at Duke University holds the records for seabirds, marine mammals, and sea turtles, including records gathered during NOAA cruises.

Environmental predictor variablesBathymetry:ETOPO2V2 from NOAA NGDCSST:Monthly climatological 4km AVHRR Pathfinder from NOAA NODCChlorophyll:Monthly climatological SeaWiFS chlorophyll-a from NASA GSFCImages shown above are for month of March

Step 1: Download species pointsDownload points using MGET tool:Presence: Records of Stenella frontalisAbsence: Records of other cetaceansThe tool uses the DiGIR protocol to retrieve data from OBIS servers

Red: PresenceGreen: Absence

Step 2: Convert oceanography to Arc rastersDownload with FTP from NOAA and NASA:ETOPO2 bathymetry 1 binary fileAVHRR Pathfinder monthly climatological SST 12 HDF filesSeaWiFS monthly climatological chlorophyll 12 HDF filesConvert to ArcGIS rasters using MGET tools:

Step 3: Sample oceanography at points Need to sample rasters and populate fields Must sample SST and chlorophyll by date

Step 3: Sample oceanography at pointsSampling bathymetry is easy because it is staticTo sample dynamic data such as SST and chlorophyll, you must first calculate the paths to rasters to sample from the points datesThen use an MGET batch sampling tool

Step 4: Create exploratory plotsBest predictors: SST and Chl

Step 5: Fit, evaluate, and predict modelPresence ~ s(SST) + s(log10(Chlorophyll))

Partial plots produced by the Fit GAM toolSSTSSTlog10(Chlorophyll)s(SST,8.97)s(log10(Chlorophyll),5.6)Presence more likely at higher SSTPresence more likely at lower Chl

Plotting a receiver operating characteristic curve

The ROC plotFalse positive rateTrue positive rateCutoff = 0.020By default, tool selects the cutoff closest to the point of perfect classification (0, 1)Model summary statistics:

Area under the ROC curve (auc) = 0.960779Mean cross-entropy (mxe) = 0.030566Precision-recall break-even point (prbe) = 0.001866Root-mean square error (rmse) = 0.087781

Contingency table for cutoff = 0.019638:

Actual P Actual N TotalPredicted P 287 3541 3828Predicted N 26 32408 32434 Total 313 35949 36262

Accuracy (acc) = 0.901633Error rate (err) = 0.098367Rate of positive predictions (rpp) = 0.105565Rate of negative predictions (rnp) = 0.894435

True positive rate (tpr, or sensitivity) = 0.916933False positive rate (fpr, or fallout) = 0.098501True negative rate (tnr, or specificity) = 0.901499False negative rate (fnr, or miss) = 0.083067

Positive prediction value (ppv, or precision) = 0.074974Negative prediction value (npv) = 0.999198Prediction-conditioned fallout (pcfall) = 0.925026Prediction-conditioned miss (pcmiss) = 0.000802

Matthews correlation coefficient (mcc) = 0.246384Odds ratio (odds) = 101.026394SAR = 0.650065ROC summary stats for cutoff:

Predicting presence for oceanographic rasters

Rasters output by the Predict GAM toolPredicted presence:Range: 0 - 0.25Standard errors:Range: 0 - 0.11Binary classification:Species range map produced by classifying presence into 0 or 1 according to ROC cutoffSimilar to OBIS-SEAMAP range map?Predictions for October

AcknowledgementsA special thanks to the many developers of the open source software that MGET is built upon, including:Guido van Rossum and his many collaborators; Mark Hammond; Travis Oliphant and his collaborators; Walt