Mining For Lost TreasureMining For Lost Treasure

National Geospatial Data ClearinghouseNational Geospatial Data Clearinghouse

Archibald WarnockArchibald WarnockU.S. Federal Geographic Data CommitteeU.S. Federal Geographic Data Committee

A/WWW EnterprisesA/WWW Enterprises

What is Clearinghouse?What is Clearinghouse?

A distributed service to locate A distributed service to locate geospatial data based on geospatial data based on characteristics expressed in characteristics expressed in metadatametadata

Clearinghouse allows a user to pose a Clearinghouse allows a user to pose a query of all or a portion of the query of all or a portion of the community in a single sessioncommunity in a single session

Like a spatial AltaVista Like a spatial AltaVista

National Geospatial Data National Geospatial Data ClearinghouseClearinghouse

Distributed data producers and Distributed data producers and users.users.

Key components:Key components:– Data documentation (metadata)Data documentation (metadata)– Networking (Internet)Networking (Internet)– Serving, searching, and accessing Serving, searching, and accessing

softwaresoftware Z39.50 Search and Retrieve ProtocolZ39.50 Search and Retrieve Protocol WWW - World Wide WebWWW - World Wide Web

Components of Components of ClearinghouseClearinghouse There are three functional areas There are three functional areas

that interact to create the that interact to create the Clearinghouse:Clearinghouse:– Metadata preparation and indexingMetadata preparation and indexing– Metadata serviceMetadata service– User Access via Gateway formsUser Access via Gateway forms

Clearinghouse MethodClearinghouse Method

Metadatapreparation

Metadatavalidation/

staging

Metadatapublication

Useraccess

Clearinghouse DesignClearinghouse Design

The Clearinghouse in its distributed The Clearinghouse in its distributed form includes a registry of servers, form includes a registry of servers, several WWW-to-Z39.50 gateways, several WWW-to-Z39.50 gateways, and many Z39.50 serversand many Z39.50 servers

A primary goal of Clearinghouse is A primary goal of Clearinghouse is to provide the ability to find spatial to provide the ability to find spatial data throughout the entire data throughout the entire community, not one site at a timecommunity, not one site at a time

Essential ConfigurationEssential Configuration

FGDCFGDC

Gateways

WebClient

WebClient

NodeNode

NodeNode

NodeNode

NodeNode

Clearinghouse Sites

UserUser downloads query downloads query formform

FGDCFGDC

Gateways

WebClient

WebClient

NodeNode

NodeNode

NodeNode

NodeNode

Clearinghouse Sites

User sends query to web serverUser sends query to web server

FGDCFGDC

Gateways

WebClient

WebClient

NodeNode

NodeNode

NodeNode

NodeNode

Clearinghouse Sites

Gateway passes query to Clearinghouse Gateway passes query to Clearinghouse ServersServers

FGDCFGDC

Gateways

WebClient

WebClient

NodeNode

NodeNode

NodeNode

NodeNode

Clearinghouse Sites

Gateway receives and collates “hits”Gateway receives and collates “hits”

FGDCFGDC

Gateways

WebClient

WebClient

NodeNode

NodeNode

NodeNode

NodeNode

Clearinghouse Sites

Client receives results summary as HTMLClient receives results summary as HTML

FGDCFGDC

Gateways

WebClient

WebClient

NodeNode

NodeNode

NodeNode

NodeNode

Clearinghouse Sites

Client can request a specific metadata record Client can request a specific metadata record for viewingfor viewing

FGDCFGDC

Gateways

WebClient

WebClient

NodeNode

NodeNode

NodeNode

NodeNode

Clearinghouse Sites

Node in More DetailNode in More Detail

MetadataIndex/DBZ39.50server

Internet Data

DataData

The most expensive investment for The most expensive investment for an organizationan organization

Created by many different Created by many different organizationsorganizations

To solve many different problemsTo solve many different problems Using many different methods and Using many different methods and

technologiestechnologies

But . . .But . . .

Data are hard to findData are hard to find Data are difficult to accessData are difficult to access Data are hard to integrateData are hard to integrate Data are not currentData are not current Data are undocumentedData are undocumented Data are incompleteData are incomplete

The uses of metadataThe uses of metadata

Provides documentation of existing Provides documentation of existing internal geospatial data resources internal geospatial data resources within an organizationwithin an organization (inventory)(inventory)

Permits structured search and Permits structured search and comparison of held spatial data by comparison of held spatial data by othersothers (advertising)(advertising)

Provides end-users with adequate Provides end-users with adequate information to take the data and use it information to take the data and use it in an appropriate contextin an appropriate context (liability)(liability)

Metadata SolutionsMetadata Solutions

Numerous software solutions Numerous software solutions availableavailable

Commercial and free-wareCommercial and free-ware Standalone, DB-linked, GIS-linkedStandalone, DB-linked, GIS-linked Permit collection and structuring of Permit collection and structuring of

FGDC-compatible metadataFGDC-compatible metadata Present metadata as HTML, XML, or Present metadata as HTML, XML, or

texttext

GILS, Dublin Core and GILS, Dublin Core and OthersOthers Dublin Core is a minimal (15 fields) generic Dublin Core is a minimal (15 fields) generic

metadata scheme for virtually any kind of metadata scheme for virtually any kind of documentdocument

GILS represents a more detailed approach, GILS represents a more detailed approach, including most of DC, providing greater including most of DC, providing greater interoperabilityinteroperability

GILS is less bibliographically oriented than GILS is less bibliographically oriented than (Z39.50) BIB-1(Z39.50) BIB-1

GILS is lightweight compared to GEO (FGDC) GILS is lightweight compared to GEO (FGDC) and EOS/CIP (which have specific functional and EOS/CIP (which have specific functional requirements)requirements)

What Structured Metadata What Structured Metadata Means -1Means -1

GILS - Fewer GILS - Fewer fieldsfieldsMore documentsMore documentsMore metadata More metadata

recordsrecordsSkinnier metadata Skinnier metadata

recordsrecordsEasier abstractionEasier abstraction

FGDC - More FGDC - More fieldsfieldsFewer documentsFewer documentsFewer metadata Fewer metadata

recordsrecordsFatter metadata Fatter metadata

recordsrecordsLess abstractionLess abstraction

GILS is a good, general compromise

What Structured Metadata What Structured Metadata Means - 2Means - 2

A Z39.50 profile as defines a languageA Z39.50 profile as defines a language At some level, Z39.50 is a detailAt some level, Z39.50 is a detail Protocols are about communication, profiles are about Protocols are about communication, profiles are about

abstraction and GILS is about contentabstraction and GILS is about content Z39.50 guarantees that the user’s query can be Z39.50 guarantees that the user’s query can be

unambiguously decoded - no guarantees about contentunambiguously decoded - no guarantees about content We could implement the profile over any protocol - We could implement the profile over any protocol -

http, CORBA, etc.http, CORBA, etc.

Do we have to use Z39.50?Do we have to use Z39.50? No, but the abstraction is requiredNo, but the abstraction is required Z39.50 already includes the abstraction modelZ39.50 already includes the abstraction model

How much metadata is How much metadata is enough?enough? Internal documentation for local use Internal documentation for local use

(local inventory)(local inventory) Basic documentation for discovery of Basic documentation for discovery of

information holdings information holdings (catalog/search)(catalog/search) Detailed documentation to provide Detailed documentation to provide

end-users with adequate information end-users with adequate information for re-use for re-use (asset management)(asset management)

Server SolutionsServer Solutions

Z39.50 Protocol is usedZ39.50 Protocol is used ““GEO” Geospatial Metadata Profile is GEO” Geospatial Metadata Profile is

published for Z39.50 implementors to published for Z39.50 implementors to understand FGDC metadata understand FGDC metadata structuresstructures

Supports search across numeric, text, Supports search across numeric, text, date, and spatial extent and full-textdate, and spatial extent and full-text

Freeware and commercial solutions Freeware and commercial solutions

Gateway in more Gateway in more detaildetail

Nodes

GatewayGatewayWeb

serverinterface

Z39.50clients

Web Gateway Case

Webclient

Webclient

User InterfacesUser Interfaces

HTML-based forms hosted at HTML-based forms hosted at Gateways are the primary access Gateways are the primary access methodmethod

Java map-based interface from Java map-based interface from MEL allows more sophisticated MEL allows more sophisticated searchsearch

Inclusion of search capabilities in Inclusion of search capabilities in GIS client software is possibleGIS client software is possible

Who’s in Clearinghouse?Who’s in Clearinghouse?

109 Nodes (servers) online as of 109 Nodes (servers) online as of 3/1/993/1/99– 28 Federal, national scope28 Federal, national scope– 35 State/University state-wide scope35 State/University state-wide scope– 28 International scope or location28 International scope or location– 18 Local or Regional scope18 Local or Regional scope

US Federal ParticipationUS Federal Participation

NOAA (10)NOAA (10) USGS (6)USGS (6) FEMA (sampler)FEMA (sampler) NRCS climate and NRCS climate and

soilssoils CIESIN/EPACIESIN/EPA CIESIN/NASACIESIN/NASA DOT NTADDOT NTAD

National Park ServiceNational Park Service Army Corps of Army Corps of

EngineersEngineers Tri-Services CenterTri-Services Center National Wetlands National Wetlands

InventoryInventory Census (sampler)Census (sampler) Minerals Minerals

Management ServiceManagement Service

State ParticipationState Participation

New York (2)New York (2) North CarolinaNorth Carolina OklahomaOklahoma KansasKansas TexasTexas Montana (3)Montana (3) VermontVermont PennsylvaniaPennsylvania

West VirginiaWest Virginia WashingtonWashington WisconsinWisconsin Wyoming (2)Wyoming (2) FloridaFlorida AlabamaAlabama New MexicoNew Mexico ArizonaArizona

GeorgiaIllinoisMinnesotaAlaskaCaliforniaDelawareNebraska (2)New Jersey

Regional/Local Regional/Local ParticipationParticipation McKinley Co, NMMcKinley Co, NM City of Santa Fe, NMCity of Santa Fe, NM North Texas GISNorth Texas GIS Research PlanningResearch Planning Sabine R Authority, TXSabine R Authority, TX San Francisco BaySan Francisco Bay S Florida EcosystemS Florida Ecosystem SW Natural ResourcesSW Natural Resources

Olympic Peninsula, WAOlympic Peninsula, WA Greater YellowstoneGreater Yellowstone Helena NFHelena NF Ecological Reserves, KSEcological Reserves, KS MIT/Mass Boston DOQsMIT/Mass Boston DOQs Great Lakes EISGreat Lakes EIS Eastern SierraEastern Sierra

International ParticipationInternational Participation

NOAA/Japan GOINNOAA/Japan GOIN South Africa (2)South Africa (2) ESA AVHRR samplerESA AVHRR sampler GELOS, ItalyGELOS, Italy PAIGH, MexicoPAIGH, Mexico S57 Hydrography, CanadaS57 Hydrography, Canada NRL MELNRL MEL Africa DDSAfrica DDS Inter-American Geospatial Data Inter-American Geospatial Data

NetworkNetwork Hong KongHong Kong CIESIN/USDA Global CIESIN/USDA Global

Environmental ChangeEnvironmental Change Australia (10+)Australia (10+) Costa RicaCosta Rica Caribbean CEPNET, JamaicaCaribbean CEPNET, Jamaica

Planned or Funded NodesPlanned or Funded Nodes

Mt Desert Island, MEMt Desert Island, ME SW Washington COGSW Washington COG NASA GCMDNASA GCMD CODEPLAN, BrazilCODEPLAN, Brazil IowaIowa Missouri Missouri KentuckyKentucky

South DakotaSouth Dakota OregonOregon LouisianaLouisiana OhioOhio Connecticut MAGICConnecticut MAGIC ColoradoColorado NW EcosystemsNW Ecosystems

Clearinghouse provides...Clearinghouse provides...

Discovery Discovery of spatial data of spatial data Distributed Distributed search worldwidesearch worldwide Uniform interfaceUniform interface for spatial datafor spatial data

searchessearches Advertising Advertising for your data holdingsfor your data holdings

For more information:For more information:

Visit the FGDC website: http://www.fgdc.gov

Contact the Clearinghouse Coordinator, Doug Nebert (ddnebert@usgs.gov) or Archie Warnock (warnock@awcubed.com)