Download - CEOS WGISS 24 - IDN Task Team
CEOS WGISS 24 - IDN Task Team
Task leader: Lola Olsen, represented by Tyler Stevens
Agenda1. GCMD/IDN Science User Working Group2. Review of Minutes and Action Items3. Update on Population and Usage Statistics4. Development Plans for MD9.8 - MD105. GEOSS Activities and other Collaborations6. Node Reports7. Evolution of Controlled Vocabularies8. Re-emergence of the Interop and Communication9. The CEOS IDN Newsletter
Briefing on the GCMD IDN Science User Working Group
Meeting, June 19-20 2007 by Dr. Wyn
Cudlip
Key Topics from WGISS 23Key Topics from WGISS 23
New IDN Features in MD 9.7New IDN Features in MD 9.7 Evolution of the DIFEvolution of the DIF Future DevelopmentFuture Development Information System and Data Center
(ISDC) for Geo-scientific Data User Working Group (UWG) MeetingUser Working Group (UWG) Meeting IDN Contributions to IDN Contributions to
Interoperability HandbookInteroperability Handbook
WGISS Action ItemsWGISS Action Items
No. 22-13: When JAXA is ready, add a link to JAXA multilingual prototype to the GCMD page. No. 23-8: Arturo to report back to WGISS-All on the awareness of the GEO task teams of the new IDN portals that have been created for those GEO tasks, and clarify what reviews are currently underway for the new GEO portals.
Status: 23 June 2007: Lola and Arturo distributed a memo describing the portals that are currently in place, and they have been register in GEO registry. CLOSED
No. 23-10: Arturo (IDN Team) to assess the differences between the platform and instrument descriptions in the IDN and those contained in the CEOS database (CEOS Earth Observation Handbook).
No. 23-22: The IDN team (Lola Olsen) to perform a detailed analysis of user query patterns and present a summary of usage patterns and lessons learned at WGISS-24.
Characteristics of IDN UsersCharacteristics of IDN Usersthrough Population and Usage through Population and Usage
Statistics Statistics
Most Popular Keyword Search
December 2005 - October 2007
DIF Population by Earth Science Topic
Most Popular Full-Text Searches
December 2005 - October 2007
Searches by Ancillary Keywords
Platforms
12%Instruments
14%
DIF Population Growth
Deletion and eventual
replacementof Antarctic DIFs
DIF Population by IDN Node
New, Revised, and Deleted DIFs - 2007
Quality Control is on-going
Problem with Google Cache
Web Page Hits
Summer Declines
Users by Continent
North America
Europe
Asia
Australia
South America
Africa
Other
January 2007 - October 2007
Users by European Country
Google (50%)
GCMD Direct Access (20%)
Yahoo
(6%)
NA
SA
(4%
)
Others (20%)
wikipedia.orgnoaa.govusgs.gov
The IDN allows robots to crawl the site to help broaden user access.
How the Users Access the IDN
When Users Access the IDN
Lets us know best time to schedule server
maintenance.
January 2007 - October 2007
Most Popular Browsers and Platforms
Important for cross-platform and browser
support of IDN software.January 2007 - October 2007
Metadata Contributions January-September 2007
From docBUILDER45%
55 %
Improved translation technology has increased contributions from sources using other “mandated” standards.
Lessons Learned from User Stats
Concentrate resources on keyword interface Concentrate resources on keyword interface because most consistent usage come from there.because most consistent usage come from there.
Consider adding spell-corrector for full-text Consider adding spell-corrector for full-text search to give users more chances of finding search to give users more chances of finding data.data.
Improve functionality of docBUILDER and Improve functionality of docBUILDER and metadata ingestion tools.metadata ingestion tools.
Capitalize on Google cache and provide additional Capitalize on Google cache and provide additional mechanisms for accessing metadata (metadata in mechanisms for accessing metadata (metadata in KML).KML).
Use semantic web technology to offer the option Use semantic web technology to offer the option to search on related words to their original query.to search on related words to their original query.
Development Plans forDevelopment Plans forMD 9.8-MD 10MD 9.8-MD 10
MD 9.8 - Migrate from MD 9.8 - Migrate from Oracle to “Open Source” Oracle to “Open Source”
Software (MySQL)Software (MySQL)
Staff evaluated alternative databases: Staff evaluated alternative databases: • PostgreSQL, Mckoi, MySQL, and XML PostgreSQL, Mckoi, MySQL, and XML
databases.databases.
MySQL chosen as alternative to Oracle.MySQL chosen as alternative to Oracle.
Staff is now in the process of migrating to Staff is now in the process of migrating to MySQL.MySQL.
Oracle: 4610 ms. MySQL: 1582 ms.
(# of milliseconds to query 150 Atmospheric Keywords)
Oracle Vs. MySQL Database
•GCMD Oracle: 4610 ms.•GCMD XML DB: 6361 ms.•OS X MySQL: 1582 ms.•OS X XML DB: 2978 ms.•GCMD BDB (Cache): 2378 ms.•OS X BDB (Cache): 1589 ms.
(# of milliseconds to query 150 Atmospheric
Keywords)
Benchmark Report
MD 9.9 - Quality UpgradeMD 9.9 - Quality Upgrade
To improve user experience To improve user experience To enhance interoperabilityTo enhance interoperability To reduce maintenanceTo reduce maintenance To streamline software testing processTo streamline software testing process
MD 10 - Next Generation IDN
Empower data providers to manage Empower data providers to manage their “own” metadata (load, query, their “own” metadata (load, query, share, etc.) through a distributed share, etc.) through a distributed input capability.input capability.
Represent the GCMD vocabulary as Represent the GCMD vocabulary as an ontology.an ontology.
Re-engineer server to support rising Re-engineer server to support rising levels of data sets and increasing levels of data sets and increasing demands for services.demands for services.
GEOSS Activities and Other GEOSS Activities and Other Activities/CollaborationActivities/Collaboration
IDN Registered in GEOSS IDN Registered in GEOSS Registry Publication PortalRegistry Publication Portal
GEOSS PortalsGEOSS Portals
IDN Architecture Within GEOSS
Import metadata into GEOSS Portal
cataloguesMetadata content harvest
from FGDC clearinghouse
node
IDN
IDN Components in
GEOSS registry
IPY Data & Services Portals
Find IPY DIFs in
New Portals Added Since May 2007
NASA DataNASA Data THREDDSTHREDDS OBIS-SEAMAPOBIS-SEAMAP EBA-AMD SCAR ProjectEBA-AMD SCAR Project GEOSS WildfiresGEOSS Wildfires GEOSS Wildfires GEOSS Wildfires
ServicesServices GEOSS InSitu GEOSS InSitu
InstrumentsInstruments European Space AgencyEuropean Space Agency Essential Climate Essential Climate
VariablesVariables
The “King” Request
Mapped IDN Keywords to
Essential Climate
Variables (ECV)
Refine by Platforms Under
Ozone To Get Detailed Platform
Description
Enhanced Platform & Instrument Descriptions
Expanded database table structures for Instrument and Platform
hierarchical keywords
10 Platform Categories66 Platform Series/Entities
3 Instrument Categories25 Instrument_Classes29 Instrument_Types 22 Instrument_Subtypes
Requirements for a Requirements for a Platform/Instrument Search Platform/Instrument Search
Interface Interface
Capitalize on existing platform and instrument Capitalize on existing platform and instrument ancillary information. ancillary information.
CEOS Earth Observation HandbookCEOS Earth Observation Handbook NASA Science Reference HandbookNASA Science Reference Handbook IDN DescriptionsIDN Descriptions
Utilize new technology to create an intuitive Utilize new technology to create an intuitive interface.interface.
Provide capability to sort platform and instrument Provide capability to sort platform and instrument information by columns in sequential order.information by columns in sequential order.
Enhance capability to refine search within interface.Enhance capability to refine search within interface. Directly link from instrument to platform Directly link from instrument to platform
descriptions and vice-versa, and link directly to descriptions and vice-versa, and link directly to associated metadata.associated metadata.
Left Hand Navigation
NASA Earth Observation Platforms
Launch Date
TERRA Polar Sun-Synchronous 1999-12-18 TERRA METADATA
UARS ACRIMCLAESHALOEHRDIIMLSPEM
SAMSSOLSTICE
SUSIMWINDII
Polar Non-Sun-
Synchronous 1991-09-12 UARS METADATA
ASTERCERESMISR
MODISMOPITT
Platform Name Orbit Type Associated Instruments Metadata
MOCK UP
Left Hand Navigation
NASA Earth Observation Instruments
MODIS AQUATERRA
2002-06-25 MODIS METADATA
IIR CALIPSO IIR METADATA
Instrument Name Associated Platforms MetadataStart Date
Passive
Instrument Class
Passive 2006-04-28
MOCK UP
docBUILDER Tool for Platforms - docBUILDER Tool for Platforms - DemoDemo
Fields for Platform Description in GCMD
Fields for Platform Descriptions in CEOS Earth Observation Handbook
Platform IDPlatform ID Mission (Short Name)Mission (Short Name)
Platform IdentificationPlatform Identification
(Associated to GCMD controlled (Associated to GCMD controlled keywords )keywords )
+ Platform Category+ Platform Category
+ Platform Series/Entity+ Platform Series/Entity
+ Platform Short Name+ Platform Short Name
+ Platform Long Name+ Platform Long Name
Mission (Long name) Mission (Long name)
ApplicationsApplications
Platform DescriptionPlatform Description
(Summary)(Summary)ApplicationsApplications
Platform LogisticsPlatform Logistics
+ Launch date+ Launch date
+ Launch Site ( Pull down list + Launch Site ( Pull down list database)database)
+ Estimated Design Life (time)+ Estimated Design Life (time)
+ Primary Sponsor ( option to add + Primary Sponsor ( option to add various sponsors )various sponsors )
+ Launch Date+ Launch Date
+EOL : End of Life+EOL : End of Life
PLATFORMS COMPARATIVE TABLE IN RESPONSE TO TASK WGISS 23-10
Note: Description information provided in the document task were created by Olsen, L.M., G. Major, S. Ritz, T. Stevens, M. Meaux, A. Restrepo, 2007 . NASA/Global Change Master Directory (GCMD) Earth Science Keywords.
Fields for Platform Descriptions in GCMD
Fields for Platform Descriptions in CEOS Earth Observation Handbook
Online ResourceOnline Resource(URL field provided)(URL field provided)
URLURL
OrbitOrbit+ Orbit Type ( Pull down list menu to + Orbit Type ( Pull down list menu to choose orbit type)choose orbit type)+ Orbit Altitude ( km)+ Orbit Altitude ( km)+ Orbit Inclination (degrees)+ Orbit Inclination (degrees)+ Equator Crossing (Local time )+ Equator Crossing (Local time )+ Period (minutes)+ Period (minutes)+ Repeat Cycle (days)+ Repeat Cycle (days)+ Perigee (km)+ Perigee (km)+ Apogee (km) + Apogee (km)
Orbit Details and URLOrbit Details and URL+ Type+ Type
+ Altitude+ Altitude+ Inclination+ Inclination
+ Period+ Period+ Repeat Cycle+ Repeat Cycle
Platform-based InstrumentsPlatform-based Instruments InstrumentsInstruments
Synonymous Platform NamesSynonymous Platform Names ( Short ( Short names, Acronyms, Abbreviations of names, Acronyms, Abbreviations of alternative names)alternative names)
Fields for Platform Descriptions in GCMD
Fields for Platform Descriptions in CEOS Earth Observation Handbook
Creation DateCreation Date( Date this description was created)( Date this description was created)
Revision DateRevision Date (Future date to review (Future date to review this description)this description)
Sample ImageSample Image( To provide URL with an image of the ( To provide URL with an image of the platform) platform)
IDN NodeIDN Node( Platform owner or Agency sponsor)( Platform owner or Agency sponsor)
Platform LogisticsPlatform Logistics
+ Launch Date+ Launch Date
+ Design Life+ Design LifeStatusStatus (i.e., Operational, Proposed, (i.e., Operational, Proposed, Being developed )Being developed )
INSTRUMENTS COMPARATIVE TABLE IN RESPONSE TO TASK COMPARATIVE TABLE IN RESPONSE TO TASK WGISS 23-10WGISS 23-10Note: Description information provided in the document task were created by Note: Description information provided in the document task were created by Olsen, L.M., Olsen, L.M., G. Major, S. Ritz, T. Stevens, M. Meaux, A. Restrepo, 2007 . NASA/Global Change Master G. Major, S. Ritz, T. Stevens, M. Meaux, A. Restrepo, 2007 . NASA/Global Change Master Directory (GCMD) Earth Science Keywords.Directory (GCMD) Earth Science Keywords.
Fields for InstrumentsFields for InstrumentsDescriptions in GCMDDescriptions in GCMD
Fields for InstrumentsDescriptions in CEOS EarthObservation Handbook
Instrument IDInstrument ID Instruments (Short Name)Instruments (Short Name)
Instrument IdentificationInstrument Identification(Associated to GCMD controlled (Associated to GCMD controlled
keywords)keywords)+ Instrument Category+ Instrument Category+ Instrument Class+ Instrument Class+ Instrument Type+ Instrument Type+ Instrument Subtype+ Instrument Subtype
Instruments (Long name)Instruments (Long name)Measurements and ApplicationsMeasurements and Applications
TypeType
Instrument DescriptionInstrument Description
Integrated Sensors Integrated Sensors (Associated to GCMD controlled (Associated to GCMD controlled
keywords)keywords)
Associated PlatformsAssociated Platforms(Associated to GCMD controlled (Associated to GCMD controlled
keywords)keywords)
MissionsMissions
Spectral/Frequency DomainSpectral/Frequency Domain+ Wavelength Keyword+ Wavelength Keyword+ Number Channels+ Number Channels+ Spectral/Frequency+ Spectral/Frequency+ Coverage/Range+ Coverage/Range+ Spectral/Frequency Resolution+ Spectral/Frequency Resolution
Technical CharacteristicsTechnical Characteristics Waveband (included channels)Waveband (included channels) Swath widthSwath widthAccuracyAccuracy
Fields for InstrumentsDescriptions in GCMD
Fields for InstrumentsDescriptions in CEOS EarthObservation Handbook
Instrument LogisticsInstrument Logistics+ Data Rate+ Data Rate
(bps/kbps/Mbps/Gbps/Pbps)(bps/kbps/Mbps/Gbps/Pbps)+ Instrument Start Date+ Instrument Start Date+ Instrument Stop Date+ Instrument Stop Date+ Instrument Owner+ Instrument Owner AgencyAgency
Online ResourceOnline Resource
Sample ImageSample Image
Creation DateCreation Date
Revision Date info Revision Date info
IDN Node (IDN Node (Platform owner or Agency Platform owner or Agency sponsor)sponsor)
AgencyAgency
Instrument Start DateInstrument Start Date
Instrument Stop DateInstrument Stop DateStatusStatus
Follow-up-up
Evaluate the addition of fields within the IDN Instruments description.
Swath width Accuracy
Continued addition of Instruments and Platforms from CEOS Handbook.
Online Demo to add Platforms and Instruments descriptions – previously viewed.
Suggestions and questions from WGISS-24 participants.
Node ReportsNode Reports
Bernd Ritschel - GFZ ISDC
AMDAMD
AMD Search Page Hits
Over 10,000 Hits in 2005
~ 14, 000 Hits in 2006
~ 40 % increase in 2006 from 2005
AMD Search Page Hits vs. Retrievals
Over 5,500 Retrievals in 2005Over 6,500 Retrievals in 2006
Over 10,000 Retrievals in 2007 (Jan - Jul.)
Launching of IPY 2007-2008?(over 500 unique IPs)
XXIX SCAR/JCADM Meeting
Note: Stats shown only represents 55% of total portal stats from July 2006 – August 2007
Hits by Country
Average visitors per day 19Total repeat visitors 1,235Most frequent domain: AustraliaVisit site between 9pm-10pmHighest volume day of week: Tuesday
Evolution of Controlled Vocabularies
Controlled Keywords Lists List is dynamic: keywords may be changed, deleted, or added.
Changes are implemented according to the Keyword Rules http://gcmd.gsfc.nasa.gov/Resources/valids/rules.html
In addition to Science Keywords, the GCMD maintains the following controlled keyword lists:
Services Keywords, Data Resolution, Locations, Instruments, Wavelength Keyword, Platforms, Orbit Type, Launch Site, Projects, Data Centers, URL Content Types, Chronostratigraphic Units (geologic time), IDN Nodes
Controlled Keywords Recent/Upcoming Changes
Keyword hierarchy extended to 5-Levels to assist in refinement of the search.
Recent Changes:• Atmospheric Chemistry
• Biological Classification (New Topic)• Ecological Dynamics (Revised)
•Terrestrial Hydrosphere (Renamed)
Upcoming Changes:• Atmosphere
• Human Dimensions• Oceans
• Biosphere
NEW
Atmospheric Science Keyword Revision (February - October 2007)
“The Long Journey to the GOLDMIND: Steps Towards Developing a Comprehensive Keyword Ontology”
2. Do the Research
3. Collaborate
4. Persevere
The GOLDMIND
1. Set a Goal
1. Set a Goal
A Complete Revision of the IDN Atmospheric Science Keyword list in 2008
Science Keyword Revision
“The Long Journey to the GOLDMIND”
2. Do the Research
• Review DIFs in the GCMD
• AMS Glossary Online
• Online publications through “Google Scholar”
• Consult data providers and experts
• Review Text Books:
• Critchfield, “General Climatology”
• Holton, “An Introduction to Dynamic Meteorology”
• Wallace/Hobbs, “Atmospheric Science: An Introductory Survey”
Science Keyword Revision
“The Long Journey to the GOLDMIND”
3. Collaborate
• Meet with project manager - Lola Olsen
• Consult Data Providers and Experts
Science Keyword Revision
“The Long Journey to the GOLDMIND”
4. Persevere
• Be willing to accept small steps towards a larger goal.
• Be patient when a trail leads to a roadblock and cheerfully backtrack and try a new trail.
• Never give up!
Science Keyword Revision
“The Long Journey to the GOLDMIND”
The Journey is almost over! The GOLDMIND is in sight.
Science Keyword Revision
“The Long Journey to the GOLDMIND”
GCMD Ontology Language for Discovery of Metadata INformation and Data
Reemergence of the Interop Reemergence of the Interop and Communicationsand Communications
Need for Communication
With increased usage, we must With increased usage, we must communicate:communicate: DIF/SERF modifications DIF/SERF modifications Keyword changes Keyword changes Software changes Software changes User Support/FAQ User Support/FAQ
Technologies to help with Communication:Technologies to help with Communication: Mailing ListsMailing Lists ForumsForums WikisWikis Content Management Systems (CMS)Content Management Systems (CMS) RSSRSS
CEOS IDN Standards Interoperability Forum
(INTEROP)
Moving to [email protected]
Continue to use GNU MailmanContinue as a moderated list
Possible future split of list:ceos-idn-interop (for general questions/discussions)
ceos-idn-keywords (for keyword announcements)
Communication Tools: Mailing Lists
Communication Tools: RSS Feeds
The IDN RSS feed allows users to view the most current Data Set and Service descriptions added to the IDN. Once a data set or service description is added, it will show up in your RSS feed.
Communication Tools: Communication Tools: Forums/Bulletin BoardsForums/Bulletin Boards
Communication Tools: Wikis
Communication Tools: Content Management
Systems/Collaboration
Communication Tools:Cyber
Infrastructure/Community
One Final Communication…. Final Communication….
The IDN/GCMD Strategic Plan
Thank You
[email protected]/Services Coordinator
[email protected] Leader
Special Thanks to the GCMD stafffor their assistance and contributions.