Luquillo LTER Information Management

Luquillo LTER Information Management2009 MidTerm ReviewEl Verde Field StationJune 4, 20091introductionHistory ProgressChallengeVision

Present the progress throughout the past 3 years taking into consideration the course of events of the past 20 years which have delineated the path that LUQ IM has followed.

Present what the main challenge to achieve present goals andThe vision that have emerged as a product of all the processes experienced as we have made this progress to achieve our goals.21989 on1995 on2001 onevents in the lterNeed to team up and publish metadata

Organizing, cataloguing; Develop LUQ documentation standards and protocolsNeedto documentNeed to have searchable data in the web siteDocumenting and publishing the data on our first webDecadal Plan and the adoption of the EML standardHistoryIn our history of 20 years as an IM:First task was organizing and cataloguing dataAt this time we catalogued in two different catalogues mostly legacy data, specially those from Odums time. - a pre-LUQ LTER data catalogue that we called LEFDSET - an LTER data catalogue that we called LTERDBAS;

Also at this time we, started and finished developing our first version of metadata standards.

--BIG FIRST LESSON LEARN: WE NEED TO DOCUMENT OUR DATA

Around 1993-5 we received the mandate to share data on the internet, so we published our first web site (from 1988 to early 90s all the telecommunication infrastructure had been by our Pis with the help of Network people). At this time we had part of our LTER data documented with the LUQ documentation standards.

BIG LESSONS LEARN: -DOCUMENTING IS TIME CONSUMINGRESEARHCERS NEED ASSISTANCE TO DO THIS -DATA SHOULD BE ACCOMPANIED WITH THE METADATA WHEN PUBLISHED IN THE WEB, OTHERWISE THE CALLS ARE ENDLESSASKING FOR VARIABLES ATTRIBUTES, METHODOLOGY, ETC.

In the decade of synthesis, once again Information Management was included in the general scheme of the LTER program but this time its goals was more explicitly described and the web site became the main scenario for the system. In addition, the network adopted a metadata standard, EML

LESSONS LAERNED: - INVESTIGATORS NEED PRIVATE SPACE TO SHARE DOCUMENT - THE LUQ IMS NEEDS TO GO FROM A LOCAL FRAMEWORK TO A WEB-DATA BASED SYSTEM, IN ORDER TO MEET THE DEMANDS OF THE DECADAL PLAN AND ITS CYBERINFRASRUCTURE PLAN DESCRIBED IN IT.3Historyluq LTER online data sets

Main points here:Difference between catalogued and online: - Before 2000 : Legacy data catalogued; limited information was extracted from it; LTER data was documented with the developed standards.In 2000 : we archived the legacy data and saved the file cabinets that we still keep.

Between 2000 to 2002: - of the 115 catalogued : - LTER data sets: 35 were documented and online; 3 where merged into other data sets by the scientist - NON LTER information that could be extracted from the hardcopies was too scarce, previous to LTER scientists not accessibleToday (2009):two already catalogued metadata is being revised by IM: Charlie Hall (David Murphy) (Katherine Smith dissertation, Katherine is. Pringle students)- two CTE data sets, Treatment and Litterfall, are ready and pending for publishing-ONE IS PAPER ONLY SOIL Char of LFDP--Of these 127 online: 39 are long term, 1 is from a Costa Rica Project, STREAMS,

42005 onMetadata standards and developed ; data sets catalogued; Data filing protocol established1989-1991LUQ EMLLevel 32001-20051995 on20 years as a learning and evolving processBill MichenerJohn PorterMarsh WhiteInigo San GilLUQ EML Level 5 QChistory1989- on Management of dataBEFORE THIS DESCRIPTION OF EML LEVEL

Establishing the basic elements of the LUQ Information Management System (IMS):As we developed the metadata standards we also catalogued the existing data sets.Cataloguing means to : -File the data; by that time it meant in a file cabinet, now it means in the computer.-Enter basic information of the data set in the database: +In 1989 we used DBASE III as our RDBMS; we migrated to Paradox in 1992 and + it has been used as the official LUQ IM software ever since.-The data set assumes as the official ID number, the record number of the row in which its information is entered in this Paradox table. Each record number of the data set will never belong to any other data set; so the official number of each data set is LTERDBAS #n, where n is the number of the Paradox record in which the main information of the data set is entered. Main information includes Identifier, Project title, start and ending dates, list of investigators, and keywords, among other information.

Metadata forms and guidelines to complete them were made available first as hardcopies in our annual reports and then online, in our web site.

Many things have to be done at the local level in order to be ready for cyberinfrastrucure and EML. At LUQ we dedicated ourselves to: - Manage the data files such that each column of data contained only one data type - Convert each data file into a Paradox table - Clean and enhance the quality of the metadata such that the general information related to project and the data set is readily available - Make sure that all variables are well defined and documented

==============So with that lessons learned what did we responded:First 3 years we dedicated to: Develop Metadata standards 1991: first version of LUQ Documentation (or metadata)

-1995: We published our first version of a web site with 30 data sets on line; It was then when we decided that no data would be published without metadata

so we started to enhance our metadata standards which eventually lead us to EML , which in turn help us to enhance the metadata even further

FOR AT LEAST THE JUNE 2 PRESENTATION: ALL THESE HAVE BEEN EXAMPLES OF COLLABORATION WITH OTHER SITES IMS AND SINCE 2003 LNO STAFF!==================5Dataset Design -PI -Information ManagerData CollectionData EntryMetadata PreparationQuality Control and AssuranceReviewData Publication on WWWRevisionRevisiondata filing protocolhistory ALL DATA SETS ARE CONVERTED TO A RELATIONAL DATABASE MANAGEMENT SYSTEM TABLE

AT A LOCAL LEVEL, THE LUQ IMS HAS THE CAPABILITY OF QUERYING RECORDS OF DATA OUT OF THEIR DATA BASES AND APPENDING OR JOINNING TABLES TOGETHER

HISTORYMANAGING DATAShow new web sites searching capability pubs by MarshMetadata online forms designed for LUQ by InigoAchieve a true integration of the different soft elements of the LUQ IMS: data sets, publications, and people profiles with the use of EML while making them available to public thorugh the Internet

7Plans for the LUQ IM

Progress new mechanisms to allow investigators to remotely deposit their data in the LUQ IM system. Achieved for special groups in a private web site; new web site will provide forms to upload both data and its metadata.

progress2006 PROPOSALS IM GOALS

LUQ LTER Intranet functionality

9Plans for the LUQ IM

ProgressA scroll down Web site map with links to other U.S. LTER sites, and other Best Web Practices features; Will be available in the new web site; Most of the Best Web practices features implemented in the current web site

progress2006 PROPOSALS IM GOALS

Model Web site 11Plans for the LUQ IM

ProgressTranslation of the already existing metadata to EML level 5 standards to allow data as well as its already harvested metadata retrieval by the LTER Network Offices data warehouse: Metacat Approximately 87% of the online metadata is in EML, 59% of which is in Level 5 QC

progress2006 PROPOSALS IM GOALSprogressLUQ EML METADATA PACKAGES DEVELOPED

POINTS:From 2003 to 2005: --Production of LUQs EML Level 3-4 started in 2005From 2005 to 2007-Two data sets were merged into two others-Additions of Level 3 stopped up to until 2008 when ~16 where added-Production of LUQs EML QC Level 5 started in 2005 --A -a LNO-LUQ Effort2006: Visit from Inigo development of scripts-collaborative work that served as a model to other sites13Plans for the LUQ IM

ProgressCompletion of a database containing GIS referenced list of all LUQ project sites (LUQ GIS database)Simplified online version is being implemented in the current web site

progress2006 PROPOSALS IM GOALS

LTER Web Site DesignLTER Web Site ContentLTER Network IdentityLOGO AND IDlink to siteDBThe LTER network domain, luq.ternet.edu,

Navigation Repeated navigation menu Hyperlinked section in each main menu pageData accessHome page link to data set catalogSignature data setsLocal data sets search: LTER core areas, LUQ CategoriesLink to the LNO MetaCat system: owner and by keywordProminent link to network-wide databases: ClimDB/HydroDB, SiteDBProminent link to EcoTrendsprogressNETWORK STANDARDS Links reach content in few mouse clicks All recommended links the home page in place: contact information, site bibliography: contact information, Searchable site personnel directory, Searchable site bibliography, searchable citations, progress reports and proposals, news, Network resources, research description, education and outreach Challenges:-Providing the capability to achieve real integration of information in the INTERNET

-Providing a searching mechanism in the web site that will allow users to obtain the data they are looking for plus related data sets and informationTasks:- Complete in time the translation of all of our metadata to EML Level 5 and develop the mechanism to automatically produce this level of metadata for new filed data sets- Produce sets of keywords that will relate all the information in our web site

challengeCompleting the goals for the LUQ LTER 4Show new web sites searching capability pubs by MarshMetadata online forms designed for LUQ by InigoAchieve a true integration of the different soft elements of the LUQ IMS: data sets, publications, and people profiles with the use of EML while making them available to public thorugh the Internet

17

Our Vision

CyberInfrastructure:Describes the new research environments that support: advanced data acquisition, data storage, data management, data integration, data mining, data visualization and other computing and information processing services over the Internet. In scientific usage, cyberinfrastructure is a technological solution to the problem of:efficiently connecting data, computers, and people with the goal of enabling derivation of novel scientific theories and knowledge.

This new concept of web site:EML based supports Internet data, information, and computing servicesSuports data integration by the use of a vocabulary based infrastructure that links different parts of the information : data sets, publications, and pleople, and research locationSupports synthesis by providing related information when performing searhces

18