in43b-1691 next generation global navigation satellite ... · gnss processing system on 1 december...
TRANSCRIPT
MoreInformation/Feedback:•DataandproductsareacquiredaspartofNASA'sEarthScienceDataSystemsandarchivedanddistributedbytheCrustalDynamicsDataInformationSystem(CDDIS):
C.Noll,TheCrustalDynamicsDataInformationSystem:Aresourcetosupportscientificanalysisusingspacegeodesy,AdvancesinSpaceResearch,Volume45,Issue12,15June2010,Pages1421-1440,ISSN0273-1177,DOI:10.1016/j.asr.2010.01.018.
•ThestaffwelcomesfeedbackontheCDDISandinparticulartheideasexpressedinthisposter;contactPatrickMichael([email protected]).
1:NASAGoddardSpaceFlightCenter,Greenbelt,MD,USA2:Stinger-Ghaffarian Technologies,Lanham,MD,USA3:Science,SystemsandApplications,Lanham,MD,USA
CrustalDynamicsDataInformationSystemBackground:•IsNASA’sactivearchiveofspacegeodesydata,products,gatheringinformationfrom:
Ø GlobalNavigationSatelliteSystem(GNSS)Ø SatelliteLaserRanging(SLR)Ø VeryLongBaselineInterferometry(VLBI)Ø DopplerOrbitography andRadio-positioningIntegratedbySatellite(DORIS)
•Collectingdatafromover1000sitesworldwide.•Providingdataarchiving&distributionsupportforoverthirtyyears.•ThelargestCDDISusercommunitycomesfromtheserviceswithintheInternationalAssociationofGeodesy(IAG).•CDDIShasbeenanESDISDAACsince2007
ArchiveStatistics:•Archivesize:~17.5TB•Ingestrate:~9GB(100Kfiles)/day•Distributionrate:~500GB(~4.7Mfiles)/day•Data(L1,L1B),products(L2)andinformationaboutdataandproducts•Latenciesrangingfrom:multi-day,daily,hourly,sub-hourly,andsub-minute•Expecteddistributionin2016is1.5Bfiles,180TBandover140,000usersserved
OperationsOverview:•CDDIS’sprimaryoperationscanbecategorizedas:
Ø Providesystemstocollectdata& derivedproductsfromworldwidesuppliersØ Performqualitycontrolandmakedata&productsavailablefordistributionØ Collect&storemetadata&providedatadiscoverybothinternally&externallyØ Developvalued-addedreportsfromarchivecontentsforsciencecommunity
•CDDIS’sarchiveanddistributionhavebeengrowingatover30%annually•CDDIStransitionedtoallnewinfrastructureandimplementedanentirelynewGNSSprocessingsystemon1December2016.
GNSSProcessing:Overview:•GNSSaccountsfor95%ofallincomingfiles• ~30millionfiles/year• Deliveryratesupto100files/second•Latencyfromreceiptofdatafilestoavailabilityinthearchiveiscommonlylessthan30seconds•FilescannotjustbecopiedtothearchivebutMUSTbequalitytestedbeforemakingavailable
IngestOperations:•Allnewfileuploadprocess.SeeposterG41B-1013 andhttps://cddis.nasa.gov/About/CDDIS_File_Upload_Documentation.html•Allfilesundergoabatteryofsecuritychecksbeforeanyprocessingtakesplace.•24totalqualitycontrolchecks•5warningchecks– somecanbefixedinreal-timeExamplesinclude:
Ø IncorrectcompressionschemeØ IncorrectfiletypesubmittedØ Incorrectnamingscheme
•19“fatal”checks– errorsthatstopfileprocessingExampleincludes:
Ø UnknownfiletypeØ FilenamedoesnotmatchfilecontentsØ UnknownRINEXversionØ UnknownmonumentmarkerØ Virusinfectedfile
NextGenerationGlobalNavigationSatelliteSystem(GNSS)ProcessingatNASACDDIS
IN43B-1691
ABSTRACT: The Crustal Dynamics Data Information System (CDDIS) has been providing access to space geodesy and related data sets since 1982, and in particular, Global Navigation Satellite Systems (GNSS)data and derived products since 1992. The CDDIS became one of the Earth Observing System Data and Information System (EOSDIS) archive centers in 2007. As such, CDDIS has evolved to offer a broad range ofdata ingest services, from data upload, quality control, documentation, metadata extraction, and ancillary information. With a growing understanding of the needs and goals of its science users CDDIS continuesto improve these services. Due to the importance of GNSS data and derived products in scientific studies over the last decade, CDDIS has seen its ingest volume explode to over 30 million files per year or morethan one file per second from over hundreds of simultaneous data providers. In order to accommodate this increase and to streamline operations and fully automate the workflow, CDDIS has recently updatedthe data submission process and GNSS processing. This poster will cover this new ingest infrastructure, workflow, and current operations.
MetadataOperations:•Allincomingfilesareprocessedforbothfileandcontentmetadata•68totalmetadataparametersaregatheredforeachGNSSfile
Ø Filemetadata– 16parametersperfileØ Contentmetadata- 52parametersperfile
•Content& filemetadataarepublicallysearchableinmultipledatadiscoverytools•MetadataareprovidedtotheEarthObservingSystemDataInformationSystem(EOSDIS)throughtheirCommonMetadataRepository(CMR)fordatadiscoveryacrossall12DistributedActiveArchiveCenters(DAACs).•MetadatawilleventuallybemadeavailabletotheGlobalGeodeticObservingSystem(GGOS)fordiscoverybythegoedetic community
ReportingOperations:•Dataproviderreports
Ø ErrorreportsØ Trendingreportsovertime
• SystemlevelreportingØMetricsgeneration• Ingestrates• Dataprovidererrorrates
Ø Trendingreports•Sciencecommunityreports
Ø GNSSstatusreports
FutureCapabilities:•SystemwilleventuallyincorporateprocessingofSLR,DORIS,andVLBIdatafiles•Additionalcontentmetadatacollections•Automatedingestreportsbydataprovider•Betteranalysisandhandlingofduplicatefiles•Webinterfaceforviewingreal-timeprocessingstatistics•Additionalgeodeticcommunitydirectedreports/summaries•Transitionentireprocesstoanagilesoftwaredevelopmentpath
1PatrickMichael ([email protected])1CareyNoll([email protected])2JustineWoo ([email protected])3RebeccaLimbacher ([email protected])
Formoreinformation:https://cddis.nasa.gov
Files delivered?
Yes
No
End
General File Check
Fatal Errors?Yes
Check File Content and Collect Metadata
No
Fatal Errors?
Push errors to DB
Yes
Warnings?
NoPush
warnings to DB
Process the GNSS Files
Yes
Push Recorded Metadata to DB
Require Additional Processing?
Move Files to Archive
Copy to Additional Processing Directory
Yes
No
Database
File/Content Metadata
Tables
Error Tables
Archive
Error Report:- Duplicate Files
- Fatal errors- Warnings
GNSS Status Files
Flat Files
<driver software>
Check User Directories for FilesCheck Additional Processing
Directory for Files
Files in directory
?
Yes
No
Perform Additional Processing
Files Created?
No Yes
<report software>
CDDISOperationalOverview
SpaceGeodesySiteLocations
GNSSWorkflowDiagram
340795
24464
3642
Pass Warning Fatal
UserReportonDuplicateFiles
ProcessingReports,MetricsandGraphics•Metricsonindividualdataproviders•Systemwidestatistics•Metricsonfilequality
Ø BydataproviderØ Byfiletype
•Metricsonnumber/sizeofincomingfiles
TotalFilesProcessedReportOver6DayPeriodPass=FilesSuccessfullyProcessed
Warnings=FilesCorrectedDuringProcessingFatal=FilesNotAbletoBeProcessedDuetoErrors
TotalFileProcessingOveraSixDayPeriod