asprs hurricane katrina lessons learned brenda jones and ... · browse requirements browse examples...

U.S. Department of the InteriorU.S. Geological Survey

ASPRS Hurricane Katrina ASPRS Hurricane Katrina Lessons LearnedLessons Learned

Brenda Jones and Ron Brenda Jones and Ron RistyRistySAIC*SAIC*

USGS Center for Earth Resources Observation and ScienceUSGS Center for Earth Resources Observation and Science*Work performed under U.S. Geological Survey contract 03CRCN0001*Work performed under U.S. Geological Survey contract 03CRCN0001

EROSCenter for Earth Resources Observations and Science

Our MissionOur MissionScience:Science:To promote and conduct applications, users, and knowledge of land information to better understand our planet

Data Access:Data Access:To ensure that scientists, researchers, businesses, decision makers, and the public have ready access to land information

Data Archives:Data Archives:To safeguard and expand the national archive of remotely sensed land data

Current Data Ingest FlowsData Supported - Restricted

IKONOSQuickBirdOrbimageSPOTRadarsatIRSHomeland Security Infrastructure Program dataEnvisatGIS DataGraphicsDisaster Monitoring Constellation

Data Supported - PublicLandsatAsterHyperionAliAerial data

DOQNAIPNOAA AerialUSACE/FEMA

NEDNHDDeclass DataLidarGIS DataGraphics

Quickbird/Ikonos Data Flow

Aerial Data Flow

Aster Data Flow

SPOT Data Flow

Current statusEROS is investigating the re-engineering of its Emergency Response data ingest routines

AutomationShould this task be distributedHow much can vendors provideNeed users to define requirements

Convert data to standard and useable format Create browse

Provided in some cases, but inconsistentControlled and reliable format if automated

Create compressed data

Current status (continued)

Shapefiles/KML files EventData TypeExport files by area of interest or extent

Metadata Requirements for CSV, XML, or EXCEL spreadsheet Define minimal FGDC compliant metadata

Package data for deliveryProvide tiered web access capabilities

Data format requirementsDelivery data format

Datatype - 8 bit, 11 bit, 16 bitFormat – GeoTiff, NITF, flat binaryProcessing level – raw, geo-referenced, orthoBands – color, NIR, pan, merged, everythingOthers?

Browse requirements

Browse examples and analysisBrowse generated from the 8-bit image with consistent resolution down-sampling for each datasetSome Imagery is not consistent size (X by Y pixels; real estate)

Therefore, browse will also be various sizesGoals:

Fit on users screenFile sizes reduced for quick viewsRetain reasonable resolution for minimal detail reduction while earning benefits of a browse

Browse requirementsAnalyze browse for usability

What browse are needed?Multi-spectral true color Pan bandNIR

Previously made available any browse packaged with the productNow recommending we create the browse in most casesWhat is appropriate data size reduction?Determine common acceptable format for each dataset

Quickbird Browse ExamplesRGB Multi-spectral bands true color browse:

Notice the slight size differenceBoth have same resolution down-sampling applied

This is about 61m resolution

IKONOS Browse Examples

RGB Multi-spectralbands true colorbrowse:

Notice the sizedifferenceBoth have sameresolutiondown-samplingapplied

This is about 80mresolution

Landsat Browse Examples

720m true colorbrowse

Simply reduced size(and resolution) ofexisting browse createdby the EROS LAMsystem

Aerial PhotographyBrowse Examples

500x572 pixels

Compression requirements

Type of compressionMrSIDJpegJpeg2000ECW

Analyze statistics for acceptable compressionDetermine compression that is acceptable for all

Can automate if a base compression that will work for all is foundIf other options for compression are needed, may be able to provide real-time, but this introduces wait time for users that must be balanced

Compression overview

MrSID compression analysisLossy vs. lossless examples; quality loss analysis8-bit vs. native pixel depthStatistics regarding:

File size reductionTime to compress (real-time generation viability)Are World files desired with the Sid images?

Mr Sid Statistics

Both resolution reductions creates browse generally <1% of the original GeoTIFF full resolution browse file size

Quick download time to a user interface viewerExample scenes file sizes:

Image Type

Original product size

Mr Sid Lossless size

Encoding Ratio

EncodingTime

Aerial 323 M 187M16M4MB5.8M

2M 0SAerial 323 M

Lossless (1.71:1)Lossy (20:1)Lossy (20:1)

6M 4SQuickbird 81 M 1M 29S

Lossy (20:1)Ikonos 239M 1M 23S

Aerial PhotographyMr Sid Examples

Original Image Mr Sid Lossy

Aerial photography – lossy Mr Sid

Original Image – 4x Lossy – 4x

Lossy – 8xOriginal Image – 8x

Aerial Photography – Mr Sid

Original Image Mr Sid LossLess

Aerial Photography – lossless Mr Sid

Original Image – 4x Lossless – 4x

Original Image – 8x Lossless – 8x

Mr Sid results

Lossy vs LosslessImagery looked very similarSize of data

Lossless size approximately ½ size of original imageLossy – 20 times smallerSmaller size may be more advantageous for first responders

Encoding timeVarious between 1-5 minutes per imageShould be able to include this into the normal production flow as an additional product

Requirements for Managing DataEach data type will require a unique index to reference the image

2 Character data type 6 Character K/J or Path/RowYYYY - YearMM - MonthDD – DayHH -- HourMM – MinuteSS – SecondP/M – Pan or Multispectral+/-ddd_ddd+/-ddd_ddd - centerpoint00 – Unique number for duplicates or frames covering the same area

Requirements for Managing Data

Additional processing may be requiredMosaicsPan mergesOrthorectificationGeo-referencingOthers

The data will need to be grouped for downloading

TarTar-gzZip

Requirements for Managing DataMake data available for FTP and HTTP downloadGenerate status graphics

Jpgspdfshapefileskml

Pull or copy all data by event for customersCopy data to firewire or other media for delivery to customersCapture metadata and generate minimal FGDC metadataDump metadata in a comma delimited, csv, or XML format

Requirements for Managing DataStore the metadata in a database for managing the dataStop light web page to understand the current data flow, predicted taskingsCustomer notification of new data (email, web page, RSS service)Viewer integration…GIS files tied to the event

Data Management requirementsArchiving

Data will move from the disk cache to 9840 tape on siloAfter 60 days of no use, the data will be moved to the long term archiveData received on firewire will also be added to the long term archive strategy

Event specification requirementsName of eventLat/Long extentDate of the eventAny data loaded to the archive will be related to this event based on the date loaded and the extent

Data download requirements

Users unable to download data with current flows – need user requirements

Need to determine bottleneckSilo?Users network?Size of image?User interface?Others?

Data download requirements

Tiered web accessSimple web page with different access capabilities

ShapefileKML fileNon-ims indexmapIMS lite (vector based) with a tile toolFull IMS

Google EarthOthers?

Contact info

Brenda JonesWork – 605.594.6503Cell – 605.321.3995Email – bkjones@usgs.gov

Ron RistyWork – 605.594.6969Cell – 605.321.3698Email – risty@usgs.gov

Questions?Questions?