11 noaa’s climate database modernization program- data rescue activities program manager- tom ross

11

NOAA’s Climate Database Modernization Program- Data Rescue Activities

Program Manager- Tom Ross

22

Climate Database Modernization Program

• ←

The Climate Database Modernization Program (CDMP) goal is to preserve and make major climate and environmental data available via the World Wide Web

The CDMP supports NOAA’s data stewardship through all of its activities

33

Climate Data Base ModernizationOutcomes & Benefits to the Nation

Outcomes• Improved access and digital conversion of data stored on aging paper, film & obsolete digital media• Provides partnerships and jobs through private sector companies in 4 states

Benefits to the NationHuman Dimension: Reduced loss of life & impacts due to environmental events

Business Dimension: Enhanced opportunities and socio-economic gains by considering climate information in decisions

Policy Dimension: More effective government policy by appropriately using climate information in decisions

Tenth year of the program (began 2000)

44

Over 11 terabytes of climate data now digitized‣ 53 million weather and environmental images online‣ Hundreds of millions of records digitized now online ‣ International data access and rescue activities‣ 86 current NOAA climate/environmental rescue projects

July 1st, 1842 hourly weather data from Washington, DC, imaged and

digitized through the CDMP Program

Imaged Records Example:Glacial Pairs – Muir Glacier, Alaska

1941 2004

Climate Database Modernization Program: 2000-2009Climate Database Modernization Program: 2000-2009

Keying and Imaging the data increases data accessibility and data integration- Work must be done by CDMP contractors in KY, MD, WV

55

CDMP Proposal Process has NOAA Working Together on Data Rescue and Recovery

The proposals judged against the following criteria:

1. Supports NOAA’s Strategic Goals2. Contribution to improved data access & rescue3. Value to climate community4 General merit to overall program5. Cost effectiveness6. Ease of digitization by the contractors

Call for papers- Issued Each July/AugustData Access Workshop – Held November each Year

60 Proposals were submitted for FY 2009, 53 in 2006 but delayed until FY 2008 funding


66

NCDC ArchivesPaper Archives Room

77

CDMP Digitization Partners

IMC handles paper scanning, COOP scanning from microfiche, and priority scanning

HOV Services handles bound records, publications, and fragile records

SourceCorp handles incoming records, COOP/SAO keying projects

88

35mm & 16mm Film 125,129 Rolls

Microfiche860K fiche containing 51 million pages

Manuscript / Autograph*55 Million Pages stored in 120,000 boxes

* Located at Asheville; additional paper records located at the Federal Records Center in Georgia that will be inventoried and prioritized for digitization

Percent digitized

(Keyed or imaged)

50%(55 million)

1.7%(2,105 reels)

1.0%(8,600 fiche)

Non-Digital Data Archive

99

Contents of the Archives: Micrographics

1010

Fragile, bound document scanningCDMP contractors have specialized equipment to scan difficult media while preserving its physical state.

1111

Large Document ScanningIn addition to standard document sizes, contractors can also handle large paper records, such as maps and charts.

1212

What’s being digitized at present?“High Volume” Projects, 2009

SWO daily forms: 897,961 images 3,592 per day

SWO 1001 monthly forms: 19,368

documents 77 per day

COOP microfiche: 369,625 images 1,479 per day

1313

Typical Camera and Imaging Setup for International Projects

High tech digital cameras and camera stands are set up in a well lit environment- sponsored by NOAA/NWS

Technicians are trained to image the records using the camera equipment they check for quality, exposure and completeness

Digital jpeg images are stored in the camera’s hard drive then transferred to the computer

Images on the computer are then written to CD-ROM or DVD and sent to NCDC for storage and keying

Keying format developed at NCDC based on data captured from host country

Data must be keyed in U.S. by CDMP contractors

End to end process can take 1-3 years depending on complexity and amount of data to be imaged/keyed

A technical leader responsible for each step in the process at each international location is critical !

International Activities


1414

NOAA & the Uruguayan Meteorological Service have entered into a cooperative project on data rescue In coordination with the U.S. Embassy. The U.S. National Oceanic and Atmospheric Administration (NOAA) donated digital equipment used for the project.

72,000 images received from the Uruguayan Meteorological Service and the Uruguayan Navy

7 synoptic stations (4 civilian and 3 military) keyed with observations as early as the 1930’s up to 2005 for a total of 9 million keystrokes.

Total imaging/keying for on-going project involves data from over 30+ stations totaling nearly 1,000,000 records

Keying data for Punta Arenas, Chile data record begins in 1896

Integrate data into NCDC global databases

International Activities-Uruguay and Chile 2004-2008


1515

CDMP African Upper Air (Pibal) Project

32,000

30,000

12,600

5,400

51,500

3,500 20,000Kenya

Malawi

Mozambique

Niger

Senegal

Tanzania

Zambia

Over 150,000 records imaged under CDMP

1616

Making Data User Friendly and Accessible

The index on the CD-ROM will be the main point for searching for related data.

The index page provides links to all available data types (both images of recorded data and actual keyed data) associated with the CD-ROM

• Station:67423 Day/Mo/Year Time:07/11/1970 04Z• Keyed Image ID: CDMP06MA\MA0001\0009.jpg• Source Image ID: .\MA0001\1\P6160006.jpg • • COMPUTED OBSERVED• TIME HGHT Azimuth Elev DIR SPD DIR SPD• MINS FEET DEG DEG DEG KTS DEG KTS

• > 0 0 130.0 5.0• 1 500 327.2 21.2 147 13 0 0• 2 1000 337.9 22.4 170 12 0 0• 3 1500 239.7 24.9 165 8 0 0 • 4 2000 342.7 26.1 174 9 0 0• 5 2500 337.4 25.5 140 12 0 0• Note: > Marks the Beginning of each Record in the Launch

• The highlighted .jpg has a link connecting it to the original image from which the data was keyed

The data above is an upper air pibal, due to the limited wind data available a keying format was developed to key the azimuth/elevation data along with the surface wind data, found in the header section. The wind data, to the middle right of the image, we found, was often determined by the average of multiple levels of wind data.

1717


1818


One of the earliest forms with detailed hourly data

1919

Selected NOAA ProjectsClimate Database Modernization Program

Hurricane Celia 1962- Hurricane Wallet Project

2020

Electronic Hurricane Wallet Project

Tropical Cyclone 'Storm Wallet' Electronic Archive

After the dissipation of every tropical cyclone occurring in the Atlantic and eastern north Pacific basins, all of the data and relevant materials related to that cyclone are collected by the NHC staff. The materials are placed in a "storm wallet" which currently takes the form of an expandable binder, or series of binders. These storm wallets have proven to be extremely useful in the post-analysis of many tropical cyclones, both near-term and in some cases, decades later.

The procedure for storing this data dates back to well before the routine use of computers in the office environment. In the Atlantic, the wallet series begins in 1958 and proceeds continuously through the present. In the eastern north Pacific, wallets begin in 1988, the year in which operational responsibility for that basin was assumed by NHC.

NHC Central Pacific is developing the same “Storm Wallet” archive

2121

Electronic Hurricane Wallet ProjectHurricane Camille 1969

2222

Typhoon/Hurricane Database Keying task

New Delhi Tropical Cyclone Keying Project

2323


Tropical cyclone best track position and intensity from New Delhi, India need to be keyed. Photocopies will be provided of data provided by New Delhi. The data are primarily made up of tables of cyclone position and intensity.

Format

The data are requested in a spreadsheet format (e.g., MS Excel). This format was chosen since the data originates as tables. The data should be on one worksheet beginning with the earliest data and working chronologically downward with one blank row between cyclones. The column names are only required to be present once at the top of the worksheet.

Columns

The following columns are requested. They are present in most tables. In the event they are absent, the column should be left blank. It is imperative that the data for each storm have the same columns to facilitate inputting the data into analysis programs. The columns are:

Date

Time (UTC)

Latitude (lat.)

Longitude (lon.)

CI No [or “T. No.”]

Central Pressure (hPa) [or “E.C.P.”]

Maximum Surface Wind (kt) [or “E.M.S.” or “E. Max”]

Pressure Drop (hPa) [ or “Δ P” ]

Grade (text)

Outermost closed isobar (hPa)

Diameter / Size of outermost close isobar

Remarks

2424


Caveats

Times should be made consistent as HHMM (e.g., some only report the hour (18) which should be adjusted by zero-filling (1800)).

Landfall is often noted by text in the table taking up an entire line. When keying, this should be moved to the “Remarks” column for the time referenced.

Example: “The system crossed the Orissa coast … around 1500 UTC” should be put in the remarks for the 1500 UTC observation

There is often text following a table describing how the system dissipated. This should be recorded in the Remark column for the last reported position of the storm.

Tables which span a page should be included in one table in the keyed file.

Table numbers and caption are not required.

Damage estimates and other information above the table caption should not be keyed.

In the tables, the date is recorded only on the first observation of the day, with other cells left blank. It is requested that no blank cells be left in the date row, but be filled with the actual date by entering the date in each row.

Systems which are over land do not generally have intensity estimates and are often denoted with “over land” in some columns. In these cases, the text “over land” should be entered in the remarks section with affected cells left blank.

If some columns are missing (e.g., the ΔP column in 1996), it should be left blank.

2525

• WWII Royal Navy Ship logs from 1938-1947 were microfilmed and scanned at the British National Archives. The records have been keyed by CDMP and will be QA/QC’d then merged into the International Comprehensive Ocean Atmosphere Data Set (ICOADS).

•Expanded to East India Company Logbooks in 2008- Joint UK Met Office Project


International Activities

2626

EIC visualization visualization by Philip Brohan, UK Met Office - Hadley Center

http://imma.appspot.com/interfaces/imma_edit_test.html

2727

Voyage of the EIC ‘William Fairlie’ from the UK to India to China and back to UK, 1824-1825

2828

Voyage of the EIC ‘Windham’ from the UK to China and back to UK, 1816-1817

2929

Voyage of the EIC ‘Astell’ from the UK to India and back to UK, 1824

3030

Remarks from voyage of the EIC ‘Thomas Coutts’ 1818. Although the remarks are in regards to a nasty winter North Atlantic extra-tropical cyclone, additional logbooks from this collection may contain similar entries that may give tropical cyclone information for the Indian Ocean (from remarks and/or from barometric pressures).Notice the pressure from the Barraud’s barometer increasing 0.89” (30+ mb) in 4 hours.

3131

The Foreign Data LibraryThe Vast Unknown…

3232


Both recent and archived publications are well-indexed

The problem is with the actual data in the library

Larry Nicodemus is working to create a better inventory and also to identify data that should be keyed

3333


The best index we have for our foreign data:

3434

Summary

Current efforts Surface weather observations

Cooperative observations

Tropical Publications-Based on Conference Input !

Next few years

Publications for customer service and IPS

Charts

Historic journals

Down the road

Barograms, thermograms, radar

Saving data only available on film

3535

• Password secure system access to Data Cabinets with various data (observations, pubs, climate journals, manuals)

•Login http://edads.imcwv.com/default.aspx

• e-mail: [email protected] for login and password

•Mention ‘Tropical Cyclone Project”

•Other NOAA Web Sites (e.g. NOAA Central Library)


3636


Questions ?

Contact Info

NOAA/ Climate Database Modernization Program

151 Patton Avenue- Room 516-D

Asheville, NC 28801

Tom Ross- Program Manager

828 271 4499

[email protected]

http://www.ncdc.noaa.gov/oa/climate/cdmp/cdmp.html

11 noaa’s climate database modernization program- data rescue activities program manager- tom ross

Documents