1

Build EPA’s BP Oil Spill Data Tools in the Cloud

Brand NiemannUS EPA

July 10, 2010http://semanticommunity.net

Disclaimer: These slides do not reflect the views of the U.S. Environmental Protection Agencyand does not constitute endorsement by the EPA of the standards or products mentioned.

2

Overview• The Challenge• The Socrata and Google Earth Programs• The Expert and His Advice• The Cloud Tools• The Inspiration• The Data Sources• Other Sources of Data• The Process• The Results• Comments• Acknowledgements• References• Footnotes

3

The Challenge

• BP Oil Spill Data Tools:– As part of our response to the ongoing BP oil spill in the Gulf of

Mexico, we’ve been analyzing environmental conditions, including air, water, sediment, and oily wastes.  Right from the beginning, we’ve followed our open government approach, which means providing the data as soon as we have it.  We’ve posted data in CSV files, a format that people could download and open in a spreadsheet.  We’ve also provided printable PDF tables and summaries on our BP spill site.

– But we knew that people needed other ways to get the data, and we’ve been working on several options.  We launched Google Earth a few weeks ago, and today we launched Socrata.

– Source: Jeffrey Levy, EPA’s Director of Web Communications, Greenversations, July 8, 2010.

http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Challenge

4

The Challenge

• BP Oil Spill Data Tools (continued):– I’m excited to share these tools, but we can always

improve them.  In the comments section of this post, please give us your suggestions.  Some examples:

• Filtered views to provide beyond just what we’ve detected• Different ways of sorting the data• Mashups (ways to combine the data with other information)

– And if you create your own views or download the data and produce interesting stuff, let us know!

– Source: Jeffrey Levy, EPA’s Director of Web Communications, Greenversations, July 8, 2010.

http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Challenge

5

The Socrata Program

See next twoslides

http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Socrata_and_Google_Earth_Programs

6

The Socrata Program

http://www.epa.gov/bpspill/socrata.html

7

The Socrata Program

http://www.socrata.com/profile/U-S-Environmental-Protection-Agency/yahm-crqu

See next slide

8

The Socrata Program

Download data

http://www.socrata.com/Government/EPA-Air-Sampling-Response-to-BP-Oil-Spill/498c-fwrs

9

The Socrata Program

http://www.socrata.com/Government/EPA-Air-Sampling-Response-to-BP-Oil-Spill/498c-fwrs/about

Also downloaddata

Metadata

10

The Google Earth Program

See nextslide

http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Socrata_and_Google_Earth_Programs

11

The Google Earth Program

http://www.epa.gov/bpspill/popups/googledatapopup.html

Click to downloadFile in Google Earth.Note: Need to installGoogle Earth client first.

http://earth.google.com/

12

The Google Earth Program

http://www.epa.gov/bpspill/mapfiles/BPGulfSpillResponse.kmz in Google Earth

13

The Expert and His Advice

• Edward Tufte Presidential appointment announced by White House, March 5, 2010.

• Tufte Comment on iPhone interface design: Better to have users looking over material adjacent in space within our eyespan rather than stacked in time. This is especially the case for statistical data, where the fundamental analytical task is to make comparisons. Also see page 159 in the book reference below.– Edward R. Tufte, Beautiful Evidence (2006), 

Graphics Press LLC.

http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Expert_and_His_Advice

14

The Cloud Tools

http://cloud.mindtouch.com/

15

The Cloud Tools

http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools

16

The Cloud Tools

http://spotfire.tibco.com/

17

The Cloud Tools

http://ondemand.spotfire.com/public/Help/index.htm

18

The Inspiration

H1N1 Spread Courtesy of TIBCO Spotfire. See Web Player.http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Inspiration

19

The Data Sources

http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Data_Sources

20

Other Sources of Data• NOAA interactive map tracks Gulf oil spill • Online mapping tool allows public to follow closures, spill trajectory• The public can now track Deepwater Horizon-BP oil spill recovery data online via a near-real-time

interactive map at a new Web site created by the National Oceanic and Atmospheric Administration.

• The GeoPlatform Web site, launched June 15, includes regularly updated geospatial data from several federal and state agencies on the oil spill trajectory, closed fishery areas, impact on wildlife and Gulf resources, daily position of research ships, and affected shorelines.

• Data published on the site is received from NOAA, the U.S. Coast Guard, the Environmental Protection Agency, the U.S. Fish and Wildlife Service, the U.S. Geological Survey, the Homeland Security Department, NASA and several states.

• “This Web site provides users with an expansive, yet detailed geographic picture of what’s going on with the spill,” Jane Lubchenco, NOAA administrator, said in the news release June 15. “It’s a common operational picture that allows the American people to see how their government is responding to the crisis.”

• NOAA worked with the University of New Hampshire's Coastal Response Research Center to develop the Web-based geospatial platform designed for federal and local response activities and adapted for public use.

• A separate public Web site, Deepwater Horizon Response sponsored by federal agencies, has been operating for several weeks offering news, announcements and information about the disaster. It is operated by the Deepwater Horizon Unified Command, which consists of DHS and the Defense and Interior departments, as well as other federal agencies, BP and other private entities. 

http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#Other_Sources_of_Data

21

Other Sources of Data

http://www.geoplatform.gov/gulfresponse/

22

Other Sources of Data

http://www.geoplatform.gov/gulfresponse/

23

The Process

• The Basic Steps:– Inventory Data Sources and Plan Application– Prepare and Import Data and Metadata– Implement Layout and Analytics– Add Bookmarks and Create Data Stories– Publish and Test in Web Player– Get Feedback and Improve

• First create visualizations, faceted search (filters), and analytics for each individual data source and then look for relationships between the data sources.

http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Process

24

The Results

http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Results

Spotfire on the PC

25

The Results

http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Results

Spotfire on the PC

26

The Results

http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Results

Spotfireon the PC

27

The Results

http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#The_Results

SpotfireWebplayer

28

Comments• The initial objective to see how fast one could create this basic

application using Spotfire's Multiple Visualizations (Cross Table, Table, and Scatter Plot) and Hyperlinks (Between Multiple Tabs and to the Web) and mash it up with the the NOAA GeoPlatform layers. It provides the same results as Socrata's More Views and builds on the NOAA GeoPlatform layers by adding statistical visualizations of the data (see The Challenge and The Socrata and Google Programs previously).

• Please use the Add Comment feature at the bottom of this wiki page to provide feedback and suggest additional analyses you would like to see. To use the Add Comment feature you first need to register by providing your email address. Your privacy will be respected and your email address will not be available to others or used for any other purpose. You can also download the Spotfire File (about 2 MB) from this Wiki and a 30-day free evaluation copy from http://spotfire.tibco.com/ and reuse these analyses, add your own data to this file or new Spotfire files that you create. Have fun and give us your feedback!

http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#Comments

29

Acknowledgements

• The author acknowledges gratefully Dean Allemang, Cory Casanave, Sean Connors, Mills Davis, Li Ding, David Eng, Lee Feigenbaum, Aaron Fulkerson, Jim Hendler, Ralph Hodgson, Kevin Kirby, Kevin Jackson, Bob Marcus, John McMahon, Richard Murphy, Brand Niemann, Jr., Barry Nussbaum, Matthew Phoenix, Tony Shaw, Jeff Stein, George Strawn, George Thomas, Pete Tseronis, and Edward Tufte.

http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#Acknowledgements

30

References• Brand L. Niemann, Put Your Desktop in the Cloud to Support the Open Government

Directive and Data.gov/semantic, Semantic Universe, April 19, 2010. • Brand L. Niemann, Build Your Own Data.gov (Spotfire) and EPA Microsite (Spotfire) with

Semantics and Statistics in the Cloud, Slides, May 15, 2010. • Brand L. Niemann, Build Your Community Health Information "Design for America" Using

Mindtouch and Spotfire, Slides, May 17, 2010. • Brand L. Niemann, Build EPA's CASTNET in less that two hours, see Mindtouch and Spotfire

, Slides, May 21, 2010. Completed in 8 more hours by May 30, 2010. • Brand L. Niemann, Build Your Own Data.gov/semantic with Mindtouch and Spotfire in the

Cloud: The White House Visitor Database, Slides, May 22, 2010. See Data.gov takes the 'Mumsy' test, FCW, May 26, 2010. See Mindtouch and Spotfire, Updated July 7, 2010 (see below). Compare to Findthebest's WH Visitor Logs Comparison Tool.

• Brand L. Niemann, Build EPA's EPA's Facility Registry System (FRS) and Locational Reference Database with Mindtouch and Spotfire in the Cloud: Virginia, No Slides, June 1, 2010.

• Brand L. Niemann, Build the UK’s COINS in the Data Science Library Cloud (Mindtouch and Spotfire), Slides, June 9, 2010.

• Brand L. Niemann, Build EPA's Envirofacts in the Cloud: Virginia FRS, NPL, and TRI (Mindtouch and Spotfire), Slides. June 14, 2010.

• Brand L. Niemann, Build the SemTech 2010 in the Cloud (Mindtouch and Spotfire), No Slides, July 2, 2010.

• Brand L. Niemann, Build the White House Staff Salaries in the Cloud (Mindtouch and Spotfire). Slides, July 3, 2010.

http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#References

31

Footnotes 1• From: Jeffrey Levy/DC/USEPA/US@EPA• To: CONTENT-MANAGERS-L@HUDLIST.HUD.GOV• Date: Thursday, July 08, 2010 06:21PM• Subject: We're using Google Earth and Socrata to provide data about the oil spill

– Hi everyone. On our BP oil spill site, we've been providing CSV and PDF files from the beginning.  We're working on some enhancements to our database's capabilities, too, especially around providing context and summaries.

– But we're also using two tools to expand how people can access and download our data, epsecially raw data in the spirit of open government.

– Here's the blog post I just wrote about it:• http://blog.epa.gov/blog/2010/07/08/...ll-data-tools/

• Jeffrey Levy• Director of Web Communications• Office of Public Affairs• U.S. Environmental Protection Agency• levy.jeffrey@epa.gov• 202-564-9727 (cell for emergencies: 202-309-9445)• EPA's blog: http://blog.epa.gov | Twitter:• http://twitter.com/greenversations• EPA news: http://www.epa.gov/newsroom | mobile: http://m.epa.gov/news |• Twitter: http://twitter.com/epanews

http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#Footnotes

32

Footnotes 2

• EPA's Work on the Gulf Coast– http://epa.wik.is/EPA's_Work_on_the_Gulf_Coast

http://epadata.wik.is/EPA's_BP_Oil_Spill_Data_Tools#Footnotes