1

USDA Open Data / Big DataPresentation

for the

Federal Big Data Working Group Meetup

Presented by the Office of the

Chief Information OfficerMarch 16, 2015

Overview

• Open Data– Implementation of OMB’s M-13-13- Open Data Policy

– USDA Last Quarter’s Accomplishments

– Open Data Projects

– Open Data Success Stories

– Data Quality

– Open Data Summary

• Big Data– Intersection of Open and Big Data

– Why Is It Important To Agriculture

– USDA Big Data Projects• Food Resilience Theme of the Climate Data Toolkit Initiative

• Using Administrative Data for Statistical Purposes

– Big Data Summary

2

Open Data

3

USDA Implementation of M-13-13• Developed 3-Step M-13-13 Implementation Strategy

• Dataset Collection/Publishing• Created and maintained an Enterprise Data Inventory (EDI)

• Submit EDI to OMB only (not to public)

• Maintain a Public Data Listing (PDL)

• Published datasets to PDL to USDA.gov/data and Data.gov

• Developed and documented the USDA Data Publishing Process• Develop Dataset Prioritization, Privacy/Legal Review and IT Security Review Processes

• Developed Customer Feedback and Engagement Plan• Conduct Internal/External Engagement Sessions

 

4

Enterprise Data Inventory

Public Data

listing

Last Quarter’s Accomplishments

November 2014 – Submitted 61 Datasets February 2015 – Submitted Datasets

14 New Datasets 61 Updates

The Enterprise Data Inventory (EDI) consists of 651 datasets

506 Release to the Public (Public Data Listing) 145 Non-Public

December 2014 – Received Government/Industry Roundtable Final Results

Continue to support the USDA Science Council Committee on Statistics, we formed a cross Agency Administrative Data Categorization Working Group (ADCWG)

Determine ways to categorize USDA Data in a standardize manner

Determine how to collect, disseminate and access scientific statistical data for use within the Department

5

Open Data Projects• Office of the Secretary (OSEC) Disaster Clearinghouse

Project• Develop a one-stop application for disaster victims to find assistance

before, during and after a disaster

• Project sponsor - Dr. Ann Mills and Ms. Rebecca Shively of OSEC

• Successfully completed requirements and preliminary design

• User’s Test and Web-design will occur in March 2015

• Census Find it/Connect it Project• Develop a pilot using USDA, Census and NOAA data to determine

the economic impact of a flood on Farmers, Ranchers, Producers and Consumers

• Working with the National Institute Standards and Technology (NIST) to establish a common set of standards that can be shared across the federal government

• Census One City, One Problem – Software Development Kit (SDK) Project• Unlocking the value of open data to promote economic growth, fuel

innovation and entrepreneurship

• Enabling cities and counties to use data to deliver tangible and transformative results for their communities

• Select one city and one problem and solve it through the use of Census, USDA and NOAA’s open data

• Census Center for Applied Technology (CAT) Lab • Real-Time Disaster Intelligence Map with Crowdsourcing

• Pollinator Field Data Collection with Citizen Science

• Farm Service Administration Common Land Unit (CLU) Hackathon for Open Application Programming Interface (APIs) 6

Open Data Success Stories

. • WhyHunger– The WhyHunger organization is

using the Food Nutrition Service (FNS) data as one of the resources in providing information for the Summer Feeding Program for School Children, as well as healthy food sourcing in emergency food programs. Featuring program profiles, resources and strategies employed by food banks and agencies, including building relationships with farmers, creating voucher programs, teaching people how to grow their own food, and supporting an organizational culture that prioritizes nutrition. http://www.whyhunger.org/getInfo/showArticle?articleId=3674&utm_source=Clearinghouse+Connection+-+May+2014&utm_campaign=CCMay2014&utm_medium=email

7

Open Data Success Stories

. • Farmers Market Directory– The Open Data concept was used

to develop applications such as the Farmers Market Directory, which gives developers and designers direct access to the wealth of farmer’s market information housed in the online database.

• Rails-to-Trails– The Rails-to-Trails Project

extensive mapping tool mixes data from the National Household Travel Survey and USDA’s Economic Research Service (ERS) to provide a better understand pedestrian and bicycle mode patterns in rural areas.

8

Data Quality• When is the Data Good Enough for Release

• Request for Forest Service for Trails Data

• Should we consider a Disclaimer?

• Data Quality Improvement (Data Cleansing)• Cleansing of Mata data – we’ve improved the

quality of our datasets by 36% since our last submission.

• Spelling

• Broken links

• Duplicate entries (etc)

• Cleansing of the Actual Data to make it more open. We are adopting the five star maturity methodology

• Stage 1 - PDF Files

• Stage 2 – Excel Files

• Stage 3 – CSV Files

• Stage 4 – XML Files

• Stage 5 - Data that can be link to other data

• Data Management Policy• FY2015 – develop a policy to govern the data

management of USDA data

9

10

• USDA Open Data Team has made tremendous progress – Data Collection

– Process Development

– Customer Feedback and Engagement

• Challenges Facing the Open Data Team– Increasing Agencies and Administrative Offices participation in submitting

datasets to be published (Internal and External)

– Engaging the public to improve the quality and quantity of USDA datasets

– Data Quality and Data Management

– Dedicating resources to move the Open Data Initiative to the next level

• Permanent Chief Data Officer

• Data Scientist

Open Data Summary

Big Data

11

12

Intersection of Open and Big Data

13

Sensor Technology and Networks (measuring all kinds of data)

Scientific Instruments(collecting all sorts of data)

Mobile Devices(tracking all objects all the time)

What Is Generating Big Data?

Making the link between

On-Farm Optimization• Data-based decision-making for many more decisions• Early problem identification for management response• Custom solutions to minimize inputs and maximize

yields

Input Product Innovation for Unique Conditions

Value Generated by Ag Big Data

• Biotech / seed research• Equipment R&D• Other input supplies

Market Linkage• Improves transparency and

predictability of markets16

Value of AgBig Data

15

• Big data starts with precision ag, but is much larger than on-farm data alone– On-Farm Precision Ag (fertilizer, planting, crop

protection)

– 3rd Party Data (weather, satellite / aerial imagery, soil fertility, topographic data)

– Research & Development (Academic / Land grant data, biotech and equipment company publications, software and analytics

– Commodity Markets (local, regional, global)

Big Data In Agriculture

16

• Why is Big Data in agriculture emerging now?– Computer capacity has now expanded to process vast

volumes of data

• Why is Big Data an important issue?– Enables the promise of Precision Agriculture to be realized

– Makes critical agronomic decisions on small areas within each field

• What is the level of urgency?– Several major Ag companies have just launched or soon

will launch Big Data

Importance of This Issue

Enabling Farmers to Make 40 Critical Decisions

Critical Decision Sets:

SeedFactors

• Seed is lynchpin decision – key to establishing yield potential

Planting Factors•Focus on best field configuration, preparation and planting elements

Pest-Control Factors

• Focus on insect and weed control regimes

In-Season Decisions•Diseases, nutrient, etc. approaches based on in-field environment

Goal: Maximize

Net Return Per Acre

Productivity Tools:

Planning Data

• Seed Selection

• Weed- Control Program

• Insect- Control Program

Pre-Planting Data

• Fertility Program

• Tillage Program

Planting Data

• Plant Population Dynamics

• Seed Depth

In-Season Data

• Post- Emergence Pest Control

• Crop Diagnostics

Harvest Data

• Equipment• Crop

Marketing

19

Decision Support for Farmers

Whole Farm Profitability Services

Current Services Offering

Pioneer FarmerProfitability Strategy

• Field-by-field crop plans• Planting maps• Harvest maps• Crop insurance• Financial services• Grain marketing

Tools to deliver additional insights for advanced farmer profitability.

Harvest / Evaluate

Plan

PlantMonitor

20

Farmer Profitability Strategy

USDA Big Data Projects Food Resilience of the Climate Data Toolkit Initiative

Led by the USDA OSEC and OSTP Tool Kit Focuses on four types of Data;

Production:

How will crop yield and production of other food products be affected by changes in climate? What crops and locations will be most vulnerable to changes in climate? How will changes in climate affect the types of crops that can be successfully grown in a location? Under what conditions (i.e., for what crops and in what locations) will indirect effects of climate change

(e.g., pests, pathogens, invasive species, fire) overwhelm direct effects of increasing temperature and changes in precipitation?

Supply What parts of the food supply chain will be most vulnerable to climate change? How will processing, storage, and transportation need to be modified to continue to supply safe food

products under a changing climate? Nutrition

How will climate change impact food borne illness that includes chemical as well as microbial sources? How will an increase in pesticide or veterinary drug use with climate change impact the contamination

of food? Security

How will climate change affect the nutrient content of staple foods? How will the cost of a balanced diet be impacted under a changing climate? What segment of the population will be most vulnerable to changes in food prices? What countries are most vulnerable to climate change that affects food yield, quality, and availability?

Launch of the Food Resilience Theme – July 2014

19

USDA Big Data Projects

Using Administrative Data for Statistical Purposes

Driven by Memo M-14-06 – Guidance For Providing and Using Administrative Data for Statistical Purposes

Led by USDA Science Council Committee on Statistics Convenes monthly to discuss the progress on each Agency’s initiatives

Improving how agencies leverage existing and Federal data to facilitate agencies' programmatic work and enhance the value of those data to the American public

Facilitating the collection and publishing of USDA Agency administrative data to enable full, appropriate use of non-public administrative data for statistical purposes

20

21

• The Value of Investing in Open and Big Data:– Open and Big Data will create the next major technological

“sea change” in agriculture

– Will enable completely different ways to innovate and invent new business models

– Enable the mass market to source, supply and consume it with better outcomes

– Innovators and inventors can build stuff that matters and build stuff that works

– Unlock more value in the existing data market and has the potential to create a new industry of businesses

Big Data Summary

Questions

22

Joyce HunterUSDA Acting Chief Information Officer (CIO)

and Deputy CIO for Policy and Planning

202-720-8833joyce.hunter@ocio.usda.gov

Bobby JonesUSDA Senior Advisory to the

Deputy CIO for Policy and Planningand

Acting Chief Data Officer202-690-4305

bobby.jones@ocio.usda.gov

Open Data Points-Of-Contact

Special Events

• Wharton School DC Club Innovation Summit - http://whartondcinnovation.com/ or http://whartondc.com/

• GITEC Summit - http://www.gitec.org/summit-2015

23