Crowdsourcing Innovation: Open Data and APIs

Jonathan YoungDecember 10, 2012

About the Presenter

Currently: • Financial Analyst,

Oakland Housing Authority

• MPA Student, SF State

Experience:• 7 years of consulting

and public sector work in infrastructure planning and growth management finance

• jmwyoung@gmail.com

What is the management innovation?• Public agencies at all levels of government

generate and collect data on all kinds of activities (everything from bus arrivals to weather forecasts to crime reports to medical treatment outcomes).

• By making this data increasingly open and available, people can use the data to develop applications that can benefit businesses, citizens, and public agencies.

• The data can be more usable if agencies provide it in standardized formats and develop application programming interfaces (APIs) for new software to interact with the data feeds.

Big Picture

• Implementation of ideas found in emerging public administration theories: governance, public value, Government 2.0– Crowdsourcing– Networks & Coproduction– New ways to create public value

Definitions

Open data is data that meets these criteria:• accessible (ideally via the internet) at no more than

the cost of reproduction, without limitations based on user identity or intent;

• in a digital, machine readable format for interoperation with other data; and

• free of restriction on use or redistribution in its licensing conditions.

– UK Open Data White Paper

DefinitionsAn application programming interface (API) is a particular set of

rules and specifications that a software program can follow to access and make use of the services and resources provided by another software program that implements that API.

– 3Scale Networks

Examples of Applications Using Open Data

transparency on 401(k) fees and profiles of financial advisors using SEC data

web and mobile app providing data on building permits pulled in San Francisco

identifies sites in New York City for potential community

project – public vacant lots, community gardens, etc.

History & Origin

• No clear starting point – government has always been creating and publishing data used by businesses and individuals.– e.g. maps, geologic surveys, property tax assessments

• Early 2000s: with Web 2.0 technologies increased interactivity through the web– Increased interest in app development, data mashups and

crowdsourcing • 2007: iPhone introduced. With mobile devices, more

situations when data can provide value• 2009: www.data.gov and data.gov.uk launched as

clearinghouses for government data in machine-readable formats

API Explosion (mostly private sector)

The Future

• Data storage and computer power continue to get cheaper

• Computing moves to the cloud• More and more data will be created by

government, and developers will have the capability to do more and more useful things with the data

• Shift from government to governance, coproduction of public value

Significance

• “Open data is the raw material of the new industrial revolution.” – Francis Maude, UK Minister for the Cabinet Office

• Open data used in health, life sciences, transport, real estate, weather, GPS, etc.

• Government data systems generate billions of dollars of value, with the potential for more– NOAA weather data: $10 billion annually– U.S. GPS systems: $90 billion annually– Potential value from open health data: $350 billion

Source: Howard, 2012

Significance

• New data applications can increase the value of public services (e.g. next bus apps, parking spot locators)

• Public agencies do not have the resources to develop every app that could be useful or the creativity to envision every useful application of data – open data and APIs allows others to create value with the data

• U.S. government, states (Missouri, Colorado), and cities (Baltimore, Chicago, NYC, Seattle, SF, Palo Alto, etc.) have established data policies and websites to collect standardized, open data sets and make them available to application developers

Stakeholders

Data generators•Public agencies•Effort and resources required to make

data available and usable

Application developers•Some companies’ business model is

organizing data and creating useful applications

•e.g. Google, BrightScope

Application users•Business•Citizens•Government

TrendsPolitical• Open data is a newly

prominent goal in federal government and large cities

• Open Data Initiatives launched by feds, states, cities

• Establishment of CTO and CIO offices in governments

Economic• Increasingly data-driven

economy• Web 2.0• Silicon Valley

weathered Great Recession relatively well

Social• Customers joining in

to coproduction of useful data

• e.g. Yelp.com

Technical• Cheaper processing

and storage• Cloud computing• Mobile devices

Environmental• Big data applications

driving energy efficiency• Smart meters and smart

grids• e.g. Nest learning

thermostat

DesignSteps in the process from data generation to open data app:

1. Agency generates a data set or a stream of data2. Formatting standards (for data sets) or APIs (for

continuous interactions between data systems and apps) are developed and published

A. May be developed by agency or jointly with an app developer

3. Agency publishes data in a stable location4. Developers write applications that use data or

incorporate it to enrich existing apps (e.g. Google Maps and traffic congestion data)

A. Ideas for apps may come from outside developers, discussions between developers and agency, or stimulated by developer challenges and cash prizes

Data Standardization

• Data can vary from city to city because they don’t always collect the same information, their databases do not use the same format, and they use different codes to categorize information.

• Governments need to make it easier for companies to use their numbers by creating data standards.

• e.g. General Transit Feed Specification, developed in 2005 by Portland’s TriMet transit agency and Google, now used by 500+ cities. Allows web developers to integrate transit information.

8 Principles of Open Government DataData shall be considered open if it is made public in a way that complies

with these principles:1. Complete All public data is made available. Public data is data that is not subject

to valid privacy, security or privilege limitations.2. Primary Data is as collected at the source, with the highest possible level of

granularity, not in aggregate or modified forms.3. Timely Data is made available as quickly as necessary to preserve the value of the

data.4. Accessible Data is available to the widest range of users for the widest range of

purposes.5. Machine processable Data is reasonably structured to allow automated

processing.6. Non-discriminatory Data is available to anyone, with no requirement of

registration.7. Non-proprietary Data is available in a format over which no entity has exclusive

control.8. License-free Data is not subject to any copyright, patent, trademark or trade

secret regulation. Reasonable privacy, security and privilege restrictions may be allowed.

Sour

ce: O

pen

Gov

ernm

ent W

orki

ng G

roup

What resources are required/helpful stimulate app development from open data?

Required• Raw data• Formatting rules • Web hosting• IT expertise to publish

data• Interested developers

Helpful• Developer collaboration

to design useful data specs

• Formatting follows interagency standards

• An API• Developer challenges

and prizes

Metrics and Timeframe

• Relevant• Accessible• Reliable

• Well formatted• Developers know it exists

• Open data initiatives are successful if the data are actually incorporated into applications that are used by citizens, businesses or public agencies.

• Success depends on having data that are:

• Data can sit waiting for developers to use it for an indefinite amount of time. But if the agency has specific goals for the data, it needs to become a developer itself or proactively engage developers.

Interorganizational Validity – Someone has to care

• Unless there is a payoff, you can release the data but nothing will happen. There needs to be– Commercial interest– Nonprofit willing to invest resources, or– Volunteers wanting to solve a problem (Code for America)

• Scale matters. For some types of data and some agencies, the scope may be too small for it to be worthwhile to invest effort in using data.– Initiatives to standardize data formats and create APIs that

work across jurisdictions build scale and can make development of apps using data from smaller cities/agencies more feasible

Intraorganizational Validity – Not going to magically solve your agency’s IT problems

• Opening up data allows the public to benefit from your agency’s data.

BUT…• Just releasing data does not mean someone will build a useful

application with it.

• If your organization or your clients have a need for a particular application, you still need to built it yourself, contract it out, or actively engage with an organization that might want to do it.

• Agencies have used developer challenges and cash prizes to encourage people to build desired apps.

Sources of Resistance

• Public vs. private value– Developers may want exclusive use of data to

increase paybacks of their efforts– Citizens and agencies may dislike using public data

for commercial benefit• Privacy: concerns about private data from

government services becoming widely available• Sunshine: cities and other agencies may

hesitate to publish data that facilitates better performance evaluation

References• UK Government. (2012). Open data white paper: Unleashing the

potential• Black, A. (2012, September 5). The most important information

missing from yelp. Slate.com.• The Presidential Innovation Fellows: Open Data Initiatives.

http://www.whitehouse.gov/innovationfellows/opendata• Howard, A. (2012, February 12). Data for the public good.

http://strata.oreilly.com/2012/02/data-public-good.html• Howard, A. B. (2012, September 10). Here comes the data

economy. Slate.com.• What is an API? 3Scale USA.

http://www.3scale.net/wp-content/uploads/2012/06/What-is-an-API-1.0.pdf

References

• data.baltimorecity.gov• data.cityofchicago.org• data.cityofnewyork.us• data.hawaii.gov• data.honolulu.gov• data.mo.gov• data.seattle.gov• data.sfgov.org• data.somervillema.gov

• data.gov.uk• explore.data.gov• opencolorado.org• paloalto.opendata.junar.

com• www.data.gov

Frequently Asked Questions

Q: Why should private companies be allowed to profit from public data?

A: If companies are able to develop uses of data that generate a profit, it means they are creating value for someone, and that is generally good for society. The use of open data is non-exclusive, so they aren’t stopping anyone else from benefitting from the use of the data.

Q: Why should agencies invest resources in developing APIs?A: APIs allow separate computer programs to work together. If

data isn’t just a static list, but instead is a dynamic stream generated by government systems, and API might be needed to facilitate beneficial uses of the data by third party developers.