open innovation - winter 2014 - socrata, inc
DESCRIPTION
As innovators around the world push the open data movement forward, Socrata features their stories, successes, advice, and ideas in our quarterly magazine, “Open Innovation.” The Winter 2014 issue of Open Innovation is out. This special year-in-review edition contains stories about some of the biggest open data achievements in 2013, as well as expert insights into how open data can grow and where it may go in 2014.TRANSCRIPT
Designing the Future: Code for America Makes an Impact
Data-Driven Government in Action
Codelescence: Engineering Comes of Age
Socrata 2013 Highlights
WINTER 2014
Fixing the Federal IT Procurement Process
Dear Fellow Innovator,
Thank you for picking up our Winter 2014 edition of Open Innovation magazine. In this issue we take a look back at the most exciting year in open data history, and share some thoughts on what’s to come in 2014.
As part of our look back, we’ll review the civic hacking events and conferences Socrata had the honor of attending and awards our customers received. Were you part of the National Day of Civic Hacking? We highlight the group that played a central role in its success, Code for America.
Looking more closely at stories from the open data movement, Socrata Director of GovStat Beth Blauer features three government organizations that are bringing more data to their decisions. And, Socrata VP of Worldwide Markets, Erika Smith, gives us a snapshot of how open data is expanding in Europe.
Thanks to the exponential growth of the movement, there are important issues the open data community needs to address. Socrata Director of Open Data Ian Kalin calls for establishing data standards and soon, while longtime thought leader David Eaves reminds us to engage in the debate on how open data will be put to use in organizations around the world. And, you’ll hear my suggestions for cleaning up the broken federal government procurement process.
For the tech enthusiasts in the group, Socrata’s Director of Product, Ben McInnis describes how the demands on and uses of open data are evolving quickly towards bigger datasets, faster data movement, and even greater accessibility. Data artist Thomas Levine asks us to consider the benefits of open data metadata analysis. And, Chris Whong, a Socrata Data Solutions Architect and geodata specialist, describes how OpenStreetMap is helping data flow from the citizens of New York back into the city’s data stores.
Plus, you get to meet our engineering team, and learn about their hobbies and habits that keep our office full of home-brewed beer and hand-built electronics.
I am honored to be a part of the exciting time in the open data movement. I hope you enjoy the magazine and wish you the best in 2014.
Sincerely,
Kevin MerrittSocrata Founder and CEO
Editor-in-Chief
Bridget Quigg
Managing Editor
Alida Moore
Contributing Editors
Tim Cashman
Patrick Hasseries
Design/Art Direction
Corey Smith
Promotion
Steven Gottlieb
Published By
Socrata
83 S. King Street
Seattle, Wa. 98104
(206) 340-8008
www.socrata.com
16 Fixing the Federal IT Procurement ProcessBy Kevin Merritt
OPEN DATA IN FOCUS
4 Why We Need Open Data Standards Right NowBy Ian Kalin
6 Bigger, Faster, Broader: How Open Data Use Is ChangingBy Ben McInnis
8 The Importance of Engaged Open Data AdvocatesBy David Eaves
10 Open Data in Europe Leaps Beyond TransparencyBy Erika Smith
12 Making the Two-Way Street of Open Data a RealityBy Chris Whong
14 Open Data Had Better Be Data-DrivenBy Thomas Levine
22 Designing the Future: Code for America Makes an ImpactBy Alida Moore
28 Data-Driven Government in ActionBy Beth Blauer
34 Codelescence: Engineering Comes of AgeBy Patrick Hasseries
WINTER 2014
Subscribe to future issues of Open Innovation by going to www.socrata.com/magazine
OPEN DATA
IN FOCUS
OPEN INNOVATION • WINTER 2014 54 OPEN INNOVATION • WINTER 2014
OPEN DATA IN FOCUS
In today’s world of data standards, we are operating in pre-mainstream chaos, similar to that which existed with electricity in Edison and Tesla’s time. The potential for data-as-a-fuel exists as a natural resource in much the same way as magnets and copper did in the industrial revolution. As this next wave of economic opportunity takes root, the same kind of arguments are being made.
GREAT THINGS ARE POSSIBLE WITH OPEN DATAWe benefit from technical standards daily. They allow drivers to purchase gas from any station without fear of using incompatible fuels. They allow a text message from a Verizon phone to be transmitted to an AT&T phone. Data standards operate in a similar manner, but their use within modern applications is relatively new. The most popular one in use today is the General Transit Feed Specification (GTFS). GTFS allows users of Google Maps to know when the next bus is going to arrive within participating cities.
Looking to evolving data standards that deserve greater adoption, a great example at the civic level is HouseFacts. There are many businesses that help people
buy and sell homes (e.g. Trulia, Zillow, most commercial banks, etc.), but the information on the safety and health of those homes is messy and reported in
Why We Need Open Data Standards Right NowBy Ian Kalin
Socrata Director of Open Data
As the open data movement gains
momentum, more organizations, businesses,
and citizens are looking to share data and
collaborate on projects. What is essential
to them doing so? Data standards. Data
standards are a topic that deserves our
community’s full attention right now. The
government innovation movement must
address their importance across industries,
across borders, and even between
departments.
THE DIFFERENCE STANDARDS MAKE First, let’s discuss the importance of standards in familiar industries. Imagine if the electrified world we live in today had no standards; you might have to replace your toaster every time you move to a new house. When electricity first made its way into the households of the world, standards did not yet exist. To that point, two of the world’s greatest inventors, Nikola Tesla and Thomas Edison, became engaged in an epic debate between alternating-current versus direct-current. Each argument had strengths and weaknesses but Tesla made the innovative choice to back his argument up with household appliances, guaranteeing his victory in the debate and establishing our universal outlets today.
very different formats. The solution is a uniform format for reporting things like asbestos, pest infestations, and even abusive landlords. As standards like HouseFacts are adopted by city governments, businesses can aggregate the data that most cities are already collecting
and integrate that information into the websites people are using to make housing decisions. The map for standard adoption becomes a map for business growth.
The challenge lies in developing these standards. Development can be difficult because of the complexity of the data. Data reflects the world in which it is created. The maintenance alone of engineered standards is hard, as demonstrated by the huge network of organizations that deal with standards, like the International Organization for Standardization (ISO), the American National Standards Institute (ANSI), and UL.
As open data moves into our mainstream world, making it as available as possible, in a consistent, efficient manner, is essential. I encourage you to engage in the discussion of defined data standards so that public, open data truly serves the world of citizens who own it.
As open data moves into our mainstream world, making it as available as possible, in a consistent, efficient manner, is essential.
(polynomial trend line)
Data Served by Socrata 2011-2013
2011 2012 2013
OPEN INNOVATION • WINTER 2014 76 OPEN INNOVATION • WINTER 2014
OPEN DATA IN FOCUSOPEN DATA IN FOCUS
Bigger, Faster, Broader: How Open Data Use Is ChangingBy Ben McInnis
Socrata Director of Product
Imagine a state that tracks K-8 student consumption
of free and reduced lunch and puts that data
alongside test scores and attendance information.
Imagine this data is then summarized in an internal
dashboard and sent to teachers via a mobile app.
In this scenario, millions of data points updated
regularly could help with resource allocation on a
monthly, weekly, or even daily basis.
sites, and decision support systems for government.” says Socrata VP of Product Saf Rabah.
Businesses, governments, and citizens now use open data on a daily basis, and they want to look at as much of it at once as possible. For example, the City of Chicago offers a traffic dataset that is more than one million rows long. And, San Francisco’s app showing parking availability is updated in near-real-time.
Having hosted millions and millions of rows of public data since 2007, Socrata has seen data use change dramatically, particularly over the last two years. Here are three of the strongest trends we’ve noticed.
BIGGER DATATwo years ago, the average dataset stored on Socrata was under 10,000 rows. Today’s common datasets fall somewhere between 10 - 40 million rows. Socrata stores datasets as large as 100,000,000 rows and, by end of 2014, we expect to store datasets of 1 billion rows or more.
As data scales in size, two interesting things happen. First, data becomes more useful for indirect and unforeseen purposes because, within larger datasets, common keys are more likely to exist. Second, with more and more correlated records across datasets, the need to precisely clean and format data decreases.
Because of these characteristics, analysis of large datasets can often yield surprising and profound results. For example, the City of Chicago’s analytics team discovered a relationship between
streetlight outages and petty theft--correlations that were always latent within their massive 311 and crime datasets. While the connection might have seemed obvious, having the data to clearly understand this relationship helped the city strategically address the problem.
FASTER DATAHigh-frequency data (data that’s updated very often) has seen similar growth. In the last two years, as data from sensor networks and operational systems has been published as open data, the number of real-time datasets Socrata hosts has grown more than 2,000 percent. (See chart above.)
While static data is useful primarily for archival and analytical purposes, real-time or streaming data can be leveraged as an input into other systems. Snowplow locations can be fed in real-time to citizens and the media, bus locations and traffic signal state information can be
synchronized to optimize traffic flows, and less obvious connections can drive business efficiencies. A coffee vendor might synchronize the preparation of fresh coffee and warm pastries to correspond with the anticipated arrival of a commuter train - not the scheduled arrival, but the arrival time according to data coming from the train’s computer system.
By transforming data from a historical record to a dynamic feed, it can be utilized on an on-going basis to make programs and decision-making more efficient.
MACHINE-TO-MACHINE DATAData is also increasingly accessed, not by humans, but by systems and applications via application programming interfaces (API). They make data assets useful by integrating them as an input for other systems. For example, a new generation of connected and sensor-enabled municipal water systems provide data on water usage. This information lets citizens understand
and manage their consumption, public officials accurately plan for capacity needs and seasonal variance, and utility personnel pinpoint leaks and service interruptions.
A study by technology market research firm Hurwitz & Associates titled “The Benefits of APIs in the App Economy” found that organizations offering APIs, as opposed to those that didn’t, increased customer reach by 70 percent, number of apps created by 50 percent, and number of mobile platforms supported by 58 percent.
Today, nearly every dataset is accessed via API and many thousands of times per day. Socrata is the largest provider of government data APIs in the world, with more than 77,000 datasets available via API.
And, the total volume of data served by Socrata via API has grown more than 1,500 percent in the last two years. This explosion in API usage across the open data movement comes from innovative citizen apps like WasMyCarTowed. This app uses public data APIs to tell motorists, who return to their parking spot but find no car, if their car was towed and, if so, where.
FROM ARCHIVE TO FUELAs the size of datasets grows, frequency of dataset updates increases, and more machine-to-machine interactions occur, data’s usefulness is being tapped and novel uses and reuses are coming to light. Publishing data for the sake of transparency is a noble, yet shortsighted, goal. In the era of big, fast, useful data, we are challenged to do more.
This way of tracking and reporting on school programs via web and mobile apps is just one example of the many that
will emerge as government organizations offer more public data as open data.
Since its founding in 2007, Socrata has seen open data move in larger quantities, faster, and to more places than ever before.
“We’re evolving past the catalog phase of open data and into the data-as-a-platform era, where value is measured by the real-time flow and distribution of data through application ecosystems, consumer internet
OPEN INNOVATION • WINTER 2014 98 OPEN INNOVATION • WINTER 2014
OPEN DATA IN FOCUSOPEN DATA IN FOCUS
The Importance of Engaged Open Data AdvocatesBy David Eaves
Open Innovation Expert
During my 2011 keynote at Open Government Data
camp I talked about how the open data movement was
at an inflection point. For years we have been on the
outside, yelling that open data matters. Now we are
being invited inside and we have a great responsibility
to be of service. Once you have world leaders talking
about things like a G8 Open Data Charter you are no
longer on the fringes - not even remotely.
initially. From the cell phone to the car to the printing press to open source software, all these inventions have helped billions of people, but they did not distribute themselves evenly, especially at first.
So the question cannot be reduced to – will open data empower the empowered, but to what degree, and where, and with whom? I’ve seen plenty of evidence where data has enabled small groups of people to protect their communities or make more transparent the impact (or lack there of) of a government regulation. Open data expands the number of people who can use government information for their own ends – this, I believe is a good thing – but that does not mean we shouldn’t be constantly looking for ways to ensure that it does not reinforce structural inequity.
PROMOTING THE IDEA OF DATA AS A PLATFORMSome of the issues around usability I’ve addressed above in the accessibility piece – for portals that genuinely want users, the axis of evolution is pointed in the right direction with governments and companies like Socrata trying to embed more tools on the website to make the data more usable.
I also agree with a point by Professor Kitchin that, rather than creating a virtuous circle, poorly thought out and launched open data portals will create negative “doomloops” in which poor quality data begets little interest which begets less data. However, the problem is bigger than that.
One of the main reasons I have been an advocate of open data is a desire to help
citizens, nonprofits, and companies gain access to information that could help them with their missions. I also wanted to help change the way governments deal with their data, so that they can share it internally more effectively. I often cite a public servant I know who had a summer intern spend three weeks surfing the national statistical agency website to find data they knew existed but could not find because of terrible design and search. A poor open data site is not just a sign that the public can’t access or effectively use government data; it usually suggests that the government’s employees can’t access or effectively use their own data. This is often deeply frustrating to many public servants.
Thus, the most important outcome created by the open data movement may be that government organizations, save for those in the intelligence community, realize that they are not comfortable with using data to drive decisions. Getting governments to think about data as a platform (yes, I’m a fan of government as a platform for external use, but above all for internal use) is, in my mind, one way we can enable public servants to gain better access to information. Adoption of this principle will also, in many cases, obviate the need for costly solutions from huge vendors (like SAP and Oracle), whose $100 million dollar implementations often silo off data, rarely produce the results promised and are so obnoxiously expensive it boggles the mind.
The key to all this is that open data cannot be something you slap on top of a big IT stack. I try to explain this in my blog post “It’s the Icing Not the Cake” about how Washington, DC was able to effectively launch an open data program so quickly
(which was, apparently, so effective at bringing transparency to procurement data the subsequent mayor rolled it back). The point is that governments need to start thinking in terms of platforms if – over
the long term – open data is going to work. And it needs to start thinking of itself as the primary consumer of the data that is being served on that platform.
My main point is this: let’s not play at the edges and merely define this challenge as one of usability. It is a much bigger problem than that. If we get it wrong, then the big government vendors and the inertia of bureaucracy win. If we get it right, we could potentially save taxpayers millions—while enabling a more nimble, effective, and responsive government.
I try hard to be critical advocate of open data – one who engages the risks and challenges posed by open data. I’m not perfect and balancing these two goals – advocacy and a critical view – is not easy, but I hope it is how we in the open data movement see our role.
WHY IMPROVING DATA LITERACY IS IMPORTANTWe must address the reality that data, even while open to all, can be used to by the most powerful to gain more power. There are definitely cases where data can serve to further marginalize at-risk communities. For example, we should never publish publicly the locations of women’s shelters or worse, the list of families taking refuge in them.
There are two things that give me some hope in this space. The first is that, when it comes to open data, the axis of competition among providers usually centers on accessibility. For example, the Socrata platform (a provider of open data portals to government) invests heavily in creating tools that make government data accessible and usable to the broadest possible audience. This is not a claim that all communities are being engaged and that a great deal more work cannot be done, but there is a desire to show greater use, which drives some data providers to find ways to engage new communities.
The second is that if we want to create a data literate society— and I think we do, for reasons of good citizenship, social justice, and economic competitiveness— we need the data first for people to learn and play with. One of the best ways to help people become data literate is to give them more interesting data to play with. We did not build libraries after everyone knew how to read, we built them beforehand with the goal of having them as a place that could facilitate learning and education.
There are also things that often depress me. I struggle to think of technologies that did not empower the empowered – at least
Rob Kitchin, Professor of Geography at the National University of Ireland at Maynooth, recently wrote
a list of open data critiques for his blog Programmable City. His post inspired me to remind the open data community – particularly the advocates - of our responsibility to take part in the debates around open data, right now. We need to engage in the discussions on a number of topics if we want open data to reach its full potential for effecting positive change in the world. Specifically, I will address two critiques that Professor Kitchin raised: using data to empower the less powerful and how to improve utility and usability of that data.
One of the main reasons I have been an advocate of open data is a desire to help citizens, nonprofits, and companies gain access to information that could help them with their missions.
OPEN INNOVATION • WINTER 2014 1110 OPEN INNOVATION • WINTER 2014
OPEN DATA IN FOCUSOPEN DATA IN FOCUS
Open Data in Europe Leaps Beyond TransparencyBy Erika Smith
Socrata VP of Worldwide Markets
The open data movement is thriving in many parts
of Europe, including well-established programs
in the Netherlands, the United Kingdom, and
Scandinavian countries like Sweden and Norway.
But, in response to recent questions like, “What
have we achieved and what can we do with all
of this data?” the conversation has turned from
open data for transparency’s sake to open data to
strengthen the economy.
In fact, today, the United Kingdom hosts one of the most mature open data programs in the world. From this vantage point, UK’s thought leaders are talking more and more about the tremendous economic and social potential of open data, especially in machine-readable, standardized formats.
In late October 2013, Open Data Institute (ODI) founders Sir Tim Berners-Lee and Sir Nigel Shadbolt announced the
creation of a global network of “nodes” where programs aligned with ODI’s principles of openness and economic innovation would be established. As part of the announcement, they were quoted as saying, “The best way that open becomes the new default is demand: from businesses and organizations, both public and private, from individuals and corporations.”
Berners-Lee and Shadbolt assume that open data will be put to use supporting new businesses. The ODI, a not-for-profit organization that is just over one year old, has so far helped set up more than a dozen open data-based startup companies in the UK.
Socrata became an official partner of ODI in the fall of 2013. ODI lead Gavin Starks says, “We love Socrata’s view of open data as fuel for new businesses. We
delivery within Europe, Socrata recently partnered with Microsoft to make the Socrata Open Data Portal available on the Microsoft Azure platform. Azure has a number of data centers on the continent. Socrata Senior Site Reliability Engineer Paul Paradise noted, “As a growing company, we scale internationally much better by partnering with an established, trusted, international data-hosting platform like Microsoft Azure.”
MORE BUSINESS IN THE FUTURE While transparency is still a central goal for the open data movement, even traditionally transparency-focused organizations, like the Open Government Partnership (OGP), acknowledge the economic power of open data. Before its Open Government Partnership Summit in London in October, the OGP announced that 37 countries had signed commitments to, among many efforts, “radically open up government data to boost entrepreneurship, growth and accountability.”
“Open data is fuel for innovation. The world will never be the same, now that we can so easily take public data and use it to work to make people’s lives better,” says Merritt. “Businesses will inevitably put the public data shared to work for the economy. Socrata’s job is to make sure that it is easily accessed and in useful formats.”
want to collaborate with them to promote the creation of new products and services based on open data.”
Socrata CEO Kevin Merritt presented on a panel at the Open Data Summit with the title, “Open for Business – The Commercial Impact of Open Data.” Merritt commented on the growing interest in economic growth through open data, saying, “In the UK, and all of Europe, people are asking, ‘Open data, so
what?’ And, the ODI and other incubators of new businesses help demonstrate that innovation, new jobs, and better services are possible when non-private data is easy to access.”
FASTER MOVEMENT OF DATA TO SUPPORT INNOVATIONWith the growing list of businesses relying on government open data, the speed and frequency with which governments deliver that data has become more crucial. For example, Spend Network is a startup supported by ODI that “uses open spending data to create new insights for Government and its suppliers.” Spend Network’s services are only as useful and accurate as the data it gathers.
Anticipating the greater demands on datacenters across Europe and, in an effort to improve the speed of its existing data
UK AIMS FOR ECONOMIC GROWTH In the fall of 2010, when the UK launched a searchable database of business plans for government departments, Prime Minister David Cameron was quoted as saying, “We want to be the most open and transparent government in the world.” Since then, getting non-sensitive, non-personal data online has become something British citizens expect.
With the growing list of businesses relying on government open data, the speed and frequency with which governments deliver that data has become more crucial.
OPEN INNOVATION • WINTER 2014 1312 OPEN INNOVATION • WINTER 2014
OPEN DATA IN FOCUSOPEN DATA IN FOCUS
Making the Two-Way Street of Open Data a RealityBy Chris Whong
Socrata Data Solutions Architect
Department of Information Technology and Telecommunications (provider of the footprints data) and are launching a community project to systematically import the city’s treasure trove of building data into OSM. The project was announced in late September.
LEGIONS OF GEOSPATIAL ANALYSTSSo what’s in it for the city? The potential for updates that only a system like OSM can provide. If OSM users see something wrong, they can fix it. Maybe a building footprint is misaligned, or maybe the building doesn’t exist anymore. The city receives a daily update of changes to the building data, and can review those changes. If they are legitimate, DoITT can use these changes to guide updates to their own master database, making it more accurate and up to date. It’s as if the city has legions of geospatial analysts quality-checking their data and sending updates! Alex Barth, Data Lead at the web mapping company MapBox and OSM Advocate, has been a key organizer of the NYC – OSM collaboration, and has been working on the idea since early 2012. The data was already publicly available back then, but carried a license that was incompatible with OSM. NYC’s Open Data Law, passed in March 2012, cleared up the licensing issue and provided the way forward.
To Barth, the project is not simply about buildings, but is an experiment and learning experience about the impact of community-driven projects like OSM. “It’s a data improvement effort that has positive side effects and really lets us grow the community.” The longer-term vision goes beyond OSM or even geodata, and hopes
to redefine open data publishing: “This is about an open data commons, a single space in which government and citizens interact.”
The first gathering of volunteer mappers to work through the monumental task of importing the city’s data met in October. Liz Barry, another leader in the NYC-OSM collaboration, hosted the meeting at the offices of the Public Lab in Brooklyn, and 22 community members showed up to help. The data was broken down into election districts, and the team set out validating footprints against aerial imagery, checking geometries, and correcting overlapping polygons. Existing attribute data in OSM could also be merged with better polygons from the city data. Barry said the workflow is still being vetted, and is not quite ready for full-scale deployment. The idea is that once the workflow is perfected, updates won’t require a physical meetup. Volunteer OSM users will be able to import a chunk of the city’s building footprints whenever and wherever they can.
The real fun will begin when large amounts of the data have been successfully imported and the city can report back about the volume and utility of OSM-contributed changes. In many cases, there may be more information about a building in OSM than the city maintains on its own, meaning the “two-way street” of open data may not flow evenly in both directions. The OSM community has found a partner in DoITT, and this experiment will serve as an early model of the power of citizens and activists to improve government data.
OpenStreepMap allows users to view multiple regions,
from a birds-eye view of Europe to a zoomed in view of the
streets of Manhattan.
and can be just as easily undone. Users may choose to update the map for many reasons, from just knowing more about conditions on the ground than anyone else, to improving the map for a specific project such as an app.
What if a user needed some building outlines that OSM didn’t have yet? That user could manually trace over the satellite imagery, pointing and clicking lots of custom polygons into existence. But what if they needed a whole town? What if they needed New York City? They do, and NYC has an open dataset for that. The city’s detailed GIS database of building outlines and point data is freely available for download at data.cityofnewyork.us. While it will still take human effort to import and verify data for over a million buildings, creating them manually would be an unfathomable and time-consuming process. Leaders in the mapping tech community have partnered with NYC’s
Open data has by and large been a one-way conversation. Governments produce public data and
make it freely available, while citizens, journalists, researchers, and hackers consume it in whatever ways suit them. But, having more eyes on the data once it is released may be able to provide value back to the government, turning users of the data into a source of new data and quality control. This is the experiment in two-way open data that New York City is pioneering with OpenStreetMap.
OpenStreetMap (OSM) is the “Wikipedia of Maps,” where anyone can contribute changes. (Yes, if there’s a footpath or bike trail near your house that doesn’t show up on mainstream web maps, you can literally “draw” it into OSM, name it, and connect it to existing roads.) Like Wikipedia, changes to the map are subject to quality control by the rest of the community,
OPEN INNOVATION • WINTER 2014 1514 OPEN INNOVATION • WINTER 2014
OPEN DATA IN FOCUSOPEN DATA IN FOCUS
Open Data Had Better Be Data-DrivenBy Thomas Levine
“Dada” Artist
We’ve been opening government data for some
time now. Without realizing it, we’ve amassed
some rather rich data about how people publish
and consume open data. With this data about
the use of open data, it’s possible to use data-
backed benchmarks, projections, and decisions in
our open data strategies, and this can make our
approaches to open data more systematic, logical,
and obtainable.
Curious about what the data behind open data can
teach us? Read on.
Every dataset has basic, easy-to-extract properties like the number of records it contains, and the size it takes up on a hard drive. We can come up with more complicated information too, like the number of missing values and the date of the oldest record. In addition, when a dataset is published online, it creates metadata – about the dataset and its use. Metadata provides details like when it was first published, who uploaded it, and how many times people have downloaded it since then.
By collecting some of these simple properties and metadata from each dataset, it’s possible to create a dataset about the publication and use of other datasets – what I call a super-dataset.
DATASETS AS DATA POINTSA super-dataset compiles information about each dataset – when it was published, what kind of information it contains, etc. – into a single row, creating a record (i.e. a data point) about that dataset.
With this setup, you can perform all kinds of functions and analyses about the publication and use of datasets: you can look at how many datasets are on different catalogs, how data is queried and reported, what licenses datasets have, and how many there are of a certain category. Here are some examples of analyses I have done.
Number of Datasets: I compared the number of datasets on some various government data portals that run Socrata’s open data platform software. I found that New York City, Chicago, and the state of Oregon offer the most datasets.
Licensing: I also used my super-dataset of properties and metadata to look into this question: What licenses do people apply to their open data? I discovered that many portals favor public domain or some form of open license, but most list no license at all.
Groupings of Datasets: By looking at similarities in the titles, schemas, and other metadata of datasets, I determined what sorts of data government organizations were putting on their open data portals and how different datasets were related to each other.
METADATA AS DATAEven though the word “metadata” contains the word “data,” people don’t typically think of metadata as something to analyze.
Metadata is often invisible. If we could see it, it might just look like background information about web pages and their contents. For example, Socrata’s open
data platform displays metadata about each dataset, such as community rating, number of visits, number of downloads, etc.
When you use metadata to populate records in a super-dataset, that metadata becomes data that we can see, analyze, and learn from.
WHY THIS MATTERSThere are numerous case studies, by organizations such as Code for America, CKAN, Open Data Institute, about how to open up government data, but these are based strongly on personal experiences, not precise, quantitative statistics. There is comparatively little work that uses data to produce guidelines or best practices for the opening of data.
Open data experts often talk about making use of open data and building products from it, but a super-dataset accomplishes something different. It reveals information
A visualization of the
datasets available on
data.cityofnewyork.us by
category, showing both
raw datasets, as well as
derivative charts, maps, and
filtered views.
about how people publish and use open data. It paints the bigger picture of what’s going on, what’s working, and what isn’t.
Imagine what we can do with this data-backed understanding of our open data! We can find out what has been done before and what has worked, allowing open data publishers to plan their releases more strategically. We can measure release strategies against solid, quantitative statistics to make sure that they are helping us achieve our goals. We can even use these findings to build products that help people interact with open data. These are just some of the possibilities that emerge when we think of data catalogs as a datasets of datasets.
DATA, DATA, EVERYWHEREWe have data about open data, but it always starts out in formats that are not convenient for this sort of study of open data. So, we first need to turn this raw data into a dataset. In my mind, a dataset is a collection of things, with some consistent properties describing each of the things. (We often represent datasets as tables.) We’re going to treat each dataset as a thing inside our collection of many datasets.
OPEN INNOVATION • WINTER 2014 1716 OPEN INNOVATION • WINTER 2014
Fixing the Federal IT
Procurement Process
By Kevin Merritt
Socrata Founder and CEO
Sometimes it takes a disaster for people to recognize the need for change. The botched rollout of HealthCare.gov—
something I would actually describe as a systemic catastrophe—provides a historic opportunity to overhaul the federal government technology procurement system, which is so clearly broken.
DIAGNOSING THE PROBLEMGovernment technology projects have long been characterized by significant delays, mammoth cost overruns, and software products that routinely underperform. But, before the HealthCare.gov fiasco, relatively few people were aware of the magnitude of the problems plaguing the federal IT purchasing system. Now that the story has made front-page news, everyone wants to know what went wrong. Just as any good software engineer looks to uncover the root causes of bugs in a computer program, it is important to meticulously diagnose the underlying causes that continue to produce IT disasters at the federal level.
SELLING TO THE FEDERAL GOVERNMENT IS A BYZANTINE PROCESSFederal IT procurement is overly complex and opaque. The process is full of arcane rules and oppressive costs that discourage many technology providers from entering the market. The Federal Information Security Management Act (FISMA) is a prime example. This legislation was enacted at a time when applications were almost always hosted on premise and the “waterfall” methodology (not “agile”) was the predominant approach to software development. Times have changed, and we have learned a lot since then.
Yet, because FISMA is still in place, vendors looking to bid on federal technology projects must comply with outmoded information security requirements, like using tape backups to replicate files offsite. This is completely incongruent with modern IT best practices. More importantly, the odds of data loss in a single environment that uses tape backups is statistically much higher than in a geo-redundant environment where data is replicated across multiple servers.
OPEN INNOVATION • WINTER 2014 1918 OPEN INNOVATION • WINTER 2014
INNOVATORS NEED NOT APPLYThe net result is that many nimble, innovative companies simply forego doing business with the federal government, concluding that it is just too costly. Over the years, this has created an environment that nurtures mediocrity. Today, there are a whole class of enterprise companies that specialize in navigating the bloated and broken procurement system. Most of these businesses make their money by selling consulting hours, hiring dozens of subcontractors to do the work, and overseeing the development of expensive, customized technology solutions that often do not perform as advertised.
We admire startups because their core competency is innovation. They establish traction and momentum because they invent novel solutions to old problems, develop new technologies, or come up with creative new ways of doing business. The irony in the federal IT space, however, is that these behemoth systems integrators (SIs) are winning business not because
THE GSA JUGGERNAUT AND OTHER BURDENS To bid on federal government projects, businesses must first be listed on the General Services Administration (GSA) schedule. Maintaining this listing is practically a full-time job, requiring continuous updates, renewals, and, of course, piles of paperwork. To keep up, many smaller businesses are compelled to hire consultants just to clear all of the administrative hurdles.
In a very real sense, GSA requirements operate like an extra tax on businesses, siphoning profits from sales to federal government agencies. In addition to this “tax,” GSA vendors must offer their lowest commercial price to the government, even if the market value for their products is significantly higher. This squeezes profits further, disproportionately affecting the ability of smaller businesses to compete. As if this were not burdensome enough, any vendor that wants to sell to the federal government must be able to decipher the Federal Acquisition Regulation (FAR), the principal set of rules that govern the federal procurement process. The FAR is every bit as complex and unwieldy as the federal tax code, containing rules as picayune as the width of the margins for request for proposal (RFP) responses. And these rules change constantly. Few emerging technology firms have the wherewithal or the patience to wade into that kind of logistical labyrinth.
they have innovated. Rather, it is just the opposite. Their sole core competency has nothing to do with technology or innovation at all; their only real skill is that they have perfected the art of selling to the government!
CARVING A PATH FORWARDGovernment leaders need to introduce a serious-minded reform agenda if any real progress is to be made in revamping the federal IT procurement process. This means directly applying the open data ideals that the Obama administration has generally espoused to the specifics of the IT purchasing process.
First and foremost, there needs to be full transparency into federal contracts. All of the details for each contract—including the amounts paid and the names of the people who authorized each payment—need to be online in machine readable, fully sortable, and searchable formats.
The federal government also needs to make it simple for the public to follow the flow of money and influence between federal government agencies requesting vendor support and the contractors bidding on projects. As a country, we have made great strides in recent years in increasing visibility into campaign contributions. To weed out corruption and improve efficiency, senior government leaders need to bring that same level of effort and scrutiny to the procurement process.
Further, policy makers should engage providers of cloud and SaaS-based technology solutions to discuss ways to modernize the federal government’s information security requirements. In the wake of the failure of HealthCare.gov, there is an exciting opportunity to educate the government officials who certify these security mandates about modern software best practices.
WHO IS READY TO TAKE UP THE MANTLE OF PROCUREMENT REFORM? The once-stodgy subject of federal IT procurement has gone viral. In turn, this awareness has produced broad consensus around the need for meaningful change—a precious commodity that should not be wasted. Right here, right now, I am putting forth a clarion call to civic leaders on both sides of the aisle: the time is ripe to take up the mantle of real procurement reform. By unleashing the ingenuity of software companies that have previously been pushed to the sidelines, you can help launch a new era of technology innovation that improves the lives of Americans for generations to come.
Government leaders need to introduce a serious-minded reform agenda if any real progress is to be made in revamping the federal IT procurement process.
We admire startups because their core competency is innovation. They establish traction and momentum because they invent novel solutions to old problems, develop new technologies, or come up with creative new ways of doing business.
“I was very impressed to see the level of
civic innovation and ownership from the
Code Michigan hackathon participants.
Many coders I spoke to had never been to
a hackathon before and were very proud
to share how their products were going to
improve their city and many others.”
Ian Kalin
Director of Open Data for Socrata
at Code Michigan
“Socrata on site is really
beneficial because having
somebody like Clint (a Socrata
engineer) is going to bring a
technical element to it to help
promote all the different tools
that are on the portal, available
through Socrata.”
Tim Dupuis
Interim Director of the IT
Department and the Registrar
of Voters in Alameda County
at Alameda County Hackathon “It is so amazing to see in a
city that has such a diverse
set of problems that we can
unite to create a diverse set
of solutions.”
Beth Blauer
Director of GovStat for Socrata
at Hack for Change Baltimore
“This is an amazing display of
the power of civic hacking and
what we can all do together.”
Hannah Young
Program Coordinator of the National
CfA Brigade on the National Day of
Civic Hacking“A ‘datapalooza’ connects experts,
innovators, and entrepreneurs to
relevant data drawn from every level
of government.”
Jason Hare
Open Data Program Manager
at NC Datapalooza
“For those of us who
work in government,
this feels like our own
national holiday.”
Shannon Spanhake
City of San Francisco
Deputy Innovation Officer
on the National Day of
Civic Hacking
AWARDS
EVENTS
OPEN INNOVATION • WINTER 2014 2120 OPEN INNOVATION • WINTER 2014
SOCRATA 2013 HIGHLIGHTS2013 was a big year for the open data movement. It was filled with
hackathons, datapaloozas, and hack nights. Leaders and innovators in the
movement won awards and recognition. We here at Socrata were thrilled
to be a part of it all. Here is a look back at some of the top U.S. events
we attended, awards our customers won, and favorite things we heard
along the way.
AwardsBloomberg Philanthropies
Mayors Challenge (Chicago)
Data Liberation Award
(NYC Dept of Health)
2013 Digital Counties Survey
Award (Montgomery County,
Snohomish County, King County)
State Program Innovation Award
(Oregon)
2013 Web 2.0 Award (Raleigh, NC)
2013 Best of the Web Winner
(Alameda County)
2013 Digital Government
Achievement Award Winner
(Alameda County)
2013 CSAC Merit Award
(Alameda County)
2013 Achievement of Excellence
in Procurement Award
(Alameda County)
2013 Digital Cities Winners
(Boston, Seattle, Austin, Chicago,
Baltimore, Raleigh)
EventsATX Civic Hackathon III
(Austin, TX)
Code Michigan
Code Across America
(Philadelphia)
Code Across America (San Diego)
Vegas Hack
San Francisco Housing Data Jam
Open Science (Mountain View)
EcoHackSF (San Francisco)
Urban Data Challenge
Hackathon (San Francisco)
Data P2PU Course Sprint
(Mountain View)
Code for Seattle
NC Data Jam (Raleigh)
Hack 4 Change (Seattle)
Alameda County Apps Challenge
NYC Data Week
SpoCode (Spokane)
Apps4Halifax
API Craft Conference (Detroit)
API Strategies and Practices
Conference (San Francisco)
Code for America Summit
(San Francisco)
Code for L.A.
NC Datapaolooza (Raleigh)
Hawaii Digital Government
Summit
SXSW Eco Hackathon 2013
(Austin)
Atlanta Govathon
Code for Oakland
International Open Data Day
(Washington, D.C., New York City,
Seattle, Atlanta)
OPEN INNOVATION • WINTER 2014 2322 OPEN INNOVATION • WINTER 2014
DESIGNING THE FUTURE: CODE FOR AMERICA MAKES AN IMPACTBy Alida Moore
Socrata Content Strategist
Before 2011, parents in Boston, MA struggled to figure out
which schools their children were eligible to attend. The
confusing process, involving a 28-page manual and a lottery
system, was a source of contention and, at times, violent
conflict among parents across the city. A turning point came
when Code for America (CfA), a non-profit organization
dedicated to using data and technology to improve lives,
turned their team on the problem.
OPEN INNOVATION • WINTER 2014 2524 OPEN INNOVATION • WINTER 2014
Lauren Reid, Senior Public Affairs Manager at CfA, explains how they built a solution. “The CfA fellows in
Boston developed an app where parents could enter a few simple data points, such as address and age of child, and find out quickly which schools were available as options for their child,” Reid says. The app was hugely successful. “The City of Boston told us that this app, developed in just three short months, would have taken the city more than two years and two million dollars to create -- had it gone through the standard procurement process. Together with the City, we’re resolving a decades-old problem using modern technology and open data, and changing the conversation between parents and the schools system from one of contention to one that is positive and productive.”
Code for America was founded in 2009 by Jennifer Pahlka, the current Deputy Chief Technology Officer for the United States. The organization seeks to find out what can happen when smart, savvy researchers, developers, and designers are deployed to cities across the nation to help local governments unlock their data to create solutions and deliver services to their citizens.
THE FORMULAThe Code for America formula is simple. Developers and designers commit a year of their careers to helping a city government in need of problem solving. In exchange for a modest stipend, these participants, called fellows, live in their assigned cities, and use their skills to help move government forward to meet the needs of 21st century citizens.
CFA PROGRAMSCode for America has developed four programs to help further their mission.
The Fellowship: Code for America’s flagship program, in which developers and designers are matched with local governments to transform data into usable forms for public improvement.
The Brigade: Civic-minded volunteers come together to form brigades, bringing grassroots efforts to data use and transparency.
The Accelerator: Provides financial and logistical support to civic tech startups, from a $25K grant to mentorship and networking opportunities.
The Peer Network: A learning network for government innovators who want to work with other local governments to harness the power of open data in their cities.
Ben Berkowitz, CEO and
Co-Founder of SeeClickFix,
presents at the Code for
America Summit 2011.
Top: Andrew McLaughlin,
self-proclaimed “nerd,”
Sunlight Foundation board
member, and CEO of Digg
and Instapaper, presents
at the Code for America
Summit 2011. Bottom: Laura
Meixell, 2013 Code for
America Fellow in
San Francisco.
FELLOWS IN ACTIONMatthew Hampel, 2012 CfA Fellow (Detroit), spoke to Socrata about his experience as a fellow. One of his projects for the city of Detroit was to create a web and mobile app that would update commuters about bus schedules. Hampel told us about the app, called TextMyBus.
“Originally, bus data was tracked manually through an ancient interface and it wasn’t available to public,” Hampel explained. “So the city provided us with access to their servers and we exposed the data. With that, we built a text messaging app that helps people figure out when their buses are arriving.” TextMyBus has proven quite popular. As of December 2013, the app has served over 1.1 million users.
CfA fellows have contributed 62 apps across America so far, from helping citizens navigate the public school system to receiving text alerts when services, like food stamps, are about to expire.
We are on
the cutting
edge of Gov
2.0 and civic
hacking and
geeking.
OPEN INNOVATION • WINTER 2014 2726 OPEN INNOVATION • WINTER 2014
Foley agrees. “Your civic technical ecosystem should overlap with entrepreneurship. As part of that ecosystem, our brigade influences the civic IT department by promoting open data. We are on the cutting edge of Gov 2.0 and civic hacking and geeking,” Foley says.
You can expect to see more brigades pop up in 2014, allowing for crossover and
partnership. As Foley explains, “Your city boundaries shouldn’t limit innovation.” If, for example, you have an app that catalogues the greenways in your city, it should continue beyond city limits, says Foley.
A LOOK FORWARDIn 2014, Code for America plans to add 31 new fellows to the fellowship program.
For people like Matt Hampel, who is passionate about harnessing open data for civic improvement, Code for America has been a vehicle for positive change. “Data helps you design the future you want to see,” Hampel explains. “Data gives you the power to make informed decisions, and informed decisions are better decisions.”
Code for America by Numbers
2011
19fellows deployed to Boston, Philadelphia, and Seattle
2012
26fellows sent to eight cities: Austin, Chicago, Detroit, Honolulu,
Macon, New Orleans, Philadelphia, and Santa Cruz
2013
27fellows embedded within nine communities: Kansas City, Kan. +
Kansas City, Mo.; Las Vegas; Louisville, Ken.; New York; Oakland,
Calif.; San Francisco; San Mateo County, Calif.; South Bend, Ind.;
Summit County, Ohio
2014
31fellows will join the Code for America program. Cities to be
announced.
62apps created by CfA fellows over the years
Top: Volunteers gather at
a brigade meeting in San
Francisco. Bottom left: Former
City of Seattle Chief Technology
Officer Bill Schrier presents at
the Code for America Summit
2011. Bottom Right: CfA will send
more fellow to more cities than
ever before in 2014.
BRIGADES IN ACTIONSome of Code for America’s most passionate, committed volunteers live in Raleigh, NC. Jason Hibbets and Chad Foley are two of the Raleigh brigade’s four co-captains. Each man volunteers hours of time each month, outside of their day jobs, to the brigade.
“Being part of the Raleigh brigade allows us to have an impact on local government and our community,” says Hibbets. He believes that government transparency in data is paramount to innovation. “Open data is the foundation of civic entrepreneurship,” Hibbets explains. “Open data belongs to the people. Put data in the right hands and apps that can improve the daily life of citizens can be created.”
Data gives you
the power to
make informed
decisions,
and informed
decisions are
better decisions.
OPEN INNOVATION • WINTER 2014 2928 OPEN INNOVATION • WINTER 2014
DATA-DRIVEN GOVERNMENT IN ACTIONBy Beth Blauer
In the previous issue of Open Innovation, I
talked about the importance of fact-based
decision-making and how gut decisions
can be expensive and dangerous. For
governments to play a leading role in the
data revolution, performance measurement
and successful delivery are mandatory. This
truth is something I am passionate about.
My passion resonates from my experience
managing StateStat and the Delivery Unit in
Maryland and is affirmed in conversations I
have had with government leaders since.
Beth Blauer is a
leading expert
in implementing
“stat” programs in
government and is
Director of GovStat
at Socrata.
Fortunately, the open data movement has resulted in tools for data-driven decision-making, performance, and
delivery. Socrata’s product, GovStat, helps local governments become more transparent, engage citizens, and measure progress against initiatives and goals. It also allows stakeholders to collaborate throughout the entire process on one common platform.
In Beyond Transparency: Open Data and the Future of Civic Innovation1, I stressed the idea that it’s not a matter of if data-driven government can create the best solutions to society’s problems; it’s a matter of how soon governments will embrace the idea and reap the benefits. Using a tool like GovStat enables governments to collect and update data across departments, build beautiful data visualizations, and create both internal and citizen-facing dashboards to track progress.
I would like to highlight three government organizations that have chosen to use GovStat as their primary performance measurement solution. I will share their specific challenges in moving toward data
1 Code for America Press (2013)
transparency, why they chose GovStat as their solution, and how each organization plans to use GovStat to increase efficiency and accountability to their constituents.
TELLING THE STORIES THAT MATTER: COOK COUNTY, ILIn 2011, Cook County began a program to create a strategic plan around the use of open data for decision-making and communications. Andrew Schwarm, Chief Performance Officer of Cook County, was the project head tasked with finding a performance measurement tool.
“The county was publishing our quarterly reports in a PDF on our website, which was far from best practice,” Schwarm says.
Using a tool like GovStat enables governments to collect and update data across departments, build beautiful data visualizations, and create both internal and citizen-facing dashboards to track progress.
OPEN INNOVATION • WINTER 2014 3130 OPEN INNOVATION • WINTER 2014
“We wanted to transform our external performance reporting from a static PDF to a dynamic open data web portal. GovStat jumped out as the product that made the most sense for us.”
An early adopter of the GovStat platform, Cook County decided on a two-phase implementation process. First, the County replicated the PDF data and created reports for each department. “GovStat allowed us to take the data already gathered and put it on a more flexible, user-friendly, open, and transparent platform,” Schwarm says.
Once the data was updated and made available to the entire organization, Cook County entered phase two of implementation: using data to drive decisions. “[Data-driven decision-making] is now part of our culture and the way we do business,” explains Schwarm. Further,
accurate data allows Cook County to tell accurate stories. One goal the county tracks is lowering sick leave for county staff; Cook County hopes to reduce employee sick leave to 5.2 hours per month before December 2013. On October 25, 2013, Cook County published a GovStat report on the progress toward this goal, declaring it “on track.”
Schwarm is excited to track other county initiatives, including public safety and healthcare goals. “We plan to roll out one goal per month and continue to report on its progress,” Schwarm says. He continues, “The use of good, timely, accurate data, especially for a government facing tough fiscal situations, allows us to make decisions and prioritize at a high level.”
Citizens of Cook County can view progress on goals, like reducing
the amount of sick leave taken by government employees, using
GovStat’s citizen dashboard.
This chart, updated daily, shows citizens in Kansas City the percentage
of customers who have been satisfied with the city’s response to water
main break service requests.
DYNAMIC REPORTING AND COLLABORATION: KANSAS CITY, MOKansas City, MO has been a leader in government transparency and data-driven government. Since taking office, Mayor James has convened regular KCStat meetings with his senior team and holds them accountable to the goals and strategies that are most important for the citizens of Kansas City. The technology behind this program began as their homebuilt performance measurement platform, KCStat. A year later, the City added Socrata’s open data portal. As the program grew, the team found they were limited in their reporting abilities. So when Senior Performance Analysts Kate Bender and Julie Steenson began looking for a performance management solution to
enhance KCStat, they knew GovStat would meet those needs.
Of all GovStat’s features, the KCStat team is particularly excited about the tool’s drag-and-drop reporting capabilities.
“We already were on board with the concept of telling our story through data and improving government performance,” Bender says. “The only thing missing was the ability to create dynamic reports and data visualizations,” she continues.
Kansas City hopes to improve how they communicate their results to the larger community. The Kansas City council has always been dedicated to civic engagement. Each monthly council meeting is filmed, televised, and shared online, which attracts serious stakeholders. Still, the
city wanted a way to communicate with every citizen, quickly and easily. They found the GovStat public-facing, or “citizen,” dashboard most useful. “The GovStat dashboard is a way to engage every stakeholder,” says Steenson. “It’s a storytelling device that makes the data more accessible.”
Kansas City launched its citizen dashboard in early October and looks forward to using the tool to make progress toward the city’s initiatives. “One big step forward was having the city council adopt a set of strategic priorities, which form the backbone of our dashboard,” says Steenson. “The next step was assigning measurements to those priorities,” she continues. “The council made a public statement about their priorities and then adopted specific measures to track
The use of good, timely, accurate data, especially for a government facing tough fiscal situations, allows us to make decisions and prioritize at a high level.
OPEN INNOVATION • WINTER 2014 3332 OPEN INNOVATION • WINTER 2014
progress. These metrics for tracking progress will be the advantage of the GovStat dashboard as we build it out.”
Kansas City plans to roll out a new goal every month over the next six months, into 2014. Residents can follow each goal’s progress on the citizen dashboard. In the meantime, Bender is excited to see how the tool helps improve efficiency. “Dynamic reporting saves so much time,” she says. “It’s great to work in a system designed around government use.”
TAKING PERFORMANCE MANAGEMENT TO THE NEXT LEVEL: SAN MATEO COUNTY, CA The San Mateo County government is known for its deep commitment to transparency. The County’s Shared Vision 2025, a comprehensive community planning process designed to get direct input from citizens, is a sterling example of this open, collaborative governing style. So when county residents approved a half-cent sales tax increase in 2012, the Board of Supervisors promised that residents would be able to see how their tax dollars were being spent.
SMC Performance features a dashboard with a series of interactive tiles, each tile acting as a hub for the goals associated with a specific Measure A initiative. By clicking on the tile, users can explore performance measures, review raw data, read the actual proposal and, in some cases, look at charts, graphs, and maps created with the data. “One of my favorite things about GovStat is how easy it is to create maps and other visualizations of the data we collect,” says Shanna Collins, a Budget Analyst in San Mateo County’s Office of Budget and Performance. “These visualizations help us identify where the greatest needs are throughout the county, so we can make budget choices that are based on objective data.”
In the coming months, San Mateo County will use GovStat to track the performance of the original programs funded by Measure A to monitor progress in key areas. In addition, the County is planning to roll out a new dashboard to follow the nine community impact goals that make up its Shared Vision 2025 in early 2014.
“GovStat has been at the heart of our move toward true data-driven management,” says Farrales. “And it gives us a platform for involving the community and our employees in decision-making, which is central to our mission.”
Executives in the County Manager’s Office viewed the approval of the tax – known locally as Measure A – as an opportunity to take the county’s transparent, data-driven approach to budgeting to the next level. “The Measure A sales tax will generate about $65 million in revenue each year over the next 10 years,” says Reyna Farrales, Deputy County Manager for San Mateo County. “We have a duty to show how the services funded by the tax contribute to specific, measurable goals and how those results ultimately fit together with the priorities in our Shared Vision 2025.”
When voters approved the sales tax increase, county leaders began their search for a technology solution to keep the community informed about progress in the coming years. “We were aware of the success of the StateStat program in Maryland and were really impressed with the performance dashboard they were using,” says Farrales. “We discovered that Maryland, along with a number of other cities and states, were all using the Socrata platform. Our newly appointed Chief Innovation Officer (CIO) Jon Walton came from San Francisco and had experience with Socrata, so he and his staff were able to help us ramp up our Socrata-powered open data portal and move into GovStat right away.”
After implementing GovStat, the County Manager’s Office used the system to help departments define goals and metrics for their respective Measure A funding proposals. In September 2013, San Mateo County became the first county government in the U.S. to deploy a public-facing GovStat site. The launch of SMC Performance coincided with the Board of Supervisors’ approval of 22 projects totaling more than $50 million.
GovStat’s mapping
capabilities help
establish patterns.
The map above shows
all water leak, water
meter, and hydrant
repair requests open in
Kansas City.
This pie chart shows the number of emergency response vehicles by
category for San Mateo.
GOALS IN ACTIONEach of these organizations offers a compelling story of what is possible by putting the principles of data-driven decision-making, performance measurement, and delivery into action. Greater adoption of these practices is not a trend; it’s a fundamental shift in the way governments around the world are embracing their mission.
In September 2013, San Mateo County became the first county government in the U.S. to deploy a public-facing GovStat site.
OPEN INNOVATION • WINTER 2014 3534 OPEN INNOVATION • WINTER 2013
CODELESCENCE: ENGINEERING COMES OF AGEBy Patrick Hasseries
Staff Writer
Socrata is maturing, and it is most evident
through the evolution of its engineering—past,
present, and future.
“What’s really exciting about Socrata right now
is that we’re transitioning from a startup – in
the true sense of the word – to a company,”
says Steve Greenberg, one of Socrata’s newest
Engineering Program Managers.
When Socrata was originally founded, it began with a small team of scrappy
software engineers. Team structure was simple, and early team members quickly developed an office culture that paralleled the products they were building: open, engaging, and collaborative. They encouraged free discussion, shared a love for challenges, and held events – planned or spontaneous – that built camaraderie.
Since the earliest days of the company, dozens of engineers have joined Socrata, from part-time high school interns to veteran programmers with decades of education and experience. The engineering team has grown into an engineering department, and the original infrastructure must evolve to accommodate more members and bigger projects.
DIVIDING TEAMS WITHOUT CAUSING DIVISIONWhen companies expand, they must be careful not to let growth impede progress or dull internal culture. So how will Socrata’s engineering team restructure itself without sacrificing its liveliness?
“We’re emulating Spotify’s ‘Tribes, Guilds, Squads’ model,” says Jerome Gagner, Socrata Director of Engineering. “Through it we’ll keep the culture that we have as well as develop engineers’ careers and individual disciplines.”
Based on Spotify’s model for structuring internal teams, Socrata’s engineers are reorganizing into three types of groups. All engineers dedicated to a specific service or product form large teams called “Tribes.” All engineers who share a
discipline (front-end development, back-end development, user interface design, etc.) collaborate through inter-tribal groups called “Guilds.” Within each tribe are also “Squads,” small teams dedicated to specific disciplines within the tribe. Squads act like miniature startups in their own right, maintaining their own cultures and core values.
A CULTURE OF CODE AND CHARACTERAnyone who has visited Socrata can attest to the dedication and personality that each employee brings to the company, particularly the engineers.
“One thing I frequently talk about with friends outside of work is that Socrata’s engineering team is filled with a bunch of unique, funny people. We have a lot of characters here, but at the end of the day, everyone is deeply committed to our customers and to the company’s mission,” says Greenberg.
Socrata’s engineers are inventive and rarely bored. When they aren’t building or running code, they run marathons, make hot sauce, craft micro-brews, and use 3D printers to produce models of Iron Man’s helmet. In between tasks, they also pursue friendly feuds, from office Nerf-gun battles to internet pranks.
Tradition is another important part of Socrata’s culture, with the engineers being among its most avid followers. Since the company’s founding, engineers have eaten lunch every Friday at the same teriyaki place in Seattle’s Pioneer Square. They enjoy an office happy hour on Friday afternoons, bringing in beer or whiskey
Data is Socrata’s business, and it handles datasets of all sizes—from those with only a dozen records to others with records ranging in the millions. A search request on a small dataset takes only a few seconds, but advanced searches on large datasets naturally take much more time to process. Socrata’s engineers have recently enjoyed the challenge of developing methods to make even advanced searches in large datasets complete in less than four seconds.
Challenging Queries
Socrata’s engineering team
members have attended
dozens of hackathons
around the world, serving
as instructors, presenters,
and judges.
Anyone who has visited Socrata can attest to the dedication and personality that each employee brings to the company, particularly the engineers.
OPEN INNOVATION • WINTER 2014 3736 OPEN INNOVATION • WINTER 2014
to share. On their anniversary of employment, team members also share one pound of chocolate for every year they’ve worked at Socrata.
While these perks and traditions are part of what makes Socrata great, ask the engineers what they love most about their job and they will likely reply “learning, coding, and problem-solving.”
“We face a lot of hard engineering challenges, and those challenges are very exciting to work on,” says Anthony Nowell, a Socrata Software Developer. “I have no doubt that I’m surrounded by ridiculously bright people who can overcome those challenges, and I’m really encouraged knowing that we’re helping each other to grow technically.”
EXPONENTIAL GROWTHPioneering tech companies like Google, Spotify, and Facebook have demonstrated that well-defined culture and open organizational models are valuable growth tools. They help attract new team members, develop employee competency, and foster loyalty, which in turn inspires employees to produce innovative projects that put the company on the leading edge of entrepreneurship.
Socrata adds members to its team on a near-weekly basis, many of them engineers. Part of this growth success comes from the attractiveness of Socrata’s culture, which the company has nurtured since its founding. Moving forward, Socrata will continue to emphasize organizational models that maintain that culture.
“It’s amazing seeing Socrata grow as big as it has,” says Chris Metcalf, Director of Developer Platform and six-year Socrata employee. “I don’t have kids but, to me, this is like seeing my child go off to college.”
CHRIS “CHARMS” ARMSTRONG
Achievements unlocked: Helped
develop a number of startups
Geeks out on: Baking, beer, current
events, dev-ops, live music, tabletop
games, travel
Most likely to: Become a pastry chef
JEROME GAGNER
Achievements unlocked: Taught
himself to read and speak Arabic;
deployed numerous large-scale
applications in languages such as Java,
PHP, .Net, Ruby, etc.
Geeks out on: Camping, flight
simulators, SCALA, server-side and
front-end technologies
LILIA GUTNIK
Achievements unlocked: Talked about
her prank adventures on The Moth;
developed GovStat from the ground up
Geeks out on: Data visualization,
Keanu Reeves, stand-up comedy, tennis
Most likely to: Sleep with a Keanu
Reeves body pillow
GIACOMO FERRARI
Achievements unlocked: 3D printed
Iron Man’s helmet and arc reactor; co-
created Socrata’s charting library
Geeks out on: 3D printing, electronics,
web development
Most likely to: Invent a working flux
capacitor, blow himself up in a lab
STEVE GREENBERG
Achievements unlocked: Recently had
his first child, helped design Microsoft’s
error reporting system
Geeks out on: Cycling, family,
remodeling his house
KARIN HELLMAN
Achievements unlocked: “I’ve made
it through a year in the U.S. without
getting fat.”
Geeks out on: Climbing, colors,
painting, Photoshop
Meet Socrata’s EngineersWhen we say “geek out,” we mean
it. Socrata’s engineering team hosts
a bright, quirky cast of characters
who are known for much more than
creating great software.
OPEN INNOVATION • WINTER 2014 3938 OPEN INNOVATION • WINTER 2014
JOHN KEW
Achievements unlocked: Worked on
a number of evolutionary computation
projects; developed code used on
millions of servers around the world
Geeks out on: Family, illustrating
microorganisms, microbiology
AYN LESLIE-COOK
Achievements unlocked: “My kids
don’t seem like they’ll turn out to be
criminals.”
Geeks out on: Family, statistical
modelling, swimming, theatre
ANTHONY NOWELL
Achievements unlocked: Recently had
his first child, built a small application to
assist his mother-in-law with her daycare
business
Geeks out on: Coding, family, web
frameworks
PAUL PARADISE
Achievements unlocked: Helped
shape Socrata from the very beginning
Geeks out on: Computer hardware,
computer networking, triathlon
JEFF SCHERPELZ
Achievements unlocked: Led Socrata
through several versions of its front
end (including conversion from FLEX to
Javascript); designed and built his own
house
Geeks out on: Cars, food, knitting
JASON KROLL
Achievements unlocked: Built a
system to identify the best type of
education for over 15,000 careers and
find schools offering such education
Geeks out on: Computer science,
economics, music, running, statistics,
tennis
CHRIS METCALF
Achievements unlocked: Helped
Socrata grow from a startup into a
company
Geeks out on: APIs, developer tools,
home improvement, micro-brewing,
photography
BRIAN OLDFIELD
Achievements unlocked: Accepted to
grad school
Geeks out on: Bitcoin mining, website
design
DAN RATHBONE
Achievements unlocked: Travelled
and worked all over the world
Geeks out on: Database technologies,
hiking, running, triathlon
Most likely to: Wander off to explore
and then be found living in a hut in the
middle of nowhere
CLINT TSENG
Achievements unlocked: Taught math and English in India; worked on a project deployed to 80+ countries and the international space station
Geeks out on: Design, guitar, music, movies, sailing
Most likely to: Freak that this was written in Comic Sans
Meet Socrata’s Engineers
(206) 340-8008
83 S. King St., Ste. 107
Seattle, WA 98104
www.socrata.com
twitter.com/socrata
facebook.com/socrata
Subscribe to future issues of Open Innovation by going to www.socrata.com/magazine