open local data: challenges and opportunities
TRANSCRIPT
WMRO Open Data Event, Birmingham, July 15, 2010
Open Local Data Challenges & opportunities
It’s been a good year for open government datadata.gov.uk, Ordnance Survey, MPs expenses...
It’s been a good year for open government datadata.gov.uk, Ordnance Survey, MPs expenses...
It’s been a good year for open government datadata.gov.uk, Ordnance Survey, MPs expenses...
But what about local data?
But what about local data?
• Frankly it’s a mess
But what about local data?
• Frankly it’s a mess
• Sporadically published by central government
But what about local data?
• Frankly it’s a mess
• Sporadically published by central government
• Inaccessible & impenetrable council websites
But what about local data?
• Frankly it’s a mess
• Sporadically published by central government
• Inaccessible & impenetrable council websites
• Opaque local public bodies and quangos (goodbye RDAs, hello LEPs)
But what about local data?
• Frankly it’s a mess
• Sporadically published by central government
• Inaccessible & impenetrable council websites
• Opaque local public bodies and quangos (goodbye RDAs, hello LEPs)
• At best unclear & at worst unusable legal situation
But what about local data?
• Frankly it’s a mess
• Sporadically published by central government
• Inaccessible & impenetrable council websites
• Opaque local public bodies and quangos (goodbye RDAs, hello LEPs)
• At best unclear & at worst unusable legal situation
• Start with the basics. Who are the councillors, where do they represent and what committees do they sit on?
But what about local data?
• Frankly it’s a mess
• Sporadically published by central government
• Inaccessible & impenetrable council websites
• Opaque local public bodies and quangos (goodbye RDAs, hello LEPs)
• At best unclear & at worst unusable legal situation
• Start with the basics. Who are the councillors, where do they represent and what committees do they sit on?
• How easy it that information to find & reuse?
Simple questions, hard to answer
Simple questions, hard to answer
• How does the budget of my council compare with similar ones?
Simple questions, hard to answer
• How does the budget of my council compare with similar ones?
• What’s the background of the chief executive of my council?
Simple questions, hard to answer
• How does the budget of my council compare with similar ones?
• What’s the background of the chief executive of my council?
• Is my route to work going to affected by roadworks next week?
Simple questions, hard to answer
• How does the budget of my council compare with similar ones?
• What’s the background of the chief executive of my council?
• Is my route to work going to affected by roadworks next week?
• What are the connections between the companies awarded contracts and the councillors & senior management?
Simple questions, hard to answer
• How does the budget of my council compare with similar ones?
• What’s the background of the chief executive of my council?
• Is my route to work going to affected by roadworks next week?
• What are the connections between the companies awarded contracts and the councillors & senior management?
• How will cuts in services be decided and what can I do to influence the decisions?
TransparencyA brief example.
TransparencyA brief example.
• A story in Private Eye
TransparencyA brief example.
• A story in Private Eye
TransparencyA brief example.
• A story in Private Eye
• An investigation by the District Auditor
TransparencyA brief example.
• A story in Private Eye
• An investigation by the District Auditor
• A report buried in the nether recesses of the website...
TransparencyA brief example.
• A story in Private Eye
• An investigation by the District Auditor
• A report buried in the nether recesses of the website...
TransparencyA brief example.
• A story in Private Eye
• An investigation by the District Auditor
• A report buried in the nether recesses of the website...
• ...with no relevant heading
TransparencyA brief example.
• A story in Private Eye
• An investigation by the District Auditor
• A report buried in the nether recesses of the website...
• ...with no relevant heading
TransparencyA brief example.
• A story in Private Eye
• An investigation by the District Auditor
• A report buried in the nether recesses of the website...
• ...with no relevant heading
• Accessible only as a PDF of a scan of a document
TransparencyA brief example.
• A story in Private Eye
• An investigation by the District Auditor
• A report buried in the nether recesses of the website...
• ...with no relevant heading
• Accessible only as a PDF of a scan of a document
TransparencyA brief example.
• A story in Private Eye
• An investigation by the District Auditor
• A report buried in the nether recesses of the website...
• ...with no relevant heading
• Accessible only as a PDF of a scan of a document
• Is it any wonder suspicions are raised?
Engagement
Engagement
• Local Authorities now have a duty to engage. This means• Reducing barriers to understanding
and involvement
Engagement
• Local Authorities now have a duty to engage. This means• Reducing barriers to understanding
and involvement
• Accessible for all
Engagement
• Local Authorities now have a duty to engage. This means• Reducing barriers to understanding
and involvement
• Accessible for all
• Available to use when, where & how we want to
Engagement
• Local Authorities now have a duty to engage. This means• Reducing barriers to understanding
and involvement
• Accessible for all
• Available to use when, where & how we want to
• Publishing as data means easy to be repurposed – for mobile, for mashups, for offline use (e.g. Postcode Paper)
Equality
Equality
• At the moment, all this information is available... at a cost
Equality
• At the moment, all this information is available... at a cost
£££
Equality
• At the moment, all this information is available... at a cost
• Huge asymmetry of information (and thus power)
£££
Equality
• At the moment, all this information is available... at a cost
• Huge asymmetry of information (and thus power)
• Raises the barriers to involvement, and, critically, to challenge
£££
EfficiencyNow: cumbersome, wasteful, opaque, error prone
Typical local data flows
EfficiencyNow: cumbersome, wasteful, opaque, error prone
Typical local data flows
Council
EfficiencyNow: cumbersome, wasteful, opaque, error prone
Typical local data flows
Council
Govt Dept A
Govt Dept B
Govt Dept C
web
upload
EfficiencyNow: cumbersome, wasteful, opaque, error prone
Typical local data flows
Council
Govt Dept A
Govt Dept B
Govt Dept C
web
upload
public (maybe)
Enter OpenlyLocal...
Enter OpenlyLocal...
• Inspiration was a Manchester project, MCC Work For You. Doing something concrete to solve a problem.
Enter OpenlyLocal...
• Inspiration was a Manchester project, MCC Work For You. Doing something concrete to solve a problem.
• Screen-scrapes council websites, to turn web pages into structured data. Now over 150 councils done, with basic information on all 434
Enter OpenlyLocal...
• Inspiration was a Manchester project, MCC Work For You. Doing something concrete to solve a problem.
• Screen-scrapes council websites, to turn web pages into structured data. Now over 150 councils done, with basic information on all 434
• Pulls info from about local councils, plus ONS, OS, NPIA, CLG...
Enter OpenlyLocal...
• Inspiration was a Manchester project, MCC Work For You. Doing something concrete to solve a problem.
• Screen-scrapes council websites, to turn web pages into structured data. Now over 150 councils done, with basic information on all 434
• Pulls info from about local councils, plus ONS, OS, NPIA, CLG...
• Increasingly working with local authorities to help them publish their information as open data
Enter OpenlyLocal...
• Inspiration was a Manchester project, MCC Work For You. Doing something concrete to solve a problem.
• Screen-scrapes council websites, to turn web pages into structured data. Now over 150 councils done, with basic information on all 434
• Pulls info from about local councils, plus ONS, OS, NPIA, CLG...
• Increasingly working with local authorities to help them publish their information as open data
• All open data – free for reuse, including commercial reuse
Enter OpenlyLocal...
• Inspiration was a Manchester project, MCC Work For You. Doing something concrete to solve a problem.
• Screen-scrapes council websites, to turn web pages into structured data. Now over 150 councils done, with basic information on all 434
• Pulls info from about local councils, plus ONS, OS, NPIA, CLG...
• Increasingly working with local authorities to help them publish their information as open data
• All open data – free for reuse, including commercial reuse
• 100% accessible. Data first; bells & whistles later
RDF
RDFJSON
RDFJSONXML
OpenlyLocal: where next?
OpenlyLocal: where next?
• More data.
OpenlyLocal: where next?
• More data. • Election data
OpenlyLocal: where next?
• More data. • Election data✓
OpenlyLocal: where next?
• More data. • Election data• Spending data
✓
OpenlyLocal: where next?
• More data. • Election data• Spending data
✓✓
OpenlyLocal: where next?
• More data. • Election data• Spending data• Where I live...
✓✓
OpenlyLocal: where next?
• More data. • Election data• Spending data• Where I live...
✓✓✓
OpenlyLocal: where next?
• More data. • Election data• Spending data• Where I live...• Cabinet members
✓✓✓
OpenlyLocal: where next?
• More data. • Election data• Spending data• Where I live...• Cabinet members• Mayors
✓✓✓
OpenlyLocal: where next?
• More data. • Election data• Spending data• Where I live...• Cabinet members• Mayors• Police authorities
✓✓✓
OpenlyLocal: where next?
• More data. • Election data• Spending data• Where I live...• Cabinet members• Mayors• Police authorities
• More councils.
✓✓✓
OpenlyLocal: where next?
• More data. • Election data• Spending data• Where I live...• Cabinet members• Mayors• Police authorities
• More councils.
• More connections.
✓✓✓
OpenlyLocal: where next?
• More data. • Election data• Spending data• Where I live...• Cabinet members• Mayors• Police authorities
• More councils.
• More connections.
• More ways for councils to give us data.
✓✓✓
OpenlyLocal: where next?
• More data. • Election data• Spending data• Where I live...• Cabinet members• Mayors• Police authorities
• More councils.
• More connections.
• More ways for councils to give us data.
• More tie-ups with hyperlocal news/community sites.
✓✓✓
The OpenElectionData project
The OpenElectionData project
• Tackling the open local data problem, one set at a time & learning lessons on the way
The OpenElectionData project
• Tackling the open local data problem, one set at a time & learning lessons on the way
• Succeed, or fail forward
The OpenElectionData project
• Tackling the open local data problem, one set at a time & learning lessons on the way
• Succeed, or fail forward
• No public database of local election results, only a commercial one (subsidized by the Electoral Commission)
The OpenElectionData project
• Tackling the open local data problem, one set at a time & learning lessons on the way
• Succeed, or fail forward
• No public database of local election results, only a commercial one (subsidized by the Electoral Commission)
• Allows even those with no prior knowledge of linked data/RDF/semantic web (choose fave buzzword here) to take part
The OpenElectionData project
• Tackling the open local data problem, one set at a time & learning lessons on the way
• Succeed, or fail forward
• No public database of local election results, only a commercial one (subsidized by the Electoral Commission)
• Allows even those with no prior knowledge of linked data/RDF/semantic web (choose fave buzzword here) to take part
• Just need HTML competence
The OpenElectionData project
• Tackling the open local data problem, one set at a time & learning lessons on the way
• Succeed, or fail forward
• No public database of local election results, only a commercial one (subsidized by the Electoral Commission)
• Allows even those with no prior knowledge of linked data/RDF/semantic web (choose fave buzzword here) to take part
• Just need HTML competence
• Over 20 councils took part, and invaluable lessons learned (see report)
<tr rel='openelection:candidacy'> <!-- Here we are saying the candidate has given names of Keith William and a familyName Cockroft --> <th scope="row" rel="openelection:candidate"><span property="foaf:givenName">Keith William</span> <span property="foaf:familyName">Cockroft</span></th> <td rel="openelection:party" resource="http://openelectiondata.org/id/parties/6"><span property="rdfs:label">Labour</span></td> <!-- The candidacy got 330 votes --> <td property="openelection:candidateVoteCount" datatype="xsd:integer">330</td> <td>16.6%</td> <td property="openelection:elected" datatype="xsd:boolean" content="false">No</td> </tr> <tr rel='openelection:candidacy'> <th scope="row" rel="openelection:candidate"> <span typeof="openelection:Candidate" property="foaf:name">Brenda Lilian Constable</span> <span rel="openelection:address"> <span property="v:street-address">37 Morley Road</span>, <span property="v:locality">Burntwood</span>, <span property="v:region">Staffordshire</span> <span property="v:postal-code">WS7 2DE</span> </span> </th>
The future for local data? Local government for local people
Publish once, consume many times
Council open dataGovt Dept A
Govt Dept B
Govt Dept C
publicpublic
interest/media/
community/commercial
sites
may
be
Open data opportunities
Open data opportunities
✤ Massive efficiency savings (somewhat important when the cuts are so severe) – reducing duplication, management layers, automating regular routine tasks
Open data opportunities
✤ Massive efficiency savings (somewhat important when the cuts are so severe) – reducing duplication, management layers, automating regular routine tasks
✤ Reduction in bureaucracy (and associated frustration of frontline staff and citizens)
Open data opportunities
✤ Massive efficiency savings (somewhat important when the cuts are so severe) – reducing duplication, management layers, automating regular routine tasks
✤ Reduction in bureaucracy (and associated frustration of frontline staff and citizens)
✤ Improved online services for citizens at zero cost (provided by websites/apps that work with any council)
Open data opportunities
✤ Massive efficiency savings (somewhat important when the cuts are so severe) – reducing duplication, management layers, automating regular routine tasks
✤ Reduction in bureaucracy (and associated frustration of frontline staff and citizens)
✤ Improved online services for citizens at zero cost (provided by websites/apps that work with any council)
✤ Increased transparency
Open data opportunities
✤ Massive efficiency savings (somewhat important when the cuts are so severe) – reducing duplication, management layers, automating regular routine tasks
✤ Reduction in bureaucracy (and associated frustration of frontline staff and citizens)
✤ Improved online services for citizens at zero cost (provided by websites/apps that work with any council)
✤ Increased transparency
✤ More freedom from central government targets, auditing
Challenges
Challenges
✤ Lack of ‘corporate’ awareness/understanding of open data issues
Challenges
✤ Lack of ‘corporate’ awareness/understanding of open data issues
✤ There is a lack of even basic web skills at some councils
Challenges
✤ Lack of ‘corporate’ awareness/understanding of open data issues
✤ There is a lack of even basic web skills at some councils
✤ Many councils lack web publishing resources, never mind the resources to implement open, linked data publishing
Challenges
✤ Lack of ‘corporate’ awareness/understanding of open data issues
✤ There is a lack of even basic web skills at some councils
✤ Many councils lack web publishing resources, never mind the resources to implement open, linked data publishing
✤ The understanding of even the basics of linked data and the steps to publishing public data in this way is very, very limited, even outside of the public sector
Challenges
✤ Lack of ‘corporate’ awareness/understanding of open data issues
✤ There is a lack of even basic web skills at some councils
✤ Many councils lack web publishing resources, never mind the resources to implement open, linked data publishing
✤ The understanding of even the basics of linked data and the steps to publishing public data in this way is very, very limited, even outside of the public sector
✤ Getting councils to publish data in open, linked formats requires dedication, and a range of resources and a mix of skills
Challenges
✤ Lack of ‘corporate’ awareness/understanding of open data issues
✤ There is a lack of even basic web skills at some councils
✤ Many councils lack web publishing resources, never mind the resources to implement open, linked data publishing
✤ The understanding of even the basics of linked data and the steps to publishing public data in this way is very, very limited, even outside of the public sector
✤ Getting councils to publish data in open, linked formats requires dedication, and a range of resources and a mix of skills
✤ Outsourced web publishing contracts often make it difficult and expensive to publish open, linked data
More Challenges
More Challenges
• IDs – councils don’t generally use them; central government uses many... inconsistently
More Challenges
• IDs – councils don’t generally use them; central government uses many... inconsistently
• Data tied up in PDFs (will we ever get this back?)
More Challenges
• IDs – councils don’t generally use them; central government uses many... inconsistently
• Data tied up in PDFs (will we ever get this back?)
• Legacy systems. That nobody now understands
More Challenges
• IDs – councils don’t generally use them; central government uses many... inconsistently
• Data tied up in PDFs (will we ever get this back?)
• Legacy systems. That nobody now understands
• Private companies/JVs. No FoI. No access to data
More Challenges
• IDs – councils don’t generally use them; central government uses many... inconsistently
• Data tied up in PDFs (will we ever get this back?)
• Legacy systems. That nobody now understands
• Private companies/JVs. No FoI. No access to data
• Outsourcing (councils have outsourced the skills & knowledge and now are often dumb consumers)
More Challenges
• IDs – councils don’t generally use them; central government uses many... inconsistently
• Data tied up in PDFs (will we ever get this back?)
• Legacy systems. That nobody now understands
• Private companies/JVs. No FoI. No access to data
• Outsourcing (councils have outsourced the skills & knowledge and now are often dumb consumers)
• Silos, especially of budgets. How do you get money from one budget to another (even if the whole benefits)
Yet More Challenges
Yet More Challenges
✤ The size of the cuts mean they will be blunt, eliminating useful services, sparing wasteful ones. Who & what will be left & what will it mean for local public services?
Yet More Challenges
✤ The size of the cuts mean they will be blunt, eliminating useful services, sparing wasteful ones. Who & what will be left & what will it mean for local public services?
✤ Few resources to invest in doing this, and even less understanding of where to invest it. Danger of using usual procedures, suppliers (procurement, Capita, Serco etc)
Yet More Challenges
✤ The size of the cuts mean they will be blunt, eliminating useful services, sparing wasteful ones. Who & what will be left & what will it mean for local public services?
✤ Few resources to invest in doing this, and even less understanding of where to invest it. Danger of using usual procedures, suppliers (procurement, Capita, Serco etc)
✤ Little incentive for those organisations and people threatened by open data – for them it is a Darwinian fight for survival
An example dataset: local spending data (deadline jan 2011)
An example dataset: local spending data (deadline jan 2011)✤ Technical challenge: how to extract the data from the accounts system
without including non-suppliers (e.g. foster parents, children)✤ How many councils are working on this (hopefully all of them)
✤ How many have blogged, posted the steps/issues/solutions – zero?
✤ What are the key fields that MUST be included, and do these need to be listed somewhere?
An example dataset: local spending data (deadline jan 2011)✤ Technical challenge: how to extract the data from the accounts system
without including non-suppliers (e.g. foster parents, children)✤ How many councils are working on this (hopefully all of them)
✤ How many have blogged, posted the steps/issues/solutions – zero?
✤ What are the key fields that MUST be included, and do these need to be listed somewhere?
✤ Organizational challenge✤ What is this going show about our spending/processes/contracts?
✤ How will we compare with other similar bodies?
✤ What will be the consequences (FoI requests, changed relationship with suppliers, unpopular services)
An example dataset: local spending data (deadline jan 2011)✤ Technical challenge: how to extract the data from the accounts system
without including non-suppliers (e.g. foster parents, children)✤ How many councils are working on this (hopefully all of them)
✤ How many have blogged, posted the steps/issues/solutions – zero?
✤ What are the key fields that MUST be included, and do these need to be listed somewhere?
✤ Organizational challenge✤ What is this going show about our spending/processes/contracts?
✤ How will we compare with other similar bodies?
✤ What will be the consequences (FoI requests, changed relationship with suppliers, unpopular services)
✤ Systemic challenge✤ How will this affect the relationship with the citizens?
✤ What are the implications for our structure, processes, power...?
Power in the public sector was...
✤ Access to people
✤ Access to information
✤ Understanding structure/how things worked
Power in the public sector was...
✤ Access to people
✤ Access to information
✤ Understanding structure/how things worked
undermined by social networking
Power in the public sector was...
✤ Access to people
✤ Access to information
✤ Understanding structure/how things worked
undermined by social networking
democratized by open data
Power in the public sector was...
✤ Access to people
✤ Access to information
✤ Understanding structure/how things worked
undermined by social networking
democratized by open data
crowdsourced? bypassed?
All this is a proxy for the real battle: relevance
All this is a proxy for the real battle: relevance
✤ How relevant is your local government to you?
All this is a proxy for the real battle: relevance
✤ How relevant is your local government to you?
✤ How well do the ‘proper channels’ work for you?
All this is a proxy for the real battle: relevance
✤ How relevant is your local government to you?
✤ How well do the ‘proper channels’ work for you?
✤ What is your ability to achieve things outside the existing structures?
All this is a proxy for the real battle: relevance
✤ How relevant is your local government to you?
✤ How well do the ‘proper channels’ work for you?
✤ What is your ability to achieve things outside the existing structures?
✤ What difference does your council make to your life?
Open data is local government’s chance to stay relevant to their
citizens/economy(but it’ll have its work cut out)
Local Climate Change Challenges and Opportunities: Understanding NYS Municipal Official Perspectives
2015 FOSS4G Track: FOSS4G Adoption in Local Government – Opportunities and Challenges by Dave Murray
Indian Chemical Industry Challenges and Opportunities - Issues, Challenges & Opportunities - Part -5