about opendata

Lorenzino Vaccari 22/06/2016

About Open Data(Please, interrupt me and make questions!)

1

[email protected]

[email protected]

Lorenzino Vaccari

Seminar at POLIMI@Lecco

mailto:[email protected]





Who am I?

2


Part 1: for data prosumers• What are Open Data useful for?• What is Open Data?• Why are Open Data useful? • How is Open Data related to Open

Government Data and Big Data? • The Open Data movement

3

Lorenzino Vaccari 22/06/20164http://15years.morizbuesing.com/

Austrian designer Moriz Büsing created this grim interactive map of migrant and refugee deaths on the way to Europe, or trying to stay in Europe; over 32,000 deaths in 15 years.

http://15years.morizbuesing.com/







Open Source & Open Data together to tackle with humanitarian projects and economic development

5

HOT: Humanitarian OpenStreetMap Team

http://hot.openstreetmap.org

http://hot.openstreetmap.org/



Lorenzino Vaccari 22/06/20166http://www.webmapp.it/mappe/dolomiti/


Map of the pianos

7https://github.com/brunetton/OpenPianosMap

https://github.com/brunetton/OpenPianosMap

https://github.com/brunetton/OpenPianosMap

Lorenzino Vaccari 22/06/20168http://content.stamen.com/files/cartography/index_watercolor.html

http://content.stamen.com/files/cartography/index_watercolor.html





Open Data Shoes

9http://in2.ccio.co/K2/LA/G/218143175671272930BwpG7bSyc.jpg


What is Open Data?

10


“is data that can be freely used, reused and redistributed by anyone – subject only, at most, to

the requirement to attribute and sharealike.” *

*(Source: )

http://opendatahandbook.org/guide/en/what-is-open-data/ 11

http://opendatahandbook.org/guide/en/what-is-open-data/

http://opendatahandbook.org/guide/en/what-is-open-data/


• use• reuse• redistribution• commercial reuse• derivative works

BUT, may require:• attribution• share alike

J. Gray (OKF): http://www.slideshare.net/jwyg/open-government-data-what-why-how12

“open” =

http://www.slideshare.net/jwyg/open-government-data-what-why-how

http://www.slideshare.net/jwyg/open-government-data-what-why-how


“Open” data

13

• Open License• Free• Open Access, e.g.:

• No registration

• No co-authorship

• Direct access (no services)

• ….

https://unsplash.com/@ryanmoreno




Open License

14

• A license should be compatible with other open licenses.

• A license is open if its terms satisfy the following conditions...

https://unsplash.com/@rzunikoff


Open license: Required Permissions

15

The license must irrevocably permit (or allow) the following:

Use: The license must allow free use of the licensed work.

Redistribution: The license must allow redistribution of the licensed work, including sale, whether on its own or as part of a collection made from works from different sources.

Modification: The license must allow the creation of derivatives of the licensed work and allow the distribution of such derivatives under the same terms of the original licensed work.

Separation: The license must allow any part of the work to be freely used, distributed, or modified separately from any other part of the work or from any collection of works in which it was originally distributed. All parties who receive any distribution of any part of a work within the terms of the original licenseshould have the same rights as those that are granted in conjunction with the original work.

Compilation: The license must allow the licensed work to be distributed along with other distinct works without placing restrictions on these other works.

Non-discrimination: The license must not discriminate against any person or group.

Propagation: The rights attached to the work must apply to all to whom it is redistributed without the need to agree to any additional legal terms.

Application to Any Purpose: The license must allow use, redistribution, modification, and compilation for any purpose. The license must not restrict anyone from making use of the work in a specific field of endeavor.

No Charge: The license must not impose any fee arrangement, royalty, or other compensation or monetary remuneration as part of its conditions.

http://opendefinition.org/od/2.1/en/




Open license: Acceptable Conditions

16

The license must not limit, make uncertain, or otherwise diminish the permissions required in Section 2.1 except by the following allowable conditions:

Attribution: The license may require distributions of the work to include attribution of contributors, rights holders, sponsors, and creators as long as any such prescriptions are not onerous.

Integrity: The license may require that modified versions of a licensed work carry a different name or version number from the original work or otherwise indicate what changes have been made.

Share-alike: The license may require distributions of the work to remain under the same license or a similar license.

Notice: The license may require retention of copyright notices and identification of the license.

Source: The license may require that anyone distributing the work provide recipients with access to the preferred form for making modifications.

Technical Restriction Prohibition: The license may require that distributions of the work remain free of any technical measures that would restrict the exercise of otherwise allowed rights.

Non-aggression: The license may require modifiers to grant the public additional permissions (for example, patent licenses) as required for exercise of the rights allowed by the license. The license may also condition permissions on not aggressing against licensees with respect to exercising any allowed right (again, for example, patent litigation).





Open “Data”

17

Best practices:• Primary source• Timely• Open format • Updated and complete• Machine readable • ...

Lorenzino Vaccari 22/06/2016Maurizio Napolitano: http://www.youtube.com/watch?v=YlkjrVAW43Q

Primary source

18

http://www.youtube.com/watch?v=YlkjrVAW43Q

http://www.youtube.com/watch?v=YlkjrVAW43Q


Open Format & Machine Readable

19http://5stardata.info/en/

Lorenzino Vaccari 22/06/201620https://upload.wikimedia.org/wikipedia/commons/7/79/14LaAc_periodic_table_IIb.jpg

“It’s great to have the data accessible on the Web under an open license (such as PDDL, ODC-by or CC0), however, the data is locked-up in a document. Other than writing a custom scraper, it’s hard to get the data out of the document.”

make your stuff available on the Web (whatever format) under an open license

http://opendatacommons.org/licenses/pddl/

http://opendatacommons.org/licenses/by/

http://creativecommons.org/publicdomain/zero/1.0/


“Splendid! The data is accessible on the Web in a structured way (that is, machine-readable), however, the data is still locked-up in a document. To get the data out of the document you depend on proprietary software.”

make it available as structured data (e.g., Excel instead of image scan of a table)


DateTime,MC2016-01-01 00:00:00.000,58.8083312016-01-01 00:10:00.000,59.3740012016-01-01 00:20:00.000,58.7208332016-01-01 00:30:00.000,57.982016-01-01 00:40:00.000,57.6060032016-01-01 00:50:00.000,56.7620012016-01-01 01:00:00.000,55.6591842016-01-01 01:10:00.000,54.942862016-01-01 01:20:00.000,54.2632682016-01-01 01:30:00.000,52.9224512016-01-01 01:40:00.000,53.1673472016-01-01 01:50:00.000,54.8079992016-01-01 02:00:00.000,57.0632632016-01-01 02:10:00.000,58.2571412016-01-01 02:20:00.000,58.0359992016-01-01 02:30:00.000,57.8612252016-01-01 02:40:00.000,57.071432016-01-01 02:50:00.000,56.3387762016-01-01 03:00:00.000,55.452….

http://data.jrc.ec.europa.eu/dataset/jrc-abcis-ap-pm10mc-2016

“Excellent! The data is not only available via the Web but now everyone can use the data easily. On the other hand, it’s still data on the Web and not data in the Web.”

make it available in a non-proprietary open format (e.g., CSV as well as of Excel)

https://webofdata.wordpress.com/2010/03/01/data-and-the-web-choices/

Lorenzino Vaccari 22/06/201623https://data.europa.eu/euodp/en/data/dataset/jrc-names

“Wonderful! Now it’s data in the Web. The (most important) data items have a URI and can be shared on the Web. A native way to represent the data is using RDF, however other formats such as Atom can be converted/mapped, if required.”

use URIs to denote things, so that people can point at your stuff


“Brilliant! Now it’s data,in the Web linked to other data. Both the consumer and the publisher benefit from the network effect.”

https://data.europa.eu/euodp/en/data/dataset/jrc-names

link your data to other data to provide context

https://en.wikipedia.org/wiki/Network_effect

https://en.wikipedia.org/wiki/Network_effect


Why are Open Data useful?

25


The value is in its use

26http://clicnews.ie/tag/lego/

http://clicnews.ie/tag/lego/

http://clicnews.ie/tag/lego/


Open Data Benefits● The Open data are the knowledge base to:

● Improve the economic grow and the entrepreneurship based on the development of digital services reusing Public Sector Information

● Answer to social needs through the publication of innovative services and applications

● Aims at reducing the cost of the public administrative activities within Public – Private Partnerships (PPP)

● Improve the transparency of the activities of the public institutions and the participation of the citizens to these activities

27


Economic Growth“Today, the cumulative value

of products and services derived from open access to weather data is estimated at $15 billion.”

http://www.accuweather.com/

http

://w

ww

.soc

rata

.com

/blo

g/ec

onom

ic-i

mpa

ct-o

pen-

data

/



http://www.socrata.com/blog/economic-impact-open-data/

http://www.socrata.com/blog/economic-impact-open-data/


Potential value in Open Data ($billions)

29


Innovation: new visualizations

http://wheredoesmymoneygo.org/ 30

http://wheredoesmymoneygo.org/




How are Open Data related to Open Government Data and Big Data?

31


Open Government Data

32

“The three principles of transparency, participation, and collaboration form the cornerstone of an open government”

Barack Obama, 8/12/2009

https://www.whitehouse.gov/sites/default/files/omb/assets/memoranda_2010/m10-06.pdf





Open Government Data

33


Big Data & Open DataVariety

Volume Velocity

• Structured• Unstructured• Semi-structured• …

• Terabytes• Records• Transactions• Tables, Files

• Batch• Real Time• Streams• Near-time

3V’s

34

Open Data is often one of the sources for Big Data


State of the art: the Open Data movementWhat is happening around us ? Some examples...

● Globally● Europe● Italy● Locally

35


Open Data Charter - G8 (12/07/2013)The principles are:

● Open Data by Default

● Quality and Quantity

● Useable by All

● Releasing Data for

Improved

Governance

● Releasing Data for

Innovation

http://opensource.com/government/13/7/open-data-charter-g8

https://www.gov.uk/government/publications/open-data-charter/g8-open-data-charter-and-technical-annex

36





Lorenzino Vaccari 22/06/2016http://census.okfn.org/

OGD around the world

37

http://census.okfn.org/

http://census.okfn.org/


The GEOSS portal

38

The GEOSS CORE data

principles

● Full and Open Exchange of

Data, recognizing Relevant

International Instruments

and National Policies

● Data and Products at

Minimum Time delay

● Free of Charge or minimal

Cost for Research and

Education

http://www.geoportal.org/web/guest/geo_home






OpenStreetMap: OD & Crowdsourcing

39

OpenStreetMap is a free map of the world, created by someone like you

“OpenStreetMap project creates and provides geographical data, such as road

maps, freely available to anyone. Behind the establishment and growth of the project have been restrictions on use

or availability of map information across much of the world and the advent

of inexpensive portable satellite navigation devices”

https://www.openstreetmap.org


An example: Lecco

40http://tools.geofabrik.de/


OGD in Europe - Pan European

http://www.europeandataportal.eu/en/ 41

Connecting Europe Facility launches second call(16/05/2016)

The Connecting Europe Facility (CEF) in Telecom is an EU programme to facilitate cross-border interaction between public administrations, businesses and citizens, through the deployment of Digital Service Infrastructures. One of its aims is to support projects which contribute to the European ecosystem of the deployed interoperable and interconnected digital services.…

485,473 datasets found

http://www.europeandataportal.eu/en/

http://www.europeandataportal.eu/en/


OGD in Europe - EU ODP● screenshots

http://open-data.europa.eu/ 42

http://open-data.europa.eu/

http://open-data.europa.eu/


The INSPIRE geoportal

43http://inspire-geoportal.ec.europa.eu/discovery/


OGD in Italy

http://www.dati.gov.it44

http://www.dati.gov.it/dataset

http://www.dati.gov.it/dataset


OGD in Lombardy

45https://www.dati.lombardia.it/


Open Data @Lecco?

46

● Search if Lecco has an official Open Data web site: ○ Which datasets (domains)?○ Which formats?

■ How many stars?○ Which licenses?

■ Is it clear the type of license for each dataset?

● Are there any other web sites in Lecco?○ Are there any Universities which share

Open Data?


Your Open Data (data provider)

47

● Do you think you could be an Open Data provider? E.g. with the datasets of your thesis?

● Would you like to share them openly?

● If not, why?


Your Open Data (data consumer)

48

● Which data are you working on?○ Where do you get them from?

● Which data would you like to find on Internet?○ Are the dataset you download fine with

you? If not, why?


Questions?About Open Data

49

Next part: 2 - for men in the middle


Part 2: for men in the middle● Open Data Issues● Two experiences:

○ Autonomous Province of Trento

■ The story started with GeoData…

■ Now “Open Data in Trentino”: http://dati.trentino.it

■ Community building

○ European Commission: Joint Research Centre

■ http://data.jrc.ec.europa.eu

● Want to learn more?

50

http://dati.trentino.it/

http://data.jrc.ec.europa.eu



“Yeahh!!!”

https://unsplash.com/@littleppl85

51





LegalOrganizational TechnicalAdoptionBarriers

Contextual

52http://goo.gl/9dFm9v

“Ohoh!!!”

Lorenzino Vaccari, Juan Pane 22/06/201653

Organizational Barriers

● Not ready

● Lack of resources (IT, Human)

● Don’t want to be ready

http://montcomediation.org/images/MCMC_MyWayYourWay.jpg




Legal barriers

● Open the Data ○ All the data that was produced

using public money has to be

made publicly available (with

exceptions)

● vs Privacy○ You cannot open data that

could allow correlation of

private personal data

http://s177.photobucket.com/user/sealth2828/media/gavel.jpg.html


● Data is not contextualized

● Opening data is a complex task, opening

cleaned data is even more complex.

● Unclear licenses

Adoption barriers

http://www.thepadrino.com/2011/01/defendius-labyrinth-security-lock.html


Technical Barriers● Access to data:

○ Organizational○ Technical, Downtimes,

logins, ○ Payment fees

● Fragmentation, incomplete data, scattered

● Format● Cataloging,

indexing, search● Lack of explicit

semantics, metadata

● Conflicting standards, models, ontologies


● Privileged access to data● Transparency is bad for fraudulent business

Context Barriers

http://img.gawkerassets.com/img/182n8vzdlg1iojpg/original.jpg




● Zuiderwijk et al 2010

● Listed 118 socio-technical impediments for opening data in the literature such as:○ Findability○ Usability○ Understandablity○ Quality○ Linking○ Comparability and compatibility○ Metadata○ ….

Barriers


Congratulation for the presentation! I am curious about the data you used! Are these datasets freely available? Would you like to publish them as Open Data in the catalog we are creating at the JRC level? Here there is a draft version: http://data.jrc.ec.europa.eu/ . Cheers,Lorenzino

-------------------Hi Lorenzino,sorry but I am not allowed to publish my dataset.Cheers,Xyz

Meanwhile at the JRC… The Data are MINE !

http://data.jrc.ec.europa.eu/




Not Exactly...


How to deal with Open Data issues?


Autonomous Province of Trento● The story started with GeoData…● Now “Open Data in Trentino”: http://dati.

trentino.it ● Community building

63





The story started with GeoData …

http://www.territorio.provincia.tn.it

http://www.territorio.provincia.tn.it/




5 Stars Linked Geo Data Catalog

http://www.territorio.provincia.tn.it




The “Open Data in Trentino” project

66

• The “Open Data in Trentino” project is a 3 years initiative finalized to develop an open data infrastructure to enhance Service Innovation for Trentino following the PAT strategy for services innovation enabled by ICT. The project will be developed within a partnership between Trento RISE and the Autonomous Province of Trento (PAT) according to the innovation PAT model

• Goals• Improved quality of life for citizens• Open Data and local businesses• Transparency• Improved efficiency and productivity

Adopted licences

08/10/2013Juan Pane, Lorenzino Vaccari67

CC0 CCBY

Permissions: share, create, adapt Permissions: share, create, adapt

Actual interoperability! Decent interoperability

Constraints: nothing! Constraints: attribution

Catalogue

08/10/2013Juan Pane, Lorenzino Vaccari69

The Open Knowledge Foundation (OKF) is a non-profit organisation founded in 2004 and dedicated to promoting open data and open content in all their forms – including government data, publicly funded research and public domain cultural content.

http://okfn.org

http://okfn.org/

http://okfn.org/


… inspired from the guidelines of Lombardy!

72http://www.agendadigitale.regione.lombardia.it/

http://www.agendadigitale.regione.lombardia.it/

http://www.agendadigitale.regione.lombardia.it/


Services & Apps (do you remeber the cake)? Services & Apps

http://dati.trentino.it/related


Create Community

74http://media.gettyimages.com/photos/members-of-the-colla-vella-de-valls-climb-up-as-they-construct-a-picture-id153610809


Trentino Open Data (TOD)

76https://www.facebook.com/groups/todgroup/

Lorenzino Vaccari 22/06/201680 Lorenzino Vaccari


European Commission: Joint Research Centre

● http://data.jrc.ec.europa.eu

81




JRC and Open Access

82

● for scientific publications/data within Horizon 2020 and by other relevant initiatives (e.g. Research Data Alliance)

● overall trend for public move to open data (G8 charter, INSPIRE..)

As continuation of JRC's efforts to make available and transparent to the public the scientific knowledge produced, in 2014 the JRC will roll out its Open Access strategy for its publications

JRC Management Plan 2014Commission Decision on the reuse of Commission documents (2011/833/EU)

Open Access in EC and beyond

, Anders Friis-Christensen, Andrea Perego


JRC Open Data project

83

JRC Data Policy- Open Data principles- Data acquisition principles- Data management principles- Implementation principles

JRC Data CatalogueContaining JRC datasets

related to, e.g., Soil, Water, Air quality, Marine,

Biodiversity, etc. http://data.jrc.ec.europa.eu

.

EU Open Data portal

A single access point to a growing range of data from the institutions and other bodies of the EU

https://open-data.europa.eu

Commission Decision on the reuse of Commission documents (2011/833/EU)

Lorenzino Vaccari, Anders Friis-Christensen 22/06/201684 Lorenzino Vaccari, Anders Friis-Christensen 22/06/2016

Lorenzino Vaccari, Anders Friis-Christensen 22/06/201685

A JRC data infrastructure

Lorenzino Vaccari, Anders Friis-Christensen 22/06/201686

Project Scope

JRC Data Policy

Data Policy Implementation Guidelines

Software components

(e.g. data dissemination)

Data

Open Data

Applies to


http://dati.jrc.ec.europa.eu





Want to learn more?

89

Lorenzino Vaccari 22/06/201690http://opendatahandbook.org/pt_BR/

http://opendatahandbook.org/pt_BR/




The EU ODP Training

91http://www.europeandataportal.eu/elearning/

http://www.europeandataportal.eu/elearning/

http://www.europeandataportal.eu/elearning/

Lorenzino Vaccari 22/06/201692http://schoolofdata.org/

http://schoolofdata.org/




Lorenzino Vaccari 22/06/201693http://www.theodi.org/

http://www.theodi.org/

http://www.theodi.org/

Lorenzino Vaccari 22/06/201694http://www.socrata.com/open-data-field-guide/

http://www.socrata.com/open-data-field-guide/





Open Data event in Lecco (8/6/2016)

95http://www.comune.lecco.it/index.php/archivio-news/23-news-dal-comune/2437-convegno-open-data-e-sharing-economy

http://www.comune.lecco.it/index.php/archivio-news/23-news-dal-comune/2437-convegno-open-data-e-sharing-economy


Open Data & Smart Cities (EU ODP)

96

Analytical Report 4: Open Data in Cities

http://www.europeandataportal.eu/sites/default/files/edp_analytical_report_n4_-_open_data_in_cities_v1.0_final.pdf


A question for you: is it difficult to use a data catalogue? Why?

97

From the user point of view (what I found):● I do not known about it● I cannot found what I need

○ “Spaghetti” catalogues■ many records■ not clear what is inside (no clear classification)■ too few datasets

● I do not receive updates○ On datasets I am interested in

● Even if I found it ○ I cannot access it

■ Broken links, access barriers (registrations,…)○ Is the dataset the last version?


Questions?

98

Thanks For Your Attention!!!!

Acknowledgments: ● Anders Friis-Christensen● Maurizio Napolitano● Juan Pane● Andrea Perego

Lorenzino Vaccari [email protected]

[email protected]





about opendata

Science