towards a frictionless data future
TRANSCRIPT
![Page 1: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/1.jpg)
Toward a Frictionless Data FuturePRESENTED BYJo [email protected] (@jobarratt/@okfnlabs)AT3rd Research Data Network - St Andrews University - 30 November 2016
Licensed under cc-by v3.0 (any jurisdiction)
![Page 2: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/2.jpg)
International non-profit founded in 2004
Who we are● Vision
o A world where open knowledge is ubiquitous, enabling citizens and organizations to create insights that drive change on global and local challenges, combat injustice and inequality and hold governments and corporations to account.
● Missiono Open up all essential, public interest information
and see it used to create insight that drives positive change
o Build communities, tools and skills to empower individuals and organizations to use open information to create insights that drive change.
![Page 3: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/3.jpg)
Widely adopted - over 20 national governments
and 60+ local governments & cities
ckan.org/instances
![Page 4: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/4.jpg)
£4m in 15mins
![Page 5: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/5.jpg)
Frictionless Data is…● Lightweight specifications for “packaging” datasets● Integrations for loading datasets into tools and platforms
relevant to researchers
The Goals...● Introduce a significant, measurable improvement in how research
data is shared, consumed, and analyzed.● Make it easier to maintain and improve data quality.
![Page 6: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/6.jpg)
The Problem
![Page 7: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/7.jpg)
Treemap of issues …
Legal barriers
(open data, sharing agreements etc)
Data Quality
Hard to find
Interoperability
No tool integration
![Page 8: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/8.jpg)
Cargo loading ~1955
Manual, Slow, Costly
(and Dangerous)
![Page 9: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/9.jpg)
Data is Shipping Pre-
Containerization
![Page 10: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/10.jpg)
Containerization
![Page 11: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/11.jpg)
Standards- Standards (a few, simple ones)- Tools (primarily for integration)
- Documentation- Datasets
Data Containerizatio
n
http://www.flickr.com/photos/photohome_uk/1494590209/
![Page 12: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/12.jpg)
Key Principles1. Simplicity
2. Web Oriented
3. Existing Tools
4. Open
5 Distributed
![Page 13: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/13.jpg)
![Page 14: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/14.jpg)
Tabular Data Package
Data Package JSON Table Schema CSV
http://frictionlessdata.io/guides/tabular-data-package/
http://frictionlessdata.io/guides/data-package/http://frictionlessdata.io/guides/json-table-schema/
Tabular Data Package
![Page 15: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/15.jpg)
Tooling …
Data Package
Tabular Data Package
JSON Table Schema CSV
Toole.g. import to R, SQL etc
Toole.g. data checking ala GoodTables
![Page 17: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/17.jpg)
Validation
![Page 18: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/18.jpg)
Continuous Validation ● If you’re working in a group, you need continuous validation…
for data!● In < 1 hour, we integrated elements (datapackage.json + Python
libraries + GoodTables API) to support continuous data validation
![Page 19: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/19.jpg)
http://uk-25k.datadashboards.io/
![Page 20: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/20.jpg)
Platform Integrations
![Page 21: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/21.jpg)
Partners
View more: http://frictionlessdata.io/partners/
Dataship
YOUR RESEARCH
ORG
![Page 22: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/22.jpg)
● Project website: http://frictionlessdata.io/● Specifications: http://specs.frictionlessdata.io/● GitHub: https://github.com/frictionlessdata/● User Stories:
http://frictionlessdata.io/user-stories/
![Page 23: Towards a frictionless data future](https://reader035.vdocuments.mx/reader035/viewer/2022081515/5871a1e61a28ab044e8b702d/html5/thumbnails/23.jpg)
● Newsletter: http://frictionlessdata.io/get-involved/#newsletter
● Follow @okfnlabs on Twitter (#frictionlessdata)