constructing a federated data distribution service for universities
DESCRIPTION
The GET application enables students and staff at all Swedish universities to download selected areas of basemaps from Lantmäteriet (the Swedish Land Survey). The presentation will contain a technical overview of the application and lessons learned. See more presentations from the FME User Conference 2014 at: www.safe.com/fmeucTRANSCRIPT
![Page 1: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/1.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Jakob Lagerstedt, Mats HögströmConstructing a Federated Data Distribution Service for Universities
![Page 2: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/2.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Overview
Background
Previous Attempts
A modern solution
Some tricks learned
Results
![Page 3: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/3.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Background
We are from the
Swedish University of Agricultural Sciences
![Page 4: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/4.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Background
Data and researchers in Sweden
![Page 5: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/5.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Background
Not free
Image courtesy of KJGarbutt (CC BY 2.0)
![Page 6: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/6.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Background
Strange formats
Image courtesy of Toronto Public Library Special Collections (CC BY 2.0)
![Page 7: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/7.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Previous attempts
Distribution was difficultRequired authorization and data preparation
Researchers would share pieces of data on USB drives (sneakernets)
Image from http://japanese.engadget.com/2005/11/12/hdd-circa-1975/
![Page 8: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/8.jpg)
CONNECT. TRANSFORM. AUTOMATE.
A modern solution
Geographic Extraction Tool
![Page 9: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/9.jpg)
CONNECT. TRANSFORM. AUTOMATE.
A modern solution
Geographic Extraction Tool
![Page 10: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/10.jpg)
CONNECT. TRANSFORM. AUTOMATE.
A modern solution
Geographic Extraction Tool
![Page 11: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/11.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade ()
Do not panic
All the code is in the handouts if you are interested!
Image courtesy of Nico Kaiser (CC BY 2.0)
![Page 12: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/12.jpg)
CONNECT. TRANSFORM. AUTOMATE.
FME server Rest API
Token service /fmetoken/service/generate
HTTP get /fmerest/repositories/" + your_fme_repo_name + ".json
Tricks of the trade (Servlet)
Image courtesy of abraham.williams (CC BY 2.0)
![Page 13: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/13.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (Shutdown Python script)
To enable unsupported functions
Zip & Email & Remove files that have been downloaded, Anything you want!
![Page 14: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/14.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (External script example 1 handout)
Use an external script to enable faster debugging
Monkey patch sys or import something from path (yes we actually use the linux version)
![Page 15: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/15.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (SpatiaLite example 2 handout)
Wonderful format for mixed environments
We have Qgis, Arcmap, FME, they all read it!
Image courtesy of juicyverve (CC BY 2.0)
![Page 16: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/16.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (SpatiaLite)
However, FME does not read as efficiently as other formats
![Page 17: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/17.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (SpatiaLite)
does not set spatial index
![Page 18: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/18.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (SpatiaLite)
Can fix
![Page 19: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/19.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (Google Fusion Tables)
Fusion tables are nice for reporting
But often writes fail (we do not know why, reported the issue)
![Page 20: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/20.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Tricks of the trade (KML network link)
KMLs are nice for reporting
![Page 21: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/21.jpg)
CONNECT. TRANSFORM. AUTOMATE.
Results
23 Universities
100K distributed data sets ( 400 * total area of country)
6000 Unique users
Image courtesy of saamiblog (CC BY 2.0)
![Page 22: Constructing a Federated Data Distribution Service for Universities](https://reader034.vdocuments.mx/reader034/viewer/2022051817/5478ac66b4af9fce158b45cf/html5/thumbnails/22.jpg)
Thank You!
Questions? Handout link
http://pluto.slu.se/drop/public.php?service=files&t=2e0c13550e0c71e963e6f6bb2c52569c
For more information: jakob.lagerstedt | mats.hogstrom @slu.se Swedish University of Agricultural Sciences
CONNECT. TRANSFORM. AUTOMATE.