kick-off meeting delft, april 9-11 2014 7fp – space no. 607131 data management @ fast first step...
TRANSCRIPT
![Page 1: Kick-off meeting Delft, April 9-11 2014 7FP – SPACE no. 607131 Data management @ Fast First step in data management – repository Gerrit Hendriksen Gerben](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c775503460f9492c065/html5/thumbnails/1.jpg)
Kick-off meetingDelft, April 9-11 2014
7FP – SPACEno. 607131
Data management
@ Fast First step in data management – repository
Gerrit Hendriksen
Gerben J. de Boer
![Page 2: Kick-off meeting Delft, April 9-11 2014 7FP – SPACE no. 607131 Data management @ Fast First step in data management – repository Gerrit Hendriksen Gerben](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c775503460f9492c065/html5/thumbnails/2.jpg)
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Preview
tailored data> WCS> WFS> SOS> SOAP
> netCDF-CF- OPeNDAP> ISO SQL– PostGIS> SVN
> GIT> http> ftp
> KML> WMS> WFS
> CSWgraphics of data
standard dataraw
data
catalogueof data
work done on server
work done on client
Exchange, develop
standards
spatio-temporal
standards
database standards
(lab and field data)
ISO standards
whatwherewhenwho whyhow…data URLs
smart phone & tablet usersscientists professionals
![Page 3: Kick-off meeting Delft, April 9-11 2014 7FP – SPACE no. 607131 Data management @ Fast First step in data management – repository Gerrit Hendriksen Gerben](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c775503460f9492c065/html5/thumbnails/3.jpg)
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
In relation with WP’s/ Tasks
D4.3D4.3
D4.1D4.1
D4.5D4.5
T5.2T5.2
D4.8D4.8
D3.3D3.3
datamanagementdatamanagement
DisseminationDissemination
D4.8D4.8D4.1D4.1 T5.2T5.2D3.3D3.3
T5.1T5.1
![Page 4: Kick-off meeting Delft, April 9-11 2014 7FP – SPACE no. 607131 Data management @ Fast First step in data management – repository Gerrit Hendriksen Gerben](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c775503460f9492c065/html5/thumbnails/4.jpg)
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Techniques in data management
Version control (SubVersion, GIT,…)
OGC netCDF+ OPeNDAP
OGC CF Conventions
OGC PostGIS
GeoServer
CSW
ncWMS, ADAGUC
Data & tools as-isvia version control
General purposedata storage:syntax standard
Geospatialsemantic standard
Server side OGCtailoring & graphics
Discoverymeta-data catalog
ISO SQL(PostgreSQL
)
geospatial semantics
servlets
ETL
![Page 5: Kick-off meeting Delft, April 9-11 2014 7FP – SPACE no. 607131 Data management @ Fast First step in data management – repository Gerrit Hendriksen Gerben](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c775503460f9492c065/html5/thumbnails/5.jpg)
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Data management systeem
(DAS)
modelling
(DDS)
(DAS)(DAS)(DMS)
Mobile applications
Monitoring Network
EO/RS
EO/RS tools
Version control
DAS – Data Acquisition Sub-system DMS - Data Management Sub-system DDS - The Data Dissemination Sub-system
Catalogueservice
![Page 6: Kick-off meeting Delft, April 9-11 2014 7FP – SPACE no. 607131 Data management @ Fast First step in data management – repository Gerrit Hendriksen Gerben](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c775503460f9492c065/html5/thumbnails/6.jpg)
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Open source and - standards
• Software• PostgreSQL• Geoserver• Geonetwork• Mapnik• (QGIS)• Python• OpenStreetMap
• OGC (exchange of data)
• Semantic vocubularies• CF• WORMS (PESI)
![Page 7: Kick-off meeting Delft, April 9-11 2014 7FP – SPACE no. 607131 Data management @ Fast First step in data management – repository Gerrit Hendriksen Gerben](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c775503460f9492c065/html5/thumbnails/7.jpg)
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Reason
?
![Page 8: Kick-off meeting Delft, April 9-11 2014 7FP – SPACE no. 607131 Data management @ Fast First step in data management – repository Gerrit Hendriksen Gerben](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c775503460f9492c065/html5/thumbnails/8.jpg)
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Quality assurance
Vessel visualisation of location and speed of fish tracks carried out during monitoring.
The red lines show deviation from the protocol (maximum speed between 5-10 km/hour). It seemed that this was due
to input errors.
![Page 9: Kick-off meeting Delft, April 9-11 2014 7FP – SPACE no. 607131 Data management @ Fast First step in data management – repository Gerrit Hendriksen Gerben](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c775503460f9492c065/html5/thumbnails/9.jpg)
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Clear standards
![Page 11: Kick-off meeting Delft, April 9-11 2014 7FP – SPACE no. 607131 Data management @ Fast First step in data management – repository Gerrit Hendriksen Gerben](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c775503460f9492c065/html5/thumbnails/11.jpg)
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
For FAST
• Data will be available on an opensource platform:– Scientists– Basis of dissemination- and calculation platform
![Page 12: Kick-off meeting Delft, April 9-11 2014 7FP – SPACE no. 607131 Data management @ Fast First step in data management – repository Gerrit Hendriksen Gerben](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c775503460f9492c065/html5/thumbnails/12.jpg)
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
First step in generalization
• Repository• Web-based system for version control• Allows a worldwide community of people
to collaborate online• Everyone (with proper credentials) can
add/alter/remove and retrieve data• Nothing can be added/altered/removed
without being noticed• Makes sure that changes are merged
without conflicts
• Bear in mind, it is password controlled, so not fully open
![Page 13: Kick-off meeting Delft, April 9-11 2014 7FP – SPACE no. 607131 Data management @ Fast First step in data management – repository Gerrit Hendriksen Gerben](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c775503460f9492c065/html5/thumbnails/13.jpg)
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Frequently used commands
Commit
Checkout (sparse)
Update
Add/Delete/Modify
Log
![Page 14: Kick-off meeting Delft, April 9-11 2014 7FP – SPACE no. 607131 Data management @ Fast First step in data management – repository Gerrit Hendriksen Gerben](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c775503460f9492c065/html5/thumbnails/14.jpg)
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Checkout (sparse)
Create a local copy (only once) Checkout Depth (Windows)
•Fully recursive•Immediate children, including folders•Only file children•Only this item
svn checkout --depth (Command line)•Infinity•Immediates•Files•Empty
![Page 15: Kick-off meeting Delft, April 9-11 2014 7FP – SPACE no. 607131 Data management @ Fast First step in data management – repository Gerrit Hendriksen Gerben](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c775503460f9492c065/html5/thumbnails/15.jpg)
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Update
Download: Server -> Local copy
Note: update before working on a file!
![Page 16: Kick-off meeting Delft, April 9-11 2014 7FP – SPACE no. 607131 Data management @ Fast First step in data management – repository Gerrit Hendriksen Gerben](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c775503460f9492c065/html5/thumbnails/16.jpg)
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Commit
Upload: Local copy –> server Always add a log message!
Command line: svn commit -m”[insert log message]”
![Page 17: Kick-off meeting Delft, April 9-11 2014 7FP – SPACE no. 607131 Data management @ Fast First step in data management – repository Gerrit Hendriksen Gerben](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c775503460f9492c065/html5/thumbnails/17.jpg)
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Add/delete
• Svn addUsed when a new file or folder is added to the repository, before committing. Otherwise, the file will only stay local
• Svn deleteUsed when a file or folder should be deleted. Otherwise, the file or folder is only deleted locally and will reappear at the following update
![Page 18: Kick-off meeting Delft, April 9-11 2014 7FP – SPACE no. 607131 Data management @ Fast First step in data management – repository Gerrit Hendriksen Gerben](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c775503460f9492c065/html5/thumbnails/18.jpg)
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
• Language: English (folder names, file names, code)
• Don’t use spaces in file/folder names, preferably no capitals
• Always add a log message (also in English)
• Update regularly!
conventions
![Page 19: Kick-off meeting Delft, April 9-11 2014 7FP – SPACE no. 607131 Data management @ Fast First step in data management – repository Gerrit Hendriksen Gerben](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c775503460f9492c065/html5/thumbnails/19.jpg)
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
In practice
• Install subversion client– Tortoise (http://tortoisesvn.net/)
• Sparse checkout of the repository of you country• Identify some data• Commit data• Checkout the data
• http://repos.deltares.nl/repos/FAST (mind the capitals)
• temporary username: oegast• temporary password: 0terra
Let’s do it!!!
![Page 20: Kick-off meeting Delft, April 9-11 2014 7FP – SPACE no. 607131 Data management @ Fast First step in data management – repository Gerrit Hendriksen Gerben](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c775503460f9492c065/html5/thumbnails/20.jpg)
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
Meta-data for cataloguing: save *.xml
• Think about copyright and ownership already now!• Think about life cycle after end of FAST: shift to library / Datacentre• DOI
![Page 21: Kick-off meeting Delft, April 9-11 2014 7FP – SPACE no. 607131 Data management @ Fast First step in data management – repository Gerrit Hendriksen Gerben](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c775503460f9492c065/html5/thumbnails/21.jpg)
KIC
K-O
FF
ME
ET
ING
. Del
ft, 9
-11
Apr
il 20
14
7FP – SPACE
OpenEarth wiki (openearth.eu)