Data Practices and RDA Agreements
Peter WittenburgMax Planck Data and Compute Center (Munich)Former: MPI for Psycholinguistics (Nijmegen)
2
what after DOBES & CLARIN?
since ~2 years retired from MPI Nijmegen in DOBES & CLARIN we created exemplary
infrastructures with proper data organizations this was very well perceived by others
thus we got invited to give advice to the EC and member states to participate in the European EUDAT data infrastructure to participate in the worldwide Research Data Alliance
3
is there a problem?
data as an issue many publications about data deluge many publications with the question whether
we are fit for the challenges of “data science” many discussions about Big Data many stories about inefficiencies and
being excluded (projects, researchers, industry)
in the last 2 years: about 50 interviews in many departments/institutes (different
disciplines, DOBES, CLARIN) in EUDAT, RDA-EU etc. about 80 intensive interactions at scientific meetings in DFT Working group about 20 data models workshop with leading scientists
2010/2011
4
which conclusions from interviews?
Data Management and Processing is too time consuming and costly due to the heterogeneity in how data is organized in particular with respect to logical information. Researchers see the need to change habitudes, but do not have agreed suggestions for solutions.
many researchers stick to file systems – often at its limits Federating data including logical layer information which is relevant
for tracing provenance, for understanding creation context, for checking identity and integrity, etc. is so costly that in practice it is not done, although most data professionals understand that this practice cannot be continued like this.
DM and DP is not ready for Big Data due to the lack of usage of automated procedures incorporating proper data organization mechanisms.
too many ad hoc scripts without proper documentation around
5
which conclusions from interviews?
Due to lack of software that is supporting proper data organizations we continue to create legacy data that cannot be integrated easily into our growing domain of accessible data.
most disciplines have massive problem with legacy, we still create
legacy When we only consider that for example one of the key biologists in
a large research institute is spending 75% of his time for manual data management, we can estimate how much money and human capital is wasted by the way we are doing.
huge waste of money and skills
6some more conclusions from interviews?
Need “chains of trust” in our increasingly anonymous data domain one corner stone is trusted and certified repositories another one is PID association incl. state information
change the way we publish/cite data so that creators are acknowledged
We need to start with training our young people in how to overcome this situation.
of course still hard work on changing data sharing culture
7
one concrete example
a great online database with brilliant data (about 40 TB) created by excellent and engaged researchers some data is in file system some derived data is in mySQL metadata (descriptions about data content) is in a CMS identified are data items by URLs (or are they cool URIs?) relational metadata is stored in Python scripts
how do you want to replicate such data at low costs? how do you want to federate such data at low costs? how do you want to scale up such a solution? how do you want to certify the repository? what when the script expert leaves the institution?
8
are there trends?
current practice is different:
many collections
have their own organization,
own query system, own management
principles, etc.
9
can we learn from Internet?
1986-1989TCP/IP
what happened in all these years?many ideas, much testing, many proprietary solutions, many complete and “closed” solutionsan abstraction towards a unifying protocol + free a lot of time
~1960
10
is there an equivalence in data?
what
Value AddedServices
DataSources
PersistentIdentifiers
PersistentReferenceAnalysis Citation
AppsCustomClients Plug-Ins
Resolution System Typing
PID
Local Storage Cloud Computed
Data Sets RDBMS Files
Digital Objects
PID recordattributes
bit sequence(instance)
metadataattributes
points to instances describes properties
describes properties& context
point toeach other
Internet Domain
nodes with IP numbers
packages being
exchanged
standardized protocols
Data Domain
objects with PID numbers
objects being exchanged
standardized protocols
can we agree on simple concepts ...
-such as an API for PID systems and simple protocols
-it would simplify our software
-could use PID attributes (identity, integrity, rights, MD ref,
data ref, etc.)
-can again focus on other aspects
11here is a role for RDA – Research Data Alliance
How to tackle big challenges if data is highly fragmented by disciplines, domains and countries?
How to tackle big challenges if data is highly fragmented by disciplines, domains and countries?
RDA is building bridges to overcome the hurdles for easy data access, data sharing and interoperability by facilitating
collaboration between experts from all over the world belonging to different disciplines and organisations
12
Global initiative with the support & funding of European Commission, Australian National Data Service and US National Science Foundation
Who supports RDA?(RDA Colloquium)
other countries to
join soon
13Why do we need RDA?
13
Have already so many important initiatives and organizations ...
Many work on policy level which was & is so importantsuch as CODATA and WDS.Others work on standards in different but related areas (ISO, IETF, W3C, IEEE, etc.)
… wanted to have an initiative that acts like early Internet initiative in a bottom-up way to overcome
concrete barriers in short time frames.real progress in practice is so slow!!!
14RDA – How does it work?
Experts and Data practitioners come together in
RDA Working and Interest Groups
to overcome concrete hurdles
Working together by f2f & virtual meetings and on the collaborative web platformWorking together by f2f & virtual meetings and on the collaborative web platform
15The RDA Engine – Working & Interest Groups
12 Working groups including Community Capability Model, Data Citation, Data Foundation and Terminology, Data Type Registries...https://rd-alliance.org/workinggroup-list.html
27 Interest groups including Agricultural Data Interoperability, Big Data Analytics...https://rd-alliance.org/interestgroup-list.html
including joint groups with CODATA and WDS
15
16
Data Foundation & Terminology
• What is a suitable data organization? What is the core of it?• 21 models evaluated and defining a common ground.
• at the core is a Digital Object having a PID and metadata• a PID record is like a passport – has a number, fingerprint, etc.• a world-wide registration system is in place (incl. an API)
Examples from WGs
17
PID information types WG
• Persistent identifiers (PID) are the core of proper data management and access• let’s look at passports: a number is not sufficient ...•…. first solution for standardized PID info types•… will design and implement an API for interaction with typed information• Automated data management across disciplines and repositories can highly benefit from standardized types
(DOI is one specific form of PID used for published data sets to make data citable)
Examples from WGs
18
Data Type Registry WG
• There are so many data types in use, and new ones are
continuously defined in science – innovation is a must• The result is that often researchers see interesting data, but don’t know
how to open, process or visualize the data
•… implementing a type registry for data, which explains how to open,
visualize and process the data
• In 2014 a first implementation for a type registry is expected to be there
• You could build your own – all must be open
Examples from WGs
19
• An Interest Group (IGs) can be established prior to a Working Group for community discussion of issues and areas that facilitate data-driven research. • IGs are longer-term groups defining common issues and interests.
RDA Interest Groups
20RDA – classification (check web site)
RDA WGs cross-disciplinary infrastructural (7)
discipline-specific (1)non-infra (6)
RDA IGs cross-disciplinary infrastructural (10)
discipline-specific (7)non-infra (6)
infra: DFT, DTR, PIT, PP, MDR, Data Citation, MDR Interoperabilitynon-infra: Data Categories & Codes, Certification, Data Publication (4)discipline: Wheat Interoperability
infra: Semantics, PIDs, MD, Provenance, Context, BD Analytics, Longtail D, Brokering, FIM, Preservation, Data Fabric
non-infra: publishing, legal Interoperability, Clouds in Dev. Countries, community capability, engagement
discipline: agriculture, toxic genomics, structural bio, biodiversity, marine data, urban data, data in History and ethnography
21RDA Plenary Meetings …
Plenary 1 – 18- 20 March 2013 Gothenburg, Sweden
240 participants
3 WG, 9 IG
Plenary 2 - 18-20 September 2013 in Washington, DC, USA
380 participants
6 WG, 17 IG, 5 BOF
Plenary 3 - 26-28 March 2014 in Dublin, Ireland
490 participants
16 WG, 35 IG and 20 BOF meetings
10 co-located workshops & meetings
Plenary 4 - 22-24 September 2014 , Amsterdam, Netherlands
come and join the work
Working & interest groups get together and hold face-to-face discussions
New groups proposals & Birds of a Feather
RDA member networking
Co-located events
22RDA Members – who’s engaged?
~1694 membersfrom
73 countries
~1694 membersfrom
73 countries
Region MARCH 2014
%
EU 823 49%
AU 62 4%
US 620 37%
Others 189 11%TOTAL 1694
24Become a member …
Member benefits: join and form Working & Interest Groups, participate in RDA elections, contribute to discussions & debates, comment on emerging groups, attend plenaries, news & updates, etc.
Register to the on-line community and become a Member of RDA - open & free
https://www.rd-alliance.org/user/register
Countries can become
organisational members
OpennessConsensus
BalanceHarmonization
Community DrivenNon-Profit
25The European plug-in to RDA …
RDA Europe Forum – strategic advice RDA Europe Science Workshops –
interaction & feedback from target audience
RDA Europe national & pan-European outreach – to engage new members & disseminate outputs
RDA Europe policy report – to support European policy-makers & funders
RDA Europe Forum – strategic advice RDA Europe Science Workshops –
interaction & feedback from target audience
RDA Europe national & pan-European outreach – to engage new members & disseminate outputs
RDA Europe policy report – to support European policy-makers & funders
RDA Europe, the European plug-in to the global RDA, supports RDA global and brings European voice to the tableRDA Europe, the European plug-in to the global RDA, supports RDA global and brings European voice to the table
26
RDA Collaborative Web Platform rd-alliance.org
Interaction with RDA [email protected]
RDA Europe - [email protected] | europe.rd-alliance.org
Twitter - @resdatall
Facebook - https://www.facebook.com/pages/Research-Data-Alliance/459608890798924
LinkedIn - www.linkedin.com/pub/research-data-alliance/77/115/7aa/
SlideShare - http://www.slideshare.net/ResearchDataAlliance
26
All the links ….
28
do you need to bother?
but ... you will have to create a Data Management Plan, since
funders will not accept current attitudes anymore you will need to make your data visible (metadata), accessible
(repository) and re-usable (metadata) you will need to demonstrate trustfulness of your data
(identity, integrity, authenticity, provenance) trusted repositories will be the anchor of our future data
domain and they will establish requirements to be auditable most results are not re-producible (some surveys:
reproducibility for articles published in scientific journals is as low as 10-30%)
... researchers hate the “overhead” involved.
29Trends relevant for DFT
• we had store and organization together (FS, Fedora, DB, etc)• see a split in function to cope with amounts and complexity • on the one side we see Clouds etc. solving the amount problems• on the other side heterogeneity in the area of the logical layer
logicallayer
components
physicallayer
(FS, Cloud)
30What is Research Data Alliance about?
30
There will be many bridges ...
… building the social and technical bridges that enable global open sharing of data.
31Can your Organisation become a member? … include R&D agencies, for-profit companies and non-
profit foundations, community organizations, institutions, etc. (Annual membership fee based on size of organisation (# persons))
Why should it become a member?
Affiliation with likeminded organisations to coordinate efforts in mutual areas of interest & to avoid unnecessary duplication ...
32
“Knowledge is the engine of our economy. And data is its fuel.”
(Neelie Kroes, Vice-President of the European Commission)
Strong engagement and impact-
Bottom-up meeting top-down
Community engagement through the researchers global involvement in the working and interest group activities
33Community-Driven RDA Groups by Focus
Reference and Sharing - focusedData Citation IG
Data Categories and Codes WG
Legal Interoperability IG
Community Needs - focusedCommunity Capability Model IGEngagement IGClouds in Developing Countries IG
34How can you become a member?
Register to the on-line community and become a Member of RDA.
No fees involved for individual participation. Membership is open to any individual who subscribes to
the RDA Guiding Principles. As a Member one may join and form Working and
Interest Groups and participate in RDA elections. https://www.rd-alliance.org/user/register
19/04/23 34
35RDA Working Groups
Form the Foundation for RDA Community Impact!
… envisioned as accelerants to data sharing practice and infrastructure in the short-term with the overarching goal of advancing global data-driven discovery and innovation
RDA Working Group profile:
Short-term: 12-18 months
Focused efforts with specific actions adopted by specific communities
International participation
Open, voluntary, consensus-driven
Complementary to effective efforts elsewhere
Outcomes / deliverables:
•New data standards or harmonization of existing standards.
•Greater data sharing, exchange, interoperability, usability and re-usability.
•Greater discoverability of research data sets.
•Better management, stewardship, and preservation of research data.
36
high volume vs. complex data V
olu
me
Rank frequency of datatype
mostly “regular”
data
Orphan data
(B. Heidorn)
“Most of the bytes are at the high end, but most of the datasets are at the low end” – Jim Gray
collectiondata
37
do you need to bother?
if you just work in your own domain it may still work still a difference between regular and complex data everywhere the problem when people leave
Info
rmat
ion
C
on
ten
t
Time
Time of publication
Specific details
General details
Accident
Retirement or career change
Death
(Michener et al. 1997)
Data EntropyBill MichenerDataOne
38
are there commonalities?
data one
can work with
organized data
sharable & re-usable
data
scientific data
fabric
often all in file system
often a lot of copying in file system
39RDA Outputs .. What’s coming in 2014 (1/2)
Data Type Registries WG Defining a system of data type registries
Defining a formal model for describing types and building a working model of a registry.
To be adopted by CNRI, International DOI Foundation, and used by the Deep Carbon Observatory and others
(working in conjunction with PID group)
Scheduled to complete Summer, 2014
Persistent Identifier Information Types Defining a minimal set of types that must be
associated with a PID (e.g. checksum, author). Specifying an API for interaction with PID types
Adopted and used by Data Conservancy and DKRZ
(working in conjunction with DTR group)
Scheduled to complete Summer, 2014
Metadata Standards Creating use cases and
prototype directory of current metadata standards from starting point of DCC directory and stakeholder contributions.
To be hosted and used by JISC, DataOne and others
Scheduled to complete Fall, 2014
40
Practical Code policies (rules) Survey of policies in production use
across data management centers. Test bed of machine-actionable policies (IRODS, DataVerse, dCache) at RENCI, DataNet Federation Consortium, CESNET, Odum Institute.
Deployment of 5 policy sets (integrity, access control, replication, provenance / event tracking, publication ) on test beds. Publication of standard policies for use as starter kits.
Scheduled to complete Summer, 2014
Language Codes Operationalization of ISO language
categories for repositories
Adopted and used by the Language Archive, PARADISEC
Proposal of data categories associated with the CMDI schema as ISO standards.
Scheduled to complete Fall, 2014
Data Foundations and Terminology Defining a common vocabulary for data
terms based on existing models. Creating formal definitions in a structured
vocabulary too which also provides an open registry for data terms.
(active input from all RDA WGs)
Tested and adopted by EUDAT, DKRZ, Deep Carbon Observatory, CLARIN, EPOS, and others
Scheduled to complete Summer, 2014
RDA Outputs .. What’s coming in 2014 (2/2)