consultation: science europe draft dmp evaluation rubric · consultation: science europe draft dmp...
Post on 23-Sep-2020
5 Views
Preview:
TRANSCRIPT
Science Europe
Rue de la Science 14, 1040 Brussels, Belgium
Tel: +32 (0)2 226 03 00 Fax: +32 (0)2 226 03 01 Email: office@scienceeurope.org www.scienceeurope.org
Consultation: Science Europe Draft DMP
Evaluation Rubric
Overview of Survey Questions
Introduction
After the publication of its Practical Guide to the International Alignment of Research Data
Management (hereafter referred to as ‘Science Europe RDM Guide’) in January 2019, Science Europe is
currently developing a rubric for the evaluation of Data Management Plans (DMPs).
This rubric is completely aligned with the Science Europe RDM Guide and will support evaluators of
DMPs. It sets out the expectations that a DMP has to meet by listing criteria for two performance
levels, “Sufficiently Addressed” and “Insufficiently Addressed”.
With the consultation, Science Europe would like to gather feedback from stakeholders involved in
research data management (RDM) activities and/or developing RDM policies on the definition of criteria
for the two performance levels. The consultation is structured as follows:
A. Information on the responding organisation and contact person
B. Evaluation criteria for the different aspects of DMPs
C. General comments
Please note: A number of organisations have already implemented the Science Europe RDM Guide on
institutional level or with partners on national level. For this reason, we only seek feedback on the criteria
for the performance levels. The Science Europe DMP guiding questions included in the rubric to facilitate
its use are a direct extract from the Science Europe RDM Guide, and therefore not subject of the
consultation.
The consultation will run until 16 September 2020. Please note that only responses submitted online on
behalf of an organisation or initiative by this deadline will be considered. Only one response per
organisation will be accepted.
If you have any questions, please do not hesitate to contact Marie Timmermann, Senior Policy Officer at
Science Europe: marie.timmermann@scienceeurope.org
CONSULTATION: SCIENCE EUROPE DRAFT DMP EVALUATION RUBRIC
2
A. Information on the Responding Organisation and
Contact Person
1. Name of organisation:
2. Stakeholder group
Multiple choice:
o Research Funding Organisation
o Research Performing Organisation
o University
o Research infrastructure
o eInfrastructure
o Repository / data curation
o Other:
3. Contact Person, replying on behalf of the organisation mentioned above
Name:
Function:
Email:
4. Do you agree to be contacted by one of our experts in case we need more information
regarding the feedback you gave on behalf of your organisation?
5. Has your organisation currently a DMP policy in place or is one being developed?
Multiple choice:
o policy in place
o policy under development
o using DMP template, but no fully fledged policy in place
o no policy in place
6. Did your organisation take part in the consultation rounds in 2018 in preparation of the Science
Europe Practical Guide?
7. Has your organisation used the Science Europe RDM Guide as basis to set up DMP policies?
7a. If yes, how has your organisation used the Guide?
7b. If no, why did your organisation not use the Guide?
CONSULTATION: SCIENCE EUROPE DRAFT DMP EVALUATION RUBRIC
3
B. Evaluation Criteria for the Different Aspects of
DMPs
The DMP core requirements detailed in the Science Europe Practical Guide are divided in six topics with
different subtopics each that are presented in the form of guiding questions.
The following consultation focuses on the criteria to assess whether the core requirements are
sufficiently addressed in a DMP. You will have the opportunity to provide your feedback to the criteria
for each subtopic.
1. Data Description and Collection or Re-use of Existing Data
1a. How will new data be collected or produced and/or how will existing data be re-used?
Performance levels
Requirement sufficiently addressed Requirement insufficiently addressed
Gives clear details of where the data come
from and how new data will be collected or
produced.
If data is reused, explains how existing data
will be accessed and any constraints to
reuse.
If applicable, explains clearly why new data
must be collected, rather than re-using
existing data.
Provides no explanation, or details
insufficient to get a clear understanding of
where the data come from and what data
will be collected or re-used.
If applicable, does not explain sufficient
rationale for generating new data.
Does your organisation agree with the evaluation criteria presented above?
o yes
o no, I would like to suggest changes to the criteria ‘sufficiently addressed’
o no, I would like to suggest changes to the criteria ‘insufficiently addressed’
CONSULTATION: SCIENCE EUROPE DRAFT DMP EVALUATION RUBRIC
4
1. Data Description and Collection or Re-use of Existing Data
1b. What data (for example the kind, formats, and volumes), will be collected or produced?
Performance levels
Requirement sufficiently addressed Requirement insufficiently addressed
Clearly describes or lists what data types will
be generated (e.g. numeric, textual, audio,
video, etc.) and their associated data
formats, including, if needed, data
conversion strategies.
Explains why certain formats have been
chosen, and indicates if they are in open and
standard format.
Provides information about the data volume.
Alternatively, it clearly states that no new
data will be produced or generated by the
project.
NB. Information derived from previously existing
data sources, namely output, processed,
analysed data – are to be considered new data
under this question.
Provides no information, without a valid reason to do so (e.g. a statement that no data will be produced or generated)
Only lists/describes data types without specifying the data formats
Only lists formats, without specifying the data types.
Does your organisation agree with the evaluation criteria presented above?
o yes
o no, I would like to suggest changes to the criteria ‘sufficiently addressed’
o no, I would like to suggest changes to the criteria ‘insufficiently addressed’
CONSULTATION: SCIENCE EUROPE DRAFT DMP EVALUATION RUBRIC
5
2. Documentation and Data Quality
2a. What metadata and documentation (for example the methodology of data collection and way of
organising data) will accompany the data?
Performance levels
Requirement sufficiently addressed Requirement insufficiently addressed
Clearly (even if brief) outlines the documentation needed to enable data re-use, stating where the information will be recorded (e.g. a database with links to each item, a ‘readme’ text file, file headers, code books, or lab notebooks).
Clearly outlines the metadata that will accompany the data, with reference to good practice in the community (e.g. metadata standards). May indicate that no metadata standards exist for the discipline.
Indicates how the data will be organised during the project.
Indicates how such information will be captured and archived with the data.
Provides no information, or only a very vague mention of documentation, without providing any detail or explanation.
Provides little or no detail on metadata that will accompany the data.
Does your organisation agree with the evaluation criteria presented above?
o yes
o no, I would like to suggest changes to the criteria ‘sufficiently addressed’
o no, I would like to suggest changes to the criteria ‘insufficiently addressed’
CONSULTATION: SCIENCE EUROPE DRAFT DMP EVALUATION RUBRIC
6
2. Documentation and Data Quality
2b. What data quality control measures will be used?
Performance levels
Requirement sufficiently addressed Requirement insufficiently addressed
Clearly describes the approach taken to ensure quality control in the collection of data
Provides only a vague mention on how data quality is controlled and documented during the lifetime of the project.
Does your organisation agree with the evaluation criteria presented above?
o yes
o no, I would like to suggest changes to the criteria ‘sufficiently addressed’
o no, I would like to suggest changes to the criteria ‘insufficiently addressed’
CONSULTATION: SCIENCE EUROPE DRAFT DMP EVALUATION RUBRIC
7
3. Storage and Backup During the Research Process
3a. How will data and metadata be stored and backed up during the research?
Performance levels
Requirement sufficiently addressed Requirement insufficiently addressed
Clearly (even if briefly) describes the location where the data and backups will be stored during the research activities.
Indicates that the data will be stored in at least two separate locations.
Clearly indicates how often backups will be performed.
Indicates the use of robust, managed storage with automatic backup, such as storage provided by the home institution.
If not, then it clearly explains why institutional storage cannot be used (and for what part of the data) and describes the (additional) locations, storage media and procedures that will be used for storing and backing up data during the project.
Provides no information or very vague mentions how data will be stored and backed up during the project.
Does your organisation agree with the evaluation criteria presented above?
o yes
o no, I would like to suggest changes to the criteria ‘sufficiently addressed’
o no, I would like to suggest changes to the criteria ‘insufficiently addressed’
CONSULTATION: SCIENCE EUROPE DRAFT DMP EVALUATION RUBRIC
8
3. Storage and Backup During the Research Process
3b. How will data security and protection of sensitive data be taken care of during the research?
Performance levels
Requirement sufficiently addressed Requirement insufficiently addressed
Clearly explains:
› how the data will be recovered in the event
of an incident
› which institutional data protection policies
are in place
› who will have access to the data during the
research.
If sensitive data are involved (for example personal data, politically sensitive information or trade secrets), then it clearly describes the additional security measures (in terms of physical security, network security, and security of computer systems and files) that will be taken to ensure that stored and transferred data are safe.
Gives details of institutional data protection policies in place and provides a link to where they can be accessed.
Provides no information or very vague mentions how the data will be recovered in the event of an incident, which institutional data protection policies are in place and who will have access to the data during the research.
Provides no explanation about data protection and risk management or the explanation is too vague, even though sensitive data (for example personal data, politically sensitive information or trade secrets) are involved.
Does your organisation agree with the evaluation criteria presented above?
o yes
o no, I would like to suggest changes to the criteria ‘sufficiently addressed’
o no, I would like to suggest changes to the criteria ‘insufficiently addressed’
CONSULTATION: SCIENCE EUROPE DRAFT DMP EVALUATION RUBRIC
9
4. Legal and Ethical Requirements, Codes of Conduct
4a. If personal data are processed, how will compliance with legislation on personal data and
security be ensured?
Performance levels
Requirement sufficiently addressed Requirement insufficiently addressed
Clearly indicates if personal data will be collected/used as part of the project, and how data will be lawfully processed.
Indicates what method will be used in order to process data, such as anonymisation
Describes the procedure to manage access to only authorised users.
Considers possible issues with sharing data with collaborators.
Provides information that is too vague.
Lacks assurance which enables reviewers to be satisfied that the project has considered the requirements linked to the processing of personal data, including access management and data sharing issues
Does your organisation agree with the evaluation criteria presented above?
o yes
o no, I would like to suggest changes to the criteria ‘sufficiently addressed’
o no, I would like to suggest changes to the criteria ‘insufficiently addressed’
CONSULTATION: SCIENCE EUROPE DRAFT DMP EVALUATION RUBRIC
10
4. Legal and Ethical Requirements, Codes of Conduct
4b. How will other legal issues, such as intellectual property rights and ownership, be managed?
What legislation is applicable?
Performance levels
Requirement sufficiently addressed Requirement insufficiently addressed
If applicable, it clearly explains › who will control access to which part of the
data
› how intellectual property rights will be
managed.
For multi-partner projects and multiple data owners, it explains how these matters are addressed in the consortium agreement. It may state that this agreement is still being discussed and that the plan will be updated accordingly.
Alternatively, there is a clear statement that there are no such restrictions on the data.
Does not address this issue or only for a subset of the data, without good reason or explanation for doing so.
It is a multi-partner project and matters of rights to control access to the data are not addressed at all, without good reason or explanation for not doing so.
Does your organisation agree with the evaluation criteria presented above?
o yes
o no, I would like to suggest changes to the criteria ‘sufficiently addressed’
o no, I would like to suggest changes to the criteria ‘insufficiently addressed’
CONSULTATION: SCIENCE EUROPE DRAFT DMP EVALUATION RUBRIC
11
4. Legal and Ethical Requirements, Codes of Conduct
4c. What ethical issues and codes of conduct are there, and how will they be taken into account?
Performance levels
Requirement sufficiently addressed Requirement insufficiently addressed
Provides evidence that an ethics committee approval has been obtained for the study (if applicable).
Gives details of what ethical issues have been considered and that adequate measures are in place to manage requirements.
Refers to relevant ethical guidelines.
Does not provide any or enough details to demonstrate that ethical implications have been considered.
Does your organisation agree with the evaluation criteria presented above?
o yes
o no, I would like to suggest changes to the criteria ‘sufficiently addressed’
o no, I would like to suggest changes to the criteria ‘insufficiently addressed’
CONSULTATION: SCIENCE EUROPE DRAFT DMP EVALUATION RUBRIC
12
5. Data Sharing and Long-term Preservation
5a. How and when will data be shared? Are there possible restrictions to data sharing or embargo
reasons?
Performance levels
Requirement sufficiently addressed Requirement insufficiently addressed
Clearly indicates how the data and / or metadata will be discoverable and shared.
Specifies when data will be shared.
Includes information on how long the data will be retained and gives precision on the timely release.
If data sharing is limited or not possible clearly explains why, and under which conditions and who can access the data (certain communities or under sharing an agreement….).
Also explains what actions, where possible, will be taken to overcome or to minimise such restrictions.
There is no explanation, or the explanation is too vague.
Does your organisation agree with the evaluation criteria presented above?
o yes
o no, I would like to suggest changes to the criteria ‘sufficiently addressed’
o no, I would like to suggest changes to the criteria ‘insufficiently addressed’
CONSULTATION: SCIENCE EUROPE DRAFT DMP EVALUATION RUBRIC
13
5. Data Sharing and Long-term Preservation
5b. How will data for preservation be selected, and where data will be preserved long-term (for
example a data repository or archive)?
Performance levels
Requirement sufficiently addressed Requirement insufficiently addressed
Provides details of what data collected or created in the project will be preserved and clearly indicates for how long, in alignment with funders, institutional, national policies or legislation, or community standards
Provides details of which (versions of) data and accompanying documentation will be retained or destroyed, and explains rationale (e.g. contractual, legal requirements or regulatory purposes).
Provides details of how the selection is made, and what possible interest the data would have for re-use (or not).
Provides details of how the data and accompanying documentation will be preserved.
Explains how data will be managed in a sustainable way beyond the lifetime of the grant.
Provides name of the repository - or the way to curate and preserve data - that will be used to make data available for re-use
Provides no further information or lacks adequate explanation.
Does your organisation agree with the evaluation criteria presented above?
o yes
o no, I would like to suggest changes to the criteria ‘sufficiently addressed’
o no, I would like to suggest changes to the criteria ‘insufficiently addressed’
CONSULTATION: SCIENCE EUROPE DRAFT DMP EVALUATION RUBRIC
14
5. Data Sharing and Long-term Preservation
5c. What methods or software tools are needed to access and use data?
Performance levels
Requirement sufficiently addressed Requirement insufficiently addressed
Clearly indicates which specific tools or software (e.g. specific scripts, codes or algorithms developed during the project) potential users may need to access, interpret and (re-)use the data.
Outlines also a strategy for making specific tools and software available. Full details may not be yet available, but there is a clearly stated intention and plan to do so, or explanation of why this may not be possible.
It is stated that software (developed during the project) will be necessary to access and interpret the data, but no explanation of how it will be made available (or why that may not be possible) is offered.
Does your organisation agree with the evaluation criteria presented above?
o yes
o no, I would like to suggest changes to the criteria ‘sufficiently addressed’
o no, I would like to suggest changes to the criteria ‘insufficiently addressed’
CONSULTATION: SCIENCE EUROPE DRAFT DMP EVALUATION RUBRIC
15
5. Data Sharing and Long-term Preservation
5d. How will the application of a unique and persistent identifier (such as a Digital Object Identifier
(DOI)) to each data set be ensured?
Performance levels
Requirement sufficiently addressed Requirement insufficiently addressed
Specifies how the data can be re-used in other contexts.
Clearly indicates if persistent identifiers are
provided for all data sets.
The approach is clearly presented and the choice of identifiers is justified and refers to international standards
Indicates whether a persistent identifier (for instance DOI, Handle, URN, ARK) will be assigned to the data (if not, why not).
Makes no mention of persistent identifiers without a valid reason for not providing them.
Does your organisation agree with the evaluation criteria presented above?
o yes
o no, I would like to suggest changes to the criteria ‘sufficiently addressed’
o no, I would like to suggest changes to the criteria ‘insufficiently addressed’
CONSULTATION: SCIENCE EUROPE DRAFT DMP EVALUATION RUBRIC
16
6. Data Management Responsibilities and Resources
6a. Who (for example role, position, and institution) will be responsible for data management (i.e.
the data steward)?
Performance levels
Requirement sufficiently addressed Requirement insufficiently addressed
Clearly outlines the roles and responsibilities for data management / stewardship (e.g. data capture, metadata production, data quality, storage and backup, data archiving, and data sharing), naming responsible individual(s) where possible.
Clearly indicates who is responsible for reviewing and revising the DMP and for its day-to-day implementation.
For collaborative projects, explains the coordination of data management responsibilities across partners.
Does not discuss responsibility for data management / stewardship activities and/or does not indicate who is responsible for reviewing and revising the DMP and for its day-to-day implementation
It is a collaborative project but there is no description of how data management responsibilities will be coordinated across partners.
Does your organisation agree with the evaluation criteria presented above?
o yes
o no, I would like to suggest changes to the criteria ‘sufficiently addressed’
o no, I would like to suggest changes to the criteria ‘insufficiently addressed’
CONSULTATION: SCIENCE EUROPE DRAFT DMP EVALUATION RUBRIC
17
6. Data Management Responsibilities and Resources
6b. What resources (for example financial and time) will be dedicated to data management and
ensuring that data will be FAIR (Findable, Accessible, Interoperable, Re-usable)?
Performance levels
Requirement sufficiently addressed Requirement insufficiently addressed
Provides clear estimates of the resources and costs (e.g. storage costs, hardware, staff time, costs of preparing data for deposit, and repository charges) that will be dedicated to data management and ensuring that data will be FAIR; describes how these costs will be covered.
Alternatively, there is a statement that no additional resources are needed.
Provides no answer, or is vague about the resources required for data management and ensuring that data will be FAIR (e.g. resources are not listed or costed inappropriately), and/or does not describe how the costs will be covered.
Does your organisation agree with the evaluation criteria presented above?
o yes
o no, I would like to suggest changes to the criteria ‘sufficiently addressed’
o no, I would like to suggest changes to the criteria ‘insufficiently addressed’
CONSULTATION: SCIENCE EUROPE DRAFT DMP EVALUATION RUBRIC
18
C. General Comments
Do you have additional comments or suggestions on the DMP evaluation criteria? [text field]
Do you have comments on the Science Europe RDM Guide in general and on the objective to
achieve a broad alignment of RDM policies across Europe?
top related