How to Manage Your DataLisa Spiro, Kathy Weimer and Jean Aroom

June 2017

This workshop draws heavily on materials from the University of Minnesota Libraries, New England Collaborative Data Management Curriculum and DataOne.

https://sites.google.com/a/umn.edu/data-management-workshop-series/http://library.umassmed.edu/necdmc/moduleshttp://library.umassmed.edu/necdmc/moduleshttps://www.dataone.org/education-moduleshttp://library.umassmed.edu/necdmc/modules

Quick Poll: Raise Your Hand If You Have Ever...● Forgotten what you called a file and/or where

you put it● Discovered unnecessary duplicates, then

struggled over which to keep● Lost data due to hardware failure, lost devices,

etc.

Objectives for This Session1. Understand the importance of managing data.2. Learn how to create a good data management

plan.3. Name and organize your files effectively.4. Document your data to promote future use

(including by you). 5. Know options for storing, backing up and

archiving your data.

1. Why Managing Your Data Matters

What Is Research Data?“the information recorded or produced in any form or media during the course of a research investigation”(Rice Research Data Management Policy)Examples:● experimental results● lab notebooks ● protocols● charts● publications

http://professor.rice.edu/uploadedFiles/Professor/Independent_Pages/Policies/Policy308_ResearchManagement.pdf

Data Life Cycle

http://www.data-archive.ac.uk/create-manage/life-cycle

http://www.data-archive.ac.uk/create-manage/life-cyclehttp://www.data-archive.ac.uk/create-manage/life-cycle

Why Is Managing Your Data Important?● Keep track of your data, working more

efficiently.● Prevent data loss.● Uphold standards of research integrity and

reproducbility.● Make it easier to share and re-use data.● Meet funder, university & increasingly journal

requirements.

http://professor.rice.edu/uploadedFiles/Professor/Independent_Pages/Policies/Policy308_ResearchManagement.pdfhttp://libraries.mit.edu/scholarly/publishing/research-funders/research-funder-open-access-requirements/http://libraries.mit.edu/scholarly/publishing/research-funders/research-funder-open-access-requirements/

2. PLAN

Create a data management plan at the beginning of a project

● Define what kind of data you will be generating and using (e.g. file types)

● Determine who will need access and how it will be provided.

● Develop common conventions, e.g. naming & file organization.

● Figure out where the data will be stored and what the costs will be.

Create a Data Management Plan Using DMP Toolhttps://dmptool.org/

https://dmptool.org/https://dmptool.org/

How to Evaluate a DMP Center for Digital Research & Scholarship,Columbia University Libraries, Reviewer’s Worksheet for NSF Data Management Plans”

Evaluate Your DMP

http://scholcomm.columbia.edu/wp-content/uploads/2014/12/Reviewer_Guide_for_DMPs_v01.pdfhttp://scholcomm.columbia.edu/wp-content/uploads/2014/12/Reviewer_Guide_for_DMPs_v01.pdfhttp://scholcomm.columbia.edu/wp-content/uploads/2014/12/Reviewer_Guide_for_DMPs_v01.pdfhttp://scholcomm.columbia.edu/wp-content/uploads/2014/12/Reviewer_Guide_for_DMPs_v01.pdfhttp://scholcomm.columbia.edu/wp-content/uploads/2014/12/Reviewer_Guide_for_DMPs_v01.pdfhttp://scholcomm.columbia.edu/wp-content/uploads/2014/12/Reviewer_Guide_for_DMPs_v01.pdfhttp://scholcomm.columbia.edu/wp-content/uploads/2014/12/Reviewer_Guide_for_DMPs_v01.pdf

Use OSF to Manage Your Research Workflow and Collaborate

• Organize files in one place

• Share with collaborators• Control files access• Integrate with tools like

Box• Track versions• Make work citable • Facilitate reproducibility• Free & open sourcehttps://osf.io/

https://osf.io/https://osf.io/https://github.com/csoderberg/OSF-Curriculum/blob/master/Project_figs/project_overview.png

Use a Data Inventory to Understand, Track & Share Your DataPlan for, monitor & prepare to share your data by recording:● what the dataset is ● who is responsible for it● how data were created● where it is● how important it is● who can access & edit it● where it is stored and preserved

http://www.data-archive.ac.uk/create-manage/strategies-for-centres/data-inventory

Exercise 1: Jot Down What Might Belong in Your Data InventoryData inventory

Courtesy U of Minnesota Library

http://guides.library.uq.edu.au/ld.php?content_id=8034756http://guides.library.uq.edu.au/ld.php?content_id=8034756https://sites.google.com/a/umn.edu/data-management-workshop-series/module2

3. ORGANIZE

Setting Up Directories● Decide conventions for folders with your team and stick

to them.● Organize your files in a predictable, easy-to-sort way.● Use a nesting structure, from broad to narrower. Don’t

make the hierarchy too deep.● Document the system you are using

Common Ways to Organize Your Files: Structuring the Directory

● Level of processing - e.g. raw, cleaned, final (separate raw from processed data)

● Topic - e.g. grant application, background research, experiment, publications

● Organizational structure - e.g. Department, sub-unit, or by individual

● Location/Geography - e.g. Country/State then subdivision of place

Example of Directory Structure

Nikola Vukovic

http://www.vukovicnikola.info/folder-structure-for-research/http://www.vukovicnikola.info/folder-structure-for-research/

File Naming Best Practices

• Be consistent: Use the same structure and terms across projects so that files fall into a useful order (for sorting) and you can easily identify them. For example, project_site_date.

• Be clear: Incorporate relevant terms such as project, place, people, date, method, subject, etc. Use shared, meaningful terminology (e.g. “photo” not “image”)

• Be concise: Software may have difficulty processing long file names.

University of Minnesota LibrariesMIT Libraries

https://sites.google.com/a/umn.edu/data-management-workshop-series/module3https://sites.google.com/a/umn.edu/data-management-workshop-series/module3https://libraries.mit.edu/data-management/files/2014/05/file-organization-july2014.pdfhttps://libraries.mit.edu/data-management/files/2014/05/file-organization-july2014.pdf

File Naming II• Avoid special characters, like / , . # ? • Don’t use blank spaces. Use CamelCharacters or _ to

link together keywords.• Date/time: yyyymmdd is preferred, rather than Dec09

Which file naming scheme works the best?

A. bridgedata1bridgedata2bridgedata3

B. bridge1_sensor2_02142013 bridge1_sensor2_02152013 bridge1_sensor2_02162013

C. madisonavebridge_sensor2_20130214 madisonavebridge_sensor2_20130215 madisonavebridge_sensor2_20130216

D. madisonavebridge_sensor2_feb142013 madisonavebridge_sensor2_02152013 madbridge_s2_feb162013 University of Minnesota Libraries

https://sites.google.com/a/umn.edu/data-management-workshop-series/module3https://sites.google.com/a/umn.edu/data-management-workshop-series/module3

Keep Your Data Tidy•Make each variable a column & each observation a row•Make column headers variable names•Atomize your data; put only a single piece of information in each cell (e.g. city, state, country)

•Be consistent how you will handle empty values (e.g. NULL, leave blank)

•Use standard (ideally non-proprietary) formats for data, e.g. CSV, .txt

Messy vs. Tidy Data

Wickham

http://www.jstatsoft.org/article/view/v059i10http://www.jstatsoft.org/article/view/v059i10

What issues do

you see with this

spreadsheet?

Stanford U Libraries

https://library.stanford.edu/research/data-management-services/case-studies/case-study-spreadsheetshttps://library.stanford.edu/research/data-management-services/case-studies/case-study-spreadsheets

Batch Renaming Tools (for free!)

•All Platformso PSRenamer

•Maco Name Changer

•Windowso Bulk Rename Utility

University of Minnesota Libraries

http://www.powersurgepub.com/products/psrenamer.htmlhttp://www.powersurgepub.com/products/psrenamer.htmlhttp://www.mrrsoftware.com/MRRSoftware/NameChanger.htmlhttp://www.mrrsoftware.com/MRRSoftware/NameChanger.htmlhttp://www.bulkrenameutility.co.uk/http://www.bulkrenameutility.co.uk/https://sites.google.com/a/umn.edu/data-management-workshop-series/module3https://sites.google.com/a/umn.edu/data-management-workshop-series/module3

Exercise

Instructions: Review the handout, then partner with 2-3 people to decide on a file naming system in order to archive all files in one folder and sort by interviewee name.

3 minutes to discuss

University of Minnesota Libraries

https://sites.google.com/a/umn.edu/data-management-workshop-series/module3https://sites.google.com/a/umn.edu/data-management-workshop-series/module3

4. DOCUMENT

Why Document Data?● Makes it easier for you to interpret your own data

● Facilitates collaboration, sharing, and reuse

● Ensures successful long-term preservation of findings

New England Collaborative Data Management Curriculum

http://library.umassmed.edu/necdmc/moduleshttp://library.umassmed.edu/necdmc/modules

Data Documentation● Who collected this data? Who/what were the subjects

under study?● What was collected, and for what purpose? What is the

content/structure of the data?● Where was this data collected? What were the

experimental conditions?● When was the data collected? Is it part of a series or

ongoing experiment?● Why was this experiment performed?

New England Collaborative Data Management Curriculum

http://library.umassmed.edu/necdmc/moduleshttp://library.umassmed.edu/necdmc/modules

Files to replicate Sean Bolks and Richard J. Stoll, “The Arms Acquisition Process: The Effect of Internal and External Constraints on Arms Race Dynamics,” The Journal of Conflict Resolution 44, no. 5 (October 1, 2000): 580–603.

File Contenttable1.dta Stata data file with data for Table 1table1.do Stata .do file with commands to replicate Table 1table2.dta Stata data file with data for Table 2table2.do Stata .do file with commands to replicate Table

Study-level documentation

https://scholarship.rice.edu/handle/1911/77661https://scholarship.rice.edu/handle/1911/77661

Data-Level

Data-level documentation

Data-Level

Variable-level documentation

Data-Level

Process documentation

Data Documentation Handout

5. STORE, BACKUP & ARCHIVE

Data Storage Definition

● The media (optical or magnetic) to which you save your data files and software.

● All storage media are vulnerable to risk and obsolescence.

● Storage media should be evaluated and updated every 2-5 years.

New England Collaborative Data Management Curriculum

http://library.umassmed.edu/necdmc/moduleshttp://library.umassmed.edu/necdmc/modules

Data Storage Considerations

● Location (Internal/External HD, Network, Remote)● Disk size or storage quota● Computing performance● Accessibility

Data Backup Definition

● Allows you to restore your data if original data is lost or damaged due to:○ Hardware or software malfunction○ Environmental disaster (fire, flood)○ Theft○ Unauthorized access

New England Collaborative Data Management Curriculum

http://library.umassmed.edu/necdmc/moduleshttp://library.umassmed.edu/necdmc/modules

Data Backup Considerations

● Location (On-site, off-site)● Procedure (Full, differential, incremental, mirror)● Frequency (Hourly, daily, weekly, monthly)● Retention (Months, years)● Performance

TEST YOUR BACKUP PLAN!

Data Archiving Definition● Entails preserving the final version of your data.● Typically requires documenting data; may involve

converting to formats that facilitate long-term use.● Sometimes data is archived in a public repository,

whether to foster reusability and reproducibility or to comply with funder/ journal requirements.

Data Archiving Considerations

● Location● File formats● Responsibility● Accessibility

Overview of Storage and Backup Options at Rice

Options for faculty/ staff: https://kb.rice.edu/page.php?id=70762Options for students: https://kb.rice.edu/page.php?id=65636

https://kb.rice.edu/page.php?id=70762https://kb.rice.edu/page.php?id=65636

Storage: storage.rice.edu● Location: Networked● Storage quotas

○ Undergraduates: 2 GB○ Graduates, Staff, Faculty: 5 GB○ Colleges, Depts, Centers, Institutes: 40 GB

● Performance - Subject to network● Accessibility

○ NetID folder: Private, not shared○ Groups: Any Rice NetID holder by request

Backup: storage.rice.edu

● Location: On-site● Procedure: Full replication● Frequency: Daily● Retention

○ Personal access: 2 weeks○ Request IT restoration: 6 months

\\storage.rice.edu\?-home\~snapshot

Storage & sharing: box.rice.com● Location: Cloud● Storage quotas: Unlimited● Performance - Can store files locally or subject to

network● Accessibility options

○ Rice users○ Outside collaborators○ General public

https://kb.rice.edu/page.php?id=68656

https://kb.rice.edu/page.php?id=68656https://kb.rice.edu/page.php?id=68656

rice.box.com

Backup: CrashPlan

● Availability: Rice-owned computers● Cost: $82.56/year/person (up to 4 devices)● Location: Off-site cloud storage● Procedure: Incremental● Frequency: Adjustable up to every minute● Retention: Adjustable up to forever

CrashPlan PROe or crashplan.rice.edu

Sharing: Rice Dropbox/Dropoff

● Size: 1 GB per file, 2 GB per dropoff● Retention: 10 days● Accessibility

○ Inside users can share with anyone○ Outside users can only share with inside users

https://dropbox.rice.edu/For more computationally intensive file transfer needs, consider Globus.

https://dropbox.rice.edu/https://dropbox.rice.edu/https://docs.rice.edu/confluence/display/CD/CRC+Data+Management+Policy

dropbox.rice.edu

Data Archiving & Sharing Options● Deposit in an appropriate disciplinary repository

○ Nature, “Recommended Data Repositories”: https://www.nature.com/sdata/policies/repositories

○ PLOS Guide: http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories

○ Re3data: http://www.re3data.org/● Share small to medium datasets through the Rice

Digital Scholarship Archive: https://scholarship.rice.edu/handle/1911/77660

https://www.nature.com/sdata/policies/repositorieshttps://www.nature.com/sdata/policies/repositorieshttp://journals.plos.org/plosone/s/data-availability#loc-recommended-repositorieshttp://journals.plos.org/plosone/s/data-availability#loc-recommended-repositorieshttp://journals.plos.org/plosone/s/data-availability#loc-recommended-repositorieshttp://www.re3data.org/https://scholarship.rice.edu/handle/1911/77660https://scholarship.rice.edu/handle/1911/77660

Data Security

● Confidential (SSN, CC#, DL#)○ Financial records○ Health records○ Education records

● Sensitive (Birth date, address, emergency contact, EID/SID)

Version Control: Manage and Access Versions of Files

https://github.com/rzach/git4phi

● Track changes to files

● Collaborate● Roll back to

earlier versions

https://github.com/rzach/git4phihttps://github.com/rzach/git4phi

Some Options for Version Control

• Subversion: supported by Rice OIT; free• GitHub: Public repositories are free. Researchers can

receive to 5 free private repos, research groups up to 20• Careful file naming: record major changes via whole

numbers (v01), minor via an additional number (v02_01)• A version control table recording the version #, the

change, who made it & the date• Through Box, Google Drive & other storage services

https://docs.rice.edu/confluence/display/ITDIY/Subversionhttps://docs.rice.edu/confluence/display/ITDIY/Subversionhttps://github.com/blog/1840-improving-github-for-sciencehttps://github.com/blog/1840-improving-github-for-sciencehttp://researchdata.wisc.edu/file-naming-and-versioning/http://www2.le.ac.uk/services/research-data/organise-data/version-controlhttps://community.box.com/t5/Managing-Your-Content/How-To-Track-Your-Files-and-File-Versions-Version-History/ta-p/329https://support.google.com/drive/answer/2409045?hl=en

Accessing Version History on Box.com

Resources● Borer, Elizabeth T., et al “Some Simple Guidelines for Effective Data

Management.” Bulletin of the Ecological Society of America (2009): 205–14.● DataOne Primer on Data Management,

https://www.dataone.org/sites/all/documents/DataONE_BP_Primer_020212.pdf● Dataverse, Data Management Plans,

http://best-practices.dataverse.org/data-management/● ICPSR Guide to Social Science Data Preparation and Archiving,

http://www.icpsr.umich.edu/icpsrweb/content/deposit/guide/● Svend Juul et al, “Take good care of your data,”

http://www.epidata.dk/downloads/takecare.pdf● UK Data Archive, Managing and Sharing Data: Best Practices for Researchers,

http://www.data-archive.ac.uk/media/2894/managingsharing.pdf

http://onlinelibrary.wiley.com.ezproxy.rice.edu/doi/10.1890/0012-9623-90.2.205/pdfhttp://onlinelibrary.wiley.com.ezproxy.rice.edu/doi/10.1890/0012-9623-90.2.205/pdfhttp://onlinelibrary.wiley.com.ezproxy.rice.edu/doi/10.1890/0012-9623-90.2.205/pdfhttps://www.dataone.org/sites/all/documents/DataONE_BP_Primer_020212.pdfhttps://www.dataone.org/sites/all/documents/DataONE_BP_Primer_020212.pdfhttp://best-practices.dataverse.org/data-management/http://best-practices.dataverse.org/data-management/http://www.icpsr.umich.edu/icpsrweb/content/deposit/guide/http://www.icpsr.umich.edu/icpsrweb/content/deposit/guide/http://www.epidata.dk/downloads/takecare.pdfhttp://www.epidata.dk/downloads/takecare.pdfhttp://www.data-archive.ac.uk/media/2894/managingsharing.pdfhttp://www.data-archive.ac.uk/media/2894/managingsharing.pdfhttp://www.data-archive.ac.uk/media/2894/managingsharing.pdf

Thanks!Please contact researchdata@rice.edu with any questions.Visit us online at http://researchdata.rice.edu/.Help us shape future workshops! Please complete this evaluation: http://goo.gl/2luM63

mailto:researchdata@rice.eduhttp://researchdata.blogs.rice.edu/https://docs.google.com/a/rice.edu/forms/d/1zG4A0POrg3JIIqcg3gOEZyOMFOLTYwSSRYorlHqDP94/viewformhttp://goo.gl/2luM63