emc metalnx - irods · title: microsoft powerpoint - 150610_metalnx_irods_ugm15_distrib.pptx...
TRANSCRIPT
![Page 1: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/1.jpg)
1© Copyright 2015 EMC Corporation. All rights reserved.
EMC MetaLnx –An iRODS Admin & Metadata UISteve Worth
Director, Global New Business Development Operations
6/10/2015iRODS User Group Meeting
![Page 2: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/2.jpg)
2© Copyright 2015 EMC Corporation. All rights reserved.
Motivation• Observations
– A data deluge is upon us – will surpass 44 Zetabytes by 2020• Big data, IOT, and mobile computing are changing how we interact with data
– Cost dynamics are changing• Cloud computing / open source feed the “race to zero” in computing
– Opex costs begin to dominate over Capex costs
• Implication– “Big Data” is now IP– Advanced metadata facilitates IT data control
• Investigation– iRODS enables next generation metadata– How will advanced metadata facilitate IT operations?
![Page 3: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/3.jpg)
3© Copyright 2015 EMC Corporation. All rights reserved.
Advanced Metadata is Key• Maximize storage assets - find what’s valuable, no matter where it’s
located– Eliminate data that is not valuable (so called “dark data”)
• Metadata supports indexing / fast search methods– Ideal for non-SQL techniques
• Automate movement and processing of data
• Metadata can be stored/transported more cheaply– An object with 1000 tags, 4096 character limit, 3 tag AVU model requires < 12MB for
the tag set.
• Securely share data with collaborators– Easier to separate PHI from non-PHI data– Can be used to ensure data is authentic and unaltered
![Page 4: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/4.jpg)
4© Copyright 2015 EMC Corporation. All rights reserved.
The Next Important IT Skill…• Advanced metadata will be
a core information tool
• With data growing at over 2 Exabytes / year the need to catalogue, structure, and organize data will increase
• The Internet of things will make long term curation of data more critical (100+ years)Information
Science(In past times this was called Library Science)
![Page 5: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/5.jpg)
5© Copyright 2015 EMC Corporation. All rights reserved.
Want Proof?This is the modern day collegiate library….
James B. Hunt Library – North Carolina State Univ.
See any books?
![Page 6: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/6.jpg)
6© Copyright 2015 EMC Corporation. All rights reserved.
IT World According to GenomesData Management:- Data collection- Archiving- Backup / protection- Access / Rights mgmt.- Auditing- Providence- Federation- Data workflow- Perpetual curation
Analysis:- Sequence generation- Variation processing- Protein analysis- Palindrome analysis
Metadata Management:- Research Annotations- Phenotype Data Tagging- Subject Data Tagging- Structural Tagging
Workflow:- Sample sequencing- Data workflow- Analysis workflow- Variant workflow- Reports
Analytics:- Population trending- Reference genome- Pattern matching Genomics leverages all aspects of
computing, with big data sets, and interesting curation needs
![Page 7: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/7.jpg)
7© Copyright 2015 EMC Corporation. All rights reserved.
• Administration– System / account management– User access and security– Sharing/federation– Data organization– Data archiving
• Users (Curation/Research)– Data access/organization– Metadata identification – Template utilities– Searching– Sharing (Internal/External)– IT is a tool only
Genome Management Design Approach
![Page 8: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/8.jpg)
8© Copyright 2015 EMC Corporation. All rights reserved.
• User commands (subset)– iinit– iput– iget– imkdir– ichmod– icp– irm– ils– ipwd– icd– irepl– iexit– ipasswd– ichksum– imv– iphymv– ireg– irmtrash– irsync
• Iadmin commands (subset)– lt– lr– ls– lz– lg– mkuser– moduser– aua– rua– rpp– rmuser– mkdir– rmdir– mkresc– modresc– modrescdatapaths– rmresc– mkzone– modzone
Working With iRODS…
![Page 9: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/9.jpg)
9© Copyright 2014 EMC Corporation. All rights reserved.© Copyright 2014 EMC Corporation. All rights reserved.
EMC MetaLnx Demo Setup
Network
swcentos6-1iCAT server,
local disk
iRODSstorage
resources
swdebian64-1iRODS resource server, 2 local storage resources
swcentos6-2MetaLnx server
& client
iRODSgrid
Virtual Machine Layout:1 core, 1 GB memory, 10
GB disk / each
Physical System:Intel i7, 4 cores, 12 GB memory
![Page 10: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/10.jpg)
10© Copyright 2014 EMC Corporation. All rights reserved.© Copyright 2014 EMC Corporation. All rights reserved.
EMC MetaLnx Demo Setup
Network
swcentos6-1iCAT server,
local disk
iRODSstorage
resources
swdebian64-1iRODS resource server, 2 local storage resources
swcentos6-2MetaLnx server
& client
iRODSgrid
Virtual Machine Layout:1 core, 1 GB memory, 10
GB disk / each
Physical System:Intel i7, 4 cores, 12 GB memory
![Page 11: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/11.jpg)
11© Copyright 2014 EMC Corporation. All rights reserved.© Copyright 2014 EMC Corporation. All rights reserved.
EMC MetaLnx Demo Setup
Network
swcentos6-1iCAT server,
local disk
iRODSstorage
resources
swdebian64-1iRODS resource server, 2 local storage resources
swcentos6-2MetaLnx server
& client
iRODSgrid
Virtual Machine Layout:1 core, 1 GB memory, 10
GB disk / each
Physical System:Intel i7, 4 cores, 12 GB memory
![Page 12: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/12.jpg)
12© Copyright 2014 EMC Corporation. All rights reserved.© Copyright 2014 EMC Corporation. All rights reserved.
EMC MetaLnx Demo Setup
Network
swcentos6-1iCAT server,
local disk
iRODSstorage
resources
swdebian64-1iRODS resource server, 2 local storage resources
swcentos6-2MetaLnx server
& client
iRODSgrid
Virtual Machine Layout:1 core, 1 GB memory, 10
GB disk / each
Physical System:Intel i7, 4 cores, 12 GB memory
![Page 13: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/13.jpg)
13© Copyright 2015 EMC Corporation. All rights reserved.
MetaLnx Live Demo
Selected images added to show functionality
![Page 14: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/14.jpg)
14© Copyright 2015 EMC Corporation. All rights reserved.
Dashboard – iRODS Server Down
![Page 15: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/15.jpg)
15© Copyright 2015 EMC Corporation. All rights reserved.
Server Details Page
![Page 16: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/16.jpg)
16© Copyright 2015 EMC Corporation. All rights reserved.
Resource Map
![Page 17: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/17.jpg)
17© Copyright 2015 EMC Corporation. All rights reserved.
User Management
![Page 18: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/18.jpg)
18© Copyright 2015 EMC Corporation. All rights reserved.
Collections - Pagination
![Page 19: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/19.jpg)
19© Copyright 2015 EMC Corporation. All rights reserved.
Collections - Upload
![Page 20: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/20.jpg)
20© Copyright 2015 EMC Corporation. All rights reserved.
Collections – Extracted Metadata
![Page 21: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/21.jpg)
21© Copyright 2015 EMC Corporation. All rights reserved.
Metadata Search
![Page 22: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/22.jpg)
22© Copyright 2015 EMC Corporation. All rights reserved.
Metadata Search Results
![Page 23: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/23.jpg)
23© Copyright 2015 EMC Corporation. All rights reserved.
MetaLnx Value Proposition:Simplified Metadata Management• Simple, easy iRODS grid administration
• Simple, intuitive tool for non-IT researchers:– Collection management– Automated embedded metadata extraction– Metadata searching / editing– Metadata templates – consistent tagging of data
• Scale out design supports PCs and tablets
• Extensible to any data vertical
![Page 24: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/24.jpg)
24© Copyright 2015 EMC Corporation. All rights reserved.
EMC iRODS Efforts
• MetaLnx– An iRODS Administration and Metadata Interface tool
• Storage Resource Drivers for iRODS– Isilon – HDFS interface– ECS – Atmos object interface
Both are in active development and beta testing
![Page 25: EMC MetaLnx - iRODS · Title: Microsoft PowerPoint - 150610_MetaLnx_iRODS_UGM15_distrib.pptx Author: worths Created Date: 6/18/2015 8:23:03 AM](https://reader033.vdocuments.mx/reader033/viewer/2022052721/5f0ac7aa7e708231d42d4dd4/html5/thumbnails/25.jpg)
25© Copyright 2015 EMC Corporation. All rights reserved.
Questions ?
Sasha PaegleSenior Business Development [email protected]
Stephen WorthDirector, Global New Business Development [email protected]
Contacts: