spectra logic's blackpearl developers summit 2016
TRANSCRIPT
![Page 1: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/1.jpg)
![Page 2: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/2.jpg)
Developer Summit Agenda:
Nathan Thompson – CEO Spectra LogicSteven Easter – BlackPearl Product ManagerJeff Braunstein – Senior Product Manager, Client DevelopmentSharon Shabtai – Senior Software Engineer, Client DevelopmentSven Howarth-Moore – Professional Services Lead for CatDV
![Page 3: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/3.jpg)
BlackPearl 2016 Developers Conference
Nathan ThompsonCEO & FounderSpectra Logic Corporation
![Page 4: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/4.jpg)
Spectra Logic HeadquartersBoulder, CO
![Page 5: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/5.jpg)
2U BlackPearl + T50e Tape Library BlackPearl + T120 Tape Library BlackPearl + T200 Tape Library BlackPearl + T380 Tape Library BlackPearl + T680 Tape Libraries
80TB Raw Disk Storage 160TB Raw Disk Storage 928TB Raw Disk Storage 1,696TB Raw Disk Storage 6.1PB+ Raw Disk Storage
300TB Uncompressed Tape Storage 720TB Uncompressed Tape Storage 1,200TB Uncompressed Tape Storage 2,280TB Uncompressed Tape Storage 4,020TB Uncompressed Tape Storage
Multi–RackFull RackHalf RackQuarter Rack
Spectra’s BlackPearl S3 Gateway To Tape
Eighth Rack
![Page 6: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/6.jpg)
System Configuration:
BlackPearl 4U Deep Storage Gateway• 6.1PB Raw Capacity Cache
5 Frame TFinity Tape Library5,590 LTO Slots• 13.9PB Uncompressed Capacity
using LTO-6 tape technology
• 33.5PB Uncompressed Capacity using LTO-7 tape technology
4,239 TS Enterprise Slots• 42.4PB Uncompressed Capacity
using TS1150 tape technology
BlackPearl™ ExaScale Edition TFinity™
![Page 7: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/7.jpg)
Exploring The New World Of Storage
• At the last conference event I spoke about two upcoming publications– Spectra’s Storage Forecast which you can download from the Spectra Logic
web site for free (http://spectra.cc/Igh)– The Book “Society’s Genome”
• Both are now published– I will tell you a few of our findings– Jeff can send you an electronic copy of the book
![Page 8: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/8.jpg)
Major Change In Tape Roadmap
• Tape is in a position to more rapidly deliver declining cost per bit over the next 10 years– Tape has plenty of potential for capacity improvement to reach at least four or five
more doublings of current capacity, entirely due to surface area and Tunneling Magnetoresistive (TMR) heads
– Fujifilm and IBM have demonstrated BaFe media has been proven to reach 220TB or more / cartridge
– IBM and HP have partnered on the same head design at LTO-7 and beyond which gives them 98% market share (at LTO-7 and beyond) with plans to double every two years• This gives users confidence in a roadmap that can span for at least four more generations of drives
– Nearly all of the large cloud companies see a significant role for tape
![Page 9: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/9.jpg)
Here Is The Economic Opportunity For Tape
• With or without HAMR, disk prices will level out at $300 for capacity 3.5” disk drives, for 16TB, or about 2 cents per Gigabyte (raw) in 2019 and will increase with inflation from there
• Assuming constant media price and the industry stays true to a two year improvement cycle– LTO-7 is about 1.6c per Gigabyte now and will reach
~1c in 2017– LTO-8 will .8c per Gigabyte in 2018– LTO-9 tape will be .4c per Gigabyte in 2020– LTO-10 in ~2022 will be .2c per Gigabyte
![Page 10: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/10.jpg)
What Does This Mean For Large BlackPearl/Tape Opportunities?
• Ultra cost effective solutions can be built• Declining cost point into the future• Let’s look at some examples:
![Page 11: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/11.jpg)
10 PB Archive System (no compression)5 PB Nearline Copy + 5 PB Ejected Copy of DataSystems Configured with 12 LTO Drives7x24 Support adds 6%-9% to total US Price, annually
![Page 12: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/12.jpg)
100 PB Archive System (no compression)50 PB Nearline Copy + 50 PB Ejected Copy of DataSystems Configured with 48 LTO Tape Drives7x24 Support adds 6%-9% to total US Price, annually
![Page 13: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/13.jpg)
1000 PB / 1 Exabyte System (no compression)500 PB Nearline Copy + 500 PB Ejected Copy of DataSystems Configured with 144 LTO Tape Drives7x24 Support adds 6%-9% to total US Price, annually
![Page 14: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/14.jpg)
Applications Written To BlackPearl Can Take Advantage Of All Storage Types
• Tape in libraries– LTO– IBM TS
• Ejected tape• SAS disk• Power-Down SMR SATA
disk• Public Cloud (late ‘16/early
‘17)• Future storage media types
![Page 15: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/15.jpg)
THANKS
Questions?
![Page 16: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/16.jpg)
BlackPearl Product UpdatesPresenter: Steven Easter – Product Manager
![Page 17: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/17.jpg)
What Makes BlackPearl RevolutionaryBlackPearl – Is a living archive, preserving digital
data
The most data durability available
Highly efficient tape writing
Replication Targets now available
![Page 18: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/18.jpg)
Two Types of Replication TargetBlackPearl to BlackPearl replication
BlackPearl to 3rd Party (public) Cloud
![Page 19: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/19.jpg)
BlackPearl Replication• Provide two copies of data – two locations• Data is more ‘available’, • if one site is down your app can point to the other BlackPearl
to GET data• Solution then adds some HA capability
• More flexible platform to fit more customer needs
![Page 20: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/20.jpg)
BlackPearl Replication ExampleProcess to setup bi-directional replication
1. At each site: Setup other BlackPearl Replication target • Use Data Port for replication• Set default read preference• Optional – Enable User Replication (access control replication)
2. At each site: Add BlackPearl Target to Data Policy• Different policy for each BP:
• HQ system has a Data Policy for: One Copy on a tape Storage Domain and One Replication Target
• Colo system has a Data Policy for: One Copy on a ArcticBlue Storage Domain and One Replication Target (pointing back at HQ system)
3. Create a Bucket using these Data Policies• Either a new PUT Job will use default Policies for the user and
replication target, or • Best to create the same bucket on both systems first (specify the
Data Policy created in step 2), before a PUT job is done
Tape LibrarySite – Primary – “HQ”
Site – Secondary – “Colo”
![Page 21: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/21.jpg)
BlackPearl Replication Example (cont.)Data flow in bi-directional replication
A. User does A Bulk PUT into Bucket “EditingArchive” at HQ BlackPearli. Creates a S3 Job – e.g. Job 1 on HQ BlackPearl, and Replication Job 1 on Colo BlackPearlii. Both databases are synced with Replicated Job 1 at time of Job 1 creation, that way the same data cannot be sent to
other BlackPearlB. The Application sends data to HQ BlackPearl
i. Data lands in HQ BlackPearl cache, stays here until the Job 1 data has been persisted to all Storage Domainsii. For Example we’ll say the data is replicated to the Colo BlackPearl first, iii. Colo BlackPearl writes it to ArcticBlue quickly
• Replication Job 1 is now complete. Data in Colo BlackPearl cache can be reclaimed when needediv. Tape Drive resources become available at HQ BlackPearl and the Data is now written to Tape
• Job 1 is now complete. Data in HQ BlackPearl can now be reclaimed. Host application can remove any file locks for deleting data from primary storage.
Site – Primary – “HQ"
Cache:
Tape Storage Domain:
Site – Secondary – “Colo”
Cache:
ArcticBlue Storage Domain:
EditingArchiveData Policy: Tape+Replication
EditingArchiveData Policy: ArcticBlue+Replication
Job 1Replication Job 1
HostUser
data datadata
datadata
![Page 22: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/22.jpg)
BlackPearl P Seriesspectralogic.com/products/BlackPearl/specs/
• Very high performance system – up to 3GBps• Customers will need scale-out application & client• Or customer may have multiple different
applications
• Also useful if BlackPearl makes multiple copies
![Page 23: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/23.jpg)
BlackPearl Developer Updates - Agenda• General Program Updates• Certification Program• SDK Updates• End to End Data Integrity• When Files are Safely Archived to BlackPearl• New Deep Storage Browser Demonstration• Java Command Line Interface Demo and Updates
![Page 24: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/24.jpg)
Developer Program Updates• Mature Program• Certification• Simulator and Remote BlackPearl• Spectra BlackPearl Lab• Forums
![Page 25: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/25.jpg)
BlackPearl Certification Program• Goal: Ensure Clients are Robust and Reliable
• Includes the Following Components:• Program Overview and Timeline
• Legal Agreements• Questionnaire
• Test Plan• Testing Performed in Spectra Lab in Boulder
• Certified Clients Able to Participate in Joint Marketing• Retesting Available for New Releases
• Currently have 3 Spectra staff doing certification testing
• Goal: Ensure Clients are Robust and Reliable• Includes the Following Components:
• Program Overview and Timeline• Questionnaire• Test Plan
• Testing Performed in Spectra Lab in Boulder• Certified Clients Able to Participate in Joint Marketing• Retesting Available for New Releases• Currently have 2 dedicated staff doing certification testing
![Page 26: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/26.jpg)
Software Development Kit (SDK) Updates• Four Complete SDKs• Java• C#/.NET• Python• C
• Java and C#/.NET Include “Helper” Functions• All SDKs Can Now Access All 250+ BlackPearl
APIs
![Page 27: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/27.jpg)
End-To-End Data Integrity• BlackPearl includes comprehensive data integrity once data
received• Developers to be concerned with “in flight” data integrity
between client and BlackPearl• Client can use checksums to guarantee integrity• Checksums done on each file or file parts sent to BlackPearl• HTTPS/SSL is a good optional checksum data integrity
method
![Page 28: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/28.jpg)
When Are Files Safely Archived to BlackPearl?• Clients archive groups of files to BlackPearl in “jobs”• Job is “Complete” when files are on all storage targets• BlackPearl’s database tracks the location of files on
storage targets• This database is typically backed up daily• Files are not truly, safely archived until database backup with
their location has been made• Clients should not purge/delete files off of their system
until safely archived to BlackPearl
![Page 29: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/29.jpg)
Deep Storage Browser
![Page 30: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/30.jpg)
Java Command Line Interface
![Page 31: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/31.jpg)
The .Net SDK and Demonstration of Avid PAM pluginSharon Shabtai - Senior Software Engineer
![Page 32: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/32.jpg)
Table Of Contents
• Spectra Logic .Net SDK
• Helper functions
• The Spectra S3 Client
• Instantiating the API example
• NuGet Package
• AVID Plugin
![Page 33: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/33.jpg)
Spectra Logic .Net SDK• Open source (GitHub) can be imported to VS and modify to you needs
• Using .Net 4.5.1
• Latest SDK release version v3.2.7
• The SDK consists of two levels of abstraction:
1. A high level interface (Ds3.Helpers.IDs3ClientHelpers)
2. The core client interface (Ds3.IDs3Client)
![Page 34: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/34.jpg)
Helper Functions• A high level interface that abstracts several very common application requirements.
• Constantly working on adding and improving the API:
o StartWriteJob
o StartReadJob
o StartReadAllJob
o StartPartialReadJob
![Page 35: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/35.jpg)
Helper Functions• A high level interface that abstracts several very common application requirements.
• Constantly working on adding and improving the API:
o StartWriteJob
o StartReadJob
o StartReadAllJob
o StartPartialReadJob
o ListObjects - with and without prefix
![Page 36: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/36.jpg)
Helper Functions• A high level interface that abstracts several very common application requirements.
• Constantly working on adding and improving the API:
o StartWriteJob
o StartReadJob
o StartReadAllJob
o StartPartialReadJob
o ListObjects - with and without prefix
o EnsureBucketExists
![Page 37: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/37.jpg)
Helper Functions• A high level interface that abstracts several very common application requirements.
• Constantly working on adding and improving the API:
o StartWriteJob
o StartReadJob
o StartReadAllJob
o StartPartialReadJob
o ListObjects - with and without prefix
o EnsureBucketExists
o RecoverWriteJob
o RecoverReadJob
![Page 38: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/38.jpg)
Helper Functions• The Helper functions support:
o Parallel Requests
o Using the right Chunking strategy:
Random access
Streaming
o Metadata
o Checksum
o Cancellation token
o Job Events (“Data transferred”, “Item completed”, “On failure”)
![Page 39: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/39.jpg)
The Spectra S3 Client• The Spectra S3 client implements all of the BlackPearl REST API and can be use when the desire
behavior is not implemented in the helper function.
• You can find the full API in our IDs3Client interface
![Page 40: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/40.jpg)
Instantiating the API exampleThe example below shows how to configure and instantiate IDs3Client and IDs3ClientHelpers.using Ds3;using Ds3.Helpers;using System.Configuration;
namespace YourApplication{ class YourClass { public void YourMethod() { // Configure and build the core client. IDs3Client client = new Ds3Builder( "http://ds3-endpoint", new Credentials("access key", "secret key") ).Build(); // Set up the high-level abstractions. IDs3ClientHelpers helpers = new Ds3ClientHelpers(client); // Use functionality from 'helpers' and 'client', preferring 'helpers'. } }}
• More detailed examples can be found in our Examples Project
![Page 41: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/41.jpg)
NuGet Package• The SDK is distributed as a NuGet package for .Net 4.5.1 and above.
• How to get the NuGet package:
1. Download the .nupkg file from the Releases page
2. Use Spectra Logic Bintray NuGet repository
![Page 42: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/42.jpg)
• Avid Technology, Inc.
• Avid 3rd party plugin API
o Archive
o Restore (including partial restore)
o Delete
o Pause and Resume
• Avid and Spectra Logic
o We Created a plugin for BlackPearl
![Page 43: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/43.jpg)
• Avid Technology, Inc.
• Avid 3rd party plugin API
o Archive
o Restore (including partial restore)
o Delete
o Pause and Resume
• Avid and Spectra Logic
o Created a plugin for BlackPearl
![Page 44: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/44.jpg)
• Avid Technology, Inc.
• Avid 3rd party plugin API
o Archive
o Restore (including partial restore)
o Delete
o Pause and Resume
• Avid and Spectra Logic
o Created a plugin for BlackPearl
![Page 45: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/45.jpg)
![Page 46: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/46.jpg)
Best of Breed Partners
![Page 47: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/47.jpg)
Application Architecture
![Page 48: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/48.jpg)
CatDV and Spectra Logic Black Pearl
![Page 49: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/49.jpg)
CatDV UI
![Page 50: Spectra Logic's BlackPearl Developers Summit 2016](https://reader035.vdocuments.mx/reader035/viewer/2022062821/58874b921a28ab5a628b6349/html5/thumbnails/50.jpg)
Configuration