Optimize IaaS VM Storage using Azure Disks and FilesAung OoPrincipal PM Manager, Azure StorageMicrosoft Corporation

BRK3209

Sirius KuttiyanPrincipal Program Manager, Azure StorageMicrosoft Corporation

Azure Storage Talks @ IgniteSession ID When/Where Attend if you are interested in Talk titleBRK2201 Tuesday 9/27

10:45-12:00C202 - C204

Azure Storage OverviewStorage Solutions: Backup and Recovery Hybrid Storage &Global Data Access

Dive into scenarios and customer success stories with Azure Storage

BRK3210 Tuesday 9/27 12:30-1:45B312 - B314

Azure Object (Blob) Storage OverviewWorking with Blob StorageBlob Storage Design Patterns

Build tiered cloud storage in Microsoft Azure

BRK3209 Wednesday 9/28 10:45-12:00B312 - B314

Overview and Deep Dive on Azure Disks Azure Files

Optimize IaaS VM Storage using Azure Disks and Files

BRK3211 Thursday 9/2910:45-12:00B304 - B305

Azure Storage Security topics Control and data plane authorization Data Encryption – Client-side, in-transit and server-side

Secure your data in the cloud with Microsoft Azure Storage

What we will cover todayAzure Disks overview + deep diveMigration, backup, best practices

Azure Files overview + deep diveSnapshot, best practices

Customer Experience

Azure StorageFoundational Building Block of AzureAzure Services: SQL Data Warehouse, HDInsight, Data Lake Store, Event Hubs, IoT Hubs…Microsoft Services: Office 365, OneDrive, XBox, Skype…

Hyper Scale >120 Trillion Objects, >19 million transactions per second

Durable Never lose your data. Multiple redundancy options. Automatic data checks

Secure Encryption at Rest. Client side Encryption. Integration with KeyVaultHighly

AvailableFault tolerance to hardware/software issues. Automatic load balancing

Open REST API, Open sourced Client Libraries – .NET, Java, C++, Python, Node.js, iOS, Android, Xamarin…

Hybrid Extensive partner ecosystem. Azure Stack for private/hosted clouds.

Azure Storage Services

FilesFully Managed File Shares in the Cloud

SMB and REST access

“Lift and shift” legacy apps

DisksPersistent disks for Azure IaaS VMs

Premium Storage Disks option: SSD based, high IOPS, low latency

Built on a unified Distributed Storage SystemDurability, Encryption at Rest, Strongly Consistent Replication, Fault Tolerance, Auto Load-Balancing

IaaSVirtual

machinesStorage

Networking

ObjectsHighly scalable, REST based cloud object store

Block Blobs: Sequential file I/OCool Tier AvailablePage Blobs: Random-write pattern dataAppend Blobs

QueuesReliable queues at scale for cloud services

Decouple and scale componentsMessage visibility timeout and update message to protect against unreliable dequeuers

TablesMassive auto-scaling NoSQL store

Dynamic scaling based on load

Scale to PBs of table data

Fast key/value lookups

PaaSExisting frameworks

Web and mobile

Serverless ComputeMicroservices

TopicsDisks Deep

Dive

Migration

Backup

Best Practices

Files Deep Dive

Snapshot

Best Practices

Files Overview

Scenarios

Benefits

Application Architecture

Disks Overview

Features

Benefits

Roadmap

Azure Disks: OverviewWhat are Azure Disks?Persistent data disks for Azure IaaS VMsTwo types of durable storage: Premium Disks (SSD) & Standard Disks (HDD)Backed by Page Blobs in Azure Storage

CapabilitiesHigh I/O performance and low latency (>80,000 IOPS and >2,000 MB/sec disk throughput per VM)Currently available in 26 Azure RegionsServer Side Encryption at Rest, Azure Disk Encryption (BitLocker/DMCrypt)

Specialized VMs with Blob Cache technologyEnterprise grade durability with three replicasSnapshots for backupAbility to expand Disks in placeREST Interface for developers

Azure Disks: Premium vs. Standard Storage DisksPremium Storage Disks Standard Storage Disks

Disks Types SSD HDDRecommended Workloads

Mission critical applications which require high-performance, high-availability, and low latency for IO-intensive Enterprise workloads

Infrequently accessed data and Dev/Test workloads

Performance Expectations

5,000 Provisioned IOPS and 200 MB/sec throughput for 1 TB disk

80,000 Provisioned IOPS and 2,000 MB/sec throughout with GS5 (largest VM)

Single digit millisecond latency

Consistency of provisioned performance

Up to 500 IOPS and 60 MB/sec for 1 TB disk

32,000 IOPS with G5 (largest VM)

Performance is not provisioned

Recently Released• Azure Backup support• Encryption at Rest• Azure Site Recovery

Preview• Incremental Snapshot

Copy

Future• More disk sizes• Disk analytics

enhancement

Azure Disks Roadmap

Demo

Premium Disks Performance

Premium Storage PerformanceIOPS and ThroughputAre separate limits, enforced independentlyThrottling kicks in when either of the limits are reached

VM Size and Disk SizeEnforce independent limits on VM and DiskChose the right configuration based on workload needs

Optimal Traffic PatternsLarger IO sizes for higher Throughput. Smaller IO sizes for higher IOPS.Use multiple parallel operations / threads to push more trafficAvoid heavy bursts by reducing the IO buffer to <5% of the IO limits

See BRK4450 (Ignite 2015) for more

Limits IOPS ThroughputP30 (1 TiB) Disk 5,000 200 MB/secDS3 VM 12,800 128 MB/sec

TopicsDisks Deep

Dive

Migration

Backup

Best Practices

Files Deep Dive

Snapshot

Best Practices

Files Overview

Scenarios

Benefits

Application Architecture

Disks Overview

Features

Benefits

Roadmap

P

Standard Storage

Migration Option 1: Offline Migration

Premium Storage

Reference: http://aka.ms/migrate-to-premium; http://aka.ms/incremental-snapshots

Using AzCopyCopy the disks to Premium Storage with VM shutdownLonger Downtime

1 Shutdown the VM 2 Snapshot Disks3AzCopy the Snapshots4Create new VM with the copied disks5Validate and delete old VM/Disks

Migration Option 2: Live MigrationAzure Site RecoveryMigrate using ASR serviceShorter downtime

Migration StepsCreate Recovery Services VaultSetup Infrastructure for MigrationReplicate the Application Test Failover (no downtime) and validateFailover (Migrate)

12345

13

245

Reference: http://aka.ms/migrate-to-premium

Azure Disks: Backup StrategyConsistent BackupReliable backups needed for DR and for Application/Data recoveriesBackup must use “consistent” snapshotsCreating “consistent” snapshots of running VMs require additional steps

1. Use a Backup Solution (Recommended Option)Azure Backup or a 3rd Party Backup solution, like Commvault, Veritas NetBackup/BackupExec etc

2. Build a Backup SolutionBuild a custom solution for your needs

3. Backup with DowntimeAlternative to create consistent snapshots with a short downtime

Option 1: Azure Backup or 3rd party solutionAzure Backup Service for Azure VMsRecommended backup solution for Premium and Standard DisksCreates consistent backups for running VMsSupports Windows and Linux VMs

CapabilitiesApplication consistent backups for Windows VMsFile System consistent backups for Linux VMs

Local and Geo-replicated Backup optionsFlexible backup frequencies and retention periodsEncrypted backup storage

Reference: http://aka.ms/azure-back-up; http://aka.ms/commvault

Option 2: Custom Backup SolutionCreate Consistent SnapshotsFreeze the disks, flush pending IOs and then create blob snapshotsFor Windows, use WSB/VolumeShadow (VSS) to create Application consistent snapshotsFor Linux, use similar tools to create File System consistent snapshots

Copy the Backup to Different Storage Account for DRTarget a Standard RA-GRS account for geo redundant backupImplement incremental copying for efficiencyUse the new Differential Page Ranges API for incremental copying

Differential API for Copying SnapshotsEnhanced Get Page Ranges APIOld: No way to copy only the changed data since previous snapshotNew: Added capability to get list of changes between any two snapshots

NOTE: Steps are numbered in sequence

Reference: http://aka.ms/incremental-snapshots

Option 3: Backup with DowntimeCreate Snapshots

Copy to Different Storage Account for DRTarget a Standard RA-GRS account for geo redundant backup

Shutdown the VM

Create blob

SnapshotsStart the

VM

Restore from Backup1. Using a Backup SolutionRestore mechanisms as provided by the backup solution

2. Using Custom BackupsCopy the backup (snapshots) to new blobsCreate new VM using the copied disksValidate the new VM, and then delete the prior VM / disks

Copy Blob

Restored Disk and VM

Best Practices: ReliabilityPremium Storage for production workloadsProvisioned disk performance for consistencyRequired for IO-intensive enterprise workloads

Standard Storage Account ManagementLimit to <40 disks per Standard Storage AccountEnable Storage Account Analytics

RedundancyUse VM availability Set with multiple VMs for redundancyUse separate storage accounts for VMs in the same Availability Set

Best Practices: VM/Application ConfigVM Host CachingUse “ReadOnly” cache for Data disks (Portal Default), ideal for Read-heavy and mixed workloads

Use “None” cache for Write-only / Write-heavy Data disks (e.g. logs)Consider “ReadWrite” caching for Data Disks if the application handles IO flushing / durability

SQL Server / DatabasesUse Premium Storage disksSeparate disks for data and log filesUse the VM’s temporary resource disk for Temp DB

GeneralUse use multiple instances for increasing throughput for single-threaded applications

Example: setup rsync per folderRefer to Application best practices documents for details

TopicsDisks Deep

Dive

Migration

Backup

Best Practices

Files Deep Dive

Snapshot

Best Practices

Files Overview

Scenarios

Benefits

Application Architecture

PDisks

Overview

Features

Benefits

Roadmap

P

What is Azure Files?Fully Managed Cloud File Storage for use with IaaS and On Premise instances

Scenarios CoveredLift and ShiftHost high availability workload dataEnable backup and disaster recovery

Azure Files: Overview

Multiple protocols & Operating SystemsSMB 2.1, 3.0, and RESTMount from both Windows & Linux

Globally accessibleAccessible from both On Premise and IaaS instancesAvailable in all Azure regions including Government Cloud & Mooncake

High availability and durabilityThree strongly consistent copies for durability99.9% availability SLA

Azure Files: Capabilities

Azure Files RoadmapFile Share SnapshotsAD IntegrationIncrease scale limitsLarger share sizeEncryption at restBackup integration

Azure Files vs. DisksAzure Files Azure Disks

Scenario Life & Shift applications which leverage native file system

Persistent disks to Azure Virtual Machines

Protocol SMB 2.1/3.0, REST VHD, REST

Accessibility SMB – Worldwide (requires Port 445)REST – Worldwide

VHD – Azure Data CenterREST (Page Blob) – Worldwide

Durability LRS, GRS LRS, GRS, RA-GRS (for Page Blob)

Object Size Up to 1 TB file Up to 1 TB Disks (Can stripe up to 64 disks on G VM)

Max IOPS (8K) 1000 5000 (Premium)500 (Standard)

Throughput Up to 60 MB/s per share Up to 200 MB/s per disks (Premium)Up to 60 MB/s per disk (Standard)

Azure Files Azure Disks

Scenario Life & Shift applications which leverage native file system

Persistent disks to Azure Virtual Machines

Protocol SMB 2.1/3.0, REST VHD, REST

Accessibility SMB – Worldwide (requires Port 445)REST – Worldwide

VHD – Azure Data CenterREST (Page Blob) – Worldwide

Customer Success: Global Collaboration with Talon CloudFAST™ and Azure FilesRobert Bird GroupGlobal Specialists in Structural, Civil & Construction Engineering

• 10 Offices Worldwide• 600+ Employees Globally• Offices in the Northern, Central,

Southern and Western region’s of Australia, United Kingdom and Middle East

Robert Bird GroupGlobal Specialists in Structural, Civil & Construction EngineeringStephen CotthamChief Technology Officerwww.robertbird.com

One Central ParkArchitects: Jean NouvelClient: Frasers Property GroupWinner of CTBUH Best Tall Building WorldwideImages courtesy of Frasers Property Australia and Sekisui House Australia. Image © Simon Wood

Robert Bird Group - Business and IT Needs

ENABLE GLOBAL COLLABORATIONNeed to be able to work in real-time with offices in Australia, Middle-East and UK

LARGE FILE SHARINGComplex engineering models = large file sets, including application data like Autodesk, Microsoft Office and image files

IMPROVE PRODUCTIVITY AND PERFORMANCEImprove application performance and increase productivity for teams globally

REDUCE STORAGE COST AND COMPLEXITYFind better ways to reduce the storage footprint, eliminate ‘doubling up’ on storage space

SHARE SPECIALIST KNOWLEDGE AND WORK BETWEEN OFFICESAccess to data, anytime, anywhere for knowledge workers, meet project deadlines

IntroductionTalon – Microsoft Azure ISV PartnerJaap van DuijvenbodeProduct Directorwww.talonstorage.com

• Software solution that helps enterprises centralize file storage into a “Single

Set of Data”• Centralizes your data set into the Microsoft Azure Cloud• Delivers instant Enterprise Global File Sharing with Global File Locking

• High performance experience at all geographically distributed locations• Eliminates complexity, storage, infrastructure and backups at the branch• Cut storage costs up to 70% using Microsoft Azure

Before: De-Centralized = Collaboration Painful

LondonStorage: EMC

SydneyStorage: NetApp

AdelaideStorage: QNAP

PerthStorage: QNAP

Kuala LumpurStorage: QNAP

MelbourneStorage: Net App

BrisbaneStorage: NetApp

DubaiStorage: Windows Server 2008

CHALLENGESLocal storage in each siteLocal backupsTrouble managing replicationHigh latency / WAN saturationCollaboration is impossible

BENEFITSCentralized storage model Eliminated local backupsIntroduced CloudFAST™Eliminated Latency / BW issuesCollaboration enabled

CHALLENGESIncrease central storage capacityExpensive storage/infrastructureMPLS connectivity

Central Storage

Step 1: Centralize Data to ‘on-premise’

LondonTalon CloudFAST™

BrisbaneTalon CloudFAST™

SydneyTalon CloudFAST™

MelbourneTalon CloudFAST™

DubaiTalon CloudFAST™

Kuala LumpurTalon CloudFAST™

PerthTalon CloudFAST™

AdelaideTalon CloudFAST™

Collaboration Data

BrisbaneTalon CloudFAST™

MelbourneTalon CloudFAST™

DubaiTalon CloudFAST™

Kuala LumpurTalon CloudFAST™

PerthTalon CloudFAST™

AdelaideTalon CloudFAST™

LondonTalon CloudFAST™EMC Storage

EXTERNAL USERSTalon CloudFAST™

Step 2: Leverage Microsoft Azure Files

Collaboration Data• Azure Files• LRS / GRS

Central Storage

• Challenges with Scale

• Reclaim Tier-1 Storage

BENEFITSCentralized storage model Eliminated local backupsHighly scalable / Cost effectiveLeverage Microsoft AzureUse VPN instead of MPLSIntegrate with Talon CloudFAST™AD / ACLs / NTFS permissionsReclaim Tier-1 storageTransparent CollaborationAlso for external contributors

Move from a traditional ‘on-premise’ environmentHow to leverage Azure IaaS optimally

• 100’s of different Azure Solutions available today, growing every month• Azure IaaS as an alternative to traditional infrastructure• OMS / LogAnalytics, Azure Automation, etc.

Improved Data Management• Moving active and archive unstructured data from ‘on-premise’ to Azure• NTFS Permissions and ACLs for internal and external collaboration

Backups & Snapshots?Scale at large > 100TB’s

Next steps for Robert Bird Group

ConclusionCollaborationEnable true group collaboration between our offices globally

Business OutsourcingExtend project collaboration to 3rd party outsourced partners

Increase ProductivityEnsure high performance to all sites globally Modernize and Streamline StorageLeveraging Azure Files as a ‘cloud’ datacenter file storage platform

Future strategy with Microsoft AzureOpportunity to migrate DR and Archive data to Azure

TopicsDisks Deep

Dive

Migration

Backup

Best Practices

Files Deep Dive

Snapshot

Best Practices

Files Overview

Scenarios

Benefits

Application Architecture

PPDisks

Overview

Features

Benefits

Roadmap

P

Azure Files Deep Dive: Share SnapshotProtect and recover your data using share snapshotsEasy to create, easy to mount and recover filesPrevious versions integration for viewing snapshot content Store up to 100 snapshots per shareManage snapshots through REST API, PowerShell and CLI

Azure Files Deep Dive: Best PracticesPerformanceUse 1MB IO Size for best throughputEnsure KB3114025 applied on Windows 8.1 or Server 2012 R2

SecurityUse Win 8.x or Server 2012 as Windows clientsUse SMB 2.1 in the same DC for Linux clientsMount file for correct user contextEnable Port 445 outbound to access Azure Files from On Premise instances

UsabilityWrite to a file periodically to keep connection open in Linux, until kernel fix is backportedMount from logged in user context to see drive in Windows Explorer

Azure Files: Lift and Shift FTP Servers

Load Balancer

App/FTP

Server

VM 1

Azure files Share 1

Folder 1

Folder 2

Sub Folder 1

App/FTP

Server

VM 2

App/FTP

Server

VM 3

Azureftp://ftp.domain-one.com/pub/

\\<storageacct1>.file.windows.net\ftpshare

Azure Scenario 1

Port 445 Port 445 Port 445

Problem: Legacy FTP servers that need to be maintained on-prem

Solution:Move FTP server to IaaSMove file share to Azure Files

Demo

Azure File Share

Learn MoreAzure Disks ResourcesPremium Storage: High-Performance Storage for Azure Virtual Machine WorkloadsDesign for High PerformanceMigrating to Azure Premium StorageBlob Service REST Operations with Azure Premium StorageBackup with Incremental Snapshot copyingAzure VM specificationsMark's Channel 9 video on Premium Storage

Azure Files ResourcesGetting started blog with many useful links SMB features currently not supported Naming restrictions for REST compatibilityScalability targetsVarious ISPs and Port 445 status

Azure Storage Talks @ IgniteSession ID When/Where Attend if you are interested in Talk titleBRK2201 Tuesday 9/27

10:45-12:00C202 - C204

Azure Storage OverviewStorage Solutions: Backup and Recovery Hybrid Storage &Global Data Access

Dive into scenarios and customer success stories with Azure Storage

BRK3210 Tuesday 9/27 12:30-1:45B312 - B314

Azure Object (Blob) Storage OverviewWorking with Blob StorageBlob Storage Design Patterns

Build tiered cloud storage in Microsoft Azure

BRK3209 Wednesday 9/28 10:45-12:00B312 - B314

Overview and Deep Dive on Azure Disks Azure Files

Optimize IaaS VM Storage using Azure Disks and Files

BRK3211 Thursday 9/2910:45-12:00B304 - B305

Azure Storage Security topics Control and data plane authorization Data Encryption – Client-side, in-transit and server-side

Secure your data in the cloud with Microsoft Azure Storage

From your PC or Tablet visit MyIgnite at http://myignite.microsoft.com

From your phone download and use the Ignite Mobile App by scanning the QR code above or visiting https://aka.ms/ignite.mobileapp

Please evaluate this sessionYour feedback is important to us!

Thank You

Question and Answers

© 2016 Microsoft Corporation. All rights reserved.