WINDOWS AZURE BLOB STORAGE DEEP DIVE

Wely Lau (wely@ncs.com.sg) Microsoft MVP, Windows AzureSolutions Architect, NCS Pte LtdBlog : http://wely-lau.net

AGENDA

• Windows Azure Storage Overview (10 mins)

• Blob Storage Basic (15 mins)• Understanding container and blob• Main web service operations • Metadata• Snapshot

• Deep Dive (20 mins)• Block blob &page blob• Shared Access Signature• Content Delivery Network

• Tip of the day (10 mins)

• Q & A (5 mins)

WINDOWS AZURE STORAGE OVERVIEW

WINDOWS AZURE STORAGE

• Storage in the Cloud• Highly available with 99.9% monthly SLA• Anywhere at anytime access• Data Redundancy

• Data is replicated for resilience and protection with 3 copies within the same data center.

• Another 3 copies geo-replicated in another data center• Cost effective

• Storage capacity: $ 0.14 per GB per month• Storage transaction: $ 0.01 per 10,000 transaction

• User specified globally unique account name• Can choose geo-location to host storage account

• US – “North Central” and “South Central”• Europe – “North” and “West”• Asia – “East” and “Southeast”

• Can co-locate storage account with compute account• Explicitly or using affinity groups

• Some Facts• 70 Petabytes raw storage today• Grows to >200 Petabytes by start of 2012

WINDOWS AZURE STORAGE ABSTRACTION

Tables

Structured storage. A table is a set of entities; an entity is a set of properties.

Queues

Reliable storage and delivery of messages for an application.

Blobs

Simple named files along with metadata for the file.

Drives

Durable NTFS volumes for Windows Azure applications to use. Based on Blobs.

STORAGE EMULATOR

• Provides a local “Mock” storage• Emulates storage in cloud• Allows offline development• Requires SQL Express 2005/2008 or above

There are some differences between Cloud and Dev Storage.http://msdn.microsoft.com/dd320275

ACCESSING WINDOWS AZURE STORAGE VIA API

• REST-based Web Service• Platform independent• Use when a library isn’t available• Lacks Intellisense• Results in opaque code

• Storage Client Libraries• .NET, Java, Python, Ruby, PHP,

Perl, JavaScript (Node), Objective-C, etc.

• More at http://tinyurl.com/was-library

• These provide an abstraction layer on top of the REST API and is the most common way to access storage.

Windows Azure Storage

REST-based API

.NET Java PHPetc..

.

Developers…

ACCESSING WINDOWS AZURE STORAGE VIA TOOLS• Cloud Storage Studio by Cerebrata• Azure Storage Explorer by Neudistic• CloudXplorer by ClumsyLeaf• CloudBerry Explorer by CloudBerry Lab• More at http://tinyurl.com/was-explorers

WINDOWS AZURE STORAGE

Creating Windows Azure Storage AccountUsing tools to access Windows Azure Storage

demo

BLOB STORAGE BASIC

WINDOWS AZURE BLOB STORAGE

• You can have as many containers and BLOBs as you want.

• Containers can be created/destroyed on the fly

Account: MovieConversion

- Job1.mpg- Job2.mpg- Header.png

Container: Originals

- Job 9.mpg- Index.docx- Job8.pdf

Container: Completed

http://<account>.blob.core.windows.net/<container>/<blobname>http://cohowinery.blob.core.windows.net/images/pic01.jpg

blobs

CONTAINERS

• Similar to a top level folder• Special $root container

• Has an unlimited capacity• Can only contain BLOBs• Associate Metadata with Container

Each container has an access level:- Private

- Default, will require the account key to access

- Public read access for blobs only- Blob data within this container can be read via anonymous request, but container data

is not available

- Full public read- Container and blob data can be read via anonymous request

http://cohowinery.blob.core.windows.net/pic01.jpghttp://cohowinery.blob.core.windows.net/$root/pic01.jpg

BLOB

• Main Web Service Operations• PutBlob• GetBlob• DeleteBlob• CopyBlob• SnapshotBlob • LeaseBlob

• Associate Metadata with Blob• Metadata is <name, value> pairs, up to 8KB per blob

• Blob always accessed by name• Can include ‘/‘ or other delimeter in name • e.g. /<container>/myblobs/blob.jpg

BLOB STORAGE BASIC

• Main Web Service Operations • Metadata• Snapshot• Upload / Download Blob

demo

BLOB STORAGE DEEP DIVE

TWO TYPES OF BLOBS UNDER THE HOOD

Block Blob• Targeted at streaming

workloads

• Each blob consists of a sequence of blocks• Each block is identified by a Block ID

• Size limit 200GB per blob

• Optimistic Concurrency via Etags

Page Blob• Targeted at random read/write

workloads

• Each blob consists of an array of pages • Each page is identified by its offset

from the start of the blob

• Size limit 1TB per blob

• Optimistic or Pessimistic (locking) concurrency via leases

TheBlob.wmv

UPLOADING A BLOCK BLOB

• Uploading a large blob with Put Block List

10 GB Movie

Blo

ck I

d 1

Blo

ck I

d 2

Blo

ck I

d 3

Blo

ck I

d N

blobName = “TheBlob.wmv”;PutBlock(blobName, blockId1, block1Bits);PutBlock(blobName, blockId2, block2Bits);…………PutBlock(blobName, blockIdN, blockNBits);PutBlockList(blobName,

blockId1,…,blockIdN);

TheBlob.wmv

BenefitEfficient continuation and retryParallel and out of order upload of blocks

THE BLOB

Windows AzureStorage

PAGE BLOB – RANDOM READ/WRITE

Create MyBlobSpecify Blob Size = 10 GbytesSparse storage - Only charged for pages with data stored in them

Fixed Page Size = 512 bytesRandom Access Operations

PutPage[512, 2048)PutPage[0, 1024)ClearPage[512, 1536)PutPage[2048,2560)

GetPageRange[0, 4096) returns valid data ranges:

[0,512) , [1536,2560)GetBlob[1000, 2048) returns

All 0 for first 536 bytesNext 512 bytes are data stored in [1536,2048)

0

10 GB

512

1024

1536

2048

2560

10 G

B A

dd

ress S

pace

BLOCK BLOB / PAGE BLOB

demo

SHARED ACCESS SIGNATURE

• Fine grain access rights to blobs and containers

• Sign URL with storage key – permit elevated rights

• Revocation• Use short time periods and re-issue• Use container level policy that can be deleted

• Two broad approaches• Ad-hoc• Policy based

AD HOC SIGNATURES

• Create Short Dated Shared Access Signature• Signedresource Blob or Container• AccessPolicy Start, Expiry and Permissions• Signature HMAC-SHA256 of above fields

• Use case• Single use URLs• E.g. Provide URL to Silverlight client to upload to container

http://...blob.../pics/image.jpg?sr=c&st=2009-02-09T08:20Z&se=2009-02-10T08:30Z&sp=w

&sig= dD80ihBh5jfNpymO5Hg1IdiJIEvHcJpCMiCMnN%2fRnbI%3d

POLICY BASED SIGNATURES

• Create Container Level Policy• Specify StartTime, ExpiryTime, Permissions

• Create Shared Access Signature URL• Signedresource Blob or Container• Signedidentifier Optional pointer to container policy• Signature HMAC-SHA256 of above fields

• Use case• Providing revocable permissions to certain users/groups• To revoke: Delete or update container policy

http://...blob.../pics/image.jpg?sr=c&si=MyUploadPolicyForUserID12345

&sig=dD80ihBh5jfNpymO5Hg1IdiJIEvHcJpCMiCMnN%2fRnbI%3d

SHARED ACCESS SIGNATURE

demo

CONTENT DELIVERY NETWORK

• High-bandwidth global blob content delivery• 24 locations globally (US, Europe, Asia, Australia and South

America), and growing• Same experience for users no matter how far they are from

the geo-location where the storage account is hosted

• Blob service URL vs. CDN URL:• Windows Azure Blob URL:

http://images.blob.core.windows.net/

• Windows Azure CDN URL: http://<id>.vo.msecnd.net/

• Custom Domain Name for CDN: http://cdn.contoso.com/

CONTENT DELIVERY NETWORK

North Central US

South Central US

North Europe

West Europe

East Asia

South East Asia

CONTENT DELIVERY NETWORK

To Enable CDN:• Register for CDN

via Dev Portal• Set container

images to publicpic1.jp

gpic1.jpg

GEThttp://guid01.vo.msecnd.net/images/pic.1jpg

http://sally.blob.core.windows.net/images/pic1.jpg

http://sally.blob.core.windows.net/ http://guid01.vo.msecnd.net/

pic1.jpg

404

TTL Content Delivery Network

Windows Azure Blob Service

EdgeLocation

EdgeLocation

EdgeLocation

CONTENT DELIVERY NETWORK

demo

URL Time No CDN AVG

http://welyncus.blob.core.windows.net/mseinstall.exe

11:23 58 seconds50 seconds11:24 49 seconds

11:27 43 seconds

http://az145199.vo.msecnd.net/mseinstall.exe

11:28 40 seconds39 seconds11:29 37 seconds

11:30 40 seconds

http://welydemostorage.blob.core.windows.net/mseinstall.exe

11:32 38 seconds37.67 seconds11:32 34 seconds

11:33 41 seconds

http://az28746.vo.msecnd.net/mseinstall.exe

11:35 36 seconds31 seconds11:36 29 seconds

11:37 28 seconds

TIP OF THE DAY

LIVE PERFORMANCE COUNTER

• Checkout: http://tinyurl.com/wely-live-perf

• Hosting web content in Blob Storage

• Query Blob Storage from the browser

Idea courtesy to Steve Marx

THANK YOU!

Wely Lau (wely@ncs.com.sg) Microsoft MVP, Windows AzureSolutions Architect, NCS Pte LtdBlog : http://wely-lau.net