analytics in the cloud flash talk

28
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. January 10, 2017 Analytics in the Cloud from AWS AWS GovCloud (US)

Upload: trinhduong

Post on 13-Feb-2017

219 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Analytics in the Cloud Flash Talk

©"2016,"Amazon"Web"Services,"Inc."or"its"Affiliates."All"rights"reserved.

January'10,'2017

Analytics"in"the"Cloud"from"AWS

AWS GovCloud (US)

Page 2: Analytics in the Cloud Flash Talk

Global&Infrastructure

Page 3: Analytics in the Cloud Flash Talk

Global&Infrastructure&of&AWS

17* Regions – 44 Availability'Zones – 68 Edge'Locations

Region &.Number.of.Availability.Zones

AWS.GovCloud (2) EU

*C2S.(2) Ireland'(3)US West Frankfurt (2)Oregon (3) London'(2)Northern California'(3)US.East Asia.Pacific

N. Virginia'(5),'Ohio'(3) Singapore'(2)South America Sydney (2),'Tokyo'(3),São'Paulo'(3) Seoul (2),'Mumbai'(2)Canada

Central'(2) China

Beijing'(2)Announced.Regions

Paris,'Ningxia

AWS GovCloud (US)

Page 4: Analytics in the Cloud Flash Talk

AWS&GovCloud&(US)&is&an&isolated&AWS&region

Intended&for&customers&with&strict&regulatory&and&compliance&requirements&and&sensitive&data&or&workloads

August"2011Available'to'qualified'customers

ComplianceSafeguard'sensitive'data/systems

Addresses'multiple'US'Government'regulations'and'security'requirements

Page 5: Analytics in the Cloud Flash Talk

AWS&GovCloud&(US)&distinguishing&features

Separate'IAM'(unique'credentials)

Data,'network,'and'machine'isolation'from'other'regions

Dedicated'GovCloud'Management'Console

“Community'Cloud”'with'vetted'account'holders

Managed'by'US'Persons'on'US'soil

Page 6: Analytics in the Cloud Flash Talk

Requirements&for&access&to&AWS&GovCloud&(US)

Account&holder&must&be&a&US&Person&(defined&as&a&US&citizen&or&a&Green&Card&holder)

US&entity&incorporated&to&do&business&in&the&United&States&and&is&based&on&US&soil

Can&handle&export&control&data

Learn.more:'https://aws.amazon.com/govcloud`us/getting`started/

Page 7: Analytics in the Cloud Flash Talk

AWS&GovCloud&(US)&Summary

AWS&GovCloud&(US)&is&physically&and&logically&isolatedSeparate&AZs,&Management&Console,&IAM&stack,&and&service&endpointsAWS&GovCloud&(US)&is&not&just&for&the&US&GovernmentAvailable&to&qualified&contractors&and&organizations&in&regulated&industriesRemember&the&AWS&Shared&Responsibility&ModelLeverage&services&like&VPC,&IAM,&KMS&and&CloudHSM&to&secure&sensitive&workloads&and&manage&access&to&sensitive&data

AWS&GovCloud&(US)&webpage:&https://aws.amazon.com/govcloudQus/Contact&AWS&GovCloud&team&at&[email protected]

AWS GovCloud (US)

Page 8: Analytics in the Cloud Flash Talk

C2S'brings'Amazon'Web'Services'(AWS)'“over'the'fence”'and'into'the'IC.''

C2S'is'an'“air`gapped”'region'that'utilizes'the'same'innovation'as'AWS'commercial'regions.''

AWS'will'bring'in'cutting'edge'products'and'services'for'you'to'utilize'as'technology'grows'and'changes.'And'you'only'pay'for'what'you'use.

IC'ITE'Commercial'Cloud:'AWS'C2S

Page 9: Analytics in the Cloud Flash Talk

Compute,&Analytic&and&Big&Data&Services

Page 10: Analytics in the Cloud Flash Talk

Compute:&EC2&GPU&and&FPGA&Instance&Types• G2'` optimized'for'graphics`intensive'applications.• P2'` designed'for'general`purpose'GPU'compute'applications'using'CUDA'and'OpenCL• Newly.announced• Amazon'Ec2'F1'Instances` Custom'FPGAs• Amazon'EC2'Elastic'GPUs'for'attachable'graphics'Acceleration

Page 11: Analytics in the Cloud Flash Talk

Serverless'compute'service'that'runs'your'code'in'response'to'events.

Extend'AWS'services'with'user:defined'custom'logic.

Pay'only'for'the'requests'served'and'compute'time'required': billing'in'increments'of'100'milliseconds'

AWS'Lambda

AWS'Lambda:'Serverless event'processing

Page 12: Analytics in the Cloud Flash Talk

Take&advantage&of&new&computing&models

Develop&without&worrying&about&servers&

to&manage

Setup&your&code&to&run&in&response&to&a&variety&of&triggers

Code&is&executed&on&demand&with&

continuous&scaling

Pay&only&for&the&time&used&with&subQsecond&

metering

Serverless architectures&free&you&from&managing&infrastructure

New:%Lambda@Edge lets%you%run%code%at%CloudFront edge%locations%without%provisioning%or%managing%servers.%This%allows%developers%to%deliver%a%low%latency%user%experience%for%customized%web%applications.

AWS&Lambda

Page 13: Analytics in the Cloud Flash Talk

Amazon&Kinesis:&streaming&data&made&easyServices&make&it&easy&to&capture,&deliver,&and&process&streams&on&AWS

Kinesis"Analytics"For$all$developers,$data$scientists$

Easily&analyze&data&streams&using&standard&SQL&queries

Kinesis"FirehoseFor$all$developers,$data$scientists$

Easily&load&massive&volumes&of&streaming&data&into&S3,&Amazon&Redshift,&or&Amazon&ES

Kinesis"StreamsFor$Technical$Developers

Collect&and&stream&data&for&ordered,&replayable,&realQtime&processing

Page 14: Analytics in the Cloud Flash Talk

Distributed'search'and'analytics'engine

Managed'service'using'Elasticsearch'and'Kibana

Fully'managed': zero'admin

Highly'available'and'reliable

Tightly'integrated'with'other'AWS'servicesAmazon'Elasticsearch'

Service

Amazon'Elasticsearch ServiceSemi:structured'/'unstructured'data'processing'

Page 15: Analytics in the Cloud Flash Talk

ETL&with&AWS&Data&Pipeline

Page 16: Analytics in the Cloud Flash Talk

Glue'automates'the'undifferentiated'heavy`lifting'of'ETL

! Cataloging&data&sources&

! Identifying&data&formats&and&data&types

! Generating&Extract,&Transform,&Load&code

! Executing&ETL&jobs;&managing&dependencies

! Handling&errors

! Managing&and&scaling&resources

AWS&Glue

Page 17: Analytics in the Cloud Flash Talk

AWS.Glue.Components

Data.Catalog

" Hive'metastore'compatible'metadata'repository'of'data'sources.'

" Crawls'data'source'to'infer'table,'data'type,'partition'format.'

Job.Execution

" Runs'jobs'in'Spark'containers – automatic'scaling'based'on'SLA.

" Glue'is'serverless'` only'pay'for'the'resources'you'consume.'

Job.Authoring

" Generates'Python'code'to'move'data'from'source'to'destination.'

" Edit'with'your'favorite'IDE;'share'code'snippets'using'Git.

Page 18: Analytics in the Cloud Flash Talk

Elastic&MapReduce&(EMR)• Reduces'complexity'of'big'data'processing

• Managed'Hadoop'2.7'and'2.4'infrastructure• Handles'node'provisioning,'customization,'and'shutdown• Provides'tools'to'debug'and'monitor'your'Hadoop'clusters

• Provides'tight'integration'with'AWS'services'(S3,'DynamoDB)

18

Page 19: Analytics in the Cloud Flash Talk

Amazon&Machine&Learning

• Easy`to`use'service'built'for'developers• Robust,'powerful,'and'technology`based• Ability'to'create'models'using'your'data• Deployable'to'production'in'seconds

Page 20: Analytics in the Cloud Flash Talk

Fast,'powered'by'SPICE,'automatically'scales.'

Explore,'analyze,'share'insights'with'anyone.

1/10th'the'cost'of'traditional'BI'solutions.

Broad'connectivity'with'AWS'data'services,'on:premises'data,'files'and'business'applications.

Amazon'QuickSight

Amazon'QuickSight

Amazon"RDS

Amazon"S3 Amazon"Redshift

Visualize)and)explore

Page 21: Analytics in the Cloud Flash Talk

Amazon&Athena:&Serverless Query&Processing&with&S3

• Serverless'query'service'for'querying'data'in'S3'using'standard'SQL'with'no'infrastructure'to'manage

• No'data'loading'required;'query'directly'from'Amazon'S3

• Use'standard'ANSI'SQL'queries'with'support'for'joins,'JSON,'and'window'functions

• Support'for'multiple'data'formats'include'text,'CSV,'TSV,'JSON,'Avro,'ORC,'Parquet

• Pay'per'query'only'when'you’re'running'queries'based'on'data'scanned.'If'you'compress'your'data,'you'pay'less'and'your'queries'run'faster

AmazonAthena

Page 22: Analytics in the Cloud Flash Talk

Other&re:Invent 2016&Announcements• Amazon'Lex` build'conversational'interfaces'using'voice'and'text• Amazon'Polly` turn'text'into'lifelike'speech• Amazon'Rekognition` add'image'analysis'to'your'applications• AWS'Batch` Enables'users'to'easily'and'efficiently'run'hundreds'of'thousands'of'batch'computing'jobs'on'AWS• Amazon'Lightsail` deploy'a'virtual'private'server'using'preconfigured'templates• AWS'Greengrass` run'compute,'messaging'&'data'caching'for'connected'devices'in'a'secure'way,'even'when'not'connected'to'the'Internet.• AWS'Snowball.Edge` 100TB'data'transfer'device'with'on`board'storage'and'compute• AWS'Snowmobile` Migrate'data'securely'via'a'truck'storing'up'to'100'PB'of'data

Page 23: Analytics in the Cloud Flash Talk

Implementing'a'Data'Lake'on'AWS

Elasticsearch

Page 24: Analytics in the Cloud Flash Talk

AWS&and&IC&Marketplace

24

• 3,500+'software'listings• Over'1,100'participating'ISVs

• Open'source'and'commercial'software''

• Try'before'you'buy,'experiment'cheaply

• Procure'new'or'Bring`your`own`license

• Deployed'in'Commercial'and'C2S'Regions

http://aws.amazon.com/marketplace

Page 25: Analytics in the Cloud Flash Talk

Networking AnalyticsCompute

Storage&&&Content&Delivery

Developer&Tools Management&Tools Security&&&Identity

Application&Services

Mobile&Services Database

S3 EBS Storage"Gateway Glacier EFS API"Gateway AppStream CloudSearch Elastic"

Transcoder SES SQS SWF

Device"Farm Mobile"AnalyticsCognito

SNS

RDS DynamoDB ElastiCache RedShift WorkSpaces WorkDocs WorkMail

Lambda EC2"Container"Service

Elastic"BeanstalkEC2 VPC ELB Direct"

Connect EMR Kinesis Data"Pipeline Machine"LearningRoute"53 QuickSight Elasticsearch

Service

CodeCommit CodeDeploy CodePipeline CloudWatch CloudFormation CloudTrail Config OpsWorks Service"

CatalogIdentity"&"

Access"MgmtDirectory"Service

Trusted"Advisor Cloud"HSM Key"

Mgmt"ServiceWeb"App"Firewall

Snowball

Simple"DB DB"Migration"Service

IOTIoT

HubsMobile"Hub

AWS"Mgmt"Console

Command"Line"Interface

Interaction

Libraries&and&SDKsJava Javascript Python

(boto) PHP .NET Ruby Node.js iOS Android Toolkit"for"VS

Toolkit"for"Eclipse

Tools"for"Powershell AWS"CLI

CloudFront

Enterprise&Applications

AWS'Commercial'Platform'Services

Marketplace

AWS"Marketplace

Page 26: Analytics in the Cloud Flash Talk

AWS&GovCloud&(US)&Services

Kinesis'Streams

Page 27: Analytics in the Cloud Flash Talk

Requesting&New&C2S&Services• C2S&page&on&IC&ITE&home&page:

• C2S&Roadmap&• Service&launch&announcements

• New&Service&Requests:• On&the&right&side&of&the&C2S&page,&click&“Request&New&Service”• Fill&out&the&form&with&POC&info,&list&of&new&services&or&features&

requested,&estimated&users&and&usage,&and&potential&useQcase

27

Page 28: Analytics in the Cloud Flash Talk

Migrating&existing&applications&as&quickly&as&possible

Deploying&new&applications&to&the&cloud&by&default

Cloud"computing"has"become"the"new"normal

Public&Sector&and&especially&the&US&IC&is&moving&to&AWS.

AWS&services&more&than&a&million&active&customers&every&month&in&190&countries.

Through&customer&obsession&and&an&increasing&pace&of&innovation,&AWS&is&the&fastest&technology&vendor&in&history&to&reach&a&$10B&run&rate.