1. AWS 201473

2. Who am I ? ( ) 1978 AWS 140 60 250 Amazon Web Services http://aws.typepad.com/aws_japan/ 10+ years web engineer in startups Director of V-cube (perl), 2001 - 2006 CTO of FlipClip (perl), 2006 - 2009 CTO of gumi (python), 2009 - 2012 @horiuchi horiyasu ! 3. Amazon.com 4. amazon.com, 1995 Copyright 2012 Amazon Web Services 5. E 15000 amazon.com, today 6. 7. AWS Amazon API 8. 9. Amazon 10. 11. Amazon No. 1 12. Amazon 13. 14. 15. Amazon S3 - 16. Amazon No. 2 17. Amazon () Amazon.co.jp 3.5-8 = 18. Amazon Order DB Payments Service C++App C++App C++App Amazon.com 2010 19. Amazon Elastic MapReduce - 1 20. ! AWSAmazon IT 21. AWS EBS, S3, Glacier, Storage Gateway CloudFront VPC, Route 53, Direct Connect IAM, CloudTrail, CloudHSM Cloud Watch Web Management Console Elastic Beanstalk, Cloud Formation, OpsWorks CLI & SDKs Java, PHP, .NET, Python, Ruby AZRegion EC2, Auto Scaling, Elastic Load Balancing, Workspaces RDS, DynamoDB, Redshift, ElastiCache Elastic MapReduce, Kinesis, Data Pipeline AppStream, Cloud Search, SWF, SQS, SES, SNS, Elastic Transcoder 22. 23. AWS 24. Start Simple Write Events File on Device Periodically Upload to S3 Process into Redshift Point GUI Tool to Redshift 2014-01-24,nateware,e4df,login 2014-01-24,nateware,e4df,gamestart 2014-01-24,nateware,e4df,gameend 2014-01-25,nateware,a88c,login 2014-01-25,nateware,a88c,friendlist 2014-01-25,nateware,a88c,gamestart Profit! 25. Amazon Redshift 10 GigE (HPC) Ingestion Backup Restore SQL Clients/BI Tools 128GB RAM 16TB disk 16 cores Amazon S3/DynamoDB JDBC/ODBC 128GB RAM 16TB disk 16 coresCompute Node 128GB RAM 16TB disk 16 coresCompute Node 128GB RAM 16TB disk 16 coresCompute Node Leader Node - SQL - - ! - - Amazon S3 - Amazon DynamoDB ! 26. Tableau + Redshift 27. More Data Sources Also Collect Server Logs Periodically Upload to S3 Stuff into Redshift External Analytics Data Too External Analytics EC2 28. Dealing With Messy Data Different File Formats Device vs Apache vs CDN Cleanup with EMR Job Output to Clean Bucket Load into Redshift EC2 29. Direct From DynamoDB Integrate Game DB Load Directly into Redshift Redshift does Intelligent Merge Tracks Hash Keys, Columns EC2 30. Direct From DynamoDB Integrate Game DB Load Directly into Redshift Redshift does Intelligent Merge Tracks Hash Keys, Columns Or Stream into EMR EC2 31. 32. Back To Basics 2014-01-24,nateware,e4df,login 2014-01-24,nateware,e4df,gamestart 2014-01-24,nateware,e4df,gameend 2014-01-25,nateware,a88c,login 2014-01-25,nateware,a88c,friendlist 2014-01-25,nateware,a88c,gamestart 33. Back To Basics [Dubstep Remix] Always Batch Due to S3 EC2 34. Need Data Faster! Stream Data With Kinesis Multiple Writers and Readers Still Output to Redshift EC2 35. Lots of Ins and Outs Stream Data With Kinesis Multiple Writers and Readers Still Output to Redshift Stream to Spark on EMR Storm via Kinesis Spout Custom EC2 Workers EC2 EC2 36. Amazon Kinesis Data Sources App.4 ! [Machine Learning] ! ! ! A WS En dp oin t App.1 ! [Aggregate& De-Duplicate] Data Sources Data Sources Data Sources App.2 ! [Metric Extraction] S3 DynamoDB Redshift App.3 [Sliding Window Analysis] Data Sources Availability Zone Shard 1 Shard 2 Shard N Availability Zone Availability Zone 37. 38. Clash of Clans Amazon Kinesis Redshift Clickstream archive EC2: In-game engagement trends dashboard Real-time clickstream processing app Kinesis: Real-time data stream of in-game activity Multiple Kinesis applications: Dashboards, analytics and storage Redshift: Business intelligence reporting and interactive queries S3 and Glacier: Data storage and long term archival In-game activity S3 Aggregate statistics Business-intelligence user Kinesis-enabled apps on EC2 39. S3EMRRedshift ~ ~ (), (), () http://amzn.to/1ih9n9M 40. AWS 41. CPU 42. GPU IO CR1M2CC2C1 HI1 HS1 CG1M1 43. GPU IO CR1M2CC2C1 HI1 CG1M1 G2M3 R3C3 I2 HS1 44. M3 - Intel Xeon E5-2670 (Sandy Bridge) SSD CPU vCPU (GiB) SSD (GB) ) m3.medium 1 3.75 1 x 4 $0.101 m3.large 2 7.5 1 x 32 $0.203 m3.xlarge 4 15 2 x 40 $0.405 m3.2xlarge 8 30 2 x 80 $0.810 45. C3 - CPU Intel Xeon E5-2670 v2 (Ivy Bridge) SSD (SR-IOV, VPC) CPUCPU vCPU (GiB) SSD (GB) ) c3.large 2 7 2 x 16 $0.128 c3.xlarge 4 14 2 x 40 $0.255 c3.2xlarge 8 28 2 x 80 $0.511 c3.4xlarge 16 55 2 x 160 $1.021 c3.8xlarge 32 108 2 x 320 $2.043 46. R3 - Intel Xeon E5-2670 v2 (Ivy Bridge) SSD (SR-IOV, VPC) GiB vCPU (GiB) SSD (GB) ) r3.large 2 15 1 x 32 $0.210 r3.xlarge 4 30.05 1 x 80 $0.420 r3.2xlarge 8 61 1 x 160 $0.840 r3.4xlarge 16 122 1 x 320 $1.680 r3.8xlarge 32 244 2 x 320 $3.360 47. I2 - Intel Xeon E5-2670 v2 (Ivy Bridge) SSD TRIM (SR-IOV, VPC) I/OIOPS i2.8xlarge365,000315,000 vCPU (GiB) SSD (GB) ) i2.xlarge 4 30.05 1 x 800 $1.051 i2.2xlarge 8 61 2 x 800 $2.101 i2.4xlarge 16 122 4 x 800 $4.202 i2.8xlarge 32 244 8 x 800 $8.404 48. NEW!! T2 1 6CPU 144CPU 1CPU=1CPU t2.micro (10%) 49. 50. ! 51. IT ! IT 52. AWS Summit Tokyo 2014 2014717() 18() () () http://www.awssummittokyo.com/