1. Amazon Kinesis

2. ! ohtani@amazon.co.jp @shot6 ! !

3. ! : ( ) ! : Partner Solution Architect ! : ! 3 AWS ! AWS : KinesisS3

4. Kinesis

5. AWS Kinesis DynamoDB RedshiftRDS EMR Map-Reduce) Kinesis ! ! ! ! ! ! !

6. Amazon Kinesis Data Sources App.4 [Machine Learning] AWS Endpoint App.1 [Aggregate & De-Duplicate] Data Sources Data Sources Data Sources App.2 [Metric Extrac=on] S3 DynamoDB Redshift App.3 [Sliding Window Analysis] Data Sources Availability Zone Shard 1 Shard 2 Shard N Availability Zone Availability Zone SDK

7. ! Management Console Stream Cloud Watch API

8. Amazon Kinesis

9. Kinesis Kinesis Kinesis Kinesis Kinesis

10.

11. MD5 Max: 256Byte Max: 50KB Kinesis

12. PutDC 24

13. READ : 5 TPS or 2MB/sec WRITE : 1000 TPS or 1MB/sec

14. 0 MD5 Range 2128 -1 (2128/2 - 2128) MD5 () -0 (0 - 2128/2)

15. (14) (15) (17) (19) (20)

16. Kinesis(KCL) Kinesis (KCL) Kinesis (KCL) AAA BBB 12345 98765 24680 (24680) (12345) (98765) AAA BBB DynamoDB

17. Kinesis(KCL) Kinesis (KCL) Kinesis (KCL) AAA BBB 12345 98765 24680 (24680) (12345) (98765) AAA BBB DynamoDB 1. KCL 2. ID DynamoDB

18. Kinesis Kinesis S3 Kinesis S3 (24680) (12345) (98765) S3-1 S3-2 Kinesis DynamoDB Kinesis DynamoDB D-1 D-2

19. Amazon Kinesis

20. Raspberry pi +MQTT Kinesis App [Real=me ETL] Frontend [MQTT Proxy] Frontend [MQTT Proxy]

21. Raspberry pi ! Raspberry pi + (ADT7410) 21 ! I2CRaspberry pi ! 101 ! Raspberry piKinesisPut ! Kinesis

22. MQTT ! M2Mpub/sub ! ! http://public.dhe.ibm.com/software/dw/webservices/ws- mqtt/mqtt-v3r1.html ! Mosquitto : MQTT http://mosquitto.org/ ! Raspberry pi MQTT 22

23. Raspberry pi +MQTT Kinesis App [Real=me ETL] Frontend [MQTT Proxy] Frontend [MQTT Proxy] import paho.mqtt.publish as publish publish.single("mqttkb/test", "boo", hostname=xxxx")

24. Raspberry pi +MQTT Kinesis App [Real=me ETL] Frontend [MQTT Proxy] Frontend [MQTT Proxy] MQTT-Kinesis

25. Raspberry pi +MQTT Kinesis App [Real=me ETL] Frontend [MQTT Proxy] Frontend [MQTT Proxy]

26. Raspberry pi +MQTT Kinesis App [Real=me ETL] Frontend [MQTT Proxy] Frontend [MQTT Proxy] KinesisDynamoDB

27. Raspberry pi +MQTT Kinesis App [Real=me ETL] Frontend [MQTT Proxy] Frontend [MQTT Proxy] Kinesis Client LibraryPython

28. 28

29. Amazon Kinesis

30. Kinesis ! Kinesis SDK SDK ! Kinesis Client Library(KCL) https://github.com/awslabs/amazon-kinesis-client ! Kinesis Connectors https://github.com/awslabs/amazon-kinesis-connectors ! Kinesis Storm Spout https://github.com/awslabs/kinesis-storm-spout ! EMR Connector EMRAMI 30

31. Kinesis Client Library(KCL) ! KinesisJava DynamoDB, CloudWatch, AutoScaling + ! IRecordProcessorFactory IRecordProcessor 31

32. 32

33. Kinesis WebHDFS ! KinesisEMRHDFS ! WebHDFSREST ! TO 33 REST HTTP PUT http://xxx.9101/webhdfs/v1/user/ohtani/le ?op=create&user.name=hadoop Kinesis EMR Client

34. 34 Kinesis WebHDFS (EC2)

35. 35 Kinesis WebHDFS (EC2) EMR HDFS

36. Kinesis Connectors ! KinesisAWS 36 ITransformer Kinesis IFilter IBuer IEmitter AWS S3 DynamoDB Redshift Kinesis

37. Amazon Kinesis

38. AWS AWS Endpoint Kinesis App.1 Kinesis App.2 Redshift DynamoDB Kinesis App.3 Availabilit y Zone Shard 1 Shard 2 Shard N Availabi lity Zone Availabi lity Zone RDS BI as a Service S3 Kinesis

39. ! Stream Data Sources Data Sources Data Sources Kinesis App Kinesis App Kinesis App Data Sources Data Sources Data Sources Kinesis App Kinesis App Kinesis App Kinesis App Data SourceA Data SourceB Data SourceAETL Data SourceBETL

40. KinesisEMR DynamoDB RedshiftRDS EMR Map-Reduce) Kinesis ! ! ! ! ! ! !

41. Kinesis Kinesis EMR EMR DynamoDB ElastiCache Redis S3DynamoDB ElastiCache Redis Redshift S3 Redshift EMR KinesisEMR

42. ! Kinesis ! Kinesis + AWS ! 42

43.