1. AWS CloudTrail & CloudWatch LogsAWS Black Belt Tech Webinar 2014 () 2014.12.10

2. Agenda CloudTrail CloudTrail CloudWatch Logs CloudWatch Logs CloudWatch Logs Tips 3. AWS CloudTrail CloudWatchLogs 4. CloudTrail 5. AWS CloudTrailAPIAWSCloudTrailAPI 6. CloudTrail1/2 AWS 3rd partyAPI S3 SNS(Simple Notification Service) 7. CloudTrail2/2 20147 CloudTrail Amazon S3 / SNS 8. CloudTrail 7 8Quarter /Year QUARTER / YEAR162127302520151050Q4 2013 Q1 2014 Q2 2014 Q3 2014 Q4 2013 9. CloudTrailEC2VPCRDSRedshiftIAMSTSCloudTrailEBSCompute and NetworkingDatabaseDeployment and ManagementStorage and Content DeliveryApplication ServicesAnalyticsELBAutoScalingDirectConnectCloud FrontCloudFormationElasticBeanstalkOps WorksCloudwatchSQSSimpleWorkflowKinesisEMRhttp://docs.aws.amazon.com/awscloudtrail/latest/userguide/what_is_cloud_trail_supported_services.htmlElastiCacheCloudSearchDataPipelineElasticTranscoderSNSZocalo 10. 11. CloudTrailS3S3SNSSNS TopicIAMSTS 12. CloudTrail 13. One AWS Account With Resources in Multiple AWS RegionsMultiple AWS Accounts with Resources in One AWS RegionMultiple AWS Accounts With Resources in Multiple AWS Regions 14. CloudTrailAPI call EventNon-API call Event APIv StartInstances v CreateKeyPair v AWS v AWS 15. CloudTrail API API API AWS AWS 16. APIIAM BobuserIdentity:{accessKeyId:AKEXAMPLE123EJVA,accountId:123456789012,arn:arn:aws:iam::123456789012:user/Bob,principalId:AIEXAMPLE987ZKLALD3HS,type:IAMUser,userName:Bob} 17. API AliceuserIdentity:{type:FederatedUser,principalId:123456789012:Alice,arn:arn:aws:sts::123456789012:federated-user/Alice,accountId:123456789012,accessKeyId:ASEXAMPLE1234WTROX8F,sessionIssuer:{type:IAMUser,accountId:123456789012,userName:Bob}} 18. APIAPIISO8601eventTime:2014-11-29T05:58:13ZAPIeventSource:signin.amazonaws.comeventName:ConsoleLogin 19. AWSIPAPIawsRegion:us-east-1,sourceIPAddress:ec2.amazonaws.com 20. CloudTrail Tips CloudTrail JSON??? 21. CloudTrail + CloudSearch CloudTrail Amazon SNSTopicsAmazon SQSQueueAmazonElasticBeanstalkWorker RoleAmazon S3BacketAmazonCloudSearch https://medium.com/aws-activate-startup-blog/searching-cloudtrail-logs-easily-with-amazon-cloudsearch-2d716e23efee 22. CloudTrail + Amazon LambdaCloudTrail AmazonLambdaAmazon S3BacketAmazonCloudSearch 23. CloudTrail + ElasticSearch + Kibana CloudTrail Amazon SNSTopicsAmazon SQSQueueElasticSearchAmazon S3Backet https://blogs.amazon.com/aws_solutions/archive/2014/10/processing-cloudtrail-logs-into-logstashelasticsearchkibana.html 24. CloudTrail Processing Library AWS SDK for Java CloudTrail CloudTrail Processing Libraryre:Invent2014 Reads messages delivered to SNS/SQS Downloads and reads S3 log files continuously Serializes the events into a Plain Old Java Object Allows you to implement your own custom logic to process Fault tolerant and supports multi threading 25. CloudTrail + CloudTrail Processing LibraryCloudTrail Amazon SNSTopicsAmazon SQSQueuehttp://www.slideshare.net/AmazonWebServices/sec306-turn-on-cloudtrail-log-api-activity-in-your-aws-account-aws-reinvent-2014AmazonCloudTrailProcessingLiibraryAmazon S3Backet3rd Party AmazonDynamoDBAmazonRedshiftAmazonCloudWatchAmazonSNS 26. CloudTrail Processing Library public void process(ListCloudTrailEvent events) {for (CloudTrailEvent event : events) {CloudTrailEventData data = event.getEventData();if (data.getEventSource().equals(ec2.amazonaws.com) data.getEventName().equals(ModifyVpcAttribute)) {System.out.println(Processing event:+ data.getRequestId());sns.publish(myQueueArn, {+'requestId'= ' + data.getRequestId() + ', +'request' = ' + data.getRequestParameters() + ',+'response' = ' + data.getResponseElements() + ', +'source' = ' + data.getEventSource() + ', +'eventName'= ' + data.getEventName() + ' +});}}} 27. CloudTrail integration with CloudWatch Logs CloudTrailCloudWatch LogsClouWatch LogsCloudWatch LogsLog Group 28. CloudTrail 29. CloudTrail 30. CloudWatchLogs 31. CloudWatch Logs 20147 OS 3 EU 32. CloudWatch Logs OS SNS CloudWatch Logs http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/WhatIsCloudWatchLogs.html 33. CloudWatch LogsAmazonLinuxUbuntuServer WindowsRed HatEnterprise LinuxCloudWatch Logsapi call to endpointSNS 34. CloudWatch Log Log Group (Apache Log Group/var/log/messages) Log Stream Instance-id Log Event UTF-8 Log Agent Log Event Metric Filters Simple Notification Service(SNS) Retention Policies 1day Never Expired Log GroupLog Stream 35. CloudWatch LogsLog GroupLog StreamLog EventWeb Serverweb001.ap-northeast-1web002.ap-northeast-1web003.ap-northeast-1 36. UTF-8 37. CloudWatch Logs 38. CloudWatch Logs (CloudWatch) 5GB5GB$0.50 / GB$0.03 GB / (1)10 10 100 API http://aws.amazon.com/jp/cloudwatch/pricing/ 39. CloudWatch Logs 40. Cloudwatch Logs Agent AgentItemDescriptionAWS Access Key IDAWSIDIAMAWS Secret Access KeyAWSIAMDefault region nameus-east-1, us-west-2, or eu-west-1us-east-1Default output formatOKPath of log file to uploadDestination Log Group name Destination Log Stream name Timestamp format Initial positionstart_of_file end_of_file 41. CloudWatch LogsLinux 42. CloudWatch LogsLinux1/ LinuxPython[ec2-user@ip-10-0-10-104 ~]$ wget https://s3.amazonaws.com/aws-cloudwatch/downloads/awslogs-agent-setup-v1.0.py[ec2-user@ip-10-0-10-104 ~]$ sudo python ./awslogs-agent-setup-v1.0.py --region us-east-1Launching interactive setup of CloudWatch Logs agent ...Step 1 of 5: Installing pip ...DONEStep 2 of 5: Downloading the latest CloudWatch Logs agent bits ... DONEStep 3 of 5: Configuring AWS CLI ...AWS Access Key ID [****************WLGA]:AWS Secret Access Key [****************qVIu]:Default region name [None]:Default output format [None]:Step 4 of 5: Configuring the CloudWatch Logs Agent ...Path of log file to upload [/var/log/messages]:Destination Log Group name [Linux Syslog Group]:PythonAccess Key/Secret Access KeyIAMRegionLog Group 43. CloudWatch LogsLinux2/Choose Log Stream name:1. Use EC2 instance id.2. Use hostname.3. Custom.Enter choice [1]:Choose Log Event timestamp format:1. %b %d %H:%M:%S (Dec 31 23:59:59)2. %d/%b/%Y:%H:%M:%S (10/Oct/2000:13:55:36)3. %Y-%m-%d %H:%M:%S (2008-09-08 11:52:54)4. CustomEnter choice [1]: 3Choose initial position of upload:1. From start of file.2. From end of file.Enter choice [1]: 1More log files to configure? [Y]: nStep 5 of 5: Setting up agent as a daemon ...DONEID 44. CloudWatch LogsLinux/Log GroupLog StreamLog EventLinux Syslog Groupi-170ad10e(instance-id)/var/log/messages 45. CloudWatch LogsWindows 46. CloudWatch LogsWindows/ Windows AgentEC2Config Amazon Windows EC2Config Service 2.2.5 (2.2.12)CloudWatch Logs EC2 Config Windows ServicesEC2 Config Service EC2Config Service Update https://aws.amazon.com/developertools/5562082477397515 47. CloudWatch LogsWindows/2. EC2Config1. CloudWatchLogs-rolehttp://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/QuickStartEC2Instance.html 48. CloudWatch LogsWindows/ CWL JSON AWS.EC2.Windows.CloudWatch.jsonC:Program FilesAmazonEc2ConfigServiceSettings Windows Event Tracing for Windows IIS 49. Cloudwatch logs Agent Agent Linux: /var/log/awslogs.log, /var/log/awslogs-agent-setup.log Windows: C:Program FilesAmazonEc2ConfigServiceLogsEc2ConfigLog.txt Agent Requirement Python version 2.6, 2.7, 3.0, or 3.3 Amazon Linux version 2014.03.02 Ubuntu Server version 12.04, or 14.04 CentOS version 6, 6.3, 6.4, or 6.5 Red Hat Enterprise Linux (RHEL) version 6.5 or 7.0 Windowshttp://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/CWL_GettingStarted.html 50. Cloudwatch Logs CLI CloudWatch Logs CLI create-log-group / create-log-stream delete-log-group / delete-log-stream /delete-metric-filter /delete-retention-policy describe-log-groups / describe-log-streams /describe-metric-filters get-log-events put-log-events / put-metric-filter / put-retention-policy test-metric-filterhttp://docs.aws.amazon.com/cli/latest/reference/logs/index.html 51. CLI, APIhttp://docs.aws.amazon.com/cli/latest/reference/logs/index.html{nextForwardToken: f/31361289460258459128311697783906457332057505625559597056,events: [{ingestionTime: 1406284504094,timestamp: 1406284498741,message: Jul 24 08:45:19 ip-10-0-10-193useradd[1435]: add 'ec2-user' to group 'wheel'},{ingestionTime: 1406284504094,timestamp: 1406284498741,message: Jul 24 08:45:19 ip-10-0-10-193useradd[1435]: add 'ec2-user' to shadow group'wheel'},JSON 52. CloudWatch logs Metric Filter1/3 53. CloudWatch Logs Metric Filter2/3 error3errorerror 54. CloudWatch Logs Metric Filter3/3 Metric FilterSNSMetric FilterCloudWatch 55. CloudTrail integration with CloudWatch Logs CloudTrailCloudWatch LogsCloudTrail CloudWatch LogsCloudWatch LogsLog Group 56. CloudTrail JSON 57. CloudWatch Logs Tips 58. CWL Agent1v 1 Log Event 32KB32KB1,000,000 byte#!/usr/bin/rubystr=a*1000000open(/var/log/messages,a){|f|f.writestr}2014-10-18 10:29:28,770 - cwlogs.push - WARNING - 31955 - Thread-6 Truncate event: {'source_id': '4c0bbd10e46d9c06707aada0cd3e2cd8', 'timestamp': None,'start_position': 748331L, 'end_position': 1748331L},reason: single event exceeds 32KB limit./var/log/awslogs/log 59. NATNAT 60. CloudWatch Logs API/CLIv GetLogEvents, PutLogEvents API EC2CloudWatch LogsCloudWatch LogsAPI : get-log-evensAPI : put-log-evenshttp://docs.aws.amazon.com/cli/latest/reference/logs/index.html 61. Log Event buffer_duration 32KB Log Event 1000 Log Event 62. CloudWatch Logs 1AWSLog Group500 1 Log Group100Metric Filter 1 Log Stream5PutLogEvents API 1AWS10GetLogEvents API 63. CloudTrailAWS CloudWatch LogsOSCloudWatch Logs CloudTrailCloudWatch LogsAWS 64. QA 65. Webinar AWS http://aws.amazon.com/jp/aws-jp-introduction/