(ism311) how finra gains visibility into its aws environment
TRANSCRIPT
FINRA Information Security Engineering Copyright 2015 FINRAcopy 2015 Amazon Web Services Inc or its Affiliates All rights reserved
Gary Mikula Sr Director of Information Security
Siddhartha Dadana Senior Security Engineer
ISM311
How FINRA Gains End-to-End Visibility
into a Large-Scale AWS Environment
Gaining Transparency into Cloud Computing
October 2015
FINRA Information Security Engineering Copyright 2015 FINRA
Who We Are
FINRAmdashthe Financial Industry Regulatory Authoritymdashis an
independent nongovernmental regulator for all securities
firms doing business with the public in the United States
FINRA protects investors by regulating brokers and
brokerage firms and by monitoring trading on US stock
markets
FINRA monitor over 6 billion shares traded on the stock
market each day
FINRA handles more ldquobig datardquo on a daily basis than the
Library of Congress or Visaregmdashto build a holistic picture of
the trading market
FINRA Deter Detect Discipline
Investor Protection
FINRA Information Security Engineering Copyright 2015 FINRA
Historical View
Cyclic Processes
bull POC ndash Budget Approval ndash SDLC - Maintenance
Defined Roles
bull Coders Code
bull Managers Manage
bull Administrators Administer
Agile DevelopmentCloud Computing
bull Developers Make These Decisions
ndash Security
ndash Financial
ndash Architecture
bull And Itrsquos All Point and Click
Hacking Redefined Security
bull Defensive Coding
bull Baked In Not Painted On
ldquoYou guys start
coding Irsquoll go find out
what the users
wantrdquo
FINRA Information Security Engineering Copyright 2015 FINRA
Same Challenges Different Environment
Security
bull Engaged All Necessary AWS Security Features
bull Are We Firewalled Correctly
Compliance
bull Followed All Published Standards
Networking
bull Placed Servers on the Correct Network
Finance
bull Stayed Within Budget
Capacity Planning
bull Used Resources Optimally
But Now in a Decentralized Model
bull Itrsquos deacutejagrave vu all over againhellipYogi Berra
ldquoWith great power
comes great
responsibilityrdquo
FINRA Information Security Engineering Copyright 2015 FINRA
Security ndash AWS Identity and Access Management
AWS Shared Responsibility Model
bull AWS ndash ldquoSecurity of the Cloudrdquo
bull YOU ndash ldquoSecurity in the Cloudrdquo
AWS Identity and Access Management (IAM)
bull Critical Security-related ldquoServicerdquo
Best Practices
bull Lock Away Your AWS Account (Root) Access Keys
ndash Console Access
bull Rotate Credentials Regularly
ndash Time vs Event Driven
bull Remove Unnecessary Credentials
ndash Unused
bull Grant Least Privilege
bull Keep a History of Activity in Your AWS Account
FINRA Information Security Engineering Copyright 2015 FINRA
Lock Away Your AWS Account (Root) Access Keys
index=aws-cloudtrail sourcetype=aws-cloudtrail requestParametersuserName=root eventName=CreateAccessKey
index=aws-cloudtrail userIdentityarn=root eventName=Console | table _time eventName responseElementsConsoleLogin awsRegion sourceIPAddress eventSource userIdentityarn userIdentitytype userIdentityaccountId userAgent
IAMAWS CloudTrail
FINRA Information Security Engineering Copyright 2015 FINRA
IAMAWS CloudTrail
Credentials (Rotation Unnecessary)
Python (boto)
usrbinpython
import boto
from botoiamconnection import IAMConnection
iam = IAMConnection( aws_access_key_id = xxxxx
aws_secret_access_key = xxxxxxxx
proxy = xxxxx
proxy_port = xxxx
)
users = iamget_all_users()list_users_responselist_users_resultusers
for user in users
if password_last_used in user
print (ststs) ( user[create_date] user[user_name]
user[password_last_used] )
else
print (ststs) ( user[create_date] user[user_name] NEVER)
FINRA Information Security Engineering Copyright 2015 FINRA
Grant Least Privilege
Python (boto)
Version ControlledRepositorymiddot Security
middot Operationsmiddot Development
ROLE ABC
autoscalingDeletePolicyDeny
autoscalingDeleteScheduledActionDeny
autoscalingDeleteTagsDeny
autoscalingDescribeAdjustmentTypesAllow
autoscalingDescribeAutoScalingGroupsDontCare
autoscalingDescribeAutoScalingInstancesAllow
IAMAWS CloudTrail
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA Information Security Engineering Copyright 2015 FINRA
Beyond IAM Best Practices
Tagging Compliance
Security Group
Logging
Amazon S3Amazon Elastic Block Store (Amazon EBS) Encryption
Amazon S3 Bucket Policies
AMI WhiteBlacklisting
Amazon EC2 Role
Naming Conventions
New AWS Features Will Supplant
bull VPC Flows
bull Config
Project Cost Management in AWS
Harnessing the Power of Splunk
FINRA Information Security Engineering Copyright 2015 FINRA
Where We Were
Traditional Financial Review Cycles Too Long
bull Quarterly Reviews
AWS Detailed Billing Reports Are Daunting
bull Over 10 Million Line Items
Project Managers Need Focus
bull Am I Below My Budget
bull Where Are My Costs Going
bull Whorsquos Spending Them
Manual Compilation of Reports
bull Integrate FINRA Data
FINRA Information Security Engineering Copyright 2015 FINRA
Approach Chosen
Use Splunk as Process Delivery System
bull Ability to Collect Analyze Visualize
Collect AWS Billing Data in Splunk
bull Billing Data from S3 Bucket (Daily Load)
bull Detailed Line Items with Resources and Tags
Data Enrichment
bull Project Code Lookups
bull Forecast Projections
bull Billing Adjustments
Build Interfaces
bull FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
How We Did It
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
Impact ndash Reduced Costs
Focus on Low Hanging Fruit
bull Shutting Down Services over Weekends Evenings
bull Storage Sun Setting Dormant EC2
bull Identify AWS Services with Highest Spending
bull Projects Over Budget
Results
bull 135 Reduction in Billing Line Items in 1 Month
Better Forecast Projections
bull Feedback and Control
FINRA Information Security Engineering Copyright 2015 FINRA
Futures
Users Want Even Shorter Cycles
Back Tagging
ldquoFree Riderrdquo Services
Drill Down Analytics
Splunking in Amazon EMRGaining Transparency into PaaS
FINRA Information Security Engineering Copyright 2015 FINRA
What Stood in Our Way
PaaS and IaaS Are Not Equal
Instance Fingerprinting Identify Nodes
Instance Role
User Tags
Data Retention
Collection Delay
Bootstrap Splunk Agent
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Who We Are
FINRAmdashthe Financial Industry Regulatory Authoritymdashis an
independent nongovernmental regulator for all securities
firms doing business with the public in the United States
FINRA protects investors by regulating brokers and
brokerage firms and by monitoring trading on US stock
markets
FINRA monitor over 6 billion shares traded on the stock
market each day
FINRA handles more ldquobig datardquo on a daily basis than the
Library of Congress or Visaregmdashto build a holistic picture of
the trading market
FINRA Deter Detect Discipline
Investor Protection
FINRA Information Security Engineering Copyright 2015 FINRA
Historical View
Cyclic Processes
bull POC ndash Budget Approval ndash SDLC - Maintenance
Defined Roles
bull Coders Code
bull Managers Manage
bull Administrators Administer
Agile DevelopmentCloud Computing
bull Developers Make These Decisions
ndash Security
ndash Financial
ndash Architecture
bull And Itrsquos All Point and Click
Hacking Redefined Security
bull Defensive Coding
bull Baked In Not Painted On
ldquoYou guys start
coding Irsquoll go find out
what the users
wantrdquo
FINRA Information Security Engineering Copyright 2015 FINRA
Same Challenges Different Environment
Security
bull Engaged All Necessary AWS Security Features
bull Are We Firewalled Correctly
Compliance
bull Followed All Published Standards
Networking
bull Placed Servers on the Correct Network
Finance
bull Stayed Within Budget
Capacity Planning
bull Used Resources Optimally
But Now in a Decentralized Model
bull Itrsquos deacutejagrave vu all over againhellipYogi Berra
ldquoWith great power
comes great
responsibilityrdquo
FINRA Information Security Engineering Copyright 2015 FINRA
Security ndash AWS Identity and Access Management
AWS Shared Responsibility Model
bull AWS ndash ldquoSecurity of the Cloudrdquo
bull YOU ndash ldquoSecurity in the Cloudrdquo
AWS Identity and Access Management (IAM)
bull Critical Security-related ldquoServicerdquo
Best Practices
bull Lock Away Your AWS Account (Root) Access Keys
ndash Console Access
bull Rotate Credentials Regularly
ndash Time vs Event Driven
bull Remove Unnecessary Credentials
ndash Unused
bull Grant Least Privilege
bull Keep a History of Activity in Your AWS Account
FINRA Information Security Engineering Copyright 2015 FINRA
Lock Away Your AWS Account (Root) Access Keys
index=aws-cloudtrail sourcetype=aws-cloudtrail requestParametersuserName=root eventName=CreateAccessKey
index=aws-cloudtrail userIdentityarn=root eventName=Console | table _time eventName responseElementsConsoleLogin awsRegion sourceIPAddress eventSource userIdentityarn userIdentitytype userIdentityaccountId userAgent
IAMAWS CloudTrail
FINRA Information Security Engineering Copyright 2015 FINRA
IAMAWS CloudTrail
Credentials (Rotation Unnecessary)
Python (boto)
usrbinpython
import boto
from botoiamconnection import IAMConnection
iam = IAMConnection( aws_access_key_id = xxxxx
aws_secret_access_key = xxxxxxxx
proxy = xxxxx
proxy_port = xxxx
)
users = iamget_all_users()list_users_responselist_users_resultusers
for user in users
if password_last_used in user
print (ststs) ( user[create_date] user[user_name]
user[password_last_used] )
else
print (ststs) ( user[create_date] user[user_name] NEVER)
FINRA Information Security Engineering Copyright 2015 FINRA
Grant Least Privilege
Python (boto)
Version ControlledRepositorymiddot Security
middot Operationsmiddot Development
ROLE ABC
autoscalingDeletePolicyDeny
autoscalingDeleteScheduledActionDeny
autoscalingDeleteTagsDeny
autoscalingDescribeAdjustmentTypesAllow
autoscalingDescribeAutoScalingGroupsDontCare
autoscalingDescribeAutoScalingInstancesAllow
IAMAWS CloudTrail
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA Information Security Engineering Copyright 2015 FINRA
Beyond IAM Best Practices
Tagging Compliance
Security Group
Logging
Amazon S3Amazon Elastic Block Store (Amazon EBS) Encryption
Amazon S3 Bucket Policies
AMI WhiteBlacklisting
Amazon EC2 Role
Naming Conventions
New AWS Features Will Supplant
bull VPC Flows
bull Config
Project Cost Management in AWS
Harnessing the Power of Splunk
FINRA Information Security Engineering Copyright 2015 FINRA
Where We Were
Traditional Financial Review Cycles Too Long
bull Quarterly Reviews
AWS Detailed Billing Reports Are Daunting
bull Over 10 Million Line Items
Project Managers Need Focus
bull Am I Below My Budget
bull Where Are My Costs Going
bull Whorsquos Spending Them
Manual Compilation of Reports
bull Integrate FINRA Data
FINRA Information Security Engineering Copyright 2015 FINRA
Approach Chosen
Use Splunk as Process Delivery System
bull Ability to Collect Analyze Visualize
Collect AWS Billing Data in Splunk
bull Billing Data from S3 Bucket (Daily Load)
bull Detailed Line Items with Resources and Tags
Data Enrichment
bull Project Code Lookups
bull Forecast Projections
bull Billing Adjustments
Build Interfaces
bull FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
How We Did It
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
Impact ndash Reduced Costs
Focus on Low Hanging Fruit
bull Shutting Down Services over Weekends Evenings
bull Storage Sun Setting Dormant EC2
bull Identify AWS Services with Highest Spending
bull Projects Over Budget
Results
bull 135 Reduction in Billing Line Items in 1 Month
Better Forecast Projections
bull Feedback and Control
FINRA Information Security Engineering Copyright 2015 FINRA
Futures
Users Want Even Shorter Cycles
Back Tagging
ldquoFree Riderrdquo Services
Drill Down Analytics
Splunking in Amazon EMRGaining Transparency into PaaS
FINRA Information Security Engineering Copyright 2015 FINRA
What Stood in Our Way
PaaS and IaaS Are Not Equal
Instance Fingerprinting Identify Nodes
Instance Role
User Tags
Data Retention
Collection Delay
Bootstrap Splunk Agent
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Historical View
Cyclic Processes
bull POC ndash Budget Approval ndash SDLC - Maintenance
Defined Roles
bull Coders Code
bull Managers Manage
bull Administrators Administer
Agile DevelopmentCloud Computing
bull Developers Make These Decisions
ndash Security
ndash Financial
ndash Architecture
bull And Itrsquos All Point and Click
Hacking Redefined Security
bull Defensive Coding
bull Baked In Not Painted On
ldquoYou guys start
coding Irsquoll go find out
what the users
wantrdquo
FINRA Information Security Engineering Copyright 2015 FINRA
Same Challenges Different Environment
Security
bull Engaged All Necessary AWS Security Features
bull Are We Firewalled Correctly
Compliance
bull Followed All Published Standards
Networking
bull Placed Servers on the Correct Network
Finance
bull Stayed Within Budget
Capacity Planning
bull Used Resources Optimally
But Now in a Decentralized Model
bull Itrsquos deacutejagrave vu all over againhellipYogi Berra
ldquoWith great power
comes great
responsibilityrdquo
FINRA Information Security Engineering Copyright 2015 FINRA
Security ndash AWS Identity and Access Management
AWS Shared Responsibility Model
bull AWS ndash ldquoSecurity of the Cloudrdquo
bull YOU ndash ldquoSecurity in the Cloudrdquo
AWS Identity and Access Management (IAM)
bull Critical Security-related ldquoServicerdquo
Best Practices
bull Lock Away Your AWS Account (Root) Access Keys
ndash Console Access
bull Rotate Credentials Regularly
ndash Time vs Event Driven
bull Remove Unnecessary Credentials
ndash Unused
bull Grant Least Privilege
bull Keep a History of Activity in Your AWS Account
FINRA Information Security Engineering Copyright 2015 FINRA
Lock Away Your AWS Account (Root) Access Keys
index=aws-cloudtrail sourcetype=aws-cloudtrail requestParametersuserName=root eventName=CreateAccessKey
index=aws-cloudtrail userIdentityarn=root eventName=Console | table _time eventName responseElementsConsoleLogin awsRegion sourceIPAddress eventSource userIdentityarn userIdentitytype userIdentityaccountId userAgent
IAMAWS CloudTrail
FINRA Information Security Engineering Copyright 2015 FINRA
IAMAWS CloudTrail
Credentials (Rotation Unnecessary)
Python (boto)
usrbinpython
import boto
from botoiamconnection import IAMConnection
iam = IAMConnection( aws_access_key_id = xxxxx
aws_secret_access_key = xxxxxxxx
proxy = xxxxx
proxy_port = xxxx
)
users = iamget_all_users()list_users_responselist_users_resultusers
for user in users
if password_last_used in user
print (ststs) ( user[create_date] user[user_name]
user[password_last_used] )
else
print (ststs) ( user[create_date] user[user_name] NEVER)
FINRA Information Security Engineering Copyright 2015 FINRA
Grant Least Privilege
Python (boto)
Version ControlledRepositorymiddot Security
middot Operationsmiddot Development
ROLE ABC
autoscalingDeletePolicyDeny
autoscalingDeleteScheduledActionDeny
autoscalingDeleteTagsDeny
autoscalingDescribeAdjustmentTypesAllow
autoscalingDescribeAutoScalingGroupsDontCare
autoscalingDescribeAutoScalingInstancesAllow
IAMAWS CloudTrail
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA Information Security Engineering Copyright 2015 FINRA
Beyond IAM Best Practices
Tagging Compliance
Security Group
Logging
Amazon S3Amazon Elastic Block Store (Amazon EBS) Encryption
Amazon S3 Bucket Policies
AMI WhiteBlacklisting
Amazon EC2 Role
Naming Conventions
New AWS Features Will Supplant
bull VPC Flows
bull Config
Project Cost Management in AWS
Harnessing the Power of Splunk
FINRA Information Security Engineering Copyright 2015 FINRA
Where We Were
Traditional Financial Review Cycles Too Long
bull Quarterly Reviews
AWS Detailed Billing Reports Are Daunting
bull Over 10 Million Line Items
Project Managers Need Focus
bull Am I Below My Budget
bull Where Are My Costs Going
bull Whorsquos Spending Them
Manual Compilation of Reports
bull Integrate FINRA Data
FINRA Information Security Engineering Copyright 2015 FINRA
Approach Chosen
Use Splunk as Process Delivery System
bull Ability to Collect Analyze Visualize
Collect AWS Billing Data in Splunk
bull Billing Data from S3 Bucket (Daily Load)
bull Detailed Line Items with Resources and Tags
Data Enrichment
bull Project Code Lookups
bull Forecast Projections
bull Billing Adjustments
Build Interfaces
bull FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
How We Did It
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
Impact ndash Reduced Costs
Focus on Low Hanging Fruit
bull Shutting Down Services over Weekends Evenings
bull Storage Sun Setting Dormant EC2
bull Identify AWS Services with Highest Spending
bull Projects Over Budget
Results
bull 135 Reduction in Billing Line Items in 1 Month
Better Forecast Projections
bull Feedback and Control
FINRA Information Security Engineering Copyright 2015 FINRA
Futures
Users Want Even Shorter Cycles
Back Tagging
ldquoFree Riderrdquo Services
Drill Down Analytics
Splunking in Amazon EMRGaining Transparency into PaaS
FINRA Information Security Engineering Copyright 2015 FINRA
What Stood in Our Way
PaaS and IaaS Are Not Equal
Instance Fingerprinting Identify Nodes
Instance Role
User Tags
Data Retention
Collection Delay
Bootstrap Splunk Agent
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Same Challenges Different Environment
Security
bull Engaged All Necessary AWS Security Features
bull Are We Firewalled Correctly
Compliance
bull Followed All Published Standards
Networking
bull Placed Servers on the Correct Network
Finance
bull Stayed Within Budget
Capacity Planning
bull Used Resources Optimally
But Now in a Decentralized Model
bull Itrsquos deacutejagrave vu all over againhellipYogi Berra
ldquoWith great power
comes great
responsibilityrdquo
FINRA Information Security Engineering Copyright 2015 FINRA
Security ndash AWS Identity and Access Management
AWS Shared Responsibility Model
bull AWS ndash ldquoSecurity of the Cloudrdquo
bull YOU ndash ldquoSecurity in the Cloudrdquo
AWS Identity and Access Management (IAM)
bull Critical Security-related ldquoServicerdquo
Best Practices
bull Lock Away Your AWS Account (Root) Access Keys
ndash Console Access
bull Rotate Credentials Regularly
ndash Time vs Event Driven
bull Remove Unnecessary Credentials
ndash Unused
bull Grant Least Privilege
bull Keep a History of Activity in Your AWS Account
FINRA Information Security Engineering Copyright 2015 FINRA
Lock Away Your AWS Account (Root) Access Keys
index=aws-cloudtrail sourcetype=aws-cloudtrail requestParametersuserName=root eventName=CreateAccessKey
index=aws-cloudtrail userIdentityarn=root eventName=Console | table _time eventName responseElementsConsoleLogin awsRegion sourceIPAddress eventSource userIdentityarn userIdentitytype userIdentityaccountId userAgent
IAMAWS CloudTrail
FINRA Information Security Engineering Copyright 2015 FINRA
IAMAWS CloudTrail
Credentials (Rotation Unnecessary)
Python (boto)
usrbinpython
import boto
from botoiamconnection import IAMConnection
iam = IAMConnection( aws_access_key_id = xxxxx
aws_secret_access_key = xxxxxxxx
proxy = xxxxx
proxy_port = xxxx
)
users = iamget_all_users()list_users_responselist_users_resultusers
for user in users
if password_last_used in user
print (ststs) ( user[create_date] user[user_name]
user[password_last_used] )
else
print (ststs) ( user[create_date] user[user_name] NEVER)
FINRA Information Security Engineering Copyright 2015 FINRA
Grant Least Privilege
Python (boto)
Version ControlledRepositorymiddot Security
middot Operationsmiddot Development
ROLE ABC
autoscalingDeletePolicyDeny
autoscalingDeleteScheduledActionDeny
autoscalingDeleteTagsDeny
autoscalingDescribeAdjustmentTypesAllow
autoscalingDescribeAutoScalingGroupsDontCare
autoscalingDescribeAutoScalingInstancesAllow
IAMAWS CloudTrail
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA Information Security Engineering Copyright 2015 FINRA
Beyond IAM Best Practices
Tagging Compliance
Security Group
Logging
Amazon S3Amazon Elastic Block Store (Amazon EBS) Encryption
Amazon S3 Bucket Policies
AMI WhiteBlacklisting
Amazon EC2 Role
Naming Conventions
New AWS Features Will Supplant
bull VPC Flows
bull Config
Project Cost Management in AWS
Harnessing the Power of Splunk
FINRA Information Security Engineering Copyright 2015 FINRA
Where We Were
Traditional Financial Review Cycles Too Long
bull Quarterly Reviews
AWS Detailed Billing Reports Are Daunting
bull Over 10 Million Line Items
Project Managers Need Focus
bull Am I Below My Budget
bull Where Are My Costs Going
bull Whorsquos Spending Them
Manual Compilation of Reports
bull Integrate FINRA Data
FINRA Information Security Engineering Copyright 2015 FINRA
Approach Chosen
Use Splunk as Process Delivery System
bull Ability to Collect Analyze Visualize
Collect AWS Billing Data in Splunk
bull Billing Data from S3 Bucket (Daily Load)
bull Detailed Line Items with Resources and Tags
Data Enrichment
bull Project Code Lookups
bull Forecast Projections
bull Billing Adjustments
Build Interfaces
bull FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
How We Did It
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
Impact ndash Reduced Costs
Focus on Low Hanging Fruit
bull Shutting Down Services over Weekends Evenings
bull Storage Sun Setting Dormant EC2
bull Identify AWS Services with Highest Spending
bull Projects Over Budget
Results
bull 135 Reduction in Billing Line Items in 1 Month
Better Forecast Projections
bull Feedback and Control
FINRA Information Security Engineering Copyright 2015 FINRA
Futures
Users Want Even Shorter Cycles
Back Tagging
ldquoFree Riderrdquo Services
Drill Down Analytics
Splunking in Amazon EMRGaining Transparency into PaaS
FINRA Information Security Engineering Copyright 2015 FINRA
What Stood in Our Way
PaaS and IaaS Are Not Equal
Instance Fingerprinting Identify Nodes
Instance Role
User Tags
Data Retention
Collection Delay
Bootstrap Splunk Agent
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Security ndash AWS Identity and Access Management
AWS Shared Responsibility Model
bull AWS ndash ldquoSecurity of the Cloudrdquo
bull YOU ndash ldquoSecurity in the Cloudrdquo
AWS Identity and Access Management (IAM)
bull Critical Security-related ldquoServicerdquo
Best Practices
bull Lock Away Your AWS Account (Root) Access Keys
ndash Console Access
bull Rotate Credentials Regularly
ndash Time vs Event Driven
bull Remove Unnecessary Credentials
ndash Unused
bull Grant Least Privilege
bull Keep a History of Activity in Your AWS Account
FINRA Information Security Engineering Copyright 2015 FINRA
Lock Away Your AWS Account (Root) Access Keys
index=aws-cloudtrail sourcetype=aws-cloudtrail requestParametersuserName=root eventName=CreateAccessKey
index=aws-cloudtrail userIdentityarn=root eventName=Console | table _time eventName responseElementsConsoleLogin awsRegion sourceIPAddress eventSource userIdentityarn userIdentitytype userIdentityaccountId userAgent
IAMAWS CloudTrail
FINRA Information Security Engineering Copyright 2015 FINRA
IAMAWS CloudTrail
Credentials (Rotation Unnecessary)
Python (boto)
usrbinpython
import boto
from botoiamconnection import IAMConnection
iam = IAMConnection( aws_access_key_id = xxxxx
aws_secret_access_key = xxxxxxxx
proxy = xxxxx
proxy_port = xxxx
)
users = iamget_all_users()list_users_responselist_users_resultusers
for user in users
if password_last_used in user
print (ststs) ( user[create_date] user[user_name]
user[password_last_used] )
else
print (ststs) ( user[create_date] user[user_name] NEVER)
FINRA Information Security Engineering Copyright 2015 FINRA
Grant Least Privilege
Python (boto)
Version ControlledRepositorymiddot Security
middot Operationsmiddot Development
ROLE ABC
autoscalingDeletePolicyDeny
autoscalingDeleteScheduledActionDeny
autoscalingDeleteTagsDeny
autoscalingDescribeAdjustmentTypesAllow
autoscalingDescribeAutoScalingGroupsDontCare
autoscalingDescribeAutoScalingInstancesAllow
IAMAWS CloudTrail
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA Information Security Engineering Copyright 2015 FINRA
Beyond IAM Best Practices
Tagging Compliance
Security Group
Logging
Amazon S3Amazon Elastic Block Store (Amazon EBS) Encryption
Amazon S3 Bucket Policies
AMI WhiteBlacklisting
Amazon EC2 Role
Naming Conventions
New AWS Features Will Supplant
bull VPC Flows
bull Config
Project Cost Management in AWS
Harnessing the Power of Splunk
FINRA Information Security Engineering Copyright 2015 FINRA
Where We Were
Traditional Financial Review Cycles Too Long
bull Quarterly Reviews
AWS Detailed Billing Reports Are Daunting
bull Over 10 Million Line Items
Project Managers Need Focus
bull Am I Below My Budget
bull Where Are My Costs Going
bull Whorsquos Spending Them
Manual Compilation of Reports
bull Integrate FINRA Data
FINRA Information Security Engineering Copyright 2015 FINRA
Approach Chosen
Use Splunk as Process Delivery System
bull Ability to Collect Analyze Visualize
Collect AWS Billing Data in Splunk
bull Billing Data from S3 Bucket (Daily Load)
bull Detailed Line Items with Resources and Tags
Data Enrichment
bull Project Code Lookups
bull Forecast Projections
bull Billing Adjustments
Build Interfaces
bull FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
How We Did It
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
Impact ndash Reduced Costs
Focus on Low Hanging Fruit
bull Shutting Down Services over Weekends Evenings
bull Storage Sun Setting Dormant EC2
bull Identify AWS Services with Highest Spending
bull Projects Over Budget
Results
bull 135 Reduction in Billing Line Items in 1 Month
Better Forecast Projections
bull Feedback and Control
FINRA Information Security Engineering Copyright 2015 FINRA
Futures
Users Want Even Shorter Cycles
Back Tagging
ldquoFree Riderrdquo Services
Drill Down Analytics
Splunking in Amazon EMRGaining Transparency into PaaS
FINRA Information Security Engineering Copyright 2015 FINRA
What Stood in Our Way
PaaS and IaaS Are Not Equal
Instance Fingerprinting Identify Nodes
Instance Role
User Tags
Data Retention
Collection Delay
Bootstrap Splunk Agent
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Lock Away Your AWS Account (Root) Access Keys
index=aws-cloudtrail sourcetype=aws-cloudtrail requestParametersuserName=root eventName=CreateAccessKey
index=aws-cloudtrail userIdentityarn=root eventName=Console | table _time eventName responseElementsConsoleLogin awsRegion sourceIPAddress eventSource userIdentityarn userIdentitytype userIdentityaccountId userAgent
IAMAWS CloudTrail
FINRA Information Security Engineering Copyright 2015 FINRA
IAMAWS CloudTrail
Credentials (Rotation Unnecessary)
Python (boto)
usrbinpython
import boto
from botoiamconnection import IAMConnection
iam = IAMConnection( aws_access_key_id = xxxxx
aws_secret_access_key = xxxxxxxx
proxy = xxxxx
proxy_port = xxxx
)
users = iamget_all_users()list_users_responselist_users_resultusers
for user in users
if password_last_used in user
print (ststs) ( user[create_date] user[user_name]
user[password_last_used] )
else
print (ststs) ( user[create_date] user[user_name] NEVER)
FINRA Information Security Engineering Copyright 2015 FINRA
Grant Least Privilege
Python (boto)
Version ControlledRepositorymiddot Security
middot Operationsmiddot Development
ROLE ABC
autoscalingDeletePolicyDeny
autoscalingDeleteScheduledActionDeny
autoscalingDeleteTagsDeny
autoscalingDescribeAdjustmentTypesAllow
autoscalingDescribeAutoScalingGroupsDontCare
autoscalingDescribeAutoScalingInstancesAllow
IAMAWS CloudTrail
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA Information Security Engineering Copyright 2015 FINRA
Beyond IAM Best Practices
Tagging Compliance
Security Group
Logging
Amazon S3Amazon Elastic Block Store (Amazon EBS) Encryption
Amazon S3 Bucket Policies
AMI WhiteBlacklisting
Amazon EC2 Role
Naming Conventions
New AWS Features Will Supplant
bull VPC Flows
bull Config
Project Cost Management in AWS
Harnessing the Power of Splunk
FINRA Information Security Engineering Copyright 2015 FINRA
Where We Were
Traditional Financial Review Cycles Too Long
bull Quarterly Reviews
AWS Detailed Billing Reports Are Daunting
bull Over 10 Million Line Items
Project Managers Need Focus
bull Am I Below My Budget
bull Where Are My Costs Going
bull Whorsquos Spending Them
Manual Compilation of Reports
bull Integrate FINRA Data
FINRA Information Security Engineering Copyright 2015 FINRA
Approach Chosen
Use Splunk as Process Delivery System
bull Ability to Collect Analyze Visualize
Collect AWS Billing Data in Splunk
bull Billing Data from S3 Bucket (Daily Load)
bull Detailed Line Items with Resources and Tags
Data Enrichment
bull Project Code Lookups
bull Forecast Projections
bull Billing Adjustments
Build Interfaces
bull FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
How We Did It
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
Impact ndash Reduced Costs
Focus on Low Hanging Fruit
bull Shutting Down Services over Weekends Evenings
bull Storage Sun Setting Dormant EC2
bull Identify AWS Services with Highest Spending
bull Projects Over Budget
Results
bull 135 Reduction in Billing Line Items in 1 Month
Better Forecast Projections
bull Feedback and Control
FINRA Information Security Engineering Copyright 2015 FINRA
Futures
Users Want Even Shorter Cycles
Back Tagging
ldquoFree Riderrdquo Services
Drill Down Analytics
Splunking in Amazon EMRGaining Transparency into PaaS
FINRA Information Security Engineering Copyright 2015 FINRA
What Stood in Our Way
PaaS and IaaS Are Not Equal
Instance Fingerprinting Identify Nodes
Instance Role
User Tags
Data Retention
Collection Delay
Bootstrap Splunk Agent
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
IAMAWS CloudTrail
Credentials (Rotation Unnecessary)
Python (boto)
usrbinpython
import boto
from botoiamconnection import IAMConnection
iam = IAMConnection( aws_access_key_id = xxxxx
aws_secret_access_key = xxxxxxxx
proxy = xxxxx
proxy_port = xxxx
)
users = iamget_all_users()list_users_responselist_users_resultusers
for user in users
if password_last_used in user
print (ststs) ( user[create_date] user[user_name]
user[password_last_used] )
else
print (ststs) ( user[create_date] user[user_name] NEVER)
FINRA Information Security Engineering Copyright 2015 FINRA
Grant Least Privilege
Python (boto)
Version ControlledRepositorymiddot Security
middot Operationsmiddot Development
ROLE ABC
autoscalingDeletePolicyDeny
autoscalingDeleteScheduledActionDeny
autoscalingDeleteTagsDeny
autoscalingDescribeAdjustmentTypesAllow
autoscalingDescribeAutoScalingGroupsDontCare
autoscalingDescribeAutoScalingInstancesAllow
IAMAWS CloudTrail
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA Information Security Engineering Copyright 2015 FINRA
Beyond IAM Best Practices
Tagging Compliance
Security Group
Logging
Amazon S3Amazon Elastic Block Store (Amazon EBS) Encryption
Amazon S3 Bucket Policies
AMI WhiteBlacklisting
Amazon EC2 Role
Naming Conventions
New AWS Features Will Supplant
bull VPC Flows
bull Config
Project Cost Management in AWS
Harnessing the Power of Splunk
FINRA Information Security Engineering Copyright 2015 FINRA
Where We Were
Traditional Financial Review Cycles Too Long
bull Quarterly Reviews
AWS Detailed Billing Reports Are Daunting
bull Over 10 Million Line Items
Project Managers Need Focus
bull Am I Below My Budget
bull Where Are My Costs Going
bull Whorsquos Spending Them
Manual Compilation of Reports
bull Integrate FINRA Data
FINRA Information Security Engineering Copyright 2015 FINRA
Approach Chosen
Use Splunk as Process Delivery System
bull Ability to Collect Analyze Visualize
Collect AWS Billing Data in Splunk
bull Billing Data from S3 Bucket (Daily Load)
bull Detailed Line Items with Resources and Tags
Data Enrichment
bull Project Code Lookups
bull Forecast Projections
bull Billing Adjustments
Build Interfaces
bull FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
How We Did It
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
Impact ndash Reduced Costs
Focus on Low Hanging Fruit
bull Shutting Down Services over Weekends Evenings
bull Storage Sun Setting Dormant EC2
bull Identify AWS Services with Highest Spending
bull Projects Over Budget
Results
bull 135 Reduction in Billing Line Items in 1 Month
Better Forecast Projections
bull Feedback and Control
FINRA Information Security Engineering Copyright 2015 FINRA
Futures
Users Want Even Shorter Cycles
Back Tagging
ldquoFree Riderrdquo Services
Drill Down Analytics
Splunking in Amazon EMRGaining Transparency into PaaS
FINRA Information Security Engineering Copyright 2015 FINRA
What Stood in Our Way
PaaS and IaaS Are Not Equal
Instance Fingerprinting Identify Nodes
Instance Role
User Tags
Data Retention
Collection Delay
Bootstrap Splunk Agent
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Grant Least Privilege
Python (boto)
Version ControlledRepositorymiddot Security
middot Operationsmiddot Development
ROLE ABC
autoscalingDeletePolicyDeny
autoscalingDeleteScheduledActionDeny
autoscalingDeleteTagsDeny
autoscalingDescribeAdjustmentTypesAllow
autoscalingDescribeAutoScalingGroupsDontCare
autoscalingDescribeAutoScalingInstancesAllow
IAMAWS CloudTrail
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA Information Security Engineering Copyright 2015 FINRA
Beyond IAM Best Practices
Tagging Compliance
Security Group
Logging
Amazon S3Amazon Elastic Block Store (Amazon EBS) Encryption
Amazon S3 Bucket Policies
AMI WhiteBlacklisting
Amazon EC2 Role
Naming Conventions
New AWS Features Will Supplant
bull VPC Flows
bull Config
Project Cost Management in AWS
Harnessing the Power of Splunk
FINRA Information Security Engineering Copyright 2015 FINRA
Where We Were
Traditional Financial Review Cycles Too Long
bull Quarterly Reviews
AWS Detailed Billing Reports Are Daunting
bull Over 10 Million Line Items
Project Managers Need Focus
bull Am I Below My Budget
bull Where Are My Costs Going
bull Whorsquos Spending Them
Manual Compilation of Reports
bull Integrate FINRA Data
FINRA Information Security Engineering Copyright 2015 FINRA
Approach Chosen
Use Splunk as Process Delivery System
bull Ability to Collect Analyze Visualize
Collect AWS Billing Data in Splunk
bull Billing Data from S3 Bucket (Daily Load)
bull Detailed Line Items with Resources and Tags
Data Enrichment
bull Project Code Lookups
bull Forecast Projections
bull Billing Adjustments
Build Interfaces
bull FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
How We Did It
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
Impact ndash Reduced Costs
Focus on Low Hanging Fruit
bull Shutting Down Services over Weekends Evenings
bull Storage Sun Setting Dormant EC2
bull Identify AWS Services with Highest Spending
bull Projects Over Budget
Results
bull 135 Reduction in Billing Line Items in 1 Month
Better Forecast Projections
bull Feedback and Control
FINRA Information Security Engineering Copyright 2015 FINRA
Futures
Users Want Even Shorter Cycles
Back Tagging
ldquoFree Riderrdquo Services
Drill Down Analytics
Splunking in Amazon EMRGaining Transparency into PaaS
FINRA Information Security Engineering Copyright 2015 FINRA
What Stood in Our Way
PaaS and IaaS Are Not Equal
Instance Fingerprinting Identify Nodes
Instance Role
User Tags
Data Retention
Collection Delay
Bootstrap Splunk Agent
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA Information Security Engineering Copyright 2015 FINRA
Beyond IAM Best Practices
Tagging Compliance
Security Group
Logging
Amazon S3Amazon Elastic Block Store (Amazon EBS) Encryption
Amazon S3 Bucket Policies
AMI WhiteBlacklisting
Amazon EC2 Role
Naming Conventions
New AWS Features Will Supplant
bull VPC Flows
bull Config
Project Cost Management in AWS
Harnessing the Power of Splunk
FINRA Information Security Engineering Copyright 2015 FINRA
Where We Were
Traditional Financial Review Cycles Too Long
bull Quarterly Reviews
AWS Detailed Billing Reports Are Daunting
bull Over 10 Million Line Items
Project Managers Need Focus
bull Am I Below My Budget
bull Where Are My Costs Going
bull Whorsquos Spending Them
Manual Compilation of Reports
bull Integrate FINRA Data
FINRA Information Security Engineering Copyright 2015 FINRA
Approach Chosen
Use Splunk as Process Delivery System
bull Ability to Collect Analyze Visualize
Collect AWS Billing Data in Splunk
bull Billing Data from S3 Bucket (Daily Load)
bull Detailed Line Items with Resources and Tags
Data Enrichment
bull Project Code Lookups
bull Forecast Projections
bull Billing Adjustments
Build Interfaces
bull FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
How We Did It
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
Impact ndash Reduced Costs
Focus on Low Hanging Fruit
bull Shutting Down Services over Weekends Evenings
bull Storage Sun Setting Dormant EC2
bull Identify AWS Services with Highest Spending
bull Projects Over Budget
Results
bull 135 Reduction in Billing Line Items in 1 Month
Better Forecast Projections
bull Feedback and Control
FINRA Information Security Engineering Copyright 2015 FINRA
Futures
Users Want Even Shorter Cycles
Back Tagging
ldquoFree Riderrdquo Services
Drill Down Analytics
Splunking in Amazon EMRGaining Transparency into PaaS
FINRA Information Security Engineering Copyright 2015 FINRA
What Stood in Our Way
PaaS and IaaS Are Not Equal
Instance Fingerprinting Identify Nodes
Instance Role
User Tags
Data Retention
Collection Delay
Bootstrap Splunk Agent
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Beyond IAM Best Practices
Tagging Compliance
Security Group
Logging
Amazon S3Amazon Elastic Block Store (Amazon EBS) Encryption
Amazon S3 Bucket Policies
AMI WhiteBlacklisting
Amazon EC2 Role
Naming Conventions
New AWS Features Will Supplant
bull VPC Flows
bull Config
Project Cost Management in AWS
Harnessing the Power of Splunk
FINRA Information Security Engineering Copyright 2015 FINRA
Where We Were
Traditional Financial Review Cycles Too Long
bull Quarterly Reviews
AWS Detailed Billing Reports Are Daunting
bull Over 10 Million Line Items
Project Managers Need Focus
bull Am I Below My Budget
bull Where Are My Costs Going
bull Whorsquos Spending Them
Manual Compilation of Reports
bull Integrate FINRA Data
FINRA Information Security Engineering Copyright 2015 FINRA
Approach Chosen
Use Splunk as Process Delivery System
bull Ability to Collect Analyze Visualize
Collect AWS Billing Data in Splunk
bull Billing Data from S3 Bucket (Daily Load)
bull Detailed Line Items with Resources and Tags
Data Enrichment
bull Project Code Lookups
bull Forecast Projections
bull Billing Adjustments
Build Interfaces
bull FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
How We Did It
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
Impact ndash Reduced Costs
Focus on Low Hanging Fruit
bull Shutting Down Services over Weekends Evenings
bull Storage Sun Setting Dormant EC2
bull Identify AWS Services with Highest Spending
bull Projects Over Budget
Results
bull 135 Reduction in Billing Line Items in 1 Month
Better Forecast Projections
bull Feedback and Control
FINRA Information Security Engineering Copyright 2015 FINRA
Futures
Users Want Even Shorter Cycles
Back Tagging
ldquoFree Riderrdquo Services
Drill Down Analytics
Splunking in Amazon EMRGaining Transparency into PaaS
FINRA Information Security Engineering Copyright 2015 FINRA
What Stood in Our Way
PaaS and IaaS Are Not Equal
Instance Fingerprinting Identify Nodes
Instance Role
User Tags
Data Retention
Collection Delay
Bootstrap Splunk Agent
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
Project Cost Management in AWS
Harnessing the Power of Splunk
FINRA Information Security Engineering Copyright 2015 FINRA
Where We Were
Traditional Financial Review Cycles Too Long
bull Quarterly Reviews
AWS Detailed Billing Reports Are Daunting
bull Over 10 Million Line Items
Project Managers Need Focus
bull Am I Below My Budget
bull Where Are My Costs Going
bull Whorsquos Spending Them
Manual Compilation of Reports
bull Integrate FINRA Data
FINRA Information Security Engineering Copyright 2015 FINRA
Approach Chosen
Use Splunk as Process Delivery System
bull Ability to Collect Analyze Visualize
Collect AWS Billing Data in Splunk
bull Billing Data from S3 Bucket (Daily Load)
bull Detailed Line Items with Resources and Tags
Data Enrichment
bull Project Code Lookups
bull Forecast Projections
bull Billing Adjustments
Build Interfaces
bull FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
How We Did It
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
Impact ndash Reduced Costs
Focus on Low Hanging Fruit
bull Shutting Down Services over Weekends Evenings
bull Storage Sun Setting Dormant EC2
bull Identify AWS Services with Highest Spending
bull Projects Over Budget
Results
bull 135 Reduction in Billing Line Items in 1 Month
Better Forecast Projections
bull Feedback and Control
FINRA Information Security Engineering Copyright 2015 FINRA
Futures
Users Want Even Shorter Cycles
Back Tagging
ldquoFree Riderrdquo Services
Drill Down Analytics
Splunking in Amazon EMRGaining Transparency into PaaS
FINRA Information Security Engineering Copyright 2015 FINRA
What Stood in Our Way
PaaS and IaaS Are Not Equal
Instance Fingerprinting Identify Nodes
Instance Role
User Tags
Data Retention
Collection Delay
Bootstrap Splunk Agent
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Where We Were
Traditional Financial Review Cycles Too Long
bull Quarterly Reviews
AWS Detailed Billing Reports Are Daunting
bull Over 10 Million Line Items
Project Managers Need Focus
bull Am I Below My Budget
bull Where Are My Costs Going
bull Whorsquos Spending Them
Manual Compilation of Reports
bull Integrate FINRA Data
FINRA Information Security Engineering Copyright 2015 FINRA
Approach Chosen
Use Splunk as Process Delivery System
bull Ability to Collect Analyze Visualize
Collect AWS Billing Data in Splunk
bull Billing Data from S3 Bucket (Daily Load)
bull Detailed Line Items with Resources and Tags
Data Enrichment
bull Project Code Lookups
bull Forecast Projections
bull Billing Adjustments
Build Interfaces
bull FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
How We Did It
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
Impact ndash Reduced Costs
Focus on Low Hanging Fruit
bull Shutting Down Services over Weekends Evenings
bull Storage Sun Setting Dormant EC2
bull Identify AWS Services with Highest Spending
bull Projects Over Budget
Results
bull 135 Reduction in Billing Line Items in 1 Month
Better Forecast Projections
bull Feedback and Control
FINRA Information Security Engineering Copyright 2015 FINRA
Futures
Users Want Even Shorter Cycles
Back Tagging
ldquoFree Riderrdquo Services
Drill Down Analytics
Splunking in Amazon EMRGaining Transparency into PaaS
FINRA Information Security Engineering Copyright 2015 FINRA
What Stood in Our Way
PaaS and IaaS Are Not Equal
Instance Fingerprinting Identify Nodes
Instance Role
User Tags
Data Retention
Collection Delay
Bootstrap Splunk Agent
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Approach Chosen
Use Splunk as Process Delivery System
bull Ability to Collect Analyze Visualize
Collect AWS Billing Data in Splunk
bull Billing Data from S3 Bucket (Daily Load)
bull Detailed Line Items with Resources and Tags
Data Enrichment
bull Project Code Lookups
bull Forecast Projections
bull Billing Adjustments
Build Interfaces
bull FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
How We Did It
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
Impact ndash Reduced Costs
Focus on Low Hanging Fruit
bull Shutting Down Services over Weekends Evenings
bull Storage Sun Setting Dormant EC2
bull Identify AWS Services with Highest Spending
bull Projects Over Budget
Results
bull 135 Reduction in Billing Line Items in 1 Month
Better Forecast Projections
bull Feedback and Control
FINRA Information Security Engineering Copyright 2015 FINRA
Futures
Users Want Even Shorter Cycles
Back Tagging
ldquoFree Riderrdquo Services
Drill Down Analytics
Splunking in Amazon EMRGaining Transparency into PaaS
FINRA Information Security Engineering Copyright 2015 FINRA
What Stood in Our Way
PaaS and IaaS Are Not Equal
Instance Fingerprinting Identify Nodes
Instance Role
User Tags
Data Retention
Collection Delay
Bootstrap Splunk Agent
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
How We Did It
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
Impact ndash Reduced Costs
Focus on Low Hanging Fruit
bull Shutting Down Services over Weekends Evenings
bull Storage Sun Setting Dormant EC2
bull Identify AWS Services with Highest Spending
bull Projects Over Budget
Results
bull 135 Reduction in Billing Line Items in 1 Month
Better Forecast Projections
bull Feedback and Control
FINRA Information Security Engineering Copyright 2015 FINRA
Futures
Users Want Even Shorter Cycles
Back Tagging
ldquoFree Riderrdquo Services
Drill Down Analytics
Splunking in Amazon EMRGaining Transparency into PaaS
FINRA Information Security Engineering Copyright 2015 FINRA
What Stood in Our Way
PaaS and IaaS Are Not Equal
Instance Fingerprinting Identify Nodes
Instance Role
User Tags
Data Retention
Collection Delay
Bootstrap Splunk Agent
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
FINRA AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
Impact ndash Reduced Costs
Focus on Low Hanging Fruit
bull Shutting Down Services over Weekends Evenings
bull Storage Sun Setting Dormant EC2
bull Identify AWS Services with Highest Spending
bull Projects Over Budget
Results
bull 135 Reduction in Billing Line Items in 1 Month
Better Forecast Projections
bull Feedback and Control
FINRA Information Security Engineering Copyright 2015 FINRA
Futures
Users Want Even Shorter Cycles
Back Tagging
ldquoFree Riderrdquo Services
Drill Down Analytics
Splunking in Amazon EMRGaining Transparency into PaaS
FINRA Information Security Engineering Copyright 2015 FINRA
What Stood in Our Way
PaaS and IaaS Are Not Equal
Instance Fingerprinting Identify Nodes
Instance Role
User Tags
Data Retention
Collection Delay
Bootstrap Splunk Agent
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
Impact ndash Reduced Costs
Focus on Low Hanging Fruit
bull Shutting Down Services over Weekends Evenings
bull Storage Sun Setting Dormant EC2
bull Identify AWS Services with Highest Spending
bull Projects Over Budget
Results
bull 135 Reduction in Billing Line Items in 1 Month
Better Forecast Projections
bull Feedback and Control
FINRA Information Security Engineering Copyright 2015 FINRA
Futures
Users Want Even Shorter Cycles
Back Tagging
ldquoFree Riderrdquo Services
Drill Down Analytics
Splunking in Amazon EMRGaining Transparency into PaaS
FINRA Information Security Engineering Copyright 2015 FINRA
What Stood in Our Way
PaaS and IaaS Are Not Equal
Instance Fingerprinting Identify Nodes
Instance Role
User Tags
Data Retention
Collection Delay
Bootstrap Splunk Agent
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
AWS Billing App
FINRA Information Security Engineering Copyright 2015 FINRA
Impact ndash Reduced Costs
Focus on Low Hanging Fruit
bull Shutting Down Services over Weekends Evenings
bull Storage Sun Setting Dormant EC2
bull Identify AWS Services with Highest Spending
bull Projects Over Budget
Results
bull 135 Reduction in Billing Line Items in 1 Month
Better Forecast Projections
bull Feedback and Control
FINRA Information Security Engineering Copyright 2015 FINRA
Futures
Users Want Even Shorter Cycles
Back Tagging
ldquoFree Riderrdquo Services
Drill Down Analytics
Splunking in Amazon EMRGaining Transparency into PaaS
FINRA Information Security Engineering Copyright 2015 FINRA
What Stood in Our Way
PaaS and IaaS Are Not Equal
Instance Fingerprinting Identify Nodes
Instance Role
User Tags
Data Retention
Collection Delay
Bootstrap Splunk Agent
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Impact ndash Reduced Costs
Focus on Low Hanging Fruit
bull Shutting Down Services over Weekends Evenings
bull Storage Sun Setting Dormant EC2
bull Identify AWS Services with Highest Spending
bull Projects Over Budget
Results
bull 135 Reduction in Billing Line Items in 1 Month
Better Forecast Projections
bull Feedback and Control
FINRA Information Security Engineering Copyright 2015 FINRA
Futures
Users Want Even Shorter Cycles
Back Tagging
ldquoFree Riderrdquo Services
Drill Down Analytics
Splunking in Amazon EMRGaining Transparency into PaaS
FINRA Information Security Engineering Copyright 2015 FINRA
What Stood in Our Way
PaaS and IaaS Are Not Equal
Instance Fingerprinting Identify Nodes
Instance Role
User Tags
Data Retention
Collection Delay
Bootstrap Splunk Agent
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Futures
Users Want Even Shorter Cycles
Back Tagging
ldquoFree Riderrdquo Services
Drill Down Analytics
Splunking in Amazon EMRGaining Transparency into PaaS
FINRA Information Security Engineering Copyright 2015 FINRA
What Stood in Our Way
PaaS and IaaS Are Not Equal
Instance Fingerprinting Identify Nodes
Instance Role
User Tags
Data Retention
Collection Delay
Bootstrap Splunk Agent
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
Splunking in Amazon EMRGaining Transparency into PaaS
FINRA Information Security Engineering Copyright 2015 FINRA
What Stood in Our Way
PaaS and IaaS Are Not Equal
Instance Fingerprinting Identify Nodes
Instance Role
User Tags
Data Retention
Collection Delay
Bootstrap Splunk Agent
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
What Stood in Our Way
PaaS and IaaS Are Not Equal
Instance Fingerprinting Identify Nodes
Instance Role
User Tags
Data Retention
Collection Delay
Bootstrap Splunk Agent
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Bootstrap Splunk Agent
Store bsx in S3
bull Splunk rpm
bull Deploymentclientconf
bull Discovery Scripts
Execute Bootstrap
Master Installation
Core and Task Installation
AWS was Extremely Supportive
of This Method
Amazon EMR
CORES
MASTER
TASK
Availability Zone
Amazon S3
BOOTSTR
AP
CLUSTER
forwarder
forwarder
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
1 Determine the Identity of the Node
oEC2 nodes have ldquometadatardquo service http169254169254
o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`
o echo $INSTANCE_ID i-8f0d4c75
o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`
2 Grab the User Defined Tags
o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep
$INSTANCE_ID|grep AGS|cut -f5`
o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep
SDLC|cut -f5|awk print substr($011)`
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Instance Fingerprinting
3 Get the jobFlowID (EMR Cluster ID) for the Node
o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)
mntvarlibinfojob-flowjson|tr -d [ ]`
4 Determine Role Node in the EMR Cluster
o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require
json require pp igs=JSON [STDINread][instanceGroups]
igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs
ig[instanceRole]|awk print toupper($0)`
5 Update Splunk Config Files
o Deploymentclientconf
o Inputsconf
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Inputsconf Example
inputsconf
[default]
host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111
_meta=jobflow-idj-YQQU43YEHG4X instancegroup-
idig-3JBIDJGYVSD4N instance-idi-22f7f3f0
instance-roleMASTER ags_hostnameEMRAPP1-
MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD
cost_centerCLDABC creatorldquoAPP1_Team
nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC
purposeldquoAPP1_QUERY_CL sdlcDEV
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
EMR Cluster Analyzer ndashgt Summary Dashboard
AWS Billing
Splunk for NIX
Select the
Cluster
Jobs Running
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Potential EC2 Provisioning Cost Savings
m1xlarge
bull 4 vCPUs with 8 ECUs
bull 15 GB memory
bull 4 x 420 GB disk
bull $035 per hour
c3xlarge
bull 4 vCPUs with 14 ECUs
bull 75 GB memory
bull 2 x 40 GB disk (SSD)
bull $027 per hour
Potential 23
Savings in EMR
Costs
With additional ECUs and SSD disks the c3xlarge may be more
performant than the m1xlarge instances at a better price pointRESIZING
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis
Summary provides
the information on number of nodes
1 Less than 80 utilization overall
2 After 70 mins utilization at less than 50 (cpu)
Conclusion
The cluster be resized after the two hours
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 1
1st Hour 2nd Hour 3rd Hour 4th Hour
40 40 40 40
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = 035 40 4 = $56
All instances running for full duration of the job
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Resizing Analysis ndash Scenario 2
1st Hour 2nd Hour 3rd Hour 4th Hour
40 35 20 20
Cost Analysis
Cost = (Price per Instance) (No of Instances) (No of Hours)
Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025
Savings Compared to Scenario 1 = 282
Resizing after the 1st hour and 2nd hour
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Combined EC2 Provisioning and Resizing Analysis
Combined Savings
bull Original Cost = $56
bull Price After Combined Analysis = $3105
bull Job Savings = $2495 = 5544
Job Runs 5x Day ( $2495 5 = $12475 )
Every Business Day Week ( $12475 5 = $62375 )
Every Week of the Year ( $62375 52 = $32435 )
AndhellipWe Havenrsquot Affected Performance
bull Just More Efficient Provisioning
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Job History
jhist File Analysisbull Cluster Level Statistics
bull Run Time Map Reduce Stats per Steps
Cluster Summary
Per Cluster
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
FINRA Information Security Engineering Copyright 2015 FINRA
Futures Other Uses
Grade Clusters
bull Identify Underutilized Clusters for Faster Resizing
ITSI Integration
bull KPIs based Auto Analysis on Cloud
Additional Input Variables
bull Size of Data Sets
bull Number of Runs
Metrics Correlation Analyze Steps
Jobs Impact on System
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
Copyright copy 2015 Splunk Inc
Splunk Overview
Praveen Rangnath
Sr Dir of Cloud Product Marketing
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
35
Turning Machine Data Into Business Value
Index Untapped Data Any Source Type Volume
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
On-Premises
Private Cloud
Public Cloud
End-to-End Visibility
Application Delivery
Security Compliance and Fraud
IT Operations
Business Analytics
Industrial Data andthe Internet of Things
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
36
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
Splunk Runs on AWS
37
Bring Your Own License
Self-Managed Software on AWS
Analytics for EMR and S3
Available Hourly from EMR Console
Software
100 Uptime SLA
SOC2 Type II Certified
SaaS
For Small IT Teams
Starts at $90 Month
App for AWS
Apps and Integrations
Integrations
AWS CloudTrail AWS Config Billing S3
Amazon VPC Flow Logs Amazon CloudWatch
Amazon Kinesis Amazon DynamoDB S3
AWS Lambda
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
Why is Splunk Important to You
38
ldquoYou canrsquot protect what
you canrsquot seerdquo
Best Practices for Securing
Workloads in Amazon Web Services
Gartner April 2015
Neil MacDonald Greg Young
ldquoSecurity monitoring will
make or break a
technology risk
management programrdquo
ldquoSecurity requires
visibilityrdquo
Assessing the Risk Yes the Cloud
Can Be More Secure Than Your On-
Premises Environment
IDC July 2015
Pete Lindstrom
Amazon Web Services
ldquoIntro to AWS Securityrdquo
2015 AWS Summit Series
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
39
Sample CloudTrail Dashboard
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
40
Sample Amazon VPC Flow Dashboard
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
41
Sample Topology Dashboard
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
42
Collecting AWS Data in Splunk Is Easy
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
NEWAWS IoT Integration
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS
44
Splunk App for AWSEC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State End-to-End AWS Visibility
Your Next Step To AWS Visibility
45
App for AWS
Get Started for Free
Integrations
Watch Splunk in AWS
reInvent 2014 Keynote
httpswwwyoutubecomwatchv=vfRS1LUHgJM
Visit Us at
Booth 400
You Bet Your
Sweet SaaS