(ism311) how finra gains visibility into its aws environment

45
FINRA Information Security Engineering Copyright 2015 FINRA © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Gary Mikula, Sr. Director of Information Security Siddhartha Dadana, Senior Security Engineer ISM311 How FINRA Gains End-to-End Visibility into a Large-Scale AWS Environment Gaining Transparency into Cloud Computing October 2015

Upload: amazon-web-services

Post on 16-Apr-2017

1.594 views

Category:

Technology


1 download

TRANSCRIPT

FINRA Information Security Engineering Copyright 2015 FINRAcopy 2015 Amazon Web Services Inc or its Affiliates All rights reserved

Gary Mikula Sr Director of Information Security

Siddhartha Dadana Senior Security Engineer

ISM311

How FINRA Gains End-to-End Visibility

into a Large-Scale AWS Environment

Gaining Transparency into Cloud Computing

October 2015

FINRA Information Security Engineering Copyright 2015 FINRA

Who We Are

FINRAmdashthe Financial Industry Regulatory Authoritymdashis an

independent nongovernmental regulator for all securities

firms doing business with the public in the United States

FINRA protects investors by regulating brokers and

brokerage firms and by monitoring trading on US stock

markets

FINRA monitor over 6 billion shares traded on the stock

market each day

FINRA handles more ldquobig datardquo on a daily basis than the

Library of Congress or Visaregmdashto build a holistic picture of

the trading market

FINRA Deter Detect Discipline

Investor Protection

FINRA Information Security Engineering Copyright 2015 FINRA

Historical View

Cyclic Processes

bull POC ndash Budget Approval ndash SDLC - Maintenance

Defined Roles

bull Coders Code

bull Managers Manage

bull Administrators Administer

Agile DevelopmentCloud Computing

bull Developers Make These Decisions

ndash Security

ndash Financial

ndash Architecture

bull And Itrsquos All Point and Click

Hacking Redefined Security

bull Defensive Coding

bull Baked In Not Painted On

ldquoYou guys start

coding Irsquoll go find out

what the users

wantrdquo

FINRA Information Security Engineering Copyright 2015 FINRA

Same Challenges Different Environment

Security

bull Engaged All Necessary AWS Security Features

bull Are We Firewalled Correctly

Compliance

bull Followed All Published Standards

Networking

bull Placed Servers on the Correct Network

Finance

bull Stayed Within Budget

Capacity Planning

bull Used Resources Optimally

But Now in a Decentralized Model

bull Itrsquos deacutejagrave vu all over againhellipYogi Berra

ldquoWith great power

comes great

responsibilityrdquo

FINRA Information Security Engineering Copyright 2015 FINRA

Security ndash AWS Identity and Access Management

AWS Shared Responsibility Model

bull AWS ndash ldquoSecurity of the Cloudrdquo

bull YOU ndash ldquoSecurity in the Cloudrdquo

AWS Identity and Access Management (IAM)

bull Critical Security-related ldquoServicerdquo

Best Practices

bull Lock Away Your AWS Account (Root) Access Keys

ndash Console Access

bull Rotate Credentials Regularly

ndash Time vs Event Driven

bull Remove Unnecessary Credentials

ndash Unused

bull Grant Least Privilege

bull Keep a History of Activity in Your AWS Account

FINRA Information Security Engineering Copyright 2015 FINRA

Lock Away Your AWS Account (Root) Access Keys

index=aws-cloudtrail sourcetype=aws-cloudtrail requestParametersuserName=root eventName=CreateAccessKey

index=aws-cloudtrail userIdentityarn=root eventName=Console | table _time eventName responseElementsConsoleLogin awsRegion sourceIPAddress eventSource userIdentityarn userIdentitytype userIdentityaccountId userAgent

IAMAWS CloudTrail

FINRA Information Security Engineering Copyright 2015 FINRA

IAMAWS CloudTrail

Credentials (Rotation Unnecessary)

Python (boto)

usrbinpython

import boto

from botoiamconnection import IAMConnection

iam = IAMConnection( aws_access_key_id = xxxxx

aws_secret_access_key = xxxxxxxx

proxy = xxxxx

proxy_port = xxxx

)

users = iamget_all_users()list_users_responselist_users_resultusers

for user in users

if password_last_used in user

print (ststs) ( user[create_date] user[user_name]

user[password_last_used] )

else

print (ststs) ( user[create_date] user[user_name] NEVER)

FINRA Information Security Engineering Copyright 2015 FINRA

Grant Least Privilege

Python (boto)

Version ControlledRepositorymiddot Security

middot Operationsmiddot Development

ROLE ABC

autoscalingDeletePolicyDeny

autoscalingDeleteScheduledActionDeny

autoscalingDeleteTagsDeny

autoscalingDescribeAdjustmentTypesAllow

autoscalingDescribeAutoScalingGroupsDontCare

autoscalingDescribeAutoScalingInstancesAllow

IAMAWS CloudTrail

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA Information Security Engineering Copyright 2015 FINRA

Beyond IAM Best Practices

Tagging Compliance

Security Group

Logging

Amazon S3Amazon Elastic Block Store (Amazon EBS) Encryption

Amazon S3 Bucket Policies

AMI WhiteBlacklisting

Amazon EC2 Role

Naming Conventions

New AWS Features Will Supplant

bull VPC Flows

bull Config

Project Cost Management in AWS

Harnessing the Power of Splunk

FINRA Information Security Engineering Copyright 2015 FINRA

Where We Were

Traditional Financial Review Cycles Too Long

bull Quarterly Reviews

AWS Detailed Billing Reports Are Daunting

bull Over 10 Million Line Items

Project Managers Need Focus

bull Am I Below My Budget

bull Where Are My Costs Going

bull Whorsquos Spending Them

Manual Compilation of Reports

bull Integrate FINRA Data

FINRA Information Security Engineering Copyright 2015 FINRA

Approach Chosen

Use Splunk as Process Delivery System

bull Ability to Collect Analyze Visualize

Collect AWS Billing Data in Splunk

bull Billing Data from S3 Bucket (Daily Load)

bull Detailed Line Items with Resources and Tags

Data Enrichment

bull Project Code Lookups

bull Forecast Projections

bull Billing Adjustments

Build Interfaces

bull FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

How We Did It

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

Impact ndash Reduced Costs

Focus on Low Hanging Fruit

bull Shutting Down Services over Weekends Evenings

bull Storage Sun Setting Dormant EC2

bull Identify AWS Services with Highest Spending

bull Projects Over Budget

Results

bull 135 Reduction in Billing Line Items in 1 Month

Better Forecast Projections

bull Feedback and Control

FINRA Information Security Engineering Copyright 2015 FINRA

Futures

Users Want Even Shorter Cycles

Back Tagging

ldquoFree Riderrdquo Services

Drill Down Analytics

Splunking in Amazon EMRGaining Transparency into PaaS

FINRA Information Security Engineering Copyright 2015 FINRA

What Stood in Our Way

PaaS and IaaS Are Not Equal

Instance Fingerprinting Identify Nodes

Instance Role

User Tags

Data Retention

Collection Delay

Bootstrap Splunk Agent

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Who We Are

FINRAmdashthe Financial Industry Regulatory Authoritymdashis an

independent nongovernmental regulator for all securities

firms doing business with the public in the United States

FINRA protects investors by regulating brokers and

brokerage firms and by monitoring trading on US stock

markets

FINRA monitor over 6 billion shares traded on the stock

market each day

FINRA handles more ldquobig datardquo on a daily basis than the

Library of Congress or Visaregmdashto build a holistic picture of

the trading market

FINRA Deter Detect Discipline

Investor Protection

FINRA Information Security Engineering Copyright 2015 FINRA

Historical View

Cyclic Processes

bull POC ndash Budget Approval ndash SDLC - Maintenance

Defined Roles

bull Coders Code

bull Managers Manage

bull Administrators Administer

Agile DevelopmentCloud Computing

bull Developers Make These Decisions

ndash Security

ndash Financial

ndash Architecture

bull And Itrsquos All Point and Click

Hacking Redefined Security

bull Defensive Coding

bull Baked In Not Painted On

ldquoYou guys start

coding Irsquoll go find out

what the users

wantrdquo

FINRA Information Security Engineering Copyright 2015 FINRA

Same Challenges Different Environment

Security

bull Engaged All Necessary AWS Security Features

bull Are We Firewalled Correctly

Compliance

bull Followed All Published Standards

Networking

bull Placed Servers on the Correct Network

Finance

bull Stayed Within Budget

Capacity Planning

bull Used Resources Optimally

But Now in a Decentralized Model

bull Itrsquos deacutejagrave vu all over againhellipYogi Berra

ldquoWith great power

comes great

responsibilityrdquo

FINRA Information Security Engineering Copyright 2015 FINRA

Security ndash AWS Identity and Access Management

AWS Shared Responsibility Model

bull AWS ndash ldquoSecurity of the Cloudrdquo

bull YOU ndash ldquoSecurity in the Cloudrdquo

AWS Identity and Access Management (IAM)

bull Critical Security-related ldquoServicerdquo

Best Practices

bull Lock Away Your AWS Account (Root) Access Keys

ndash Console Access

bull Rotate Credentials Regularly

ndash Time vs Event Driven

bull Remove Unnecessary Credentials

ndash Unused

bull Grant Least Privilege

bull Keep a History of Activity in Your AWS Account

FINRA Information Security Engineering Copyright 2015 FINRA

Lock Away Your AWS Account (Root) Access Keys

index=aws-cloudtrail sourcetype=aws-cloudtrail requestParametersuserName=root eventName=CreateAccessKey

index=aws-cloudtrail userIdentityarn=root eventName=Console | table _time eventName responseElementsConsoleLogin awsRegion sourceIPAddress eventSource userIdentityarn userIdentitytype userIdentityaccountId userAgent

IAMAWS CloudTrail

FINRA Information Security Engineering Copyright 2015 FINRA

IAMAWS CloudTrail

Credentials (Rotation Unnecessary)

Python (boto)

usrbinpython

import boto

from botoiamconnection import IAMConnection

iam = IAMConnection( aws_access_key_id = xxxxx

aws_secret_access_key = xxxxxxxx

proxy = xxxxx

proxy_port = xxxx

)

users = iamget_all_users()list_users_responselist_users_resultusers

for user in users

if password_last_used in user

print (ststs) ( user[create_date] user[user_name]

user[password_last_used] )

else

print (ststs) ( user[create_date] user[user_name] NEVER)

FINRA Information Security Engineering Copyright 2015 FINRA

Grant Least Privilege

Python (boto)

Version ControlledRepositorymiddot Security

middot Operationsmiddot Development

ROLE ABC

autoscalingDeletePolicyDeny

autoscalingDeleteScheduledActionDeny

autoscalingDeleteTagsDeny

autoscalingDescribeAdjustmentTypesAllow

autoscalingDescribeAutoScalingGroupsDontCare

autoscalingDescribeAutoScalingInstancesAllow

IAMAWS CloudTrail

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA Information Security Engineering Copyright 2015 FINRA

Beyond IAM Best Practices

Tagging Compliance

Security Group

Logging

Amazon S3Amazon Elastic Block Store (Amazon EBS) Encryption

Amazon S3 Bucket Policies

AMI WhiteBlacklisting

Amazon EC2 Role

Naming Conventions

New AWS Features Will Supplant

bull VPC Flows

bull Config

Project Cost Management in AWS

Harnessing the Power of Splunk

FINRA Information Security Engineering Copyright 2015 FINRA

Where We Were

Traditional Financial Review Cycles Too Long

bull Quarterly Reviews

AWS Detailed Billing Reports Are Daunting

bull Over 10 Million Line Items

Project Managers Need Focus

bull Am I Below My Budget

bull Where Are My Costs Going

bull Whorsquos Spending Them

Manual Compilation of Reports

bull Integrate FINRA Data

FINRA Information Security Engineering Copyright 2015 FINRA

Approach Chosen

Use Splunk as Process Delivery System

bull Ability to Collect Analyze Visualize

Collect AWS Billing Data in Splunk

bull Billing Data from S3 Bucket (Daily Load)

bull Detailed Line Items with Resources and Tags

Data Enrichment

bull Project Code Lookups

bull Forecast Projections

bull Billing Adjustments

Build Interfaces

bull FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

How We Did It

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

Impact ndash Reduced Costs

Focus on Low Hanging Fruit

bull Shutting Down Services over Weekends Evenings

bull Storage Sun Setting Dormant EC2

bull Identify AWS Services with Highest Spending

bull Projects Over Budget

Results

bull 135 Reduction in Billing Line Items in 1 Month

Better Forecast Projections

bull Feedback and Control

FINRA Information Security Engineering Copyright 2015 FINRA

Futures

Users Want Even Shorter Cycles

Back Tagging

ldquoFree Riderrdquo Services

Drill Down Analytics

Splunking in Amazon EMRGaining Transparency into PaaS

FINRA Information Security Engineering Copyright 2015 FINRA

What Stood in Our Way

PaaS and IaaS Are Not Equal

Instance Fingerprinting Identify Nodes

Instance Role

User Tags

Data Retention

Collection Delay

Bootstrap Splunk Agent

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Historical View

Cyclic Processes

bull POC ndash Budget Approval ndash SDLC - Maintenance

Defined Roles

bull Coders Code

bull Managers Manage

bull Administrators Administer

Agile DevelopmentCloud Computing

bull Developers Make These Decisions

ndash Security

ndash Financial

ndash Architecture

bull And Itrsquos All Point and Click

Hacking Redefined Security

bull Defensive Coding

bull Baked In Not Painted On

ldquoYou guys start

coding Irsquoll go find out

what the users

wantrdquo

FINRA Information Security Engineering Copyright 2015 FINRA

Same Challenges Different Environment

Security

bull Engaged All Necessary AWS Security Features

bull Are We Firewalled Correctly

Compliance

bull Followed All Published Standards

Networking

bull Placed Servers on the Correct Network

Finance

bull Stayed Within Budget

Capacity Planning

bull Used Resources Optimally

But Now in a Decentralized Model

bull Itrsquos deacutejagrave vu all over againhellipYogi Berra

ldquoWith great power

comes great

responsibilityrdquo

FINRA Information Security Engineering Copyright 2015 FINRA

Security ndash AWS Identity and Access Management

AWS Shared Responsibility Model

bull AWS ndash ldquoSecurity of the Cloudrdquo

bull YOU ndash ldquoSecurity in the Cloudrdquo

AWS Identity and Access Management (IAM)

bull Critical Security-related ldquoServicerdquo

Best Practices

bull Lock Away Your AWS Account (Root) Access Keys

ndash Console Access

bull Rotate Credentials Regularly

ndash Time vs Event Driven

bull Remove Unnecessary Credentials

ndash Unused

bull Grant Least Privilege

bull Keep a History of Activity in Your AWS Account

FINRA Information Security Engineering Copyright 2015 FINRA

Lock Away Your AWS Account (Root) Access Keys

index=aws-cloudtrail sourcetype=aws-cloudtrail requestParametersuserName=root eventName=CreateAccessKey

index=aws-cloudtrail userIdentityarn=root eventName=Console | table _time eventName responseElementsConsoleLogin awsRegion sourceIPAddress eventSource userIdentityarn userIdentitytype userIdentityaccountId userAgent

IAMAWS CloudTrail

FINRA Information Security Engineering Copyright 2015 FINRA

IAMAWS CloudTrail

Credentials (Rotation Unnecessary)

Python (boto)

usrbinpython

import boto

from botoiamconnection import IAMConnection

iam = IAMConnection( aws_access_key_id = xxxxx

aws_secret_access_key = xxxxxxxx

proxy = xxxxx

proxy_port = xxxx

)

users = iamget_all_users()list_users_responselist_users_resultusers

for user in users

if password_last_used in user

print (ststs) ( user[create_date] user[user_name]

user[password_last_used] )

else

print (ststs) ( user[create_date] user[user_name] NEVER)

FINRA Information Security Engineering Copyright 2015 FINRA

Grant Least Privilege

Python (boto)

Version ControlledRepositorymiddot Security

middot Operationsmiddot Development

ROLE ABC

autoscalingDeletePolicyDeny

autoscalingDeleteScheduledActionDeny

autoscalingDeleteTagsDeny

autoscalingDescribeAdjustmentTypesAllow

autoscalingDescribeAutoScalingGroupsDontCare

autoscalingDescribeAutoScalingInstancesAllow

IAMAWS CloudTrail

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA Information Security Engineering Copyright 2015 FINRA

Beyond IAM Best Practices

Tagging Compliance

Security Group

Logging

Amazon S3Amazon Elastic Block Store (Amazon EBS) Encryption

Amazon S3 Bucket Policies

AMI WhiteBlacklisting

Amazon EC2 Role

Naming Conventions

New AWS Features Will Supplant

bull VPC Flows

bull Config

Project Cost Management in AWS

Harnessing the Power of Splunk

FINRA Information Security Engineering Copyright 2015 FINRA

Where We Were

Traditional Financial Review Cycles Too Long

bull Quarterly Reviews

AWS Detailed Billing Reports Are Daunting

bull Over 10 Million Line Items

Project Managers Need Focus

bull Am I Below My Budget

bull Where Are My Costs Going

bull Whorsquos Spending Them

Manual Compilation of Reports

bull Integrate FINRA Data

FINRA Information Security Engineering Copyright 2015 FINRA

Approach Chosen

Use Splunk as Process Delivery System

bull Ability to Collect Analyze Visualize

Collect AWS Billing Data in Splunk

bull Billing Data from S3 Bucket (Daily Load)

bull Detailed Line Items with Resources and Tags

Data Enrichment

bull Project Code Lookups

bull Forecast Projections

bull Billing Adjustments

Build Interfaces

bull FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

How We Did It

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

Impact ndash Reduced Costs

Focus on Low Hanging Fruit

bull Shutting Down Services over Weekends Evenings

bull Storage Sun Setting Dormant EC2

bull Identify AWS Services with Highest Spending

bull Projects Over Budget

Results

bull 135 Reduction in Billing Line Items in 1 Month

Better Forecast Projections

bull Feedback and Control

FINRA Information Security Engineering Copyright 2015 FINRA

Futures

Users Want Even Shorter Cycles

Back Tagging

ldquoFree Riderrdquo Services

Drill Down Analytics

Splunking in Amazon EMRGaining Transparency into PaaS

FINRA Information Security Engineering Copyright 2015 FINRA

What Stood in Our Way

PaaS and IaaS Are Not Equal

Instance Fingerprinting Identify Nodes

Instance Role

User Tags

Data Retention

Collection Delay

Bootstrap Splunk Agent

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Same Challenges Different Environment

Security

bull Engaged All Necessary AWS Security Features

bull Are We Firewalled Correctly

Compliance

bull Followed All Published Standards

Networking

bull Placed Servers on the Correct Network

Finance

bull Stayed Within Budget

Capacity Planning

bull Used Resources Optimally

But Now in a Decentralized Model

bull Itrsquos deacutejagrave vu all over againhellipYogi Berra

ldquoWith great power

comes great

responsibilityrdquo

FINRA Information Security Engineering Copyright 2015 FINRA

Security ndash AWS Identity and Access Management

AWS Shared Responsibility Model

bull AWS ndash ldquoSecurity of the Cloudrdquo

bull YOU ndash ldquoSecurity in the Cloudrdquo

AWS Identity and Access Management (IAM)

bull Critical Security-related ldquoServicerdquo

Best Practices

bull Lock Away Your AWS Account (Root) Access Keys

ndash Console Access

bull Rotate Credentials Regularly

ndash Time vs Event Driven

bull Remove Unnecessary Credentials

ndash Unused

bull Grant Least Privilege

bull Keep a History of Activity in Your AWS Account

FINRA Information Security Engineering Copyright 2015 FINRA

Lock Away Your AWS Account (Root) Access Keys

index=aws-cloudtrail sourcetype=aws-cloudtrail requestParametersuserName=root eventName=CreateAccessKey

index=aws-cloudtrail userIdentityarn=root eventName=Console | table _time eventName responseElementsConsoleLogin awsRegion sourceIPAddress eventSource userIdentityarn userIdentitytype userIdentityaccountId userAgent

IAMAWS CloudTrail

FINRA Information Security Engineering Copyright 2015 FINRA

IAMAWS CloudTrail

Credentials (Rotation Unnecessary)

Python (boto)

usrbinpython

import boto

from botoiamconnection import IAMConnection

iam = IAMConnection( aws_access_key_id = xxxxx

aws_secret_access_key = xxxxxxxx

proxy = xxxxx

proxy_port = xxxx

)

users = iamget_all_users()list_users_responselist_users_resultusers

for user in users

if password_last_used in user

print (ststs) ( user[create_date] user[user_name]

user[password_last_used] )

else

print (ststs) ( user[create_date] user[user_name] NEVER)

FINRA Information Security Engineering Copyright 2015 FINRA

Grant Least Privilege

Python (boto)

Version ControlledRepositorymiddot Security

middot Operationsmiddot Development

ROLE ABC

autoscalingDeletePolicyDeny

autoscalingDeleteScheduledActionDeny

autoscalingDeleteTagsDeny

autoscalingDescribeAdjustmentTypesAllow

autoscalingDescribeAutoScalingGroupsDontCare

autoscalingDescribeAutoScalingInstancesAllow

IAMAWS CloudTrail

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA Information Security Engineering Copyright 2015 FINRA

Beyond IAM Best Practices

Tagging Compliance

Security Group

Logging

Amazon S3Amazon Elastic Block Store (Amazon EBS) Encryption

Amazon S3 Bucket Policies

AMI WhiteBlacklisting

Amazon EC2 Role

Naming Conventions

New AWS Features Will Supplant

bull VPC Flows

bull Config

Project Cost Management in AWS

Harnessing the Power of Splunk

FINRA Information Security Engineering Copyright 2015 FINRA

Where We Were

Traditional Financial Review Cycles Too Long

bull Quarterly Reviews

AWS Detailed Billing Reports Are Daunting

bull Over 10 Million Line Items

Project Managers Need Focus

bull Am I Below My Budget

bull Where Are My Costs Going

bull Whorsquos Spending Them

Manual Compilation of Reports

bull Integrate FINRA Data

FINRA Information Security Engineering Copyright 2015 FINRA

Approach Chosen

Use Splunk as Process Delivery System

bull Ability to Collect Analyze Visualize

Collect AWS Billing Data in Splunk

bull Billing Data from S3 Bucket (Daily Load)

bull Detailed Line Items with Resources and Tags

Data Enrichment

bull Project Code Lookups

bull Forecast Projections

bull Billing Adjustments

Build Interfaces

bull FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

How We Did It

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

Impact ndash Reduced Costs

Focus on Low Hanging Fruit

bull Shutting Down Services over Weekends Evenings

bull Storage Sun Setting Dormant EC2

bull Identify AWS Services with Highest Spending

bull Projects Over Budget

Results

bull 135 Reduction in Billing Line Items in 1 Month

Better Forecast Projections

bull Feedback and Control

FINRA Information Security Engineering Copyright 2015 FINRA

Futures

Users Want Even Shorter Cycles

Back Tagging

ldquoFree Riderrdquo Services

Drill Down Analytics

Splunking in Amazon EMRGaining Transparency into PaaS

FINRA Information Security Engineering Copyright 2015 FINRA

What Stood in Our Way

PaaS and IaaS Are Not Equal

Instance Fingerprinting Identify Nodes

Instance Role

User Tags

Data Retention

Collection Delay

Bootstrap Splunk Agent

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Security ndash AWS Identity and Access Management

AWS Shared Responsibility Model

bull AWS ndash ldquoSecurity of the Cloudrdquo

bull YOU ndash ldquoSecurity in the Cloudrdquo

AWS Identity and Access Management (IAM)

bull Critical Security-related ldquoServicerdquo

Best Practices

bull Lock Away Your AWS Account (Root) Access Keys

ndash Console Access

bull Rotate Credentials Regularly

ndash Time vs Event Driven

bull Remove Unnecessary Credentials

ndash Unused

bull Grant Least Privilege

bull Keep a History of Activity in Your AWS Account

FINRA Information Security Engineering Copyright 2015 FINRA

Lock Away Your AWS Account (Root) Access Keys

index=aws-cloudtrail sourcetype=aws-cloudtrail requestParametersuserName=root eventName=CreateAccessKey

index=aws-cloudtrail userIdentityarn=root eventName=Console | table _time eventName responseElementsConsoleLogin awsRegion sourceIPAddress eventSource userIdentityarn userIdentitytype userIdentityaccountId userAgent

IAMAWS CloudTrail

FINRA Information Security Engineering Copyright 2015 FINRA

IAMAWS CloudTrail

Credentials (Rotation Unnecessary)

Python (boto)

usrbinpython

import boto

from botoiamconnection import IAMConnection

iam = IAMConnection( aws_access_key_id = xxxxx

aws_secret_access_key = xxxxxxxx

proxy = xxxxx

proxy_port = xxxx

)

users = iamget_all_users()list_users_responselist_users_resultusers

for user in users

if password_last_used in user

print (ststs) ( user[create_date] user[user_name]

user[password_last_used] )

else

print (ststs) ( user[create_date] user[user_name] NEVER)

FINRA Information Security Engineering Copyright 2015 FINRA

Grant Least Privilege

Python (boto)

Version ControlledRepositorymiddot Security

middot Operationsmiddot Development

ROLE ABC

autoscalingDeletePolicyDeny

autoscalingDeleteScheduledActionDeny

autoscalingDeleteTagsDeny

autoscalingDescribeAdjustmentTypesAllow

autoscalingDescribeAutoScalingGroupsDontCare

autoscalingDescribeAutoScalingInstancesAllow

IAMAWS CloudTrail

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA Information Security Engineering Copyright 2015 FINRA

Beyond IAM Best Practices

Tagging Compliance

Security Group

Logging

Amazon S3Amazon Elastic Block Store (Amazon EBS) Encryption

Amazon S3 Bucket Policies

AMI WhiteBlacklisting

Amazon EC2 Role

Naming Conventions

New AWS Features Will Supplant

bull VPC Flows

bull Config

Project Cost Management in AWS

Harnessing the Power of Splunk

FINRA Information Security Engineering Copyright 2015 FINRA

Where We Were

Traditional Financial Review Cycles Too Long

bull Quarterly Reviews

AWS Detailed Billing Reports Are Daunting

bull Over 10 Million Line Items

Project Managers Need Focus

bull Am I Below My Budget

bull Where Are My Costs Going

bull Whorsquos Spending Them

Manual Compilation of Reports

bull Integrate FINRA Data

FINRA Information Security Engineering Copyright 2015 FINRA

Approach Chosen

Use Splunk as Process Delivery System

bull Ability to Collect Analyze Visualize

Collect AWS Billing Data in Splunk

bull Billing Data from S3 Bucket (Daily Load)

bull Detailed Line Items with Resources and Tags

Data Enrichment

bull Project Code Lookups

bull Forecast Projections

bull Billing Adjustments

Build Interfaces

bull FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

How We Did It

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

Impact ndash Reduced Costs

Focus on Low Hanging Fruit

bull Shutting Down Services over Weekends Evenings

bull Storage Sun Setting Dormant EC2

bull Identify AWS Services with Highest Spending

bull Projects Over Budget

Results

bull 135 Reduction in Billing Line Items in 1 Month

Better Forecast Projections

bull Feedback and Control

FINRA Information Security Engineering Copyright 2015 FINRA

Futures

Users Want Even Shorter Cycles

Back Tagging

ldquoFree Riderrdquo Services

Drill Down Analytics

Splunking in Amazon EMRGaining Transparency into PaaS

FINRA Information Security Engineering Copyright 2015 FINRA

What Stood in Our Way

PaaS and IaaS Are Not Equal

Instance Fingerprinting Identify Nodes

Instance Role

User Tags

Data Retention

Collection Delay

Bootstrap Splunk Agent

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Lock Away Your AWS Account (Root) Access Keys

index=aws-cloudtrail sourcetype=aws-cloudtrail requestParametersuserName=root eventName=CreateAccessKey

index=aws-cloudtrail userIdentityarn=root eventName=Console | table _time eventName responseElementsConsoleLogin awsRegion sourceIPAddress eventSource userIdentityarn userIdentitytype userIdentityaccountId userAgent

IAMAWS CloudTrail

FINRA Information Security Engineering Copyright 2015 FINRA

IAMAWS CloudTrail

Credentials (Rotation Unnecessary)

Python (boto)

usrbinpython

import boto

from botoiamconnection import IAMConnection

iam = IAMConnection( aws_access_key_id = xxxxx

aws_secret_access_key = xxxxxxxx

proxy = xxxxx

proxy_port = xxxx

)

users = iamget_all_users()list_users_responselist_users_resultusers

for user in users

if password_last_used in user

print (ststs) ( user[create_date] user[user_name]

user[password_last_used] )

else

print (ststs) ( user[create_date] user[user_name] NEVER)

FINRA Information Security Engineering Copyright 2015 FINRA

Grant Least Privilege

Python (boto)

Version ControlledRepositorymiddot Security

middot Operationsmiddot Development

ROLE ABC

autoscalingDeletePolicyDeny

autoscalingDeleteScheduledActionDeny

autoscalingDeleteTagsDeny

autoscalingDescribeAdjustmentTypesAllow

autoscalingDescribeAutoScalingGroupsDontCare

autoscalingDescribeAutoScalingInstancesAllow

IAMAWS CloudTrail

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA Information Security Engineering Copyright 2015 FINRA

Beyond IAM Best Practices

Tagging Compliance

Security Group

Logging

Amazon S3Amazon Elastic Block Store (Amazon EBS) Encryption

Amazon S3 Bucket Policies

AMI WhiteBlacklisting

Amazon EC2 Role

Naming Conventions

New AWS Features Will Supplant

bull VPC Flows

bull Config

Project Cost Management in AWS

Harnessing the Power of Splunk

FINRA Information Security Engineering Copyright 2015 FINRA

Where We Were

Traditional Financial Review Cycles Too Long

bull Quarterly Reviews

AWS Detailed Billing Reports Are Daunting

bull Over 10 Million Line Items

Project Managers Need Focus

bull Am I Below My Budget

bull Where Are My Costs Going

bull Whorsquos Spending Them

Manual Compilation of Reports

bull Integrate FINRA Data

FINRA Information Security Engineering Copyright 2015 FINRA

Approach Chosen

Use Splunk as Process Delivery System

bull Ability to Collect Analyze Visualize

Collect AWS Billing Data in Splunk

bull Billing Data from S3 Bucket (Daily Load)

bull Detailed Line Items with Resources and Tags

Data Enrichment

bull Project Code Lookups

bull Forecast Projections

bull Billing Adjustments

Build Interfaces

bull FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

How We Did It

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

Impact ndash Reduced Costs

Focus on Low Hanging Fruit

bull Shutting Down Services over Weekends Evenings

bull Storage Sun Setting Dormant EC2

bull Identify AWS Services with Highest Spending

bull Projects Over Budget

Results

bull 135 Reduction in Billing Line Items in 1 Month

Better Forecast Projections

bull Feedback and Control

FINRA Information Security Engineering Copyright 2015 FINRA

Futures

Users Want Even Shorter Cycles

Back Tagging

ldquoFree Riderrdquo Services

Drill Down Analytics

Splunking in Amazon EMRGaining Transparency into PaaS

FINRA Information Security Engineering Copyright 2015 FINRA

What Stood in Our Way

PaaS and IaaS Are Not Equal

Instance Fingerprinting Identify Nodes

Instance Role

User Tags

Data Retention

Collection Delay

Bootstrap Splunk Agent

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

IAMAWS CloudTrail

Credentials (Rotation Unnecessary)

Python (boto)

usrbinpython

import boto

from botoiamconnection import IAMConnection

iam = IAMConnection( aws_access_key_id = xxxxx

aws_secret_access_key = xxxxxxxx

proxy = xxxxx

proxy_port = xxxx

)

users = iamget_all_users()list_users_responselist_users_resultusers

for user in users

if password_last_used in user

print (ststs) ( user[create_date] user[user_name]

user[password_last_used] )

else

print (ststs) ( user[create_date] user[user_name] NEVER)

FINRA Information Security Engineering Copyright 2015 FINRA

Grant Least Privilege

Python (boto)

Version ControlledRepositorymiddot Security

middot Operationsmiddot Development

ROLE ABC

autoscalingDeletePolicyDeny

autoscalingDeleteScheduledActionDeny

autoscalingDeleteTagsDeny

autoscalingDescribeAdjustmentTypesAllow

autoscalingDescribeAutoScalingGroupsDontCare

autoscalingDescribeAutoScalingInstancesAllow

IAMAWS CloudTrail

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA Information Security Engineering Copyright 2015 FINRA

Beyond IAM Best Practices

Tagging Compliance

Security Group

Logging

Amazon S3Amazon Elastic Block Store (Amazon EBS) Encryption

Amazon S3 Bucket Policies

AMI WhiteBlacklisting

Amazon EC2 Role

Naming Conventions

New AWS Features Will Supplant

bull VPC Flows

bull Config

Project Cost Management in AWS

Harnessing the Power of Splunk

FINRA Information Security Engineering Copyright 2015 FINRA

Where We Were

Traditional Financial Review Cycles Too Long

bull Quarterly Reviews

AWS Detailed Billing Reports Are Daunting

bull Over 10 Million Line Items

Project Managers Need Focus

bull Am I Below My Budget

bull Where Are My Costs Going

bull Whorsquos Spending Them

Manual Compilation of Reports

bull Integrate FINRA Data

FINRA Information Security Engineering Copyright 2015 FINRA

Approach Chosen

Use Splunk as Process Delivery System

bull Ability to Collect Analyze Visualize

Collect AWS Billing Data in Splunk

bull Billing Data from S3 Bucket (Daily Load)

bull Detailed Line Items with Resources and Tags

Data Enrichment

bull Project Code Lookups

bull Forecast Projections

bull Billing Adjustments

Build Interfaces

bull FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

How We Did It

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

Impact ndash Reduced Costs

Focus on Low Hanging Fruit

bull Shutting Down Services over Weekends Evenings

bull Storage Sun Setting Dormant EC2

bull Identify AWS Services with Highest Spending

bull Projects Over Budget

Results

bull 135 Reduction in Billing Line Items in 1 Month

Better Forecast Projections

bull Feedback and Control

FINRA Information Security Engineering Copyright 2015 FINRA

Futures

Users Want Even Shorter Cycles

Back Tagging

ldquoFree Riderrdquo Services

Drill Down Analytics

Splunking in Amazon EMRGaining Transparency into PaaS

FINRA Information Security Engineering Copyright 2015 FINRA

What Stood in Our Way

PaaS and IaaS Are Not Equal

Instance Fingerprinting Identify Nodes

Instance Role

User Tags

Data Retention

Collection Delay

Bootstrap Splunk Agent

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Grant Least Privilege

Python (boto)

Version ControlledRepositorymiddot Security

middot Operationsmiddot Development

ROLE ABC

autoscalingDeletePolicyDeny

autoscalingDeleteScheduledActionDeny

autoscalingDeleteTagsDeny

autoscalingDescribeAdjustmentTypesAllow

autoscalingDescribeAutoScalingGroupsDontCare

autoscalingDescribeAutoScalingInstancesAllow

IAMAWS CloudTrail

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA Information Security Engineering Copyright 2015 FINRA

Beyond IAM Best Practices

Tagging Compliance

Security Group

Logging

Amazon S3Amazon Elastic Block Store (Amazon EBS) Encryption

Amazon S3 Bucket Policies

AMI WhiteBlacklisting

Amazon EC2 Role

Naming Conventions

New AWS Features Will Supplant

bull VPC Flows

bull Config

Project Cost Management in AWS

Harnessing the Power of Splunk

FINRA Information Security Engineering Copyright 2015 FINRA

Where We Were

Traditional Financial Review Cycles Too Long

bull Quarterly Reviews

AWS Detailed Billing Reports Are Daunting

bull Over 10 Million Line Items

Project Managers Need Focus

bull Am I Below My Budget

bull Where Are My Costs Going

bull Whorsquos Spending Them

Manual Compilation of Reports

bull Integrate FINRA Data

FINRA Information Security Engineering Copyright 2015 FINRA

Approach Chosen

Use Splunk as Process Delivery System

bull Ability to Collect Analyze Visualize

Collect AWS Billing Data in Splunk

bull Billing Data from S3 Bucket (Daily Load)

bull Detailed Line Items with Resources and Tags

Data Enrichment

bull Project Code Lookups

bull Forecast Projections

bull Billing Adjustments

Build Interfaces

bull FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

How We Did It

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

Impact ndash Reduced Costs

Focus on Low Hanging Fruit

bull Shutting Down Services over Weekends Evenings

bull Storage Sun Setting Dormant EC2

bull Identify AWS Services with Highest Spending

bull Projects Over Budget

Results

bull 135 Reduction in Billing Line Items in 1 Month

Better Forecast Projections

bull Feedback and Control

FINRA Information Security Engineering Copyright 2015 FINRA

Futures

Users Want Even Shorter Cycles

Back Tagging

ldquoFree Riderrdquo Services

Drill Down Analytics

Splunking in Amazon EMRGaining Transparency into PaaS

FINRA Information Security Engineering Copyright 2015 FINRA

What Stood in Our Way

PaaS and IaaS Are Not Equal

Instance Fingerprinting Identify Nodes

Instance Role

User Tags

Data Retention

Collection Delay

Bootstrap Splunk Agent

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA Information Security Engineering Copyright 2015 FINRA

Beyond IAM Best Practices

Tagging Compliance

Security Group

Logging

Amazon S3Amazon Elastic Block Store (Amazon EBS) Encryption

Amazon S3 Bucket Policies

AMI WhiteBlacklisting

Amazon EC2 Role

Naming Conventions

New AWS Features Will Supplant

bull VPC Flows

bull Config

Project Cost Management in AWS

Harnessing the Power of Splunk

FINRA Information Security Engineering Copyright 2015 FINRA

Where We Were

Traditional Financial Review Cycles Too Long

bull Quarterly Reviews

AWS Detailed Billing Reports Are Daunting

bull Over 10 Million Line Items

Project Managers Need Focus

bull Am I Below My Budget

bull Where Are My Costs Going

bull Whorsquos Spending Them

Manual Compilation of Reports

bull Integrate FINRA Data

FINRA Information Security Engineering Copyright 2015 FINRA

Approach Chosen

Use Splunk as Process Delivery System

bull Ability to Collect Analyze Visualize

Collect AWS Billing Data in Splunk

bull Billing Data from S3 Bucket (Daily Load)

bull Detailed Line Items with Resources and Tags

Data Enrichment

bull Project Code Lookups

bull Forecast Projections

bull Billing Adjustments

Build Interfaces

bull FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

How We Did It

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

Impact ndash Reduced Costs

Focus on Low Hanging Fruit

bull Shutting Down Services over Weekends Evenings

bull Storage Sun Setting Dormant EC2

bull Identify AWS Services with Highest Spending

bull Projects Over Budget

Results

bull 135 Reduction in Billing Line Items in 1 Month

Better Forecast Projections

bull Feedback and Control

FINRA Information Security Engineering Copyright 2015 FINRA

Futures

Users Want Even Shorter Cycles

Back Tagging

ldquoFree Riderrdquo Services

Drill Down Analytics

Splunking in Amazon EMRGaining Transparency into PaaS

FINRA Information Security Engineering Copyright 2015 FINRA

What Stood in Our Way

PaaS and IaaS Are Not Equal

Instance Fingerprinting Identify Nodes

Instance Role

User Tags

Data Retention

Collection Delay

Bootstrap Splunk Agent

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Beyond IAM Best Practices

Tagging Compliance

Security Group

Logging

Amazon S3Amazon Elastic Block Store (Amazon EBS) Encryption

Amazon S3 Bucket Policies

AMI WhiteBlacklisting

Amazon EC2 Role

Naming Conventions

New AWS Features Will Supplant

bull VPC Flows

bull Config

Project Cost Management in AWS

Harnessing the Power of Splunk

FINRA Information Security Engineering Copyright 2015 FINRA

Where We Were

Traditional Financial Review Cycles Too Long

bull Quarterly Reviews

AWS Detailed Billing Reports Are Daunting

bull Over 10 Million Line Items

Project Managers Need Focus

bull Am I Below My Budget

bull Where Are My Costs Going

bull Whorsquos Spending Them

Manual Compilation of Reports

bull Integrate FINRA Data

FINRA Information Security Engineering Copyright 2015 FINRA

Approach Chosen

Use Splunk as Process Delivery System

bull Ability to Collect Analyze Visualize

Collect AWS Billing Data in Splunk

bull Billing Data from S3 Bucket (Daily Load)

bull Detailed Line Items with Resources and Tags

Data Enrichment

bull Project Code Lookups

bull Forecast Projections

bull Billing Adjustments

Build Interfaces

bull FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

How We Did It

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

Impact ndash Reduced Costs

Focus on Low Hanging Fruit

bull Shutting Down Services over Weekends Evenings

bull Storage Sun Setting Dormant EC2

bull Identify AWS Services with Highest Spending

bull Projects Over Budget

Results

bull 135 Reduction in Billing Line Items in 1 Month

Better Forecast Projections

bull Feedback and Control

FINRA Information Security Engineering Copyright 2015 FINRA

Futures

Users Want Even Shorter Cycles

Back Tagging

ldquoFree Riderrdquo Services

Drill Down Analytics

Splunking in Amazon EMRGaining Transparency into PaaS

FINRA Information Security Engineering Copyright 2015 FINRA

What Stood in Our Way

PaaS and IaaS Are Not Equal

Instance Fingerprinting Identify Nodes

Instance Role

User Tags

Data Retention

Collection Delay

Bootstrap Splunk Agent

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

Project Cost Management in AWS

Harnessing the Power of Splunk

FINRA Information Security Engineering Copyright 2015 FINRA

Where We Were

Traditional Financial Review Cycles Too Long

bull Quarterly Reviews

AWS Detailed Billing Reports Are Daunting

bull Over 10 Million Line Items

Project Managers Need Focus

bull Am I Below My Budget

bull Where Are My Costs Going

bull Whorsquos Spending Them

Manual Compilation of Reports

bull Integrate FINRA Data

FINRA Information Security Engineering Copyright 2015 FINRA

Approach Chosen

Use Splunk as Process Delivery System

bull Ability to Collect Analyze Visualize

Collect AWS Billing Data in Splunk

bull Billing Data from S3 Bucket (Daily Load)

bull Detailed Line Items with Resources and Tags

Data Enrichment

bull Project Code Lookups

bull Forecast Projections

bull Billing Adjustments

Build Interfaces

bull FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

How We Did It

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

Impact ndash Reduced Costs

Focus on Low Hanging Fruit

bull Shutting Down Services over Weekends Evenings

bull Storage Sun Setting Dormant EC2

bull Identify AWS Services with Highest Spending

bull Projects Over Budget

Results

bull 135 Reduction in Billing Line Items in 1 Month

Better Forecast Projections

bull Feedback and Control

FINRA Information Security Engineering Copyright 2015 FINRA

Futures

Users Want Even Shorter Cycles

Back Tagging

ldquoFree Riderrdquo Services

Drill Down Analytics

Splunking in Amazon EMRGaining Transparency into PaaS

FINRA Information Security Engineering Copyright 2015 FINRA

What Stood in Our Way

PaaS and IaaS Are Not Equal

Instance Fingerprinting Identify Nodes

Instance Role

User Tags

Data Retention

Collection Delay

Bootstrap Splunk Agent

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Where We Were

Traditional Financial Review Cycles Too Long

bull Quarterly Reviews

AWS Detailed Billing Reports Are Daunting

bull Over 10 Million Line Items

Project Managers Need Focus

bull Am I Below My Budget

bull Where Are My Costs Going

bull Whorsquos Spending Them

Manual Compilation of Reports

bull Integrate FINRA Data

FINRA Information Security Engineering Copyright 2015 FINRA

Approach Chosen

Use Splunk as Process Delivery System

bull Ability to Collect Analyze Visualize

Collect AWS Billing Data in Splunk

bull Billing Data from S3 Bucket (Daily Load)

bull Detailed Line Items with Resources and Tags

Data Enrichment

bull Project Code Lookups

bull Forecast Projections

bull Billing Adjustments

Build Interfaces

bull FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

How We Did It

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

Impact ndash Reduced Costs

Focus on Low Hanging Fruit

bull Shutting Down Services over Weekends Evenings

bull Storage Sun Setting Dormant EC2

bull Identify AWS Services with Highest Spending

bull Projects Over Budget

Results

bull 135 Reduction in Billing Line Items in 1 Month

Better Forecast Projections

bull Feedback and Control

FINRA Information Security Engineering Copyright 2015 FINRA

Futures

Users Want Even Shorter Cycles

Back Tagging

ldquoFree Riderrdquo Services

Drill Down Analytics

Splunking in Amazon EMRGaining Transparency into PaaS

FINRA Information Security Engineering Copyright 2015 FINRA

What Stood in Our Way

PaaS and IaaS Are Not Equal

Instance Fingerprinting Identify Nodes

Instance Role

User Tags

Data Retention

Collection Delay

Bootstrap Splunk Agent

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Approach Chosen

Use Splunk as Process Delivery System

bull Ability to Collect Analyze Visualize

Collect AWS Billing Data in Splunk

bull Billing Data from S3 Bucket (Daily Load)

bull Detailed Line Items with Resources and Tags

Data Enrichment

bull Project Code Lookups

bull Forecast Projections

bull Billing Adjustments

Build Interfaces

bull FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

How We Did It

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

Impact ndash Reduced Costs

Focus on Low Hanging Fruit

bull Shutting Down Services over Weekends Evenings

bull Storage Sun Setting Dormant EC2

bull Identify AWS Services with Highest Spending

bull Projects Over Budget

Results

bull 135 Reduction in Billing Line Items in 1 Month

Better Forecast Projections

bull Feedback and Control

FINRA Information Security Engineering Copyright 2015 FINRA

Futures

Users Want Even Shorter Cycles

Back Tagging

ldquoFree Riderrdquo Services

Drill Down Analytics

Splunking in Amazon EMRGaining Transparency into PaaS

FINRA Information Security Engineering Copyright 2015 FINRA

What Stood in Our Way

PaaS and IaaS Are Not Equal

Instance Fingerprinting Identify Nodes

Instance Role

User Tags

Data Retention

Collection Delay

Bootstrap Splunk Agent

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

How We Did It

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

Impact ndash Reduced Costs

Focus on Low Hanging Fruit

bull Shutting Down Services over Weekends Evenings

bull Storage Sun Setting Dormant EC2

bull Identify AWS Services with Highest Spending

bull Projects Over Budget

Results

bull 135 Reduction in Billing Line Items in 1 Month

Better Forecast Projections

bull Feedback and Control

FINRA Information Security Engineering Copyright 2015 FINRA

Futures

Users Want Even Shorter Cycles

Back Tagging

ldquoFree Riderrdquo Services

Drill Down Analytics

Splunking in Amazon EMRGaining Transparency into PaaS

FINRA Information Security Engineering Copyright 2015 FINRA

What Stood in Our Way

PaaS and IaaS Are Not Equal

Instance Fingerprinting Identify Nodes

Instance Role

User Tags

Data Retention

Collection Delay

Bootstrap Splunk Agent

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

FINRA AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

Impact ndash Reduced Costs

Focus on Low Hanging Fruit

bull Shutting Down Services over Weekends Evenings

bull Storage Sun Setting Dormant EC2

bull Identify AWS Services with Highest Spending

bull Projects Over Budget

Results

bull 135 Reduction in Billing Line Items in 1 Month

Better Forecast Projections

bull Feedback and Control

FINRA Information Security Engineering Copyright 2015 FINRA

Futures

Users Want Even Shorter Cycles

Back Tagging

ldquoFree Riderrdquo Services

Drill Down Analytics

Splunking in Amazon EMRGaining Transparency into PaaS

FINRA Information Security Engineering Copyright 2015 FINRA

What Stood in Our Way

PaaS and IaaS Are Not Equal

Instance Fingerprinting Identify Nodes

Instance Role

User Tags

Data Retention

Collection Delay

Bootstrap Splunk Agent

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

Impact ndash Reduced Costs

Focus on Low Hanging Fruit

bull Shutting Down Services over Weekends Evenings

bull Storage Sun Setting Dormant EC2

bull Identify AWS Services with Highest Spending

bull Projects Over Budget

Results

bull 135 Reduction in Billing Line Items in 1 Month

Better Forecast Projections

bull Feedback and Control

FINRA Information Security Engineering Copyright 2015 FINRA

Futures

Users Want Even Shorter Cycles

Back Tagging

ldquoFree Riderrdquo Services

Drill Down Analytics

Splunking in Amazon EMRGaining Transparency into PaaS

FINRA Information Security Engineering Copyright 2015 FINRA

What Stood in Our Way

PaaS and IaaS Are Not Equal

Instance Fingerprinting Identify Nodes

Instance Role

User Tags

Data Retention

Collection Delay

Bootstrap Splunk Agent

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

AWS Billing App

FINRA Information Security Engineering Copyright 2015 FINRA

Impact ndash Reduced Costs

Focus on Low Hanging Fruit

bull Shutting Down Services over Weekends Evenings

bull Storage Sun Setting Dormant EC2

bull Identify AWS Services with Highest Spending

bull Projects Over Budget

Results

bull 135 Reduction in Billing Line Items in 1 Month

Better Forecast Projections

bull Feedback and Control

FINRA Information Security Engineering Copyright 2015 FINRA

Futures

Users Want Even Shorter Cycles

Back Tagging

ldquoFree Riderrdquo Services

Drill Down Analytics

Splunking in Amazon EMRGaining Transparency into PaaS

FINRA Information Security Engineering Copyright 2015 FINRA

What Stood in Our Way

PaaS and IaaS Are Not Equal

Instance Fingerprinting Identify Nodes

Instance Role

User Tags

Data Retention

Collection Delay

Bootstrap Splunk Agent

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Impact ndash Reduced Costs

Focus on Low Hanging Fruit

bull Shutting Down Services over Weekends Evenings

bull Storage Sun Setting Dormant EC2

bull Identify AWS Services with Highest Spending

bull Projects Over Budget

Results

bull 135 Reduction in Billing Line Items in 1 Month

Better Forecast Projections

bull Feedback and Control

FINRA Information Security Engineering Copyright 2015 FINRA

Futures

Users Want Even Shorter Cycles

Back Tagging

ldquoFree Riderrdquo Services

Drill Down Analytics

Splunking in Amazon EMRGaining Transparency into PaaS

FINRA Information Security Engineering Copyright 2015 FINRA

What Stood in Our Way

PaaS and IaaS Are Not Equal

Instance Fingerprinting Identify Nodes

Instance Role

User Tags

Data Retention

Collection Delay

Bootstrap Splunk Agent

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Futures

Users Want Even Shorter Cycles

Back Tagging

ldquoFree Riderrdquo Services

Drill Down Analytics

Splunking in Amazon EMRGaining Transparency into PaaS

FINRA Information Security Engineering Copyright 2015 FINRA

What Stood in Our Way

PaaS and IaaS Are Not Equal

Instance Fingerprinting Identify Nodes

Instance Role

User Tags

Data Retention

Collection Delay

Bootstrap Splunk Agent

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

Splunking in Amazon EMRGaining Transparency into PaaS

FINRA Information Security Engineering Copyright 2015 FINRA

What Stood in Our Way

PaaS and IaaS Are Not Equal

Instance Fingerprinting Identify Nodes

Instance Role

User Tags

Data Retention

Collection Delay

Bootstrap Splunk Agent

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

What Stood in Our Way

PaaS and IaaS Are Not Equal

Instance Fingerprinting Identify Nodes

Instance Role

User Tags

Data Retention

Collection Delay

Bootstrap Splunk Agent

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Bootstrap Splunk Agent

Store bsx in S3

bull Splunk rpm

bull Deploymentclientconf

bull Discovery Scripts

Execute Bootstrap

Master Installation

Core and Task Installation

AWS was Extremely Supportive

of This Method

Amazon EMR

CORES

MASTER

TASK

Availability Zone

Amazon S3

BOOTSTR

AP

CLUSTER

forwarder

forwarder

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

1 Determine the Identity of the Node

oEC2 nodes have ldquometadatardquo service http169254169254

o INSTANCE_ID=`curl http16925416925410meta-datainstance-id`

o echo $INSTANCE_ID i-8f0d4c75

o IP=`curl -s http1692541692542014-11-05meta-datalocal-ipv4`

2 Grab the User Defined Tags

o AGS_TAG=`optawsapitoolsec2binec2-describe-tags|grep

$INSTANCE_ID|grep AGS|cut -f5`

o SDLC=`optawsapitoolsec2binec2-describe-tags|grep $INSTANCE_ID|grep

SDLC|cut -f5|awk print substr($011)`

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Instance Fingerprinting

3 Get the jobFlowID (EMR Cluster ID) for the Node

o JOBFLOW_ID=`awk -F[] for(i=1ilt=NFi++)if($i~jobFlowId042)print $(i+1)

mntvarlibinfojob-flowjson|tr -d [ ]`

4 Determine Role Node in the EMR Cluster

o INSTANCE_ROLE=`cat mntvarlibinfojob-flowjson | ruby -e require rubygems require

json require pp igs=JSON [STDINread][instanceGroups]

igid=ENV[INSTANCEGROUP_ID]ig = igsfind |i| i[instanceGroupId] == igidputs

ig[instanceRole]|awk print toupper($0)`

5 Update Splunk Config Files

o Deploymentclientconf

o Inputsconf

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Inputsconf Example

inputsconf

[default]

host = EMRAPP1-MASTERED-j-YQQU43YEHG4X1111

_meta=jobflow-idj-YQQU43YEHG4X instancegroup-

idig-3JBIDJGYVSD4N instance-idi-22f7f3f0

instance-roleMASTER ags_hostnameEMRAPP1-

MASTERED-j-YQQU43YEHG4X1111 agsldquoABCD

cost_centerCLDABC creatorldquoAPP1_Team

nameAWSLXAPP1-CLED01-YR ownerldquoUSERIDABC

purposeldquoAPP1_QUERY_CL sdlcDEV

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

EMR Cluster Analyzer ndashgt Summary Dashboard

AWS Billing

Splunk for NIX

Select the

Cluster

Jobs Running

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Potential EC2 Provisioning Cost Savings

m1xlarge

bull 4 vCPUs with 8 ECUs

bull 15 GB memory

bull 4 x 420 GB disk

bull $035 per hour

c3xlarge

bull 4 vCPUs with 14 ECUs

bull 75 GB memory

bull 2 x 40 GB disk (SSD)

bull $027 per hour

Potential 23

Savings in EMR

Costs

With additional ECUs and SSD disks the c3xlarge may be more

performant than the m1xlarge instances at a better price pointRESIZING

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis

Summary provides

the information on number of nodes

1 Less than 80 utilization overall

2 After 70 mins utilization at less than 50 (cpu)

Conclusion

The cluster be resized after the two hours

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 1

1st Hour 2nd Hour 3rd Hour 4th Hour

40 40 40 40

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = 035 40 4 = $56

All instances running for full duration of the job

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Resizing Analysis ndash Scenario 2

1st Hour 2nd Hour 3rd Hour 4th Hour

40 35 20 20

Cost Analysis

Cost = (Price per Instance) (No of Instances) (No of Hours)

Price = (035401) + (035351)+(035202) = 14 + 1225 + 14 = $4025

Savings Compared to Scenario 1 = 282

Resizing after the 1st hour and 2nd hour

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Combined EC2 Provisioning and Resizing Analysis

Combined Savings

bull Original Cost = $56

bull Price After Combined Analysis = $3105

bull Job Savings = $2495 = 5544

Job Runs 5x Day ( $2495 5 = $12475 )

Every Business Day Week ( $12475 5 = $62375 )

Every Week of the Year ( $62375 52 = $32435 )

AndhellipWe Havenrsquot Affected Performance

bull Just More Efficient Provisioning

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Job History

jhist File Analysisbull Cluster Level Statistics

bull Run Time Map Reduce Stats per Steps

Cluster Summary

Per Cluster

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

FINRA Information Security Engineering Copyright 2015 FINRA

Futures Other Uses

Grade Clusters

bull Identify Underutilized Clusters for Faster Resizing

ITSI Integration

bull KPIs based Auto Analysis on Cloud

Additional Input Variables

bull Size of Data Sets

bull Number of Runs

Metrics Correlation Analyze Steps

Jobs Impact on System

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

Copyright copy 2015 Splunk Inc

Splunk Overview

Praveen Rangnath

Sr Dir of Cloud Product Marketing

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

35

Turning Machine Data Into Business Value

Index Untapped Data Any Source Type Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility

Application Delivery

Security Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

Technology Telecommunications Travel and Leisure

Education

Healthcare

Energy and Utilities

Manufacturing

Financial Services and Insurance

Media

Proven at 10000+ Customers in 100 CountriesMore than 80 of the Fortune 100

Retail

Cloud and Online Services

Government

36

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

Splunk Runs on AWS

37

Bring Your Own License

Self-Managed Software on AWS

Analytics for EMR and S3

Available Hourly from EMR Console

Software

100 Uptime SLA

SOC2 Type II Certified

SaaS

For Small IT Teams

Starts at $90 Month

App for AWS

Apps and Integrations

Integrations

AWS CloudTrail AWS Config Billing S3

Amazon VPC Flow Logs Amazon CloudWatch

Amazon Kinesis Amazon DynamoDB S3

AWS Lambda

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

Why is Splunk Important to You

38

ldquoYou canrsquot protect what

you canrsquot seerdquo

Best Practices for Securing

Workloads in Amazon Web Services

Gartner April 2015

Neil MacDonald Greg Young

ldquoSecurity monitoring will

make or break a

technology risk

management programrdquo

ldquoSecurity requires

visibilityrdquo

Assessing the Risk Yes the Cloud

Can Be More Secure Than Your On-

Premises Environment

IDC July 2015

Pete Lindstrom

Amazon Web Services

ldquoIntro to AWS Securityrdquo

2015 AWS Summit Series

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

39

Sample CloudTrail Dashboard

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

40

Sample Amazon VPC Flow Dashboard

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

41

Sample Topology Dashboard

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

42

Collecting AWS Data in Splunk Is Easy

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

NEWAWS IoT Integration

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

44

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State End-to-End AWS Visibility

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS

Your Next Step To AWS Visibility

45

App for AWS

Get Started for Free

Integrations

Watch Splunk in AWS

reInvent 2014 Keynote

httpswwwyoutubecomwatchv=vfRS1LUHgJM

Visit Us at

Booth 400

You Bet Your

Sweet SaaS