aws summit auckland - sponsor presentation - splunk

Copyright © 2015 Splunk Inc.

Splunk For the Cloud, in the Cloud

Richard Smith

Manager, Global Strategic Alliances

Australia and New Zealand

2

Make machine data accessible,usable and valuable to everyone.

2

Big Data Comes from MachinesVolume | Velocity | Variety | Variability

GPS,RFID,

Hypervisor,Web Servers,

Email, Messaging,Clickstreams, Mobile,

Telephony, IVR, Databases,Sensors, Telematics, Storage,

Servers, Security Devices, Desktops 3

Building a Big Data Platform

HA / DR Admin Data Security Apps SDKs/APIScale

CollectData

IndexData

Enrich Data

Search & Explore

Analyze& Predict

Report &Visualize

Alert & Action

4

Fully Integrated Enterprise Platform

HA / DR Admin Data Security Apps SDKs/APIScale

CollectData

IndexData

Enrich Data

Search & Explore

Analyze& Predict

Report &Visualize

Alert & Action

5

StructuredRDBMS

SQL Search

Schema at Write Schema at Read

Traditional Splunk

Splunk Approach to Machine Data

Copyright © 2014 Splunk Inc.6

ETL Universal Indexing

Volume Velocity Variety

Unstructured

7

Turning Machine Data Into Business Value

Index Untapped Data: Any Source, Type, Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Packaged Applications

CustomApplicationsMessaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

Call Detail Records

Smartphones and Devices

RFID

On-Premises

Private Cloud

Public Cloud

Ask Any Question

Application Delivery

Security, Compliance and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

8

Splunk App for AWSEC2

EMR

Kinesis

R53

VPC

ELB

S3

CloudFront

CloudTrail

CloudWatch

Redshift

SNS

API Gateway

Config

RDS

CF

IAM

Lambda

Explore Analyze Dashboard Alert Act

AWS Data Sources

End State: Comprehensive AWS Visibility

9

True End State: Complete Hybrid Visibility

Index Untapped Data: Any Source, Type, Volume

Online Services Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

Messaging

TelecomsOnline

Shopping Cart

Web Clickstreams

Databases

Energy Meters

RFID

On-Premises

Private Cloud

Public Cloud

End-to-End Visibility


Security, Compliance, and Fraud

IT Operations

Business Analytics

Industrial Data andthe Internet of Things

CloudTrail

Config

Lambda

EC2

ITOperations


Developer Platform (REST API, SDKs)

Business Analytics

Industrial Data and Internet of

Things

10

Delivers Value Across IT and the Business

Business Analytics

Industrial Data and Internet of

Things

Security, Complianceand Fraud

11

Platform for Application Delivery and IT Operations

ROOT CAUSE AND ISSUE

RESOLUTION

PROACTIVEMONITORING

AND REAL-TIME ALERTING

DELIVER BETTER QUALITY CODE

FASTER

CLOUD APP AND INFRASTRUCTURE

MONITORING

MOBILE APPTROUBLESHOOTING

USER & USAGE ANALYTICS

IT SERVICE INTELLIGENCE

Fault Management

& Alerting

Infrastructure Monitoring App

Website and API Monitoring

Application Management Data Fabric

Ticket Analytics AppMachine Learning and Social Media

Sentiment

Web Analytics App

Capacity Management App

Copyright © 2016 Accenture. All rights reserved.

Better Code, Faster Development and Migration to Cloud

• Reduced error rates by 2 orders of magnitude in a couple of weeks

• Rapidly found and fixed one line of code responsible for 30,000+ errors

• Real-time dashboards on error rates and production impact

• In-depth visibility as they strategically migrate apps to AWS Cloud

14

Application Delivery & IT Ops Landscape

API

SDKs UI

Server, Storage, Network

Server Virtualization

Operating Systems

Custom Applications

Business Applications

Cloud Services

App Performance MonitoringTicketing/Other

Web Intelligence

Mobile Applications

Stream

15

Single Platform for Security Intelligence

SECURITY & COMPLIANCE REPORTING

REAL-TIME MONITORING OF KNOWN THREATS

DETECT UNKNOWN

THREATS

INCIDENT INVESTIGATIONS

& FORENSICS

FRAUD DETECTION

INSIDER THREAT

Splunk Complements, Replaces and Goes Beyond Existing SIEMs

How FINRA Uses Splunk Cloud for Security

• Transforms third-party threat intelligence information into security alerts

• Leverages the Splunk App for AWS

• Efficient provisioning dramatically reduces costs

“Splunk Cloud gives you applications which let you get huge amounts of value from your data.”

— Sr. Director of Information Security

API

SDKs UI

Network Traffic Analysis

Identity & Access Control

Perimeter Defense

EmailPayload Analysis

Endpoint Behavior Analysis

Endpoint Change Tracking

DLP

Security Analytics

Threat Intelligence

Cloud Security

Security & Compliance Landscape

17

18

Extending Splunk for Business Analytics

Splunk Software Complements Existing BI Solutions

CUSTOMER EXPERIENCE

PRODUCT ANALYTICS

BUSINESS PROCESS

ANALYTICS

DIGITALMARKETING

Why Domino’s uses Splunk for Application Management and Business Analytics

• Understand device and app usage trends for orders

• Real-time reNex insights from store data

• Visibility into online and mobile coupon redemption

• Refine Campaigns for higher conversion

20

Splunk for Industrial Data & the Internet of Things

REMOTETROUBLESHOOTING

& PREVENTIVE MAINTENANCE

SECURITY &COMPLIANCE

DEVICE USAGE &CUSTOMER ANALYTICS

OPERATIONALEFFICIENCY

Splunk HEC, AWS Lambda, Kinesis & IoT

Lambda

Kinesis

IoT

Other

Services

HTTP Event Collector

(data queuing system)

(serverless codeexecution)

Splunk/HEC Blueprints available on AWS Lambda Console

events > Kinesis > Lambda > Splunk Correlations/Alerts)

Saving Customers $Billions on Fuel, Operations

• Improved customer operations by mining large volumes of unstructured data

• Moved from monthly batch analysis to flexible real-time reporting

• Delivered value-added services

• Minimized in-train forces

• Optimized operational efficiency

“Thanks to Splunk, our systems allow our customers to provide engineers with real-time feedback

and use operational insight to achieve optimal runs every time.”

— Director of Engineering, Train Dynamic Systems (a division of NYAB)

23

All the features of Splunk EnterpriseAll the benefits of SaaS

Hybrid

24

Search Head(s)

Indexer(s)

On Premises Private Cloud Public

Cloud

Search Head(s)

Indexer(s)

On Premises Private Cloud Public

Cloud

Hybrid Search

Single Pane of Glass Visibility

Platform for Operational Intelligence

The Splunk Portfolio

Rich Ecosystem ofApps & Add-Ons

Splunk PremiumSolutions

MainframeData

RelationalDatabases

MobileForwarders Syslog/TCPIoTDevices

NetworkWire Data

Hadoop

Dev.splunk.com40,000+ questionsand answers

1,000+ apps Local User Groups and

SplunkLive! events

26

Thriving Community

COLLECT DATA FROM ANYWHERE

SEARCHAND ANALYZE EVERYTHING

GAIN REAL-TIME OPERATIONAL INTELLIGENCE

The Power of Splunk

27

FREE CLOUD TRIAL

FREE DOWNLOAD

FREE AMAZON MACHINE

IMAGES (AMI)

28

Easy to Try & Get Started

1 32

Thank you

HOW SJS ARE USING SPLUNK>

& AWS TO DRIVE EFFICIENCIES

PHIL TANNER @Phil_Tanner

CHIEF TECHNICAL OFFICER

Student Job Search Aotearoa

June 2016

Splunk AWS v0.05.20160616.PT

AGENDA

Who am I?

Who is SJS?

Challenges and

needs

Use cases

www.sjs.co.nz @Phil_Tanner

WHO IS SJS?

Charity – owned by students

Last year, we filled 28,000

vacancies

Receive a call every 1.7 minutes

Earning our students over

NZ$80,000,000

With an average of $5,250 for

each student

Place a student every 5 minutes


What did we need from

our data solution?

What were our core

requirements?

Performance Measurement

Understandthe data

Historical Analytics

Increase Efficiencies

Cross -Departmental

Access

Minimal CAPEX


CHALLENGES FACED BY SJS


How is the company performing?

Spot outliers

And trends

And relationships

Faster

To create

and to check

PERFORMANCE MEASUREMENT


GET THE REAL STORY

Why are Student Registrations

decreasing?

Overlaying Activated Registrations

provides a completely different story

Decreased time-wasters means it’s a good

news story, not a bad one.

Supply and demand in the marketplace

Provide historical context to current trends


Historical Analysis


INCREASED EFFICIENCIES

Helping the business run more smoothly


NOT JUST A TOOL FOR IT

Allows departments to

track their own progress

Easy to understand

Encourages friendly

competition


AVOIDING CAPEX

Better cash-flow

Higher uptime

Faster/cheaper upgrades


CONCLUSION

Performance measurement Measuring KPIs

Improving data integrity

Get the Real StoryUnderstand the story behind the numbers

SecurityTrack brute force attacks in real time

Identify website security breaches

Single-Sign-On to access reports

Systems MonitoringPerfect for log file analysis

Proactive problem identification

Post-analysis

Increased EfficienciesHelping the business run more smoothly

Cross-Departmental AccessEncourages friendly competition

Departments track their own progress

Easy to understand

Reduction in CAPEXBetter cash-flow

Higher uptime

Faster/cheaper upgrades

[email protected]

Chief Technical Officer

Student Job Search Aotearoa


THANK YOU FOR LISTENING

During this talk, SJS has placed six students in work,

generating $14,375 for the New Zealand economy

MIX DATASOURCES INTO ONE NUMBER

Pull our audited numbers

from the Splunk index

Extract data up to last night

from the reporting database

Grab today’s data from CSV

index=auditedplacementreports

| where _time >= relative_time(now(), "-6mon@y+6mon")

| eval class = "audited"

| stats count as placement by class

| append [ | dbquery sjs_reporting [ | stats count

| eval last_audited_date = round([ search index=audited_placement_reports | stats max(_time) as x | return $x],0)

| eval sql_str = " \"

SELECT

'unaudited' AS `class`,

COUNT(à`.ìd`) AS `placement`

FROM

àpplications` à`

WHERE

à`.`status` = 'accepted'

AND à`.`confirmed_date` > FROM_UNIXTIME(".last_audited_date.")

\" "

| return $sql_str ]

| appendcols [ | inputlookup todaysplacements

| stats count as today_placement ]

| eval placement = placement+today_placement

| fields class placement ]

| transpose header_field=class

| eval achieved = toString( audited+unaudited, "commas")

| fields achieved


Agenda

• This session is meant for AWS users who are setting up the Splunk App for AWS (or people who will be doing so after this session!).

• Using the Splunk App for AWS you will gain visibility across your AWS Deployment.

Agenda

• AWS Components

• Permissions for SQS, SNS and S3 buckets

• Setting up Splunk App for AWS

• Questions & Answers – at the Splunk booth after this session!

Splunk Offerings in AWS

• Splunk App for AWS: Integrates w/CloudTrail, Config and Billing, VPC Flow Logs

Integrations• Self-managed cloud deployments

• Self-deploy in AWS

• Integrated with EMR

• Search data in S3

• Hourly pricing Self-managed

• Cloud service designed for small IT environments

• $90 a month

• Splunk Enterprise as a service

• Full app, SDK, API, platform support

Cloud-service

AWS Architecture Diagram

Amazon Instances

Amazon Logging Layer

Amazon Messaging

Amazon Storage / Queues

Splunk Collects the data from the AWS SQS and the S3 bucket using the AWS SDK for python (Boto3).

Requirements For Splunk App For AWS

• Splunk

• Splunk 6.1 or later

• Splunk Add-on for Amazon Web Services

• Splunk Add-on for Amazon Web Services +1.1.0 required for AWS Config

• AWS

• AWS CloudTrail: Enable CloudTrail with SQS and SNS.

• AWS Config: Enable Config with SQS and SNS.

• Billing: Refer to the AWS documentation to turn on AWS detailed billing.

• VPC Flow Logs: Enable VPC Flow log collection.

Install the Splunk Add-on for AWS

1. Configure your AWS accounts and services, or confirm your existing configurations.

2. Configure your AWS account permissions to match those required by the add-on.

3. Install the add-on.

4. Set up the add-on on your forwarders or single instance.

5. Configure your inputs to get your AWS data into Splunk Enterprise.

6. This is all very well documented at docs.splunk.com

Permissions

S3 Storage

CloudTrailAPI Tracking

SNS NotificationSQS Message Queue

splunkuser

Sample permissions for cloudtrail

Splunk Architecture

• Distributed Splunk Deployment

Single Splunk Deployment

Splunk'serverIndexer

Heavy+Weight+Forwarder

Splunk+search

IndexerIndexer

Splunk Add-on for AWS installed on

Heavy Weight Forwarder

Splunk App for AWS

installed on all-in-one

Splunk server

Setup Interface

Add Your Account

Add your AWS

Inputs

Wait 5 – 10 Minutes

• Yes, you’ll need to wait before all the dashboards and reports populate.

Gain Visibility Into AWS Logs

THANKYOU

aws summit auckland - sponsor presentation - splunk

Technology