deploying splunk on amazon web services
TRANSCRIPT
![Page 1: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/1.jpg)
Copyright © 2015 Splunk Inc.
Simeon Yep, Strategic Alliances Nate Kwong, Senior SE Bill BartleD, Senior SE
Deploying Splunk on Amazon Web Services
![Page 2: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/2.jpg)
Disclaimer
2
During the course of this presentaKon, we may make forward looking statements regarding future events or the expected performance of the company. We cauKon you that such statements reflect our current expectaKons and esKmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements, please review our filings with the SEC. The forward-‐looking statements made in the this presentaKon are being made as of the Kme and date of its live presentaKon. If reviewed aTer its live presentaKon, this presentaKon may not contain current or
accurate informaKon. We do not assume any obligaKon to update any forward looking statements we may make.
In addiKon, any informaKon about our roadmap outlines our general product direcKon and is subject to change at any Kme without noKce. It is for informaKonal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligaKon either to develop the features
or funcKonality described or to include any such feature or funcKonality in a future release.
![Page 3: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/3.jpg)
ObjecKve:
3
Integrate your Splunk Enterprise deployment with Amazon Web Services (AWS)
![Page 4: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/4.jpg)
4
Amazon Web Services vs. Everyone Else
![Page 5: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/5.jpg)
5
#1!MQ Leader!
![Page 6: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/6.jpg)
Presenters
6
Bill BartleD
• Senior SE, AWS
• SeaDle
Nate Kwong
• Senior SE, Majors
• SF
Simeon Yep
• Director, Alliances
• SF
![Page 7: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/7.jpg)
Agenda • Infrastructure: AWS ElasKc Compute Cloud (EC2) • Deployment Examples & leveraging AWS features • AWS Provisioning and AutomaKon • Apps + Other (Time pending)
7
![Page 8: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/8.jpg)
AWS EC2 Infrastructure
![Page 9: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/9.jpg)
9
11!Worldwide Regions!
![Page 10: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/10.jpg)
10
28!Availability Zones!
![Page 11: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/11.jpg)
AWS Regions & Availability Zones
11
US Regions Global Regions
Availability Zone A
Availability Zone B
Availability Zone C
EU (Ireland)
Availability Zone A
Availability Zone B
South America (Sao Paulo)
Availability Zone A
Availability Zone B
Asia Pacific (Sydney)
Availability Zone A
Availability Zone B
GovCloud (OR)
Availability Zone A
Availability Zone B
Availability Zone C
Availability Zone D
US East (VA)
Availability Zone A
Availability Zone B
US West (CA)
Availability Zone A
Availability Zone B
Asia Pacific (Singapore)
Availability Zone A
Availability Zone B
Availability Zone C
Asia Pacific (Tokyo)
Availability Zone A
Availability Zone B
Availability Zone C
US West (OR)
Customer Decides Where Applications and Data Reside Note: Conceptual drawing only. The number of Availability Zones may vary.
Availability Zone A
Availability Zone B
EU (Frankfurt)
**China (Beijing) Region – Currently in Limited Preview with 1 AZ**
![Page 12: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/12.jpg)
Broad And Deep Services To Support Any Cloud Workload
12
AWS Global Infrastructure
Application Services
Networking
Deployment & Administration
Database Storage Compute
![Page 13: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/13.jpg)
Amazon Web Services EC2
13
" Amazon ElasKc Compute Cloud (EC2)
" Pay-‐as-‐you-‐go pricing model " Splunk is easily deployed in Amazon
![Page 14: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/14.jpg)
Broad Set of Compute Instance Types…
14
G2
GPU enabled
M3
General purpose
Memory opKmized
R3
CR1 M2
Storage and IO opKmized
C3
Compute opKmized
CC2 C1
I2
HI1
D2
CG1 M1
T2
T1 HS1
C4 Current Genera@on
Previous Genera@ons are s@ll available
M4
![Page 15: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/15.jpg)
Typical User Scenario 1. Sign-‐up for an AWS account (use AWS IAM – IdenKty and Access
Management) 2. Launch an EC2 instance (via user chosen tool such as GUI, CLI, or
external) 3. Use key credenKals to access the EC2 instance 4. Install SoTware/Splunk
15
![Page 16: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/16.jpg)
Splunk and Hardware • Splunk consumes high I/O due to indexing and searching • Load != GB/day • Search drives a large porKon of the load
– Rare vs. Sparse vs. ReporKng – Real-‐Kme vs. Historic
• Rule of thumb – up to 300 GB/day – Reference servers can index 500 GB/day with no search load
• Virtualized systems incur some overhead, but work well if tuned correctly
16
![Page 17: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/17.jpg)
Instances • Instance type
– Pricing: Spot vs. On-‐demand vs. Reserved – Family: Storage vs. Compute vs. GPU vs. Memory vs. General Purpose – GeneraKon: Current vs. Previous
• Instance size – Workload size: compute units, memory, storage – Micro, Small, Medium, Large, Extra Large (XL)
ê MulKple XL sizes: xlarge, 2xlarge, 4xlarge, 8xlarge – 4XL general purpose provides similar performance to a reference server
ê 50-‐250 GB/day indexing and searching
17
![Page 18: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/18.jpg)
Instance Storage • Instances have ephemeral storage (Current Gen has SSDs)
– General Purpose instances have GBs to TBs – Storage OpKmized instances have up to 48 TB! – Data is lost when the instance dies
• EBS – ElasKc Block Storage – Persistent block level storage volumes for use with EC2 instances – Cost associated – 1 TB costs $100/month – Data is not lost when instance dies – can be remounted with new instance – For storage needs larger than 16 TB, RAID required – Built-‐in resiliency – data is backed up
• S3 – Simple Storage Service – Online cloud storage service (files, data, etc…) – Need this for backup purposes (Snapshots) – Can also be used as a data feed for Splunk, TA available
18
![Page 19: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/19.jpg)
Storage Best PracKces • Single instances or non-‐replicated distributed deployments:
– Use EBS volumes for indexes and the OS/soTware – RAID can be an extra measure of reliability, but will consume CPU – Use snapshots to backup the instance (S3) – IOPS opKmized provides benefits – XFS preferred (customer feedback) – c4 (compute opKmized) instances will require storage
19
![Page 20: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/20.jpg)
Instance SelecKon • How can I make my deployment resilient?
– OpKon 1: EBS – OpKon 2: Index ReplicaKon – OpKon 3: Data Cloning (Index and Forward)
• Instance selecKon should factor in resiliency, use-‐case, and cost • Index ReplicaKon (IR)
– ReplicaKon requires more instances as data is stored twice – Does not require EBS for indexes – Major driver is instance cost as you leverage ephemeral storage
20
![Page 21: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/21.jpg)
Instance SelecKon Exercise • 1 TB/day Distributed Deployment
– EBS backed storage for availability – No replicaKon
• AWS Calculator spreadsheet available
21
![Page 22: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/22.jpg)
Instance SelecKon Exercise
22
" RetenKon values for EBS backed deployments significantly drive cost
![Page 23: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/23.jpg)
Instance SelecKon Exercise • 1 TB/day Distributed Deployment
– Index ReplicaKon enabled (Double the indexers and add 1 AdministraKve node)
• Index ReplicaKon offers immediate search capability with SF/RF • Differences:
– $5k – Increased availability, higher performance
23
![Page 24: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/24.jpg)
Instance SelecKon Distributed Deployments
Using EBS volumes, no IR Using Index ReplicaKon (IR)
• Local ephemeral storage (SSDs) may perform beDer than EBS
• Search/ReplicaKon Factor determines availability of data for searching
• IR adds load and requires more servers and storage
• Typically fewer instances to manage vs. IR
• Search Availability is driven by the capability to remount a volume to a new instance (automaKcally or manually)
• Cost can be largely driven by retenKon and daily volume
24
![Page 25: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/25.jpg)
Amazon Machine Image (AMI) • Amazon Machine Image (AMI) preferences for Splunk
– Amazon Linux based – Best Performance – Cost EffecKve (extra $$ for Windows)
• AMIs available for download – Splunk Enterprise – Hunk
ê Hunk + EMR baked into Marketplace
![Page 26: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/26.jpg)
Best PracKces • Custom AMI creaKon
– Create your own AMI using Linux based or Splunk provided – Leverage current configuraKon tooling with AMI (don’t have to use deployment
server, but can be very helpful) • AuthenKcaKon and AuthorizaKon
– Policies will dictate what you can or cannot use – LDAP/AD will require an SSL tunnel – Other opKons: scripted input or proxying (SSO) – SAML (Okta) – NOTE – SSO methods sKll require role informaKon
• Security – SSL everywhere + private network – Install your own cerKficates
26
![Page 27: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/27.jpg)
Best PracKces • Search Head Clustering • Deploy to the same AWS Region
– ReplicaKon and searches across Regions can be a challenge • Monitor from outside of the Region/AZ
– Offers addiKonal resiliency • Use a Virtual Private Cloud (VPC)
27
![Page 28: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/28.jpg)
General Guidelines Follow Best PracKces for ArchitecKng and Sizing: Load=Searching+Indexing
Search Heads (8+ users) Indexers (50-‐250 GB/day)
• c4.4xlarge 16 vCPU, 30 GB RAM • d2.4xlarge 16 vCPU, 122 GB RAM • c4.8xlarge 36 vCPU, 60 GB RAM
• c4.4xlarge 16 vCPU, 30 GB RAM • c4.8xlarge 36 vCPU, 60 GB RAM • c4.xlarge 4 vCPU, 7.5 GB RAM • c4.2xlarge 8 vCPU, 15 GB RAM
• c4.large 2 vCPU, 3.75 GB RAM • c4.xlarge 4 vCPU, 7.5 GB RAM
28
Cluster Master or Deployment Server
*These are all starKng points! Splunk can index and search more OR less depending on overall load.
License Master
![Page 29: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/29.jpg)
Architecture & Deployment Examples
![Page 30: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/30.jpg)
Architecture Examples • Centralized • Decentralized • Hybrid • Centralized with Index ReplicaKon
30
![Page 31: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/31.jpg)
Centralized Topology
Indexers
Forwarders
Syslog Devices
Intermediate Forwarder Forwarders
Search Heads
31
![Page 32: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/32.jpg)
Decentralized Topology
Search Heads
32
![Page 33: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/33.jpg)
Hybrid Topology
33
![Page 34: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/34.jpg)
Index Replication with Search Clustering Search Cluster
Peer Nodes
Cluster Master
Forwarders
34
![Page 35: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/35.jpg)
Deployment Examples • Single Server • MulK-‐Server
35
![Page 36: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/36.jpg)
Single Server • Use case: Searching, ReporKng and AnalyKcs • Up to 250+ GB/day indexing with common search loads
– For heavy reporKng and analyKcs, decrease indexing volume
• c4.4xlarge instance – EBS volumes configured to support retenKon needs
• Up to 16 concurrent users
36
![Page 37: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/37.jpg)
MulK-‐Server • Use Case: ApplicaKon Management, Security Forensics • Up to 1 TB/day indexing with common search loads • Distributed deployment with Index ReplicaKon (2 SF, 3 RF) • 8 -‐ d2.4xl instances with 24 TB ephemeral storage (indexers) • 3 -‐ c4.4xlarge instance (search cluster)
37
![Page 38: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/38.jpg)
Deployment B
38 38
CM + DMC + Deployer
Search Head(s)
Indexer
![Page 39: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/39.jpg)
Example Architectures Use case and requirements influence final setup, but there is no right or wrong way
Using Index ReplicaKon Using EBS backed storage • 20 GB/day
– c4.2xlarge (single instance) • 100 GB/day
– c4.4xlarge (single instance) • 300 GB/day
– c4.4xlarge – c4.8xlarge
• 500 GB/day – c4.4xlarge as indexer (3) – c4.4xlarge as search head (1)
• 100 GB/day – d2.2xlarge as indexer (2) – c4.2xlarge as search head (1) – c4.xlarge as CM/LM
• 500 GB/day – d2.4xlarge as indexer (3) – c4.4xlarge as search head (1) – c4.xlarge as CM/LM
39
![Page 40: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/40.jpg)
Self Healing Splunk Architecture
40
![Page 41: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/41.jpg)
AWS Auto Scaling • AutomaKcally replace unhealthy EC2 instances • MulKple Auto Scaling Policies
– Maintain a fixed number of EC2 Instances (recommended for Splunk Indexers) – Performance metrics – Time based – Manual Scaling
41
![Page 42: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/42.jpg)
Architecture Diagram (Splunk + AWS)
42
Availability Zone A Availability Zone C
Indexer instance
Indexer instance
Indexer instance
Indexer instance
Auto Scaling Group AZ-A
Search Head instance
Auto Scaling group – Across 3 Zones
Auto Scaling Group AZ-C
Cluster Master
Instance Auto
Scaling Group of 1 Availability Zone B
Indexer instance
Indexer instance
Auto Scaling Group AZ-B
Search Head instance
Search Head instance
![Page 43: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/43.jpg)
Architecture Diagram (Splunk + AWS)
43
Availability Zone A Availability Zone C
Indexer instance
Indexer instance
Indexer instance
Indexer instance
Auto Scaling Group AZ-A
Search Head instance
Auto Scaling group – Across 3 Zones
Auto Scaling Group AZ-C
Cluster Master
Instance Auto
Scaling Group of 1 Availability Zone B
Indexer instance
Indexer instance
Auto Scaling Group AZ-B
Search Head instance
Search Head instance
![Page 44: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/44.jpg)
Architecture Diagram (Splunk + AWS)
44
Availability Zone A Availability Zone C
Indexer instance
Indexer instance
Indexer instance
Indexer instance
Auto Scaling Group AZ-A
Search Head instance
Auto Scaling group – Across 3 Zones
Auto Scaling Group AZ-C
Cluster Master
Instance Auto
Scaling Group of 1 Availability Zone B
Indexer instance
Indexer instance
Auto Scaling Group AZ-B
Search Head instance
Search Head instance
Indexer instance
![Page 45: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/45.jpg)
Splunk Indexer Clustering with Auto Scaling • MulKsite clustering
– Replicate a copy of your data to mulKple sites – Hint: AWS Availability Zone = Splunk Site
• Separate Auto Scaling Groups for each Availability Zone
45
![Page 46: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/46.jpg)
Splunk Search Head Clustering with Auto Scaling
• Auto-‐elecKon of captain within the Search Head Cluster • Auto Scaling Policy spans across mulKple Availability Zones
46
![Page 47: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/47.jpg)
Architecture Diagram (Splunk + AWS)
47
Availability Zone A Availability Zone C
Indexer instance
Indexer instance
Indexer instance
Indexer instance
Auto Scaling Group AZ-A
Search Head instance
Auto Scaling group – Across 3 Zones
Auto Scaling Group AZ-C
Cluster Master
Instance Auto
Scaling Group of 1 Availability Zone B
Indexer instance
Indexer instance
Auto Scaling Group AZ-B
Search Head instance
Search Head instance
Indexer instance
![Page 48: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/48.jpg)
Architecture Diagram (Splunk + AWS)
48
Availability Zone A Availability Zone C
Indexer instance
Indexer instance
Indexer instance
Indexer instance
Auto Scaling Group AZ-A
Search Head instance
Auto Scaling group – Across 3 Zones
Auto Scaling Group AZ-C
Cluster Master
Instance Auto
Scaling Group of 1 Availability Zone B
Indexer instance
Indexer instance
Auto Scaling Group AZ-B
Search Head instance
Search Head instance
Indexer instance
![Page 49: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/49.jpg)
Architecture Diagram (Splunk + AWS)
49
Availability Zone A Availability Zone C
Indexer instance
Indexer instance
Indexer instance
Indexer instance
Auto Scaling Group AZ-A
Search Head instance
Auto Scaling group – Across 3 Zones
Auto Scaling Group AZ-C
Cluster Master
Instance Auto
Scaling Group of 1 Availability Zone B
Indexer instance
Indexer instance
Auto Scaling Group AZ-B
Search Head instance
Search Head instance
Indexer instance
Search Head instance
![Page 50: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/50.jpg)
Splunk + AWS Features = FTW • Self Healing Splunk Infrastructure • Splunk Clustering provides data availability and replicaKon • AWS Auto Scaling can automaKcally replace failed Splunk instances
50
![Page 51: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/51.jpg)
Splunk + AWS Auto Scaling ConsideraKons • Auto Scaling Group of 1 for Splunk Cluster Master
– Splunk Cluster Master is a stateless server
• Use DNS name instead of IP address for Splunk Cluster Master URI • Bootstrap EC2 instances to automaKcally join Splunk Indexer and Search Head Clusters
51
![Page 52: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/52.jpg)
How To Provision Deployments
![Page 53: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/53.jpg)
Cloud Provisioning Tools
53
Deployment Provisioning
• Fast template-‐based provisioning – Provision & connect resources
Server Provisioning
• Flexible recipe-‐based configuraKon – Configure machine based on role
AWS CloudFormation
AWS OpsWorks
Terraform
Scalr
![Page 54: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/54.jpg)
54
Splunk + AWS + CloudFormaKon Ready in 10 minutes.
![Page 55: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/55.jpg)
Why CloudFormaKon? • Open-‐source self-‐service tool (no cost associated) • Fast, automated, consistent Splunk deployments on AWS
![Page 56: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/56.jpg)
“Live Demo” What could go wrong?
56
![Page 57: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/57.jpg)
Example Architecture
57 57
Search Head(s) Indexer Indexer Indexer Indexer Indexer
![Page 58: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/58.jpg)
Splunk AWS CloudFormaKon
58
What can Splunk AWS CloudFormaKon do for you? " Consistent, repeatable deployment Kme cut to minutes " Incorporates Splunk best pracLces for operaKons and administraKon
" Abstracts away details of configuring distributed Splunk " Extensible and customizable templates to fit custom needs
![Page 59: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/59.jpg)
Splunk AWS CloudFormaKon
59
“What used to take days to get all configured properly, now I can do in few minutes with
Splunk [AWS] CloudFormaKon” Abdallah Mohammed, Data Architect, Intuit
![Page 60: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/60.jpg)
“Live Demo” The Results Show.
60
![Page 61: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/61.jpg)
Apps and more
61
![Page 62: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/62.jpg)
Content " Splunk Apps:
– AWS: Data CollecKon and Dashboards for Cloudtrail, Cloudwatch, Billing/Usage
" Technology Add-‐on – S3 Modular Input
ê Simplified access to your content on S3
" Hunk App: – ElasKc Load Balancer: AnalyKcs for your ELB
" Hunk+EMR – Turn Key Hunk SoluKon
![Page 63: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/63.jpg)
63
QuesKons?
![Page 65: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/65.jpg)
References • Splunk App for AWS: hDp://apps.splunk.com/app/1274/ • Hunk App for AWS ELB: hDp://apps.splunk.com/app/1731/ • Technical Brief: hDp://www.splunk.com/content/dam/splunk2/pdfs/technical-‐briefs/deploying-‐splunk-‐enterprise-‐on-‐amazon-‐web-‐services-‐technical-‐brief.pdf
65
![Page 66: Deploying Splunk on Amazon Web Services](https://reader033.vdocuments.mx/reader033/viewer/2022050900/584d977c1a28ab85739145ce/html5/thumbnails/66.jpg)
References • Blogs:
– hDp://blogs.splunk.com/2012/03/07/splunk-‐and-‐aws-‐sizing-‐revisited/ – hDp://blogs.splunk.com/2013/06/06/splunkit-‐v2-‐0-‐2-‐results-‐ec2-‐storage-‐
comparisons/ – hDp://blogs.splunk.com/2013/07/31/whats-‐going-‐on-‐with-‐aws-‐and-‐splunk/ – hDp://blogs.splunk.com/2014/05/20/deploy-‐your-‐own-‐splunk-‐cluster-‐on-‐aws-‐
in-‐minutes/
• AMIs – Splunk: hDps://aws.amazon.com/marketplace/pp/B00GIZITUO?sr=0-‐4 – Hunk: hDps://aws.amazon.com/marketplace/pp/B00GIZK2QI?sr=0-‐2
66