Reference Architecture: Splunk Enterprise with ThinkSystem ... ?· Reference Architecture: Splunk Enterprise…

Download Reference Architecture: Splunk Enterprise with ThinkSystem ... ?· Reference Architecture: Splunk Enterprise…

Post on 26-Jan-2019

212 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

Reference Architecture: Splunk Enterprise with ThinkSystem Servers

Describes reference architecture for Splunk Enterprise

Contains sizing recommendations

Includes four different deployment models from department to large enterprise

Contains detailed bill of materials for Lenovo servers and networking

Mike Perks

Kenny Bain

Last update: 30 July 2018 Version 1.0

https://lenovopress.com/updatecheck/LP0908/71756038a67f450ab4980f40d967f5d0

ii Reference Architecture: Splunk Enterprise with ThinkSystem Servers version 1.0

Table of Contents

1 Introduction ............................................................................................... 1

2 Business problem and business value ................................................... 2

2.1 Business problem .................................................................................................... 2

2.2 Business value ......................................................................................................... 3

3 Requirements ............................................................................................ 4

3.1 Functional requirements .......................................................................................... 4

3.2 Non-functional requirements .................................................................................... 4

4 Architectural overview ............................................................................. 5

5 Component Model .................................................................................... 6

6 Operational model .................................................................................... 8

6.1 Operational model scenarios ................................................................................... 8

6.2 Hardware components ............................................................................................. 9

6.3 Servers .................................................................................................................. 12

6.4 Systems management ........................................................................................... 14

6.5 Networking ............................................................................................................. 22

6.6 Racks ..................................................................................................................... 23

6.7 Operating Systems ................................................................................................ 24

7 Appendix: Bill of Materials ..................................................................... 25

7.1 Server BOM ........................................................................................................... 25

7.2 Networking BOM .................................................................................................... 27

7.3 Rack BOM.............................................................................................................. 27

Resources ..................................................................................................... 28

1 Reference Architecture: Splunk Enterprise with ThinkSystem Servers version 1.0

1 Introduction This document describes the reference architecture for Splunk Enterprise using Lenovo ThinkSystem servers and networking. The intended audience of this document is IT professionals, technical architects, sales engineers, and consultants to assist in planning, designing, and implementing Splunk Enterprise 7.1.1.

This document provides an overview of the business problem and business value that is addressed by Splunk Enterprise. A description of customer requirements is followed by an architectural overview of the solution and a description of the logical components. The operational model describes the recommended operational architecture of Splunk Enterprise and four different deployment scenarios using Lenovo ThinkSystem servers and network switches. The appendix features detailed Bill of Materials configurations that are used in the solution.

2 Reference Architecture: Splunk Enterprise with ThinkSystem Servers version 1.0

2 Business problem and business value The following section provides a summary of the business problems that this reference architecture is intended to help address, and the value that this solution can provide.

2.1 Business problem The advent of mobile data, social streams, clouds and interconnected everything signifies the "Transformation of Information" with huge shift in data usage. It delivers on the promise of analysis of big data to identify patterns in statistical populations vs. traditional reliance on data modeling tools, queries, spreadsheet dashboards and charts.

Global enterprises are under competitive pressure to expand into new markets, to find clients and build customer loyalty. To yield real-time insights, they now leverage technology to sift through their data instantaneously and not after-the-fact data processing on a monthly, quarterly, or a yearly basis which typically results in a potential loss of competitive advantage. Agility, security, cost-effectiveness, flexibility and efficiency are key deterministic priorities for their IT. Picture a bank sifting through its enormous data to recognize fraud, with a response time, of a few microseconds, during an ATM transaction, or an auto insurer receiving real-time updates on driving habits from sensors installed in clients vehicles.

While customers are faced with many business challenges, this solution highlights two specific Big Data challenges that represent significant opportunities. The first challenge focuses on real-time identification and mitigation of advanced organizational security threats to the Enterprise by leveraging vigilant analysis and response capabilities. The second challenge is highlighted by the complexity of managing the abundance of systems prevalent in a data center, and ensuring high performance and availability of these systems, daily.

2.1.1 Vigilant enterprise security intelligence Organizational security threats do not make a story line for spy thrillers anymore. Global newsfeeds abound daily, with compromised websites, stolen credit card data, abnormal HTTP traffic, financial fraud, and malware presence. Detecting advanced Enterprise Security threats require a new approach, enabled by a smart & scalable security intelligence platform (SIP). SIP makes any data security relevant, scales to tens of terabytes of data per day and provides real-time analysis and response capabilities.

2.1.2 Operations analysis of machine data in data centers It is an extremely complex effort to efficiently manage the abundance of systems, deployed in a typical data center. On a daily basis, several systems experience outages, performance issues, or missed SLAs. To ensure high performance and availability, Enterprise IT administration teams waste valuable resources accessing several management consoles, and run home-grown scripts to serially trace the valuable data they need from failed systems. This is machine data, a form of Big-Data.

3 Reference Architecture: Splunk Enterprise with ThinkSystem Servers version 1.0

2.2 Business value Splunk Enterprise provides an end-to-end, real-time solution for both of these business problems by delivering the following core capabilities:

Universal collection and indexing of machine data and security data, from virtually any source Powerful search processing language (SPL) to search and analyze real-time and historical data Real-time monitoring for patterns and thresholds; real-time alerts when specific conditions arise Powerful reporting and analysis Custom dashboards and views for different roles Resilience and horizontal scalability Granular role-based security and access controls Support for multi-tenancy and flexible, distributed deployments on-premises or in the cloud Robust, flexible platform for big data apps

In addition, the Lenovo XClarity Administrator App for Splunk enables collection, visual representation, and analysis of Lenovo hardware events from the Splunk platform. Here are some examples of the critical insights that can be gained from the XClarity Administrator App for Splunk:

The volume and types of events generated over time from all monitored hardware. This will help administrators quickly identify problem hardware and take actions.

Percentage of total events being surfaced by each end point type such as the chassis management module (CMM), switch module, server, etc.

Number of times when a power threshold has been exceeded for any XClarity-managed resource, over time. This can help identify environmental issues in the data center. If exceeding of power thresholds caused power capping, this could also explain performance slowdowns.

Number of user accounts that were created on XClarity instances over time. Spikes in the number of new accounts could help identify uncommon security activities for audit purposes.

User IDs that attempted to authenticate to XClarity, but failed. Seeing which unauthorized user IDs were used to attempt access would be useful in system audits.

Number of login attempts made outside of normal business hours. This may help identify uncommon user account activity, like a large number of login attempts in the middle of the night or on a weekend.

4 Reference Architecture: Splunk Enterprise with ThinkSystem Servers version 1.0

3 Requirements This section describes the functional and non-functional requirements for this reference architecture.

3.1 Functional requirements The key functional requirements for the Splunk Enterprise solution include:

Support for collecting, indexing and searching data Support for real-time processing of data Support for a variety of data and data types, including security data and machine data Support for large volumes of data

In addressing the functional requirements, the reference architecture and sizing for the Splunk Enterprise solution must consider the following data requirements:

The amount of incoming data. The amount of indexed data in the datastore. Data placement in relevant storage tiers (in accordance with Splunk Indexer Data Retirement &

Archiving Policies). Data indexing performance is influenced by the choices of searches, and number of concurrent users. Deployment and execution of Splunk ecosystem applications such as Lenovo XClarity App for Splunk

and Splunk App for Enterprise Security. Required storage IO capabilities of high performance, scalability, and availability to support the

creation of extremely large, compressed data indexes, and offer the ability to run Storage IO-intensive sparse searches against this data.

3.2 Non-functional requirements The key non-functional requirement is to provide superior performance with both indexing data and searching data. The following shows the minimum performance requirements for Splunk Enterprise:

Minimum performance for each Indexing Server o Up to 5.8 megabytes per second (or 500 GB per day) of raw indexing performance, provided

no other Splunk activity is occurring. Minimum performance for each Search Server

o Up to 50,000 events per second for dense searches o Up to 5,000 events per second for sparse searches o Up to 2 seconds per index bucket for super-sparse searches o From 10 to 50 buckets per second for rare searches with bloom filters

In addition, the Splunk infrastructure needs to support both scale up and scale out as well as high availability and resilience to a single point of failure.

5 Reference Architecture: Splunk Enterprise with ThinkSystem Servers version 1.0

4 Architectural overview Splunk Enterprise provides an application platform for real-time operational intelligence. It facilitates easy, fast and secure collection, analysis, and search of data from massive data streams generated by devices, applications, transactions, timed events, systems and technologies.

Figure 1 below shows the architectural overview of Splunk Enterprise. Users can access one or more search head servers through a load balancer. The search head(s) provide access to information that is collected by forwarders from a variety of data sources possibly across multiple data centers.

Figure 1: Architectural Overview of Splunk Enterprise

IndexersSearch Head

Cluster

Clients

3rd Party Load Balancer

Applications

Forwarders

Web Servers

Hypervisors, OS

Databases

App Servers

Storage

Servers

Networks

Cloud Services

Deployment and License Server

6 Reference Architecture: Splunk Enterprise with ThinkSystem Servers version 1.0

5 Component Model This section describes the component model for Splunk Enterprise. Figure 2 shows an overview of the major components.

Figure 2: Component Model of Splunk Enterprise

5.1.1 Forwarders Forwarders collect data and send it to a Splunk deployment for indexing and searching. A particular environment could have thousands of forwarders executing on all different types of hardware. A forwarder represent a more robust solution than raw network feeds, with capabilities to

Tag metadata Buffer compress and secure data Run local scripts to collect or massage the data Use any available network ports on the remote device

5.1.2 Indexers The indexer is the Splunk Enterprise component that creates and manages indexes. The primary functions of an indexer are:

Indexing incoming data.

Searching the indexed data.

Forwarder

Data Routing, Cloning and Load Balancing

Indexer

DeploymentServer

Search Head

REST Protocol

Splunk CLISplunk Web

Server

Lenovo XClarity

App

SplunkDeployment

Monitor App

HTTP Protocol

Web Browser

Other Apps

LicenseServer

7 Reference Architecture: Splunk Enterprise with ThinkSystem Servers version 1.0

5.1.3 Search heads For large amounts of indexed data and numerous users concurrently searching on the data, it can make sense to distribute the indexing load across several indexers, while offloading the search query function to a separate machine. In this type of scenario, known as distributed search, one or more Splunk Enterprise components called search heads distribute search requests across multiple indexers.

5.1.4 Deployment server Splunk Enterprise deployment server is used to update a distributed deployment. The deployment server pushes out configurations and content to sets of Splunk Enterprise instances (referred to, in this context, as deployment clients), grouped according to any useful criteria, such as OS, machine type, application area, location, and so on. The deployment clients are usually forwarders or indexers....

Recommended

View more >