cisco ucs and splunk workshop
Post on 16-Apr-2017
Embed Size (px)
Copyright 2015 Splunk Inc.
AgendaIntroduction How Cisco IT Operations Uses Splunk Operational Intelligence Splunk quick overview Splunk on UCS 6.3 and results on UCS Splunk IT Ops Demo
Ciscos Footprint with Splunk70+ Monitored Applications7+ Year RelationshipAcross 7 Global Data CentersFlexible infrastructure to accommodate new business needs
Applying Splunk to Cisco IT RequirementsAggregated multiple siloed systems into SplunkMonitoring 70+ Applications846% increase of search volume per day in one yearOperational Intelligence in minutes rather than hoursCisco IT uses Splunk to index a broad range of system logs and machine data for networking devices, operating systems, unified communications, video events, and applications. Proactive monitoring enables 50% reduction in high priority issues80% reduction in operational costs90% improvement in problem resolution & root cause analysis timesImprovements in system stability, availability and performance
Ciscos growth with Splunk has consistently grown over the past 3 years. 18 applications in 2011 and now we have over 70+ applications.
As in many Splunk deployments, as Splunk proves vital for one use case, we found additional use cases that it could help address.
Our increase in the past year was 846% for search volumes (4/1/2014 compared to 4/1/2015).
You know, we dont always know what we are looking for, but the speed and scalability of Splunk lets us iterate in minutes rather than hours, avoiding dead ends and finding solutions.
For routine operations, the Cisco Security Monitoring, Analysis, and Response System (CS-MARS) was no longer sold by Cisco.
Insights Across Cisco - Platform BusinessUnitPlatformSPLUNK AppSources and LogsSYSLOGWindowsActive DirectoryACSStorageInfra StructureIT OPSSecurityCommerceSales & MarketingChannelsEngineeringWebexCCIX (web + app)FTPRAC DBWSGPINGOBIEEACESplunk on SplunkDeployment MonitorUCS AppJMX AppUnix AppNetApp AppNetworkLinux / UnixUCSVMWare ESXiDatacenter battery / temperature logsPre-Prod Event LogsProduction Event LogsEvent LogsEvent LogsAAA LogsISE LogsEvent Logs
Search HeadsIndexersStorageData Center16 VMs (64 core X 32 GB) 20 VMs (16 core X 16 GB)70 + Unique Indexes56 TB SAN Hot & Warm28 TB NAS - ColdProd: RCDN 8 SH & 10 IndexersProd: ALLEN 8 SH & 10 IndexersDev: RTP 4 SH & 2 indexers
~2TB indexed daily40,000+ devices sending data to splunk5,000+forwarders 70+ unique indexesData from any sourcetype available- Appliances-Linux-Windows-Environmental logs from the data center such as temperature and humidity6
10 Indexers16 Search Heads 47 Search Heads20 IndexersDaily Indexing ~ 2TB20142014201520152015Ciscos IT Operations Evolving with SplunkDaily Indexing 300G2010
7High-level architecture: IT: - Grown from 10 indexers in 2014, to 20 today- By using cluster search head, with load balancing (introduced in Splunk Enterprise 6.2), Cisco was able to condense from 47 search heads to 16. - Cisco previously deployed a search head for each LOB and consolidation ensued.
Volume Growth: Daily indexing: 300G in trials circa 2010, ~2TB per day in 2015
The last version of Splunk introduced clustered search heads. This is a lotlike load balancing. Easier to scale multiple workloads and users this way. Also, I believe part of the earlier generation of the Splunk deployment at Cisco had separate search heads per customer / user. The goal was to build a multi-tenant system for different internal Cisco use cases. Each would need their own search head. I believe this is being re-engineered. The significance is that more server resources can be put toward indexersthe storage and preprocessing side. As an environment grows, you scale the two components such: Search heads like a web front-end scale by search volume/number of concurrent users (more or less) Indexers like a back-end database or storage server scale by ingest volume/amount of data being stored in SplunkIn the CSIRT example, beginning of 2014 (I think) had 650GB of data, and middle of 2015 (i.e. April 1, 2015) had almost 2000GB thats 3x growth. More important to scale the indexers for this; query volume and user base havent grown 3x in all likelihood.
Splunk Activity Daily Average
1. Interactive Searches = 55K+2. Scheduled Searches = 45K+3. Total Searches = 100K+4. Number of Users = 180+
Snapshot from earlier this year but could have really been taken anytime.
This shows the growth trend mentioned earlier
Over 500 unique users per month (note that the graph shows daily users, not monthly uniques)8
A look at our pre 6.2 environmentInitially a search head pool was deployed for each client team that was integrated. Which was fine in the beginning.47 SHs and 12 SHPsPainpoints:An administration nightmareResource availability Lots of compute dedicated overall for search heads but not its not always available where needed 9
Current 6.2 based setup was built side by side with the existing pre 6.2 environment SHPsMigrated each client team over one by oneIf we had kept out heads down and didnt know of the new features we would have continued down the same path that lead to headachesAnd with that, back to Robert, TY!10
Replacing Legacy SIEM at Cisco CSIRTEnter Splunk: Flexible SIEM and empowered team
Easy to index any type of machine data from any source Over 60 users doing investigations, correlations, reporting, advanced threat detection All the data + flexible searches and reporting = empowered team 2TB/day and searches take less than a minute. 7 global data centers with 350TB stored data Flashback Malware ExampleEstimate Splunk is 25% the cost of a traditional SIEM
Easy to index any type of machine data coming in from anywhere60 users 7x24 around the globe investigating and reportingMassive amounts of data combined with flexible searches empowered the team at Cisco2TB a day and searches take less than a 1 minute. 25% cost multi-purpose tool, its not a dedicated niche or point product, Cisco saves money, does SIEMs + Much more (Swiss Army Knife does many things).
We moved to Splunk from traditional SIEM as Splunk is designed and engineered for big data use cases. Our previous SIEM was not and simply could not scale to the data volumes we have Former Director, Cisco Computer Security Incident Response Team
Ciscos footprint of security monitoring spans across our 7 global data centers with searches taking less than a minute. A search could be anything from known exploits to uncommon error messages. 11
33 percent reduction in the time required to conduct security investigations
All security data is readily available in a single, centralized portal for faster and simpler access Ability to automate routine tasks and search log data allows CSIRT analysts to work more effectivelySubstantially easier correlation allows for more thorough investigations
HeadingCisco Security Analytics Results
12Incorporating Splunk data was key. Improvements in operations visibility, intelligence, and efficiency have produced significant results for Cisco including:
33 percent reduction in the time required to conduct security investigations
All security data is readily available in a single, centralized portal for faster and simpler access
Substantially easier correlation allows for more thorough investigations
Ability to automate routine tasks and search log data allows CSIRT analysts to work more effectively
240+ security apps & add-onsSplunk app for Enterprise SecuritySplunk Apps for Cisco Environments
Cisco ASANetFlow LogicOSSECCisco WSA
Cisco ESACisco ISESourcefire
Active DirectoryCisco Security Suite MobileIronBit9 ETDNorse Darklist
600+ apps/add-onsCisco ACI, IOS, Nexus 9000Cisco UCS
Counts are based on splunkbase.splunk.com as of 3:45pm PT 11/10/2015, searching for apps and add-ons qualified for Splunk 6.0 and later. 25 cisco matches. 242 security category entries. 631 total 6.x listings.
Splunk Enterprise, the core product. Every Splunk deployment includes this and is where the core indexing and searching resides. Many customers build their own searches/reports/dashboards on top of it. Optional Apps can be installed and these Apps are basically a collection of reports, dashboards, and searches purpose-built for a specific use case or product to extend point solutions.
Apps are great for customers who want out-of-the-box content, do NOT want to have to build it themselves, and want to extend point solutions.
Cisco is the top search term YES, Cisco is among most popular apps. 27 total Cisco related applications, 6 Security specific
Custom apps for ASA firewalls, ESA, WSA, ISE, Sourcefire plus umbrella apps for Cisco Security Suite.Just the starting point customers leverage the API and SDKs that come with Splunk to further extend the platform.
Microsoft Exchange Example This App is a good example because a lot of people use Exchange and want to monitor it but with a dedicated App/Add-on, every customer does not have to re-invent the wheel.
Over 700 total apps and 160 security apps/add-ons available in Splunk
Splunk App for Cisco UCSNEW AND IMPROVED as of May 2015Aggregates, monitors, trends and analyzes all relevant data from Cisco UCS Manager instancesEnables proactive capacity and performance monitoring/ management, fault trending, power and cooling, and moreWorks with other Splunk add-ons and data sou