security analytics and big data: what you need to know
Post on 15-Jul-2015
235 Views
Preview:
TRANSCRIPT
David MonahanResearch Director
EMA
Security Analytics and Big Data: What You Need to
Know
Sameer Nori
Senior Product Marketing
Manager
MapR
Nick Amato
Director Technical
Marketing
MapR
© 2015 MapR Technologies 2
Today’s Presenters
Slide 2
David Monahan, Research Director, Risk & Security Management, EMA
David has over 15 years of IT security experience and has organized and managed both physical and
information security programs, including Security and Network Operations (SOCs and NOCs) for
organizations ranging from Fortune 100 companies to local government and small public and private
companies.
Sameer Nori, Senior Product Marketing Manager, MapR Technologies
Sameer has over ten years of experience in the technology industry in marketing, pre-sales, and
consulting, with domain experience in business intelligence, analytics, and big data.
Nick Amato, Director, Technical Marketing, MapR Technologies
Nick works with the MapR ecosystem and technology partners to identify new opportunities where the
MapR platform can bring value to customers. His areas of focus include third-party integrations with BI
tools, benchmarking, architecture, and enabling scalable data platforms.
© 2015 MapR Technologies 3
Logistics for Today’s Webinar
Slide 3 © 2015 Enterprise
A PDF of the PowerPoint
presentation will be available
An archived version of the event
recording will be available at
www.enterprisemanagement.com
• Log questions in the Q&A panel located on
the lower right corner of your screen
• Questions will be addressed during the
Q&A session of the event
Questions
Event recording
Event presentation
David Monahan
Research Director, Security and Risk Management
Enterprise Management Associates
http://www.enterprisemanagement.com
@SecurityMonahan
The Convergence of
Security Analytics and Big Data
April 27, 2015
© 2015 MapR Technologies 5
Threats Come From Everywhere
• Hacking: The mentality has changed
• Data breaches affect every industry
• Organizations are being attacked from all sides
– External threats
– Insider threats
• All information is up for grabs
Slide 5 © 2015 Enterprise
© 2015 MapR Technologies 6
Identifying Threats is Harder Than Ever
Slide 6 © 2015 Enterprise
EMA research identified several troubling statistics about
identifying and responding to threats:
of organizations were between “Highly Doubtful”
and only “Somewhat Confident” that they could
detect an important security issue before it had a
significant impact.
of organizations believe they
are consistently successful in
in correlating security data to
business impact.
of organizations said they
were unable to stop exploits
because of outdated or
insufficient threat intelligence.
69% 22%
60%
41%
28%
33%
29%
TOO DIFFICULT SEPARATING LEGITIMATEFROM MALICIOUS ACTIVITY
TOO DIFFICULT PRIORITIZINGREMEDIATION ACTIVITIES
INABILITY TO REPORT MEANINGFULINFORMATION TO STAKEHOLDERS
INSUFFICIENT TOOLING TOSUPPORT SECURITY DUTIES
Top frustrations with IT Security Practices:
© 2015 MapR Technologies 7
The Problem Requires Better Data and Better Tools
• Data volumes are too high
– EMA research identified that 45% of organizations are collecting
more than 40GB/day of logs
– Nearly 16% are collecting over 500GB/day of logs
• Data correlation and normalization is not sufficient
– Organizations are fielding 100:1 high priority and greater alerts per
person in security
• Operations, Analysts, and Responders need better context
and Higher Fidelity (Ponemon Study)
– Actionable Intelligence within 60 seconds reduced breach resolution
costs by an average of 40%
Slide 7 © 2015 Enterprise
© 2015 MapR Technologies 8
The Problem Requires Better Data and Better Tools (cont’d)
• Persistent threats and their complexity is expanding rapidly
– Criminal organizations are creating new and better attacks
• [Gameover] Zeus (Botnet and data theft)
• Crypto-Locker/Wall, CTB-Locker (data theft)
• Dexter, POSLogr, BlackPOS (Point of Sale Terminal malware)
– The Nations states show criminals virtually anything is possible
• StuxNet malware (Supervisory Control and Data Acquisition (SCADA)
malware)
• Direct Memory Access Video RAM malware
• TAO- Micro processor embedded malware (network sniffing, key logging,
data collection, remote access, etc.)
• “nls_933w.dll”- Hard drive Firmware embedded malware (anything)
Slide 8 © 2015 Enterprise
© 2015 MapR Technologies 9
The Problem Requires Better Data and Better Tools (cont’d)
• EMA Research has identified key issues with current tools
Most Significant Frustrations with IT Security Technologies
Slide 9 © 2015 Enterprise
38%
36%
35%
LACK OF INTEGRATION/INTEROPERABILITY
TOOLS UNABLE TO RECOGNIZE EMERGING THREATS/ATTACKS
VENDORS ARE SLOW TO RESPOND TO EMERGING THREATS OR ATTACKS
© 2015 MapR Technologies 10
SIEM Limitations
Slide 10 © 2015 Enterprise
SIEM technology provides real-time analysis of security alerts generated by network hardware and applications.
This is limited “analysis” based primarily upon
correlation and normalization of alerts.SIEM only understands deltas for those things inside of
its defined rules or policies
SIEM understands network information and log entries
to correlate events at a network level and identify
system/application alerts.
SIEM does not understand human, system, and
application specific activity and patterns (behaviors) to
determine how some activities raise the threat level.
Post notification SIEM often requires manual investigation*.
* EMA research found 55% of organizations said they still conduct
manual incident investigations
© 2015 MapR Technologies 11
SIEM Limitations (cont’d)
What features is your organization not getting from SIEM tools that it is looking for in Security Analytics technology/products?
Slide 11 © 2015 Enterprise
65%
53%
51%
ADVANCED AUTOMATED RESPONSE CAPABILITIES
INCREASED ABILITY TO EASILY AGGREGATE AND CROSS
ANALYZE DATA FROM NON-SECURITY SOURCES (IENETFLOW, WEB ACCESS LOGS)
ENHANCED DATA VISUALIZATION
© 2015 MapR Technologies 12
Poll Question #1
Have you heard of Security Analytics or
Security Intelligence as a solution?
A. Have not heard of it
B. Believe they are the same as SIEM
C. Deployed a security analytics solution
D. Considering security analytics in the next 6-12 months
Slide 12 © 2015 Enterprise
© 2015 MapR Technologies 13
Moving to Security Analytics
Slide 13 © 2015 Enterprise
Security Analytics Improvements
Better context and fidelity Reduce false positives
Reduce alert volumes Provide better prioritization
Accelerate Incident Response
of organizations using Security Analytics have
seen a reduction in false positives or an
improvement in actionable alerts since they
implemented a Security Analytics technology.
of organizations that use
Security Analytics said that the
tool produced expected or
greater than expected value.
90% 95%
© 2015 MapR Technologies 14
Why Security Analytics
Which of the following are your organization’s views or reasons why it needs/uses capabilities for advanced analytics or security data management for IT/information
security?
Slide 14 © 2015 Enterprise
53%
46%
43%
36%
IMPROVES DEFENSE AGAINST TARGETED THREATS
INCREASES OPERATIONAL EFFICIENCIES DEMONSTRATING HIGHER
SECURITY EFFECTIVENESS TO THE BUSINESS
IMPROVES PRODUCTIVITY/EFFICIENCY OF IT SECURITY EFFORTS
IMPROVES STRATEGIC DECISION MAKING
© 2015 MapR Technologies 15
Why Hadoop for Security Analytics
• We need tools that can handle more data and a wider variety of data.
– When asked if they would collect more data or a wider variety of data if they
could, 66% of organizations said they would. (Only 10% said they would not.)
– EMA Research - 57% of organizations said that they expect the greatest
improvements in security through data analysis to come from innovations from
IT security technologies and their vendors.
– For true fidelity we need to be able to combine ALL information relevant to
data management.
• User, system, application, network packet/netflow, infrastructure logging, HR records,
endpoint, et. al.
• EMA Research - 32% of organizations indicated they wanted to be able to analyze
unstructured data for use in security.
Slide 15 © 2015 Enterprise
© 2015 MapR Technologies 16
Benefits of Hadoop for Security Analytics
• Purpose-built for processing large amounts of data
• Designed for unstructured data analysis
• Business Analytics can be applied to security use cases
• Increased ROI from a tool that supports both Business Intelligence and Security Operations
Slide 16 © 2015 Enterprise
47%
36%
35%
35%
MACHINE LEARNING TOOLS
FRAUD MANAGEMENT OR DETECTION SYSTEM
BUSINESS INTELLIGENCE (BI) PLATFORM
ENTERPRISE DATA WAREHOUSES
Which of the following non-traditional data sources are currently NOT included/supported by your
organizations current SIEM or log management system?
© 2015 MapR Technologies 17© 2015 MapR Technologies
Security Log Analytics on MapR
© 2015 MapR Technologies 18
Zions Bank: Security Analytics and Fraud DetectionCost effective security analytics and fraud detection on one platform
• Fraud Operations and Security Analytics team at Zions maintains data stores, builds
statistical models to detect fraud, and then uses these models to data mine and
evaluate suspicious activity
“We initially got into centralizing all of our data from an information security perspective. We then saw
that we could use this same environment to help with fraud detection”Michael Fowkes - SVP Fraud Operations and Security Analytics
• Existing technology infrastructure could not scale
• Timeliness of reports degraded over the last several years
• Chose MapR and cut storage costs by 50%
• Querying time reduced from 24 hours to 30 min on 1.2 PB of data
• Leverage MapR scale for increased model accuracy and deeper insights
OBJECTIVES
CHALLENGES
SOLUTION
Business Impact
© 2015 MapR Technologies 19
Zions Bank with MapR – Faster Operations at Lower Costs
Web Server
Data
Transactional
Data
3rd Party Real Time
Fraud Detection
Reporting and
Batch Analytics
Deeper Analysis with
Machine Learning
PRD and Dev on
MapR
N
F
S
Technical Benefits
High availability
Multi-tenancy
Snapshots
Performance
Business Benefits
Unified platform for data
Lower operating costs
Operational guarantees
Faster model development
© 2015 MapR Technologies 20
Solutionary: Managed Security Services ProviderThreat detection on real-time streaming data via platform as a service
• To address their growing customer base by processing trillions of messages (petabyte)
per year while continuing to provide reliable security services
• To improve data analytics by leveraging newer, more granular unstructured data
sources
”MapR has taken Apache Hadoop to a new level of performance and manageability. It integrates into
our systems seamlessly to help us boost the speed and capacity of data analytics for our clients.”
- Dave Caplinger, Director of Architecture, Solutionary
• Expanding existing database solution to meet demand was cost prohibitive
• The existing technology could not process unstructured data at scale
• Replaced RDBMS with MapR Enterprise Database Edition to scale Reduced time
needed to investigate security events for relevance and impact
• Improved data analytics, enabling new services and security analytics
• 2x faster performance compared to competing solutions
OBJECTIVES
CHALLENGES
SOLUTION
Business Impact
Leader in Magic Quadrant
© 2015 MapR Technologies 21
Why MapR for Security Analytics
Business
• Large scale and deep analytics on security data to reduce risk
• Early detection of advanced persistent threats and unknown threats
• React fast on any abnormal or malicious activity from internal and external actors
• Avoid fines, lawsuits, loss of business and negative PR
Technical
• Build a data vault for security event logs from multiple sources
• With more data to scrutinize, get insights into anomalous behavior and close loop with other security solutions
• Platform that enables analysis of both historical data as well as real-time analysis of large volumes of security data
Operations
• Fast ingestion of large volume of data and perform deep analytics
• Easy integration with existing IT ecosystem
• Low overhead to maintain system
• Early detection of threats and closed loop feedback with existing security solutions
© 2015 MapR Technologies 22
The MapR Advantage
• Scale Reliability Across the Enterprise– Advanced multi-tenancy
– Business continuity – HA, DR
• Speed– 2-7x faster than other Hadoop distributions
– Ultra-fast data ingest (100M data points per sec)
– NFS & R/W file system
• Real-time & Self-Service Data Exploration– On-the-fly SQL without up-front schema
– Fast lookups and queries
Best Hadoop Platform for Security Log Analytics
Security
Streaming
NoSQL & Search
Provisioning &
coordination
ML, Graph
W orkflow & Data Governance
Batch
SQL
INTEGRATED
COMMERCIAL
ENGINES
TOOLSCOMPUTE
ENGINES
Batch
Interactive
Real-time
Online
Others
Management
Operations
Governance
Audits
Security
MapR-FS MapR-DB
MapR Data Platform
© 2015 MapR Technologies 23
Poll Question #2
Do you use Hadoop for Security Analytics?
A. No, didn’t know it could be used for Security Analytics.
B. Yes, it's been 6 months or less.
C. Yes, it’s been deployed for 12 months or more.
D. No, but considering it in the next 6-12 months.
© 2015 MapR Technologies 24
What’s in the Quick Start Solution
6 nodes of
MapR software
2 week
engagement
3 Hadoop
Professional
Certifications
© 2015 MapR Technologies 25
Quick Start Service Engagement
Engagement includes:
1. Identification of data sources, transformations and reporting engines
2. Access and use of the solution template including source code
3. Training on customizing the solution template to the organization’s requirement
4. Deployment architecture document that enables a production deployment plan for the specific solution
SOLUTION
TEMPLATE
KNOWLEDGE
TRANSFERDEPLOYMENT
ARCHITECTURE
© 2015 MapR Technologies 26
Components of the Solution Template
• Data Workflows
– Read/collect input data
– Handle bulk load and streaming use cases
• Parsers and Enrichment
– Process input data (filtering and deriving additional data as needed)
– Storing in one or more data types or formats
• Machine learning
– Clustering analysis
– Reservoir sampling analysis
INTEGRATED
COMMERCIAL
ENGINES
TOOLSCOMPUTE
ENGINES
MapR Data Platform
© 2015 MapR Technologies 27
The Power of the Open Source Community
APACHE HADOOP AND OSS ECOSYSTEM
Security
YARN
Spark Streaming
Storm
StreamingNoSQL & Search
Juju
Provisioning &
Coordination
Sahara
ML, Graph
Mahout
MLLib
GraphX
EXECUTION ENGINES DATA GOVERNANCE AND OPERATIONS
Workflow & Data
Governance
Pig
Cascading
Spark
Batch
MapReduce v1 & v2
Tez
HBase
Solr
Hive
Impala
Spark SQL
Drill
SQL
Sentry Oozie ZooKeeperSqoop
Flume
Data Integration& Access
HttpFS
Hue
Data PlatformMapR-FS MapR-DB
Manag
em
ent
© 2015 MapR Technologies 28
MapR: Best Solution for Customer Success
Premier
InvestorsHigh Growth
2X Growth In Direct Customers
90%Subscription Licenses
Software Margins
140%
Dollar-based Net Expansion
700+ Customers
2X Growth In Annual
Subscriptions ( ACV)
Best Product
Apache Open
Source
© 2015 MapR Technologies 29
Security Log Analytics Template
MapR-FS
MapR-DB
© 2015 MapR Technologies 30
Resources
https://www.mapr.com/solutions/quickstart/hadoop
-security-log-analytics-quick-start
– Research Report: The Evolution of Data Driven
Security
– Solution Brief: Jump-Start Security Log Analytics
© 2015 MapR Technologies 31
Free on-demand
Hadoop training leading to certification
Start becoming an expert now
mapr.com/training
50MIn Free Training
© 2015 MapR Technologies 32
Q & A
@mapr maprtech
sales@mapr.com
Engage with us!
MapR
maprtech
mapr-technologies
© 2015 MapR Technologies 33© 2015 MapR Technologies
Security Log Analytics on MapR
© 2015 MapR Technologies 34
Zions Bank: Security Analytics and Fraud DetectionCost effective security analytics and fraud detection on one platform
• Fraud Operations and Security Analytics team at Zions maintains data stores, builds
statistical models to detect fraud, and then uses these models to data mine and
evaluate suspicious activity
“We initially got into centralizing all of our data from an information security perspective. We then saw
that we could use this same environment to help with fraud detection”Michael Fowkes - SVP Fraud Operations and Security Analytics
• Existing technology infrastructure could not scale
• Timeliness of reports degraded over the last several years
• Chose MapR and cut storage costs by 50%
• Querying time reduced from 24 hours to 30 min on 1.2 PB of data
• Leverage MapR scale for increased model accuracy and deeper insights
OBJECTIVES
CHALLENGES
SOLUTION
Business Impact
© 2015 MapR Technologies 35
Zions Bank with MapR – Faster Operations at Lower Costs
Web Server
Data
Transactional
Data
3rd Party Real Time
Fraud Detection
Reporting and
Batch Analytics
Deeper Analysis with
Machine Learning
PRD and Dev on
MapR
N
F
S
Technical Benefits
High availability
Multi-tenancy
Snapshots
Performance
Business Benefits
Unified platform for data
Lower operating costs
Operational guarantees
Faster model development
© 2015 MapR Technologies 36
Solutionary: Managed Security Services ProviderThreat detection on real-time streaming data via platform as a service
• To address their growing customer base by processing trillions of messages (petabyte)
per year while continuing to provide reliable security services
• To improve data analytics by leveraging newer, more granular unstructured data
sources
”MapR has taken Apache Hadoop to a new level of performance and manageability. It integrates into
our systems seamlessly to help us boost the speed and capacity of data analytics for our clients.”
- Dave Caplinger, Director of Architecture, Solutionary
• Expanding existing database solution to meet demand was cost prohibitive
• The existing technology could not process unstructured data at scale
• Replaced RDBMS with MapR Enterprise Database Edition to scale Reduced time
needed to investigate security events for relevance and impact
• Improved data analytics, enabling new services and security analytics
• 2x faster performance compared to competing solutions
OBJECTIVES
CHALLENGES
SOLUTION
Business Impact
Leader in Magic Quadrant
© 2015 MapR Technologies 37
Why MapR for Security Analytics
Business
• Large scale and deep analytics on security data to reduce risk
• Early detection of advanced persistent threats and unknown threats
• React fast on any abnormal or malicious activity from internal and external actors
• Avoid fines, lawsuits, loss of business and negative PR
Technical
• Build a data vault for security event logs from multiple sources
• With more data to scrutinize, get insights into anomalous behavior and close loop with other security solutions
• Platform that enables analysis of both historical data as well as real-time analysis of large volumes of security data
Operations
• Fast ingestion of large volume of data and perform deep analytics
• Easy integration with existing IT ecosystem
• Low overhead to maintain system
• Early detection of threats and closed loop feedback with existing security solutions
© 2015 MapR Technologies 38
The MapR Advantage
• Scale Reliability Across the Enterprise– Advanced multi-tenancy
– Business continuity – HA, DR
• Speed– 2-7x faster than other Hadoop distributions
– Ultra-fast data ingest (100M data points per sec)
– NFS & R/W file system
• Real-time & Self-Service Data Exploration– On-the-fly SQL without up-front schema
– Fast lookups and queries
Best Hadoop Platform for Security Log Analytics
Security
Streaming
NoSQL & Search
Provisioning &
coordination
ML, Graph
W orkflow & Data Governance
Batch
SQL
INTEGRATED
COMMERCIAL
ENGINES
TOOLSCOMPUTE
ENGINES
Batch
Interactive
Real-time
Online
Others
Management
Operations
Governance
Audits
Security
MapR-FS MapR-DB
MapR Data Platform
© 2015 MapR Technologies 39
Poll Question #2
Do you use Hadoop for Security Analytics?
A. No, didn’t know it could be used for Security Analytics.
B. Yes, it's been 6 months or less.
C. Yes, it’s been deployed for 12 months or more.
D. No, but considering it in the next 6-12 months.
© 2015 MapR Technologies 40
What’s in the Quick Start Solution
6 nodes of
MapR software
2 week
engagement
3 Hadoop
Professional
Certifications
© 2015 MapR Technologies 41
Quick Start Service Engagement
Engagement includes:
1. Identification of data sources, transformations and reporting engines
2. Access and use of the solution template including source code
3. Training on customizing the solution template to the organization’s requirement
4. Deployment architecture document that enables a production deployment plan for the specific solution
SOLUTION
TEMPLATE
KNOWLEDGE
TRANSFERDEPLOYMENT
ARCHITECTURE
© 2015 MapR Technologies 42
Components of the Solution Template
• Data Workflows
– Read/collect input data
– Handle bulk load and streaming use cases
• Parsers and Enrichment
– Process input data (filtering and deriving additional data as needed)
– Storing in one or more data types or formats
• Machine learning
– Clustering analysis
– Reservoir sampling analysis
INTEGRATED
COMMERCIAL
ENGINES
TOOLSCOMPUTE
ENGINES
MapR Data Platform
© 2015 MapR Technologies 43
The Power of the Open Source Community
APACHE HADOOP AND OSS ECOSYSTEM
Security
YARN
Spark Streaming
Storm
StreamingNoSQL & Search
Juju
Provisioning &
Coordination
Sahara
ML, Graph
Mahout
MLLib
GraphX
EXECUTION ENGINES DATA GOVERNANCE AND OPERATIONS
Workflow & Data
Governance
Pig
Cascading
Spark
Batch
MapReduce v1 & v2
Tez
HBase
Solr
Hive
Impala
Spark SQL
Drill
SQL
Sentry Oozie ZooKeeperSqoop
Flume
Data Integration& Access
HttpFS
Hue
Data PlatformMapR-FS MapR-DB
Manag
em
ent
© 2015 MapR Technologies 44
MapR: Best Solution for Customer Success
Premier
InvestorsHigh Growth
2X Growth In Direct Customers
90%Subscription Licenses
Software Margins
140% Dollar-based Net Expansion
700+ Customers
2X Growth In Annual
Subscriptions ( ACV)
Best Product
Apache Open Source
© 2015 MapR Technologies 45
Security Log Analytics Template
MapR-FS
MapR-DB
© 2015 MapR Technologies 46
Find more Resources on MapR.com or …
Research Report
The Evolution of
Data Driven
Security
Solution Brief
Jump-Start
Security Log
Analytics
Webinar Recording
Security Analytics and
Big Data: What You
Need to Know
top related