aws re:invent 2016: predictive security: using big data to fortify your defenses (sac304)
TRANSCRIPT
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Michael Capicotto, Solutions Architect
Matt Nowina, Solutions Architect
November 30, 2016
SAC304
Predictive SecurityUsing Big Data to Fortify Your Defenses
Cybersecurity headlines from 2015…
...Over 169 million personal records were exposed, stemming from 781
publicized breaches across the financial, business, education,
government and healthcare sectors.
...There were 38 percent more security incidents detected than in 2014.
...The median number of days that attackers stay dormant within a
network before detection is over 200.
... 81 percent reported they had neither a system nor a managed security
service in place to ensure they could self-detect data breaches, relying
instead on notification from an external party.
... Only 38 percent of global organizations claim they are prepared to
handle a sophisticated cyberattack.
You will learn how to…
Build a log analytics stack with Amazon Elasticsearch
Service
Utilize Amazon Machine Learning to predict bad actors
Perform forensic analysis on your network paths
Implement advanced options in your continuous,
predictive security stack
Big Data – Logs, logs everywhere
?Nobody looks at
them!
Big Data – Logs, logs everywhere…isn’t always good
Build a log analytics stack
Log sources in AWS
AWS CloudTrail logs OS and application
logs
VPC flow logs Amazon CloudWatch Logs
Setting up a log analytics stack
CloudWatch Logs Amazon Elasticsearch
Service
AWS Lambda
Demo #1 – Elasticsearch and Kibana
Awesome, we can see stuff!
Now we have real-time visualization of all logs
Great for risk scenarios we
already know about!
Example – Single user logging in from
several IP addresses
Not so great for unknown
scenarios
There are many of these!
How do we protect against these risks?
Integrating machine learning
Amazon Machine Learning
Easy to use,
managed machine
learning service built
for developers
Robust, powerful
machine learning
technology based on
Amazon’s internal
systems
One-click production
model deployment
Binary classification
Multiclass classification
Regression
Using Amazon Machine Learning’s real-time predictions, we
can drastically shorten how long it takes you to become aware
of a threat
Training your model (daily)
Amazon S3
Stores machine
learning dataset
AWS Lambda
Daily machine
learning model
training
Amazon Machine
Learning
Build model from
dataset
Log analytics
stack
AWS Lambda
Transform and
store logs in S3
Using Big Data – Example dataset
{
"datetime": "7/30/16 0:20",
"AWSregion": "aws-sa-east-1",
"IP": "69.90.60.155",
"protocol": "TCP",
"source": "6000",
"destination": "1433",
"country": ”BrVirginIslands",
"region": ”PricklyPear",
"postalcode": ”VG1120",
"Lat": ”18.5000",
"Long": ”64.3667”,
"Threat": 94
}
Real-time predictions
Amazon Machine
Learning
Endpoint for real-
time predictions
Log analytics
stackAWS Lambda
Trigger on each
new log entry
Amazon SNS
notification
Demo #2 – Real-time ML predictions
Security stack
Amazon Machine
Learning
Trained model and
endpoint for real-
time predictions
Log analytics
stackAWS Lambda
Trigger on each
new log entry
Amazon SNS
notification
Amazon S3
Stores machine
learning dataset
AWS Lambda
Daily machine
learning model
training
AWS Lambda
Transform and
store logs in S3
Close, but not perfect!
We still wont catch every potential breach Machine learning cannot predict every possible threat
Attackers are getting smarter and more sophisticated every day
When one does occur, we want to know why This helps us prevent it from happening again!
Forensic analysis
AWS Production Account
us-east-1a
us-east-1b
Pro
xie
s
NAT
RDS DB
DM
Z S
ub
ne
t
Priv
ate
Su
bn
et
Priv
ate
Su
bn
et
Pro
xie
s
Bastion
RDS DB
Priv
ate
Su
bn
et
Priv
ate
Su
bn
et
Virtual Private Cloud (VPC)
Network sprawl
AWS API Account
us-east-1a
us-east-1b
Priv
ate
Su
bn
et
Priv
ate
Su
bn
et
Virtual Private Cloud (VPC)
Reasoning about networks
Web service and CLI
available in private
beta
Answers questions
about your network
No packets sent
?
Demo #3 – Network reasoning
Demo
Advanced options
Evolving the practice of security architecture
Security architecture as a separate function can no longer
existStatic position papers,
architecture diagrams, and
documents
UI-dependent consoles and technologies
Auditing, assurance, and
compliance are decoupled,
separate processes
Current security
architecture
practice
Evolving the practice of security architecture
Architecture artifacts
(design choices, narrative,
etc.) committed to common
repositories
Complete solutions account for automation
Solution architectures are
living audit/compliance
artifacts and evidence in a
closed loop
Evolved security
architecture
practice
AWS
CodeCommit
AWS
CodePipeline Jenkins
Security architecture can now be part of the “maker” team
Continuous monitoring and auto-remediation
Self-managed AWS CloudTrail -> Amazon CloudWatch Logs -> Amazon CloudWatch Alerts
AWS CloudTrail -> Amazon SNS -> AWS Lambda -> Network reasoning
Compliance validation AWS Config Rules
Host-based compliance validation Amazon Inspector
Active change remediation Amazon CloudWatch Events
More sophisticated machine learning models
Train your model with your data Real-world data specific to your application
Previous threats you have dealt with
Considering modeling threats by clusters of logs Identify threats more accurately than just a single log entry
Build threat profiles that pattern typical attack stages Reconnaissance, scanning, gaining access, maintaining access, and
covering tracks
Tying it all together
Amazon Machine
Learning
Trained model and
endpoint for real-
time predictions
Log analytics
stackAWS Lambda
Trigger on each
new log entry
Amazon SNS
notification
Amazon S3
Stores machine
learning dataset
AWS Lambda
Daily machine
learning model
training
AWS Lambda
Transform and
store logs in S3
AWS Config Rules
Network
reasoning
VPC, security groups,
network ACLs
Next steps
Set up your log analytics stack: http://amzn.to/2dIZjIz Blog post and AWS CloudFormation template
Build your first Amazon ML machine learning model:
http://amzn.to/1K8HfRu
Stay tuned on the AWS Security Blog for more on this
topic
We’re here all week! Come chat with us.
Thank you!
Remember to complete
your evaluations!