josh patterson, principal visualizing 200m alerts mike ... · • data: query billions of alerts...

Attack Graphs: Visualizing 200M Alerts a Day with GPU Clouds and JavaScript

Josh Patterson, Principal Mike Wendt, Principal

Leo Meyerovich, CEO

Silicon Valley • Digital Experiences • Artificial Intelligence • Platforms & Systems

Washington DC •  Security

Dublin • Artificial Intelligence

Sophia Antipolis •  Industry Innovation (FS & Resources)

Beijing •  Industrial Internet

Bangalore • Software Engineering

Expanding Global Presence

Tel-Aviv • Security

For more than 20 years, Accenture Labs has served as the tip of the spear for technology innovation at Accenture.

Over the last 5 years Accenture Labs has: •  Supported 300+ client engagements and hosted 1100+ client workshops •  Published 200+ thought leadership pieces, filed 110+ patent applications, and garnered 350+ Tier-1 media hits

2 Copyright © 2016 Accenture All rights reserved.

3

•  a

a

•  GPU client/cloud acceleration to visually analyze 1M+ events or entities

•  Platform for enterprise-scale IT sec/ops analysts

•  Toolkit for data developers

•  Pilots with Federal, enterprises, & startups

3 Copyright © 2016 Graphistry All rights reserved.


Security Data Science is Hard

Pace of change

New technologies

Changing workforce models

Increased connectedness

Huge scale of software operations

Expanded attack

surfaces

More sophisticated, higher volume

of attacks

Ongoing privacy

concerns


Security Data Science is Hard

Once the security community moves beyond the mantras “encrypt everything” and “secure the perimeter,” it can begin developing intelligent prioritization and response plans to various kinds of breaches – with a strong focus on integrity. http://www.wired.com/2015/12/the-cia-secret-to-cybersecurity-that-no-one-seems-to-get/

Right now, financial services reports it takes an average of 98 days to detect an Advance Threat but retailers say it can be about seven months.

6

Enable Incubate Discover

Intellectual asset

licensing Joint Ventures

Products in-sourced for scale up

Intellectual assets insourced for development

Insourced ideas & technologies

Out to Market

Scale

ASGARD

ASGARD Rethinking Cyber Security Analytics Hunting

Streaming

Storage

Analytics

Visualization

Interaction

Copyright © 2016 Accenture All rights reserved.

7

Enable Incubate Discover

Intellectual asset

licensing Joint Ventures

Products in-sourced for scale up

Intellectual assets insourced for development

Insourced ideas & technologies

Out to Market

Scale

ASGARD

ASGARD Rethinking Cyber Security Analytics Hunting

Streaming

Storage

Analytics

Visualization

Interaction

Copyright © 2016 Accenture All rights reserved.

PEOPLE: Who Visually Analyzes & How?

SOC

“triage”

Response

“dig”

Forensics & Hunting “dig deep”

Escalation Chain



Escalation Chain

Freeform Notebooks

Premade Playbooks

Search Apps

Workflow automation


SOC

“triage”

Response

“dig”



Escalation Chain

Freeform Notebooks

Premade Playbooks

Search Apps

Workflow automation


SOC

“triage”

Response

“dig”


Today’s Topic

Security Visualization Stuck in 2000s: Dashboarding and Search

ü  Prebaked reports

x  Problem: summaries hide data

x  Can’t see entities and events

x  Can’t see patterns and anomalies

x  Hard to drill in and pivot

??


Security Visualization Stuck in 2000s: Dashboarding and Search

Search is a great starting point!

Lists don’t scale: Cannot see the 30K+ events nor

the IPs, users


Scale to modern enterprises •  Data: query billions of alerts

•  Visuals: 1 million devices under management, 1 million event hits, …

•  Intelligence: visual correlation to reveal patterns and anomalies

Explore at speed of thought •  Write fewer queries; interact visually & directly

•  Responsive: 10ms – 1s

GOAL: Security Visualization for the Data Era


Accenture Labs ASGARD Platform

Ingest

Event Processing

Storage

Notebooks Query Layer

Data Sources

Visualizations

SQL

Streaming

py



Ingest

Event Processing

Storage


Data Sources

Visualizations

SQL

Streaming

py

Today’s Topic


Two Case Studies

1.  HUNTING: Network mapping to drill into all priority 10 alerts

2.  RESPONSE: Botnet analysis to reveal full infection


Design Principles

Inspiration: A slider is worth 1000 queries

Practice: Augment the analysis loop with compute

•  See everything, & intelligently

•  Visually query, & quickly


GRAPHISTRY’S GPU PLATFORM: Pack Every Interaction with Magic

Optimized networking

GPU analysis & ML GPU rendering

(No JavaScript!)

GovCloud


Accelerate Every Component 10X+ with GPUs: Interactive Rendering

Goal •  Vector displays with perceptual features •  Rich, interactive labels •  1+ million entities @ 60 FPS Solution •  WebGL scene + managed HTML5 labels •  Client only receives geometry, changes •  Game engine tricks: bulk processing (SoA),

perceptual opts, … •  100-1000X more data than D3


.js

100-1000X Bigger graphs

Accelerate Every Component 10X+ with GPUs: Meaningful Viz

Goal: •  Informative visuals

•  Stats for clustering, coloring, sizing, …

•  Quickly respond to filters & pivots

Solution: •  GPU+JS in server via Node-OpenCL, Docker

•  Fast iterative clustering: pure GPU

•  More: tree maps, edge bundling, …

•  60X more data than Gephi


0.1

1

10

100

500K 1.0M 1.5M

Frames per

second

Graph Size: # Nodes + # Edges

Accelerate Every Component 10X+ with GPUs: Interactive Analytics

•  Fast drilldowns are essential

•  1 NVidia Tesla K80 = ~9 TFLOPS, 24 GB RAM

•  Real announcements next year J

•  Major work goes in visual querying: minimize time in SQL, Python, …


Integrating Graphistry into Spark, Notebooks


•  Binding*: Python Pandas

•  GPU dataframe

•  GPU clustering

•  Networking: geometry streaming

SQL

py

Client

*: github.com/graphistry/pygraphistry

Cloud

•  Network streaming

•  GPU big data rendering

•  HTML5 labeling, small charts: D3, JQuery, …

•  Data viz query lang; Falcor for composition

22 Copyright © 2016 Accenture & Graphistry All rights reserved.

Two Case Studies

1.  HUNTING: Network mapping to drill into all priority 10 alerts

2.  RESPONSE: Botnet analysis to reveal full infection


SIEM

Advanced Visualization

Scalable Compute

Distributed Stream

Processing

Longer Storage

Retention ASGARD

Commodity advances: GPU+CPU clusters

Unlocks workflows for incident

response & forensics

Repeatable notebooks to custom tailored apps

Beyond SIEM: Enterprise security has a data problem



Ingest

Event Processing

Storage


Data Sources

Visualizations

SQL

Streaming

py



Innovation Cycle

Architecture

Data

Visualization

Analytics

DATA SCIENCE ARCHITECTURE

Customize, create, and iterate

Thanks!

We’re piloting: Want more out of Splunk, ArcSight, …?

[email protected] We’re hiring: UX, security researchers, …

@lmeyerov @graphistry

@datametrician @mike_wendt @accenturelabs

github.com/graphistry/pygraphistry 27 Copyright © 2016 Accenture & Graphistry All rights reserved.

josh patterson, principal visualizing 200m alerts mike ... · • data: query billions of alerts...

Documents