In-kernel Analytics and Tracing with eBPF for

OpenStack CloudsOctober 2016

Brenden BlancoPLUMgrid

Ali KhayamPLUMgrid

Thank You to Sponsoring Members

2

IO Visor Project, What is in it?

• A set of development tools, IO Visor Dev Tools

• A set of IO Visor Tools for management and operations of the IO Visor Engine

• A set of Applications, Tools and open IO Modules build on top of the IO Visor framework

• A set of possible use cases & applications like Networking, Security, Tracing & others

3

The promise of Microservices: Better cloud app lifecycle …… but what about security?

4

Shared kernel Larger attack surface?

Self service Developer = Security Expert?

Shared Infrastructure Insider threats?

Fast Development & Iteration Compromised zero trust?

Where should microservices security be implemented?

All layers…. but from the app cloud provider’s perspective:best to trust what you build/operate/control

=> “Security-as-a-Service” in the cloud infrastructure

InfrastructureOperator

ApplicationDeveloper

An ideal Security-as-a-Service offeringTransparent: Application shouldn’t be aware of this layer

No new software installation/configuration

Generically applicable: Should be able to characterize microservice security profiles for diverse applications, without having visibility into service behavior

Efficient: No compromises on performance or scalability

What features can characterize a Microservice Security Profile?

APIAPI call, payload len

Trafficbytes tx/rx, packets rx/rx

Disk I/ODisk I/O rx/tx

Tenants# of active tenants

…. how to get these features without compromising transparency and efficiency

How to extract features for Microservice Security Profiles?Objectives: Transparency, Seamlessness,

Efficiency

IO Visor instrumented infra to extract features for service security profiles:

▪already present in Linux kernels▪capture API calls and resource usage▪system-call level insight

▪ real-time monitoring▪without efficiency degradation

8

Automation

Developers

IOVisor framework

Advanced MonitoringSecurityAutomation / OperationsMachine Learning

Infrastructure

Monitor

Ops/Automation

Maintain

Plugging features into an ML model to learn Microservice security profiles

9

Com

pute

Nod

e

Use

r Spa

ceK

erne

l Spa

ce

API / Traffic Data Disk/Memory Data

Microservice CollectorMachine

Learning

API Traffic(Ingress / Egress)

Microservice Security Profiles

IO Visor Code Snippet (Userspace)

IO Visor Code Snippet (Kernel)

www.iovisor.org

Preliminary Evaluation1) OpenStack Controller Services as

Microservices

12

OpenStack Controller Services as MicroservicesIO Visor instrumentation is used to build security profiles of all controller services

nova, neutron, keystone, cinder, etc.

API calls learned as they arrive on the services’ veth interfaceno pre-training of API calls

IO Visor hooks to monitor vfs_{read/write} accesses from each serviceseparated based on PIDs for each container

ML algorithm builds security profiles based on initial (training) datathen security profile deviations are used for attack detection on run-time data

Attack: Bruteforce password cracking on keystone

Lots of Background (benign) Traffic:Continuous CRUD APIs from a real-world app cloud use caseAll API calls (incl. service-to-service) must get auth_token from keystone

first

Attack Traffic:2-4 password attempts per secondAttack continued for a sustained period of time

Results of brute-force password attack on keystone

Attack Detection Rate False Positive Rate

97% 0%

• Results obtained from an ROC curve by tuning the detection threshold

• API and Traffic features are the main contributors to these results

Preliminary Evaluation2) Database container using MySQL

16

MySQL Microservice instrumentation

MySQL Docker image (MySQL version 5.7, docker 1.12 )

SQL queries (TCP packets) intercepted by IOVisor hooks on veth pairs

handshakes, teardown and acks ignored

IOVisor hooks for vfs_{read/write} for queries into a large DB (180Mb)

separated on PID and TID for docker

17

Attack: First order SQL injection

Benign traffic consisted of Simulated SQL queriesGenerated randomly and continuously

Attack results in extracting large segments of the DBSegment size varyingIn parallel to benign traffic on the microservice

18

Results of brute-force password attack on keystone

Attack Detection Rate False Positive Rate

93.5% 3.5%

• Results obtained from an ROC curve by tuning the detection threshold

• Correlating Traffic and disk access was essential for detection

Dashboard

Conclusion:Meeting the requirements of an ideal Security-as-a-Service offering

21

Transparency

Application shouldn’t be aware of this layer

IO Visor works on eBPF constructs that are present in >4.x upstream kernels

IO Visor instrumentation runs in kernel and is not visible to the developer

The only non-standard dependency is github.com/iovisor/bcc python library

Generic Applicability

Should be able to characterize microservice security profiles for diverse applications,

without having visibility into service behavior

Trained/Tested on SQL

Trained/Tested on OpenStack services

Future Work:

Train/Test for DNS attacks

Train/Test for ransomware attacks

EfficiencyNo compromises on performance or scalability

eBPF counting is done inside the kernel with little or no overhead

Main overhead is kernel to userspace interaction

Data polled by userspace every 1 minute

All data structures are reset after polling; data cannot grow

indefinitely

Data is exported by the userspace application to a collector node

Machine learning and classification is applied on the collector node

i.e. no impact to performance on computes

EfficiencyNo compromises on performance or scalabilityData structures have low overhead:vfs_read (BFP_HASH):

size at time ti = Ni x 3, where:Ni = # of read process at ti

the map has: {key: pid, value1: # of reads, value2: aggregate size of all reads

vfs_write (BFP_HASH): has the the same structure as vfs_read

traffic (BFP_HASH):size at time ti = Fi x 7, where:

Fi = # of active TCP flows at ti

the map’s key is a 5-tuple flow id, and values are the same as vfs_{read/write}

http_traffic (BPF_HISTOGRAM):size at time ti = Si x LSi x 7, where:

key is a 5-tuple flow id of http packetsSi = # of active HTTP session at ti

LSi = # of HTTP packets with unique lengths received on session S i

How to Contribute

github.com/akhayam/conmon (this presentation)

www.iovisor.org

github.com/iovisor

#iovisor at irc.oftc.net

lists.iovisor.org/mailman/listinfo/iovisor-dev

26

Questions?