page 1 © hortonworks inc. 2014 hdp with advanced security comprehensive security for enterprise...

© Hortonworks Inc. 2014

HDP with Advanced SecurityComprehensive Security for Enterprise Hadoop

Hortonworks. We do Hadoop.


Agenda

• Our approach across security pillars

• Component Deep Dive

• Questions


Security needs are changing

AdministrationCentrally management & consistent security

AuthenticationAuthenticate users and systems

AuthorizationProvision access to data

AuditMaintain a record of data access

Data ProtectionProtect data at rest and in motion

Security needs are changing• YARN unlocks the data lake

• Multi-tenant: Multiple applications for data access

• Changing and complex compliance environment

• ETL of non-sensitive data can yield sensitive data

Summer 201465% of clusters host multiple workloads

Fall 2013Largely silo’d deployments with single workload clusters

5 areas of security focus


Security in Hadoop with HDP + Argus (XA Secure)

AuthorizationRestrict access to explicit data

AuditUnderstand who did what

Data ProtectionEncrypt data at rest & in motion

• Kerberos in native Apache Hadoop

• HTTP/REST API Secured with Apache Knox Gateway

• HDFS Permissions, HDFS ACL,• Audit logs in with HDFS & MR• Hive ATZ-NG

AuthenticationWho am I/prove it?

• Wire encryption in Hadoop

• Open Source Initiatives

• Partner Solutions

• HDFS, Hive and Hbase

• Fine grain access control

• RBAC

• Centralized audit reporting

• Policy and access history

• Future Integration

HD

P 2

.1A

rgus

Centralized Security Administration

• As-Is, works with current authentication methods


Map to Nevada Energy Requirements

Questions HDP Security Component

End User Security

LDAP Integration Kerberos, Argus (XA)

Group level access Argus(XA)

Multiple level of access Argus(XA)

Multiple Environments Argus(XA)

Developer Security

Access control for creating tables Argus(XA)

Limit of creating scheme, creating folders

Argus(XA)


Security Features

HDP w/ Advanced Security

Authentication

Kerberos Support ✔

Perimeter Security – For services and rest API ✔

Authorizations

Fine grained access control HDFS, Hbase and Hive

Role base access control ✔

Column level ✔

Permission Support Create, Drop, Index, lock, user

Auditing

Resource access auditing Extensive Auditing

Policy auditing ✔


HDP w/ Advanced Security

Data Protection

Wire Encryption ✔Volume Encryption ✔File/Column Encryption Partners

Reporting

Global view of policies and audit data ✔

Manage

User/ Group mapping ✔

Global policy manager, Web UI ✔

Delegated administration ✔

Security Features


Authentication w/ Kerberos

Page 8


Kerberos Primer

Page 9

Client

KDC

NN

DN

1. kinit - Login and get Ticket Granting Ticket (TGT)

3. Get NameNode Service Ticket (NN-ST)

2. Client Stores TGT in Ticket Cache

4. Client Stores NN-ST in Ticket Cache

5. Read/write file given NN-ST and

file name; returns block locations,

block IDs and Block Access Tokens

if access permitted

6. Read/write block given

Block Access Token and block ID

Client’sKerberos

Ticket Cache


Kerberos Summary

• Provides Strong Authentication

• Establishes identity for users, services and hosts

• Prevents impersonation on unauthorized account

• Supports token delegation model

• Works with existing directory services

• Basis for Authorization

Page 10


User Management

•Most customers use LDAP for user info–LDAP guarantees that user information is consistent across the cluster–An easy way to manage users & groups–The standard user to group mapping comes from the OS on the NameNode

•Kerberos provides authentication–PAM can automatically log user into Kerberos

Page 12


Kerberos + Active Directory

Page 13

Cross Realm Trust

Client

Hadoop Cluster

AD / LDAP KDC

Users: [email protected]

Hosts: [email protected]

Services: hdfs/[email protected]

User Store

Use existing directory tools to manage users

Use Kerberos tools to manage host + service

principals

Authentication


Knox Gateway OverviewPerimeter REST API Security

Page 17


REST API

HadoopServices

What does Perimeter Security really mean?

Gateway

REST API

Firewall

User

Firewall required at perimeter(today)

Knox Gateway controls all

Hadoop REST API access

through firewall

Hadoop cluster mostly

unaffected

Firewall only allows

connections through specific ports from Knox

host


Why Knox?

Simplified Access

•Kerberos encapsulation •Extends API reach•Single access point•Multi-cluster support•Single SSL certificate

Centralized Control

• Central REST API auditing• Service-level authorization• Alternative to SSH “edge node”

Enterprise Integration

•LDAP integration•Active Directory integration•SSO integration•Apache Shiro extensibility•Custom extensibility

Enhanced Security

• Protect network details• Partial SSL for non-SSL services• WebApp vulnerability filter


Current Hadoop Client Model

• FileSystem and MapReduce Java APIs• HDFS, Pig, Hive and Oozie clients (that wrap the Java APIs)

• Typical use of APIs is via “Edge Node” that is “inside” cluster• Users SSH to Edge Node and execute API commands from shell

Page 20

HadoopUser Edge NodeSSH


Hadoop REST APIs

• Useful for connecting to Hadoop from the outside the cluster

Page 21

Service API

WebHDFS Supports HDFS user operations including reading files, writing to files, making directories, changing permissions and renaming. Learn more about WebHDFS.

WebHCat Job control for MapReduce, Pig and Hive jobs, and HCatalog DDL commands. Learn more about WebHCat.

Hive Hive REST API operations, JDBC/ODBC over HTTP

HBase HBase REST API operations

Oozie Job submission and management, and Oozie administration. Learn more about Oozie.

http://hadoop.apache.org/docs/r1.0.4/webhdfs.html

http://people.apache.org/~thejas/templeton_doc_latest/installation.html

http://oozie.apache.org/


Hadoop REST API Security: Drill-Down

Page 22

RESTClient

EnterpriseIdentityProvider

LDAP/AD

Knox Gateway

GWGW

Firew

all

Firew

all

DMZ

LB

Edge Node/Hado

op CLIs RPC

HTTP

HTTP HTTP

LDAP

Hadoop Cluster 1Masters

Slaves

RM

NN

WebHCat

Oozie

DN NM

HS2

Hadoop Cluster 2Masters

Slaves

RM

NN

WebHCat

Oozie

DN NM

HS2

HBase

HBase


Authorization and Auditing

Page 23


Authorization and Audit

AuthorizationFine grain access control

• HDFS – Folder, File

• Hive – Database, Table, Column

• HBase – Table, Column Family, Column

AuditExtensive user access auditing in HDFS, Hive and HBase

• IP Address

• Resource type/ resource

• Timestamp

• Access granted or denied

Control access into

system

Flexibility in defining

policies


Central Security Administration

HDP Advanced Security • Delivers a ‘single pane of glass’ for

the security administrator

• Centralizes administration of security policy

• Ensures consistent coverage across the entire Hadoop stack


Setup Authorization Policies

26

file level access control, flexible definition

Control permissions


Monitor through Auditing

27


Authorization and Auditing w/ XA

Hadoop distributed file system (HDFS)

XA Administration Portal

HBase

Hive Server2

XA Policy ServerXA Audit Server

XA Plugin

Had

oop

Com

pone

nts

Ent

erpr

ise

Use

rs

XA Plugin

XA Plugin

Legacy Tools

Integration API

RDBMS

HDFS

Knox

Falcon

XA Plugin*

XA Plugin*

XA Plugin*

Storm

YARN : Data Operating System

* - Future Integration


Data Protection

HDP allows you to apply data protection policy at three different layers across the Hadoop stack

Layer What? How ?

Storage Encrypt data while it is at rest Partners, OS level encrypt, Custom Code

Transmission Encrypt data as it moves Supported in HDP 2.1

Upon Access Apply restrictions when accessed Partners, Open Source Initiatives


Points of Communication

Page 31

WebHDFS

DataTransferProtocol

Nodes

M/R Shuffle

Client

1

2

4

RPC3Nodes

DataTransfer2

JDBC/ODBC

3

Hadoop Cluster

RPC

4


Data Transmission Protection in HDP 2.1

• WebHDFS– Provides read/write access to HDFS– Optionally enable HTTPS– Authenticated using SPNEGO (Kerberos for HTTP) filter– SSL based wire encryption

• RPC– Communications between NNs, DNs, etc. and Clients– SASL based wire encryption– DTP encryption with SASL

• JDBC/ODBC– SSL based wire encryption– Also available SASL based encryption

• Shuffle– Mapper to Reducer over HTTP(S) with SSL

32


Data Storage Protection

• Encrypt at the physical file system level (e.g. dm-crypt)• Encrypt via custom HDFS “compression” codec• Encrypt at Application level (including security service/device)

Page 33

HDFSABC 1a3d

ABC DEF

ETL App

ABCDEF

Security Service(Partner)

ENCRYPT DECRYPT


Current Open Source Initiatives

• HDFS Encryption– Transparent encryption of data at rest in HDFS via Encryption zones. Being worked in the community– Dependency on Key Management Server and Keyshell

• Key Management Server• Key Provider API• Hive Column Level Encryption• HBase Column Level Encryption

– Transparent Column Encryption, needs more testing/validation

• Command line Key Operations


Resources

Page 35


Security Page


Hortonworks Security Investment Plans

HDP + XA

Comprehensive Security for Enterprise Hadoop

…all IN Hadoop

Goals:

Investment themes

Central AdministrationProvide one location for administering security policies and audit reporting for entire platform

Comprehensive SecurityMeet all security requirements across Authentication, Authorization, Audit & Data Protection for all HDP components

Consistent IntegrationIntegrate with other security & identity management systems, for compliance with IT policies

XA Secure Phase• Centralized Security Admin for HDFS, Hive &

HBase• Centralized Audit Reporting• Delegated Policy Administration

Previous Phases Kerberos Authentication HDFS, Hive & Hbase authorization Wire Encryption for data in motion Knox for perimeter security Basic Audit in HDFS & MR SQL Style Hive Authorization ACLs for HDFS

Delivered

Future Phases• Encryption in HDFS, Hive & Hbase• Centralized security administration of entire

Hadoop platform• Centralized auditing of entire platform• Expand Authentication & SSO integration choices• Tag based global policies (e.g. Policy for PII)

Delivered XA Secure

page 1 © hortonworks inc. 2014 hdp with advanced security comprehensive security for enterprise...

Documents

data accesschanging

motion security needs

areas of security focuspage

ticket tgt

sensitive data summer

ticket cache5

block ids

hdfs communication