© hortonworks inc. 2015 hadoop and kerberos: the madness beyond the gate steve loughran...
TRANSCRIPT
![Page 1: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/1.jpg)
© Hortonworks Inc. 2015
Hadoop and Kerberos: The madness beyond the gate
Steve [email protected]@steveloughran2015
![Page 2: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/2.jpg)
Page 2
Me: Before Kerberos
![Page 3: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/3.jpg)
© Hortonworks Inc.Page 3
Me: After Kerberos
![Page 4: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/4.jpg)
© Hortonworks Inc.Page 4
HP Lovecraft Kerberos
Evil lurking in New England MIT Project Athena
Ancient, inhuman deities Kerberos Domain Controller
Manuscripts to drive the reader insane
IETF RFC 4120
Entities never spoken of aloud UserGroupInformation
Doomed explorers of darkness You
![Page 5: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/5.jpg)
© Hortonworks Inc. 2015
Leave now if you want to retain your life ofnaïve innocence
Page 5
![Page 6: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/6.jpg)
© Hortonworks Inc.Page 6
![Page 7: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/7.jpg)
© Hortonworks Inc. 2015
export HADOOP_USER="root"
Page 7
![Page 8: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/8.jpg)
© Hortonworks Inc. 2015
Modern Hadoop clusters are locked downthrough Kerberos
Page 8
![Page 9: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/9.jpg)
© Hortonworks Inc. 2015
Discover Kerberosbefore Kerberosdiscovers you
Page 9
![Page 10: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/10.jpg)
© Hortonworks Inc. 2015
Kerberos:the gateway to hell
Page 10
![Page 11: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/11.jpg)
© Hortonworks Inc.
This is not a metaphor
Art: Andrés Álvarez Iglesias
![Page 12: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/12.jpg)
© Hortonworks Inc. 2015
KP
Kerberos is the gateway
Page 12
Authentication Service
Ticket Granting Service
Principal
user@REALMuser/hostname@REALM
(P, TGS, n1)
{KP.TGS, n1}KP, {ticket(P,TGS)}
KTGS Ticket(P, TGS) = (TGS, P, tstart, tend, KPT)
KP
{KP.S, n2}KP, {ticket(P,S)} KS
{auth(P)}KP.TGS,{ticket(P,TGS)}KTGS,S,n2
KTGS
Kerberos Domain ControllerClient
auth(P)KP.TGS = {P, time)}KP.TGS
![Page 13: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/13.jpg)
© Hortonworks Inc
Every service is a principal
alice@REALMbob@REALM
oozie/ooziehost@REALM
namenode/nn1@REALM
hdfs/_HOST@REALMhdfs/r04s12@REALMhdfs/r04s13@REALM
yarn/_HOST@REALMyarn/r04s12@REALM
HTTP/_HOST@REALMPage 13
short names:aliceboboozienamenodehdfsyarnHTTP
![Page 14: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/14.jpg)
© Hortonworks Inc.Page 14
Entering the darkness
![Page 15: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/15.jpg)
![Page 16: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/16.jpg)
© Hortonworks Inc.Page 16
![Page 17: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/17.jpg)
© Hortonworks Inc. 2015
HDFS Bootstrap: Kerberos Login
Page 17
shared keytab in /etc/hadoop
log in to kerberos
datanode/_HOST@REALM
tickets for TGS
namenode/nn@REALM
![Page 18: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/18.jpg)
© Hortonworks Inc. 2015
HDFS Bootstrap: DNs register with NN
Page 18
shared keytab in /etc/hadoop
DN registration
Ticket for namenode/nn@REALM
ExportedBlockKeys
Request ticket for namenode/nn@REALM
namenode/nn@REALM
datanode/_HOST@REALM
![Page 19: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/19.jpg)
© Hortonworks Inc.
Hadoop Tokens
• Issued and tracked by individual services
(HDFS, WebHDFS, Timeline Server, YARN RM, …)
• Grant some form of access:
Block tokens, Delegation Tokens
• Can be passed on to other processes
• Renewable via service APIs (RPC, HTTP)
• Revocable in server via service APIs
Page 19
read: O'Malley 2009, Hadoop Security Architecture
![Page 20: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/20.jpg)
© Hortonworks Inc. 2015
HDFS IO: Block Tokens
Page 20
alice@REALM
Obtain ticket for namenode/nn@REALM
BlockToken
BlockToken
BlockToken: userId, (BlockPoolId, BlockId), keyId, expiryDate, access-modes
namenode/nn@REALM
open("file")
![Page 21: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/21.jpg)
© Hortonworks Inc. 2015
service/host@REALM
Delegation Tokens delegate access
Page 21
alice@REALM
BlockToken
HDFS Delegation Token
BlockToken
HDFS Delegation Token
HDFS Delegation Token
namenode/nn@REALM
Token
Obtain ticket for namenode/nn@REALM
Request delegation token
![Page 22: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/22.jpg)
© Hortonworks Inc. 2015
Launch Context
YARN app launch
Page 22
alice@REALM
HDFS Delegation Token
HDFS
resourcemanager/rm@REALM
nodemanager/_HOST@REALMalice
Launch Context
AM/RM
HDFS AM/RM
HDFS
HDFSHDFS
AM/RM
namenode/nn@REALM
Obtain ticket for resourcemanager/rm@REALM
Request delegation token
AM/RM
Token
Obtain tickvet for namenode/nn@REALM
AM/RM'
AM/RM'
AM/RM'
Refresh AM/RM
![Page 23: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/23.jpg)
© Hortonworks Inc
That which must not be named: UGI
if(!UserGroupInformation.isSecurityEnabled()) { stayInALifeOfNaiveInnocence();} else { sufferTheEnternalPainOfKerberos();}
UserGroupInformation.checkTGTAndReloginFromKeytab();
UserGroupInformation.getLoginUser() // principal logged in asUserGroupInformation.getCurrentUser() // principal acting as
Page 23
![Page 24: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/24.jpg)
© Hortonworks Inc
UGI.doAs()
UserGroupInformation bob = UserGroupInformation.createProxyUser("bob", UserGroupInformation.getLoginUser());
FileSystem userFS = bob.doAs( new PrivilegedExceptionAction<FileSystem>() { public FileSystem run() throws Exception { return FileSystem.get(FileSystem.getDefaultUri(), conf); } });
Page 24
![Page 25: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/25.jpg)
© Hortonworks Inc
Hadoop RPC
@KerberosInfo(serverPrincipal = "my.kerberos.principal")public interface MyRpc extends VersionedProtocol { … }
public class MyRpcPolicyProvider extends PolicyProvider { public Service[] getServices() { return new Service[] { new Service("my.protocol.acl", MyRpc.class) }; }}public class MyRpcSecurityInfo extends SecurityInfo { … }
META-INF/services/org.apache.hadoop.security.SecurityInfoorg.example.rpc.MyRpcSecurityInfo
Page 25
![Page 26: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/26.jpg)
© Hortonworks Inc
IPC Server: get the current user identity
Messages.KillResponse killContainer(Messages.KillRequest request) {
UserGroupInformation callerUGI;
try { callerUGI = UserGroupInformation.getCurrentUser();} catch (IOException ie) { LOG.info("Error getting UGI ", ie); AuditLogger.logFailure("UNKNOWN", "Error getting UGI"); throw RPCUtil.getRemoteException(ie);}
…
Page 26
![Page 27: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/27.jpg)
© Hortonworks Inc
IPC Server: Authorize
String user = callerUGI.getShortUserName();
if (!checkAccess(callerUGI, MODIFY)) { AuditLog.unauthorized(user, KILL_CONTAINER_REQUEST, "User doesn't have permissions to " + MODIFY); throw RPCUtil.getRemoteException( new AccessControlException( + user + " lacks access " + MODIFY_APP.name()));}
AuditLog.authorized(user, KILL_CONTAINER_REQUEST)
Page 27
![Page 28: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/28.jpg)
© Hortonworks Inc. 2015
SASL: RFC4422
Page 28
![Page 29: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/29.jpg)
© Hortonworks Inc.
REST: SPNEGO (+ Delegation tokens)
Page 29
• Jersey + java.net• httpclient? “if lucky it'll work”
HADOOP-11825: Move timeline client Jersey+Kerberos+UGI support into a public implementation
![Page 30: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/30.jpg)
© Hortonworks Inc.
Testing
Page 30
![Page 31: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/31.jpg)
© Hortonworks Inc.
Error messages to fear
Art: Andrés Álvarez Iglesias
Failure unspecified at GSS-API level (Checksum failed)
No valid credentials provided (Failed to find any Kerberos tgt)
Server not found in Kerberos database
Clock skew too great
Principal not found
No valid credentials provided (Illegal key size)
![Page 32: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/32.jpg)
© Hortonworks Inc.
Topics Avoided Not Covered
• Zookeeper
• JAAS
• Trying to use HTTPS in a YARN application
• Trying to use Full REST in a YARN application
• System properties to debug Kerberos & SPNEGO
• Group management
• HADOOP_PROXY_USER
Page 32
![Page 33: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/33.jpg)
© Hortonworks Inc.
gitbook.com/@steveloughran
Questions?
Art: Andrés Álvarez Iglesias
![Page 34: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/34.jpg)
© Hortonworks Inc.
Zookeeper
• SASL to negotiate security:System.setProperty("zookeeper.sasl.client", "true");
• Permissions are not transitive down the tree
Page 34
List<ACL> perms = new ArrayList<>();if (UserGroupInformation.isSecurityEnabled()) { perms(new ACL(ZooDefs.Perms.ALL, ZooDefs.Ids.AUTH_IDS)); perms.add(new ACL(ZooDefs.Perms.READ,ZooDefs.Ids.ANYONE_ID_UNSAFE));} else { perms.add(new ACL(ZooDefs.Perms.ALL, ZooDefs.Ids.ANYONE_ID_UNSAFE));}zk.createPath(path, null, perms, CreateMode.PERSISTENT);
![Page 35: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/35.jpg)
© Hortonworks Inc
System Properties for debugging
-Dsun.security.krb5.debug=true-Dsun.security.spnego.debug=true
export HADOOP_JAAS_DEBUG=true
Page 35
![Page 36: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/36.jpg)
© Hortonworks Inc.
Services
• RPC authentication via annotations & metadata in JAR• YARN Web UIs: rely on RM proxy for authentication• Authentication != Authorization• Add audit logs on service endpoints• YARN services: come up with a token refresh strategy:keytab everywhere; keytab in AM; update from client
Page 36
![Page 37: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/37.jpg)
© Hortonworks Inc.
JAAS
• Java Authentication and Authorization Service• Core Kerberos classes and types (Principal)• Text files to configure
–Different for different JVMs–Need to double escape \ for windows paths
• UGI handles setting up a JAAS context & logging in
Page 37
![Page 38: © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran stevel@hortonworks.com @steveloughran 2015](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5697bf821a28abf838c85dbc/html5/thumbnails/38.jpg)
© Hortonworks Inc.
Glossary
• Simple Authentication and Security Layer (SASL)• GSSAPI Generic Security Service Application Program Interface (RFC-2743+ others)
• JAAS: Java Authentication and Authorization Service• Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO)
Page 38