ulf troppens c se16 - secure data access to personalized patient data - 2014-04-08

© 2013 IBM Corporation1 IBM Confidential

Customer success story:

Secure data access to personalized

patient data for multiple tenant

Ulf Troppens, NAS Development Mainz

[email protected]

de.linkedin.com/pub/ulf-troppens/7/292/852/

storage-explained.blogspot.de/

IBM STU Istanbul 2014 – Lesson: cSE16 – Meeting Room 2

© 2014 IBM Corporation2

Problem Statement

� Joint research project

– Multiple independent organizations

– Researchers of different organizations need to work on the same set of data

– There is no central authentication service

� Legal obligations

– Personalized patient data

– Access to data must be identified and authorized

� Technical constraints

– Data cannot be replicated (too much data)

– Different organizations are connected via public internet

– Not trust relations ships between the AD server (independent organizations!)

– There are duplicate UIDs/GIDs

Compute

Nodes

Medical

Devices

Compute

Nodes

Medical

Devices

SONAS

LDAP

AD

AD +

Unix info

Internet

Secure access to

personalized patient data

Customer Requirement


High-level Schedule

Customer confirmed that all success criteria are metNov 2013

4th Proof-of-concept (at customer data center):

Microsoft Active Directory (AD) and kerberized NFSv4

Sep 2013 -

Oct 2013

3nd Proof-of-concept (IBM internal):

Microsoft Active Directory (AD) and kerberized NFSv3

Dec 2012 -

Jan 2013

2nd Proof-of-Concept (at customer data center):Repeat first Proof-of-Concept in customer data center to assure end-to-end

integration in the customer’s already existing IT infrastructure

Jun 2012 -Aug 2012

1st Proof-of-Concept (IBM internal):

Tivoli Directory Integrator (TDI) and kerberized NFSv3

May 2012 -

Jun 2012

Analyzed customer requirements

Identified high-level changes for target architecture:

1) Kerberized NFS for secure data access

2) Integrated authentication service

Apr 2012 -

May 2012


Agreed success criteria

1. Kerberized NFS for Tenant_A CentOS clientsFor a CentOS client who is already authenticated (kinit) to the Tenant_A AD Server:The client must be able create, read, update, delete data on a SONAS share which is exported via kerberized NFS without being prompted for a password.

2. Kerberized NFS for Tenant_B CentOS clientsFor a CentOS client who is already authenticated (kinit) to the Tenant_B AD Server:The client must be able create, read, update, delete data on a SONAS share which is exported via kerberized NFS without being prompted for a password.

3. Data sharing between users at Tenant_A and Tenant_BUsers of Tenant_A and Tenant B must be able to access the same files and directories. SONAS must be correctly allow or deny access to files and directories based on POSIX mod bits (owner, group, other) or ACLs.

4. Management of file owner shipTenant_A and Tenant_B user must be able to change the owner and the group of their files and directories using the chown and the chgrp command on a CentOS client.

5. Kerberized CIFS for Tenant_A Windows clientsFor a Windows client who is already authenticated (Windows domain logon) to the Tenant_A AD Server:The client must be able create, read, update, delete data on a SONAS share which is exported via CIFS without being prompted for a password.


1st PoC: Tivoli Directory Server (TDS) and kerberized NFSv3

� Objectives– Can TDS/TDI be used to delegate authentication to

already existing Microsoft AD server?

– Document and validate installation and configuration steps for customer proof-of-concept

– Document all LDAP entries and attributes required for SONAS with CIFS and plain NFSv3

– Understand potential limitations/constraint of the respective LDAP attribute values

� Scope– Configure SONAS with TDS for LDAP authentication

– Configure pass-through authentication in TDS, so that incoming authentication requests to TDS will be served by the remote AD or LDAP server

– Configure CentOS 5.8 clients for data access to SONAS via NFSv3

– Configure Windows 2003 clients for data access to SONAS via CIFS

� *NOT* in scope– Setup and configuration of TDI

– This pre-proof-of-concept uses only a few users and groups

– Their data will be copied manually from the remote AD and LDAP server to the central TDS.

Phase 1: TDS (LDAP)

SONAS

LDAPAD

TDI

CentOSWindows

TDS

CIFS NFSv3

LDAP


1st PoC: Tivoli Directory Server (TDS) and kerberized NFSv3

� Objectives

– Extend Phase 1 to include kerberized NFSv3

� Scope

– Same as Phase 1

– In addition:Setup kerberized NFSv3 on SONAS and CentOSclients

Phase 2: Kerberized NFSv3

SONAS

LDAPAD

TDI

CentOSWindows

TDS

CIFS kerberized

NFSv3

LDAP

KDC


� The LDAP schema of TDS must be extended, before SONAS can be configured with

LDAP authentication:

– http://pic.dhe.ibm.com/infocenter/sonasic/sonas1ic/topic/com.ibm.sonas.doc/mng_ldap_srvr_prereqs.htmlhttp://pic.dhe.ibm.com/infocenter/sonasic/sonas1ic/index.jsp?topic=%2Fcom.ibm.sonas.doc%2Fmng_ldap_srvr_prereqs_samba.html

� The “Configuring LDAP servers for use with the CIFS protocol” of the IBM Information Archive product was helpful to configure the required schema extension for TDS:

– http://www-304.ibm.com/support/docview.wss?uid=ssg1S7003490

� The LDAP schema extension for SONAS which needs to be added to TDS can be found

on any server where Samba is installed:

– /usr/share/doc/samba-doc-3.6.1_ctdb_36/LDAP/samba.schema.at.IBM-DS /usr/share/doc/samba-doc-3.6.1_ctdb_36/LDAP/samba.schema.oc.IBM-DS

� The steps for adding the schema to TDS are described in the “TDS Installation and

Configuration Guide”:

– http://publib.boulder.ibm.com/infocenter/tivihelp/v2r1/topic/com.ibm.IBMDS.doc/install163.htm?path=8_3_16_14_0#addschem

� Check ‘V7000U Authentication HOWTO’ for setting up SONAS/V7000U for LDAP

authentication and how to fill additional LDAP attributes for users and groups

– http://www-01.ibm.com/support/docview.wss?uid=ssg1S1004060

1st PoC: LDAP schema extension


LDAP Entries for Groups

•Used to differentiate different Windows group types Windows Groups typesambagrouptype

•Used by CIFS clients to refer to group, e.g. in CIFS ACLsWindows SIDsambasid

•Used by NFS clients to refer to group

•Used in GPFS ACLs and GPFS POSIX group attributes for files and directories

stored on SONAS

Unix GIDgidnumber

•Used by CIFS clients to show group name in Windows file properties

•Note: NFS clients will resolve group name directly via LDAP

group namecn

UsagePurpose

Attribute

cn

•Group names must be unique

gidnumber

•gidnumbers must be unique

sambagrouptype

•Must be set to ‘2’

sambasid

•The prefix and the domain part of the group SID must match the SID prefix and SID domain part of the SONAS domain entry

•There must be no other user, group or other object having the same SID

•As best practices the RID part of group SIDs is derived from gidnumber: RID = gidnumber * 2 + 1001


LDAP Entries for User (1/2)

•Inherently used by SambaList of previously

used CIFS passwords

sambapasswordhistory

•Used by Samba LDAP back-end which is used in SONASSamba account flagsambaacctflags

•Must be set to unlock CIFS accountdate of last password

update

sambapwdlastset

•Used for authentication, when a CIFS client maps a network drive.CIFS passwordsambantpassword

•Used by CIFS clients to refer to user, e.g. in CIFS ACLsWindows SIDsambasid

•Used by SCP, SFTP and HTPPs for user authenticationUnix passworduserpassword

•Used by NFS clients to refer to user

•Used in GPFS ACLs and GPFS POSIX user attributes for files and directories stored

on SONAS

Unix UIDuidnumber

•Used by CIFS clients to show group name in Windows file properties

•Used by SCP, SFTP and HTPPs for user authentication

•Note: NFS clients will resolve user name directly via LDAP

user nameuid

UsagePurpose

Attribute


LDAP Entries for User (2/2)

uid

•User names must be unique

uidnumber

•uidnumbers must be unique

sambasid

•The prefix and the domain part of the user SID must match the SID prefix and SID domain part of the SONAS domain entry

•There must be no other user, group or other object having the same SID

•As best practices the RID part of user SIDs is derived from uidnumber: RID = uidnumber * 2 + 1000

sambantpassword

•The CIFS password must be hashed as described here:

http://pic.dhe.ibm.com/infocenter/sonasic/sonas1ic/topic/com.ibm.sonas.doc/mng_ldap_srvr_prereqs_samba.html

sambapwdlastset

•Must be set to a value different then zero, which indicates that a CIFS account is locked.

sambaacctflags

•Must be set to:

‘[U ]’•Note: TDS GUI does not process the brackets correctly. We configured sambaacctflag via LDAP command line interface

sambapasswordhistory

•Must be a string of zeros:

‘00000000000000000000000000000000’

cn

•Not used by SONAS

gidnumber

•Not used by SONAS

•Primary Unix group


2nd PoC: TDS, TDI and kerberized NFSv3 at customer site

� Objectives

– Extend Phase 1 to include kerberized NFSv3

� Scope

– Same as Phase 1

– In addition:Setup kerberized NFSv3 on SONAS and CentOSclients

– In addition:Integrate kerberized NFSv3 CentOS client of Tenant_B

Phase 2: Kerberized NFSv3

SONAS

LDAPAD

TDI

CentOSWindows

TDS

CIFS kerberized

NFSv3

LDAP

KDC


1st PoC & 2nd PoC – Insights

� Kerberized CIFS access failed– Authentication with Kerberos ticket failed

– CIFS authentication failed back to password based authentication (NTLMv2)

� Password based CIFS access successful with limitations

– TDS proxy authentication cannot be used for NTLMv2 authentication of new CIFS session with AD password which is stored in already existing AD server

– NTLMv2 requires to maintain a Samba password in TDS (LDAP)

– This would require additional password management for CIFS access

� Kerberized NFSv3 access successful with limitations

– Kerberos ticket of already existing AD server cannot be used to authenticate to MIT KDC

– Kerberized NFSv3 could be demonstrated for users which where configured in MIT KDC

– This would require additional user and password management in MIT KDC for Kerberized NFSv3 access

� Customer Response– Success criteria not met

• Limitations for kerberized NFSv3 are not acceptable

– Architecture too complex

• Initial and operational costs are too high

– Operational processes too complex

• Operational efforts for user management are not acceptable

• How to trust an unknown user from a remote Tenant?

• Due to legal requirements new users must be identified (e.g., show passport when added to central LDAP / KDC)


3rd PoC: TDS, TDI, kerberized NFS and one-way trusts

(1) TDI pulls user information from already existing AD and LDAP feeds this into the new TDS and KDC

(2) One-way MIT KDC to Microsoft AD trust to allow incoming Kerberized NFSv3 and CIFS sessions

(3) Configure SONAS as kerberized service in above setup

(4) Configure kerberized NFSv3 and CIFS for Tenant_A and Tenant_B users

(5) Configure cross organization groups in TDS for secure data sharing between Tenant_A and Tenant_B


3rd PoC – Insights

� Kerberized CIFS access failed– Authentication with Kerberos ticket failed

– CIFS authentication failed back to password based authentication (NTLMv2)



– NTLMv2 requires to maintain a Samba password in TDI (LDAP)






� Customer Response– Success criteria not met

• Kerberized CIFS does not work=> Still need to manage new CIFS password

– Architecture too complex

• Initial and operational costs are too high

– Operational processes too complex

• Operational efforts for user management are not acceptable

• How to trust an unknown user from a remote Tenant?

• Due to legal requirements new users must be identified (e.g., show passport when added to central LDAP / KDC)


4th PoC: Kerberized NFSv4, AutoRID and one-way trusts

� Details– Establish new AD server having a one-way trust to

each tenants AD server

– Use AutoRID feature to automatically map Windows SIDs to internal UIDs/GIDs

– User are authenticated via already existing AD server in home location

� Pros– Eliminates need for TDI

– Eliminates need for additional password management

– Duplicate UIDs/GIDs are automatically resvolved

� Cons– Difficult to use plain NFSv3/4 (AUTH_SYS)

• NAS internal UIDs/GIDs must be synchronized with UIDs/GIDs on NFSv3/4 clients

– Requires

• Reconfiguration of authentication config

• Update of file/dir owners/groups/GPFS ACLs

• Impact to quotas not clear

• Triggers full TSM backup

Compute

Nodes

Medical

Devices

Compute

Nodes

Medical

Devices

NAS

w/AutoRID

AD

AD +

Unix info

Internet

Kerberized

NFSv4

Kerberized

CIFS

Kerb.

NFSv4

AD

One-way

trust

One-way

trust

LDAP


4th PoC – Insights

� Kerberized CIFS access failed

– Authentication with Kerberos ticket failed

– CIFS authentication failed back to password based

authentication (NTLMv2)



– NTLMv2 requires to maintain a Samba password in TDI (LDAP)






� Customer Response

– All success criteria met

– Architecture is OK

– Operational processes are OK

• User are authenticated by the home site


Validated success criteria

1. Kerberized NFS for Tenant_A CentOS clientsFor a CentOS client who is already authenticated (kinit) to the Tenant_A AD Server:The client must be able create, read, update, delete data on a NAS share which is exported via kerberized NFS without being prompted for a password.

2. Kerberized NFS for Tenant_B CentOS clientsFor a CentOS client who is already authenticated (kinit) to the Tenant_B AD Server:The client must be able create, read, update, delete data on a NAS share which is exported via kerberized NFS without being prompted for a password.

3. Data sharing between users at Tenant_A and Tenant_BUsers of Tenant_A and Tenant B must be able to access the same files and directories. NAS must be correctly allow or deny access to files and directories based on POSIX mod bits (owner, group, other) or ACLs.

4. Management of file owner shipTenant_A and Tenant_B user must be able to change the owner and the group of their files and directories using the chown and the chgrp command on a CentOS client-

5. Kerberized CIFS for Tenant_A Windows clientsFor a Windows client who is already authenticated (Windows domain logon) to the Tenant_A AD Server:The client must be able create, read, update, delete data on a NAS share which is exported via CIFS without being prompted for a password.


Other Considerations

� Need to establish new AD server which is just for pass-through

� Need to roll out Kerberos on NFSv4 clients

� Need to change server and client authentication configuration

� Need to change UIDs/GIDs for owner, group and file ACLs of existing files and

directories


NFSv4 Key Features

� Reference - NFSv4 Draft RFC, dated May 18, 2013:

https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rfc3530bis

Mandatory

Mandatory

Mandatory

Required

NFSv4 extends the traditional POSIX mode bits with access control lists (ACL). The ACL definition allows for specific setsof permissions for individual users and groups. ACL inheritance propagates access permissions and restrictions down a directory tree as file system objects are created.

NFSv4 mandates support for a strong security model, where client/server interactions are done using the GSS-API framework. The required security flavor is Kerberos. The quality of protection, such as which crypto techniques are used, and service, i.e., authentication only, integrity, or privacy are negotiated between the client and server. Note that while the RFC requires the implementation of the stronger security model, it does not require it's deployment. Accordingly, the NFSv4 implementation still support traditional Unix style authentication and security.

The same export may be exported via NFSv3 and NFSv4. Care should be taken, however, as the NFSv3 and NFSv4 protocols are not completely compatible. For example: NFSv3 does not support share reservations. Grace period behaviors are different.

Description

NFSv4 ACLs3

RPCSEC_GSS support for NFSv3 and NFSv4

2

Concurrent support for NFSv3 and NFSv4

1

TitleID


NFSv4 Key Features (continued)

Mandatory

Mandatory

Mandatory

Required

The NFSv4 standard defines certain requirements of a server and client during recovery. Specifically, if the server loses state (open/locking) as the result of a node fail over or IP address relocation, it must allow clients a grace period to discover this fact and re-establish the lost state. It is the NFS client's responsibility to detect the server failure and “reclaim”the lost state. In a cluster, this grace and recovery coordination must occur on all cluster nodes.

NFSv4 integrates file locking into the protocol. NFSv4 locking is also leased based which eliminates the “orphan lock”problem. Orphan locks arise when a client owning a lock never unlocks or recovers it.

NFSv4 introduced the open operation to NFS and support for share reservations. Share reservations are established through OPEN operations and can be viewed as mandatory whole file locks. Very few NFS applications utilize share reservations.

Description

Lock recovery after NFSv4 server node fail-over

6

Lease based locking5

Share reservation4

TitleID



Required

The NFSv4 pseudo-file system is a structure created by the server and provides clients with access to server exports without requiring that the client mount each underlying file system. Previous versions of NFS did not permit this.

Description

Pseudo file systems7

TitleID

� Two file systems are exported

– /gpfs/gpfsA, exported withRW_Access = clientA, Pseudo Path = /gpfsA

– /gpfs/gpfsB, exported withRW_Access = “*”, Pseudo Path = /gpfsB

� Clients mount just one file system:

mount -o vers=4 <server_name>:/ /mnt

� clientA will see:

# ls /mnt

gpfsA gpfsB

� Other clients will see:

# ls /mnt

gpfsB

� Export structure can be adjusted on server without changing the mount on the client

– On server, modify export of gpfsARW_Access = “*”, Pseudo Path = /gpfsA

� After remount other clients will see:# ls /mntgpfsA gpfsB







# ls /mnt

gpfsA gpfsB


# ls /mnt

gpfsB







# ls /mnt

gpfsA gpfsB


# ls /mnt

gpfsB



Required

Typically, NFS operations will send user and group identifiers as strings (e.g., [email protected]) and not numeric ids (e.g, UID/GID). The client and server must then map these strings to traditional POSIX user/group ids as GPFS is a POSIX based file system. It is possible that these mapping's are different between the client and server ( the cross mapping is essentially done by the id mapping service on each side ). While this works for NFSv4 with Kerberos, it is not best practice as applications which use numeric UID/GID fail. NFSv3/NFSv4 with traditional UNIX authentication is but one example.

Description

NFSv4 ids are string based instead of numbers

8

TitleID

Client 1

[usera1@client1]$ cd /mnt/NFSV4_KRB5/projecta_shared_access.txt

[usera1@client1]$ echo "test from usera1@client1" >> ./projecta_shared_access.txt

[usera1@client1]$ ls –l ./projecta_shared_access.txt

-rw-r--r--. 1 usera1 useragrp 25 Jun 26 10:03 /mnt/NFSV4_KRB5/projecta_shared_access.txt

[usera1@client1]$ ls -ln ./projecta_shared_access.txt

-rw-r--r--. 1 100001 100001 25 Jun 26 10:03 /mnt/NFSV4_KRB5/projecta_shared_access.txt

Client 2

[userb1@client2 ~]$ ls -l /mnt/NFSV4_KRB5/projecta_shared_access.txt

-rw-r--r--. 1 usera1 useragrp 25 Jun 26 10:03 /mnt/NFSV4_KRB5/projecta_shared_access.txt

[userb1@client2 ~]$ ll -n /mnt/NFSV4_KRB5/projecta_shared_access.txt

-rw-r--r--. 1 500 500 25 Jun 26 10:03 /mnt/NFSV4_KRB5/projecta_shared_access.txt



Optional

Optional

Optional

Optional

Optional

Required

Performance improvement for certain workloads. Requires special hardware typically.

Gives clients multiple servers for NFS service.

Refer client to another server for accessing certain files or directories.

Performance improvement for certain workloads. Various delegation types. Allows client to handle respective read and write requests as well as meta data operations.

Traditional NFS locking is support for advisory locks. Applications need to honor advisory locks. The NFS server enforces mandatory locks but not advisory locks.

Description

RDMA support13

Replicas, Migration12

Referrals11

Delegations10

Mandatory locking9

TitleID


Questions


IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of

International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or

other companies. A current list of IBM trademarks is available on the Web at

"Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml.

Microsoft, Windows and Active Directory are trademarks of Microsoft Corporation

in the United States, other countries, or both.

CentOS is a trademark of Red Hat, Inc.

ulf troppens c se16 - secure data access to personalized patient data - 2014-04-08

Software

tenant b

customer data center

data sharing

set of data

secure data access

data different organizations

b users of tenant

b ad server