how we ran it - gse

How we ran itz/OS data set encryption project

Lennie Dymoke-Bradshaw

Wednesday 4th November 2020

Session: 2AF

https://twitter.com/gseukc

https://www.linkedin.com/groups/1741877

https://www.facebook.com/pages/Guide-Share-Europe-UK-Region/174907039241415

Who am I?

Lennie Dymoke-Bradshaw

• Late of IBM (left in 2014)

• Been working with IBM System 370 and its successors since 1975.• Programming, System Programming, Security processes

• Currently on contract through,• my own company (Reverse Sweep Consulting Ltd.) • BMC (who acquired RSM partners earlier this year)

• I have been working in System z security since RACF 1.3 (1979) and with ICSF since 2001.

• I used to know quite a bit about JES3, but not so much now .

4th November 2020 z/OS data set encryption project - How we ran it 2

What’s this session all about then?

• At GSE in 2014, just after I left IBM, I gave a presentation at this conference challenging IBM to produce some changes.

• One of these was to produce a solution for encryption of z/OS data sets.

• IBM delivered!

• This session shows how a team tackled the implementation of this technology for a client. It may give you hints that may help you with a similar project.

• This is not a session with loads of detailed technical bits. Most of those are in the IBM manuals. • http://www.redbooks.ibm.com/abstracts/sg248410.html?Open


http://www.redbooks.ibm.com/abstracts/sg248410.html?Open

What did IBM deliver?

Originally…..

• Encryption for VSAM clusters• But they must be SMS managed• They must be Extended format

• Encryption for flat files• But they must be SMS managed• They must be Extended format

Since then……

• Support for PDSE data sets is supplied by the fix to OA56324.

• Support for JES2 Spool encrypted data sets is in z/OS 2.4

Also available (Apar OA56622)

• Support for basic format flat files (SMS managed)

• Support for LARGE format flat files (SMS managed)


Why do we want to encrypt data?

Security

• Protects data when it is accessed outside of its normal processes.• Backups accessed at another site.• Data sets that are exfiltrated via volume backups.• Volumes access from another system or sysplex.• Data flowing on PPRC links during replication .

• IBM have another solution for this now.

Compliance

• Several standards require encryption, including PCI and GDPR.• These standards require encryption to protect data that is NOT accessed using

normal processes.


How does it all work?

Please read the IBM Redbook

• Getting Started with z/OS Data Set Encryption – SG24-8410

But the quick bits are (for the way we did it)…

…….see next slides


Creating an encrypted data set: Before creation

Data set name: PAYRL.T003.MASTFILE

RACF Profile: PAYRL.T003.*

Access List:PAYRL: ALTEROTHERS: NONE

DFP Segment has Encryption Key:DSET.PLEX3.PROD.PRIM.PAYRL.T003.G01

Access List:PAYRL: READOTHERS: NONE

SYSTEM ACTIONS1. Data set name is matched to a RACF generic profile.2. RACF profile has access list used to confirm user

allowed to create this data set.3. RACF profile also has encryption key to be used to

encrypt data.4. When data set is created, label of encryption key is

stored in Catalog.

Dataset Catalog:<no-entry>

4th November 2020 7

Encryption Key defined in class CSFKEYSDSET.PLEX3.PROD.PRIM.PAYRL.T003.G01

z/OS data set encryption project - How we ran it

Creating an encrypted data set: After creation

Data set name: PAYRL.T003.MASTFILE

RACF Profile: PAYRL.T003.*

Access List:PAYRL: ALTEROTHERS: NONE

DFP Segment has Encryption Key:DSET.PLEX3.PROD.PRIM.PAYRL.T003.G01

SYSTEM STATE1. Data set access control is via Access List on RACF profile2. Encryption key access control is via Access List on

Encryption key.3. Name of Encryption Key is now stored in Catalog and

effectively “travels” with the data set.4. Data is encrypted and decrypted using “protected key”

processing.

Dataset Catalog:PAYRL.T003.MASTFILEVOL: VOL001Encrypted: YESEncryption key:DSET.PLEX3.PROD.PRIM.PAYRL.T003.G01

4th November 2020 8

Access List:PAYRL: READOTHERS: NONE

Encryption Key defined in class CSFKEYSDSET.PLEX3.PROD.PRIM.PAYRL.T003.G01

z/OS data set encryption project - How we ran it

System copies Key Label

So what bits and pieces do we need?

1. Z13, Z14 or Z15 Processor, with Crypto Express devices.

2. ICSF configured and active with AES master keys in Crypto Express(s).

3. TKE(s) for secure master key management.

4. EKMF/Web for operational key management.

5. Process: Reallocating data sets, copying, encrypting.

6. Documentation with standards, processes, procedures, ceremonies.

7. Physical: Secure rooms, safes, physical security.

NO

Yes

Yes

Yes

Yes

Yes

NO


Discussing today?


Crypto Express 5

z/OS LPAR

OSA-Express

TKE Workstation(manages master keys)

Network

CryptoApp

(CICS, IMS,

batch job)

ICSF

Specia

lIn

terfa

ce

PC interface

TKEListener

TC

P/IP

PC

inte

rface

Master keys for Crypto Domain

are stored in CEX5C

Diffie-Hellman Key Exchange

SMF

CKDSSymmetricKey store

CKDS

PKDSAsymmetricKey store

PKDS

TKDSToken

Key storeTKDS

Keys are loaded into ICSF 64-bit

memory in tree structure

EKMFWebAgent

Key Stores

SMF lo

gging

Browseraccess to

EKMF

DB2 EKMF

key database

Data

Encrypted Application data

AT-TLS Secured

Data

DataDocumentation

Standards

Processes

Procedures

Ceremonies

Roles assigned

Responsibilities assigned

Physical security

Locked rooms

Safes

Organising the (technical) work

1. ICSF implementation across all relevant LPARS and sysplexes.

2. TKE installation and deployment.

3. EKMF/Web installation and deployment.

4. Processes for encrypting the data.

5. Documenting everything…..

Five streams of work, much of which could be handled in parallel.


ICSF implementation

Things we encountered that we think are useful.

• Always look to see if there is a later version of ICSF available. Each is available as a distinct download (termed a “web deliverable”)

• If you have been using ICSF for a while, then examine all new parameters to see if they are relevant.

• If you are using ICSF in a sysplex, then use the SYSPLEXCKDS, SYSPLEXPKDS and SYSPLEXTKDS parameters to keep keys updated across sysplex.

• Examine each level of ICSF for new API services. I recommend you define each service individually to RACF.


ICSF implementation

• Look at the Key Store Policy controls. These are implemented using RACF profiles. Implement most of them if you can, especially the granular key label access controls.

• Create your keys stores nice and big. We used 20cylinders. It’s not really much disk space.

• If you are running sysplexes then keep your key stores aligned with the RACF database (or your security product of choice’s database).

• Treat your key stores in a similar manner to your RACF database in terms of access controls and backup frequency.


ICSF implementation

• Lock down the CKDS browser facility so that only your encryption team can access it.

• Update actions for the CKDS browser should be locked even tighter. Perhaps they should only be available using “break-glass” process.

• When setting up master keys for a first implementation use the ICSF panels. Migrate to the TKE once you have understood ICSF better.• We decided not to use any encryption for production data until we had the TKE

managing the master keys.

• Configure and run a TKE Listener address space. One instance is needed for each z13/z14/z15 processor. NOT one per LPAR.


So what is a TKE?

• Workstation with extra bits.

• Runs a Linux operating system that is locked.

• Runs the TKE application within Linux.

• Has its own Crypto card (IBM 476x).

• Sets up and has a crypto conversation with multiple Crypto Express devices in multiple System z processors.

• Can set up and manage master keys and some operational keys.


TKE installation and deployment

Why do we need to use TKEs? Why can’t we just use the ICSF panels?

1. Security.• TKE never lets anyone see a master key, or its key parts. • Key parts are generated on the TKE and then stored on smartcards.

2. Compliance• Master key management is an important part of the whole crypto infrastructure. You may

lose compliance if you do not have adequate controls on master keys.

3. Automation• We had 13 LPARs to set master keys on, across two sysplexes. Each LPAR can run on one of

two processors (for fast recovery). • DR site had another set of two processors.• Setting each new master key required entering each of 3 key parts (64 hex digits) into 52 (4 x

13) LPAR locations. There are 4 master keys to be set (ECC, AES, RSA, DES). Nightmare!• When set up correctly, TKE can push new master keys to all locations at the same time.



• IBM Resource link is where you will get the TKE manuals.

• Get TKE training if you don’t know the product.• See https://www.mainframecrypto.com/about/ for your training needs.• Greg Boyd covers ICSF and other crypto matters as well as TKE.

• Talk to your friendly IBM rep. Many of them know Garry Sullivan of IBM and he is very obliging and helpful.

• See the TKE sessions on Youtube for good info. These have been created by Garry Sullivan of IBM. • A few years ago now, so slightly out of date, but really useful for understanding.• Some new ones available too, mentioning z15 support.


https://www.mainframecrypto.com/about/


https://www.youtube.com/watch?v=Y8T9rSd-qrQ


• Setting up the TKE is complex and needs to be understood carefully. This is a major part of your implementation.• Get it right and it will deliver smooth operations.• However, “Our guy” who performed the TKE configuration said afterwards that it was all very logical, once you

had grasped the concepts.

• You will need at least 2 TKEs.

• Keep them physically separate for DR considerations.

• There is no remote access to a TKE• This caused us problems under COVID-19 lockdown

• TKE is a requirement for using EKMF/Web


EKMF/Web installation and deployment

• EKMF/Web provides us with a way of managing operational keys securely.

• EKMF/Web was a “Service Offering” from IBM Copenhagen, but is now a full product available on Shopz.

• This is a software only version of the EKMF workstation that IBM Copenhagen have been supplying for many years.• Previously called DKMS


EKMF = Enterprise Key Management FoundationDKMS = Distributed Key Management System


• Our team had several “teething problems” with the installation and configuration, but most of these are now resolved.

• Needs Websphere Liberty Server to provide the browser interface.• Must be z/OS 2.3 or above.

• Needs a DB2 environment for the key database.• Also needs DB2 Connect. But only for a license issue to bind one program

during installation.



Capabilities

• Can manage key definition, storage and lifecycle.

• Can set template for keys so they are named according to your standards.

• Can push keys from its repository to multiple keystore locations.

• Provides jobs and a viewer for seeing which data sets are encrypted.

• Support for keys in cloud expected soon (I think ☺).• Attend session 2AW, Weds 11th November for more details about EKMF/Web.



• This product passed all our testing and acceptance criteria.

• In our view this is a worthwhile product and we encourage IBM,• To develop it further and improve the useability of the interfaces.

• Work with IBM customers who are implementing Encryption of z/OS data sets in sysplexes to see and understand how they use the product.

• Simplify the installation and configuration.

• Add more sophisticated reporting.

• Maybe add support to write SMF records to feed to a SIEM.


Reallocating data sets, copying, encrypting

• How many data sets do you need to encrypt?• Less than 20? You can probably do it manually.• More….? You probably need some kind of automation.

• IBM supply a product called IBM z/OS Dataset Mobility Feature (zDMF).• Can handle most copying while data sets are in use.• Has some issues with blocksizes when converting to extended format.• e.g. For VSAM data sets there are 32 different Cisizes that can be used. Not all are

supported.• Works at low level I/O and converts datasets in flight.

• We developed a different solution……….


Reallocating data sets, copying, encrypting

• Requirement is to gather shape of existing data set,• From catalog,

• From DSCB (VTOC).

• Allocate a new version of the data set.

• Copy current to new.

• Rename so that new is the current.


ACE overview


ACE ProcessProduce JCL to1. Analyse data sets2. Define new data sets3. Copy old data sets to new4. Rename data sets

Application Catalog

Disk Definitions

(from VTOC)

Define new data sets

Copy data sets (and encrypt)

Rename data sets

Data set selection and

analysis

JCLDecks

ACE process builds JCL and IDCAMS statements to redefine data sets and then copy and encrypt data

CONTROLCONTROL file specifies data selection and processing options

Allocate, Copy and Encrypt (ACE)

• Primarily a REXX based suite of programs.

• Assembler module (CSIREXX) to read catalogs via IGGCSI00.• Supplies catalog values into REXX variables (similar to IRRXUTIL in that

respect).

• Produces • Reports,

• JCL and IDCAMS statements to define new datasets,

• JCL and IDCAMS statements to copy (using REPRO),

• JCL and IDCAMS statements to perform renaming (using ALTER),

• JCL and IDCAMS statements to revert back to original data sets (using ALTER).



• While doing this ACE can also,• Change SMS classes,

• Suppress Volume specifications (and many other values if needed),

• Change SPACE parameters (% up or down),

• Handle migration and recall using HSM,

• Identify “problem” data sets,

• Switch old (i.e. non-encrypted) data sets to different management class.



Just builds JCL, so,

• Application team manages copying and encryption.

• Requires data sets to NOT be in use.

• Can be stopped and restarted (normally excludes those data sets already encrypted).

• Allocates, copies and renames using standard utilities (IDCAMS).• Easy to understand and debug if required.


Other data set types

• IMS data base• Can be encrypted if correct level of IMS installed.• Need IMS 15.2 to include OSAM databases.• Unload, reallocate as extended, with encryption key in RACF DFP segment on DATACLAS.

• DB2• Can be encrypted if correct level of DB2 installed.• Unload, reallocate as extended, with RACF DFP segment on DATACLAS.

• CICS• VSAM data sets can be encrypted.

• Unsupported• Partitioned Data Sets (PDS)• Catalogs (ICF and VVDS)• VTOCs• RACF data base• Data sets accessed by EXCP (programming support available to change applications)• Temporary data sets (we have a partial solution for this – I can talk about it if we have time)• Tape data sets


Document everything

• Don’t leave documentation to the end.• Plans and pictures you draw during planning can be reused in your final

documentation.

• Make sure you have RACI plans for your processes for key management.• GDPR requires documented processes.

• Understand roles and responsibilities.

• Use the NIST recommendations for key management.• https://csrc.nist.gov/projects/key-management/key-management-guidelines


Document everything

• Have plans for your documentation.• Design should include the documents you are to produce.• They will be far better and be kept up to date if they start well.• Auditors will respect the plans.

• Details all the processes that will be needed, e.g.• Master key change.

• This will have a cast of “many” and will require a formal ceremony.• Operational key creation and deployment.• Operational key lifetime and lifecycle.• Key exposure processes.• Key register processes.

• To include details of each key and what changes are made to it.• Needs to be maintained for GDPR compliance.


Final notes

• Planning and design is everything. “Doing stuff” just follows the plan.

• Scope the project. Know how much and what type of data is to be encrypted.

• Document everything as you go. If you leave it to the end, it may never get done.

• Setting up the infrastructure may take a while, but once done, it is ready for all future applications needing encryption.

• Planning and design is everything.

• Don’t forget the documentation!


END4th November 2020 z/OS data set encryption project - How we ran it 34

Lennie Dymoke-BradshawReverse Sweep Consulting [email protected]

Please submit your session feedback!

• Do it online at http://conferences.gse.org.uk/2020/feedback/nn

• This session is 2AF


http://conferences.gse.org.uk/2020/feedback/nn

GSE UK Conference 2020 Charity

• The GSE UK Region team hope that you find this presentation and others that follow useful and help to expand your knowledge of z Systems.

• Please consider showing your appreciation by kindly donating a small sum to our charity this year, NHS Charities Together. Follow the link below or scan the QR Code:

http://uk.virginmoneygiving.com/GuideShareEuropeUKRegion


http://uk.virginmoneygiving.com/GuideShareEuropeUKRegion

Extra details on ACE

Diagrams showing ACE process in more detail.


Lots of data sets

(originals)

Lots of (empty) data

sets(marked for encryption)

Define copies

Lots of data sets

(originals)Define copiesBefore allocation

After allocation

ACE Stage 1


Lots of data sets

(originals)

Lots of (empty) data

sets(marked for encryption)

Copy data

Lots of data sets

(originals)

Lots of (full) data sets

(encrypted)

Copy data

Before copy

After copy

ACE Stage 2


Lots of data sets

(originals)


(encrypted)

Renamestep 1

Renamestep 2

Before rename


(encrypted)

Lots of data sets

(originals)

Renamestep 1

Renamestep 2

After rename

ACE Stage 3


Lots of data sets

(originals)


(encrypted)

Renamestep 2

Renamestep 1


(encrypted)

Lots of data sets

(originals)

Renamestep 2

Renamestep 1

Before recovery

After recovery

ACE Stage 4 (recovery)


Extra details onTemporary data sets

ICHRCX02 exit with controls.


Temporary data sets

• Normally deleted at end of each job or started task.

• Temporary data sets are not encrypted.

• If jobs are tasks are “blown away” they can remain.• Some types of termination.• Disk connectivity issues.• Power issues.

• Also, disk space will contain clear data unless disk space is ERASED.

• Need,• SETR CLASSACT(TEMPDSN)• Erase on Scratch


Temporary data setsQuestion: How can we set Erase-On-Scratch for temporary data sets?

Answer: SETROPTS ERASE(ALL)

This means ALL data is erased in space released on deletion or partial release of disk space for ALL dasd data sets.

Many manager are nervous about such a move.

We use a version of ICHRCX02 which will enable more granular controls of EOS for temporary data sets.


Temporary data sets

• Resource: #EOS.TEMPDSN in RACF class FACILITY.

• Access levels

• PERMIT #EOS.TEMPDSN ID(application group or id) CLASS(FACILITY) ACCESS(you-choose)


NONE No change to processing.

READ Issue messages showing what WOULD be erased if UPDATE granted.

UPDATE Erase temporary data sets and issue messages.

CONTROL Erase temporary data sets with no messages.

Extra details onExtended format data sets


Extended format data sets• Been around since mid-1990s.

• In JCL use DSNTYPE=(EXTREQ,2) or have a DATACLAS with extended format mandated. • NOTE: Cannot use DSNTYPE in IDCAMS.

• Sequential data sets can have 123 extents (per volume).

• Each physical block on DASD has 32-byte suffix.• Provides better error checking for write I/O processes over basic format.• Conversion works well if data sets are allocated with SDB (i.e. BLKSIZE=0) as these data sets

are “re-blockable”.• Can cause space issues if data sets converted to extended format if existing BLKSIZE has less

than 32 bytes slack and SDB not used.


how we ran it - gse

Documents