file-system and block-layer encryption: theory, …...data at-rest encryption file-system encryption...

2017 Storage Developer Conference. © Intel All Rights Reserved. 1

File-system and Block-layer Encryption: Theory, Practice, and Improvement

Weigang LiWenqian Yu

Data Center GroupIntel Corporation


Agenda Data at-rest encryption File-system encryption

Layered: eCryptfs, EFS Native: EXT4, ZFS

Full-disk encryption dm-crypt

Linux Kernel Crypto API Crypto Hardware Acceleration

Benchmark and Optimization Key management – Linux keyring, LUKS Summary


• Flexible to define security strategy, algorithm, key management, etc.

• Application dependent

Data at-rest EncryptionApplication-level

Encryption

File SystemEncryption

Full Disk Encryption (FDE) software

Self Encrypting Drive (SED)

Tran

spar

ency

Flex

ibilit

y

Software

Hardware

• File based key management• Transparent to application• Support multiple users / keys

• One key for whole disk (volume)• Encrypt everything on disk• OS-agnostic, Block-layer

• One Data Encryption Key (DEK) to encrypt the whole disk, protected by Authentication Key (AK).

• Hardware based, secure key is kept in hard-drive

Database

eCryptfs EFS

EXT4 ZFS

dm-crypt

LUKS


• Flexible to define security strategy, algorithm, key management, etc.

• Application dependent

Data at-rest EncryptionApplication-level

Encryption

File SystemEncryption

Full Disk Encryption (FDE) software

Self-encrypting Drive (SED)

Tran

spar

ency

Flex

ibilit

y

Software

Hardware

• File based key management• Transparent to application• Support multiple users / keys

• One key for whole disk (volume)• Encrypt everything on disk• OS-agnostic, Block-layer

• One Data Encryption Key (DEK) to encrypt the whole disk, protected by Authentication Key (AK).

• Hardware based, secure key is kept in hard-drive

Database

eCryptfs EFS

EXT4 ZFS

dm-crypt

LUKS

Our focus today


File System Encryption


Application

File System

Block Layer

Disk

kerneluser

hardware

syscalls

File-system level encryption can be implemented at different level: Fuse-based FS in User space Layered FS on top of native FS In native FS, better performance

Transparent to application. Per-file encryption and key management.

File system encryption

Layered

Native

Enc/Dec

Enc/Dec

Fuse Enc/Dec

FEK File

EFEKEnc

File

Master key


$ mount -t ecryptfs /secret /secretSelect key type to use for newly created files:1) tspi2) openssl3) passphrase4) pkcs11-helperSelection: 3Passphrase: xxxSelect cipher: xxx1) aes: blocksize = 16; min keysize = 16; max keysize = 322) blowfish: blocksize = 8; min keysize = 16; max keysize = 563) des3_ede: blocksize = 8; min keysize = 24; max keysize = 244) twofish: blocksize = 16; min keysize = 16; max keysize = 325) cast6: blocksize = 16; min keysize = 16; max keysize = 326) cast5: blocksize = 8; min keysize = 5; max keysize = 16Selection [aes]: 1Select key bytes:1) 162) 323) 24Selection [16]: 1Enable plaintext passthrough (y/n) [n]:Enable filename encryption (y/n) [n]:Attempting to mount with the following options:

ecryptfs_unlink_sigsecryptfs_key_bytes=16ecryptfs_cipher=aesecryptfs_sig=aa20c2d38cf280d5

Mounted eCryptfs

eCryptfs (upper)

Extent

Native FS (lower)

eCryptfs

auth-tokkeyring

key source

FEKEK

FEKRNG

IV (page based)

EncEnc

EFEK

eCryptfs = Enterprise Cryptographic Filesystem.

Layered file system. In Linux kernel since version

2.6.19. Mount eCryptfs on top of a

directory to protect it. Un-mount eCryptfs, the files in

lower FS are encrypted and un-readable.

The Encrypted FEK (EFEK) is stored in the lower FS.

KDF

Per page encryption

PlaintextPage

CiphertextPage

IV (page based)

Ref: http://ecryptfs.sourceforge.net/ecryptfs.pdf

http://www.linuxjournal.com/article/9400?page=0,0


EFS = Encrypting File System

Filter driver layered on top of NTFS

EFS

Source: https://technet.microsoft.com/library/bb457116.aspx#EJAA

https://technet.microsoft.com/library/bb457116.aspx#EJAA


Combination of public key (RSA) and symmetric key encryption (3DES, AES).

FEK is protected by user’s public key.

EFEK is decrypted by user’s private key to unlock the encrypted file.

FEK can be re-encrypted by other’s pub-key to share the encrypted file with other user w/o re-encrypting the file content.

EFS (Cont.)

On-disk

pub-key

Plaintext

Ciphertext

FEKRNGEncryption

RSAEncryption

EFEK

pri-key RSADecryption

FEK Decryption

ENC

DEC

Plaintext


Key-ring

In Linux kernel since version 4.1. Per-directory Master key. Per-file encryption key (FEK, derived from master key and nonce). Support AES-256-XTS.

EXT4 encryption

On-disk

e4crypt / keyctl

inodenonce

Master key

FEK

RNGIV (page based)

EncEnc

setkey

inodenonce

PlaintextPage

CiphertextPage

(Linux Kernel 4.12)

$ mkfs -t ext4 /dev/nvme1n1$ tune2fs -O encrypt /dev/nvme1n1$ mount /dev/nvme1n1 /mnt/ext4$ e4crypt add_key /mnt/ext4Enter passphrase (echo disabled): xxxKey with descriptor [775a2062517e439c] applied to /mnt/ext4.$ keyctl list @s2 keys in keyring:453787240: --alswrv 0 65534 keyring: _uid.01024073693: --alsw-v 0 0 logon: ext4:775a2062517e439c


Keystore

On-diskkey source

ZOL = ZFS On Linux http://zfsonlinux.org/

ZOL encryption is not in mainline yet PR: https://github.com/zfsonlinux/zfs/pull/5769

Support AES-CCM, AES-GCM

ZOL encryption

Masterkey

Wrapping key

FEKKDF

$ zpool create -o ashift=13 testpool /dev/..$ zfs create -o encryption=aes-128-ccm testpool/tankEnter passphrase for 'tank': xxxEnter again: xxx$ zfs get all testpool/tanktestpool/tank encryption aes-128-ccm local

passphrase

file

uri

RNG

KDF

blkptr_t

IV

Salt

E(master key)

MAC

Salt

IV

Enc

Enc

Seal master key IV

MAC

IV

PlaintextBlock

CiphertextBlock

http://zfsonlinux.org/

https://github.com/zfsonlinux/zfs/pull/5769


Block Layer Encryption


volume

Block layer encryption

Application

File system

Block layer

Disk

Encrypt Decrypt

kernel

user

hardware

syscalls

Encrypt everything on the disk – one key for whole disk (volume).

Hides file and directory information, such as name and size.

OS-agnostic.

Masterkey

WrappedkeyEnc

Password

volume


cryptsetup

In Linux kernel since version 2.6. Per-sector encryption. Support AES-XTS, AES-CBC. Single master key for whole disk. Using LUKS for key management - 8 key

slots.

dm-crypt

On-disk

Master key

passphrase

RNG

EncEncsetkey

LUKS HDR

Key Slotkey-file

$cryptsetup luksFormat -c aes-xts-plain64 -s 512 / dev/nvme3n1 $cryptsetup luksFormat -c aes-cbc-essiv:sha256 -s 256 / dev/nvme3n1 Are you sure? (Type uppercase yes): YESEnter passphrase: xxxVerify passphrase: xxx$cryptsetup open --type luks /dev/nvme3n1 dm-crypt-diskEnter passphrase for /dev/nvme3n1: xxx$ lsblknvme3n1 259:1 0 745.2G 0 disk└─dm-crypt-disk 253:3 0 745.2G 0 crypt$ mkfs -t ext4 /dev/mapper/dm-crypt-disk$ mount /dev/mapper/dm-crypt-disk /dm-crypt-disk/$ df -T/dev/mapper/dm-crypt-disk ext4 1% /dm-crypt-disk$ umount /dm-crypt-disk$ cryptsetup close dm-crypt-disk

PlaintextSector (512B)

(sector based)IV

(sector based)

CiphertextSector (512B)


Linux Kernel Crypto API


Linux Kernel Crypto APILinux Kernel sub-system


Engine Driver

API call

Crypto Hardware

Request Response

Callback

Data Transform Symmetric key ciphers: skcipher Asymmetric key ciphers:

akcipher AEAD ciphers Message digest Random number generation Compression

Crypto driver is registered and selected based on its priority.

Support asynchronous operation for hardware acceleration.

Ext4eCryptfs

dm-cryptIPsec

…

MemorySubmit

Pagesg_list

DMA

Callback

$ cat /proc/crypto:


Cryptographic Hardware Acceleration


Intel® QuickAssist Technology

1

Intel® QuickAssist Technology integrates hardware acceleration for compute intensive workloadsSuch as Bulk Cryptography, Public Key Exchange & Compression on Intel® Architecture Platforms

0

10

20

30

40

50

60

70

80

90

100

110

RSA 2KDecrypt(kOps/s)

IPSecForwarding

(Gbps)

SSLWebProxy

(Gbps)

Software-based OpenSSL with Intel® QAT

99

87

T E R A S O R T T I M E I N M I N U T E S L O W E R I S B E T T E R

SW Snappy Compression

Intel® QAT Compression

Security Benchmarks

Big Data Benchmarks with Compression

Hadoop run time reduced significantly1. NGINX* and OpenSSL* connections/second. Conducted by Intel Applications Integration Team. Claim is actual performance measurement.

Intel® microprocessor. Processor: Intel® Xeon® processor Scalable family with C6xxB0 ES2Performance tests use cores from a single CPU, Memory configuration:, DDR4–2400. Populated with 1 (16 GB) DIMM per channel, total of 6 DIMMsIntel® QuickAssist Technology driver: QAT1.7.Upstream.L.0.8.0-37 Fedora* 22 (Kernel 4.2.7) BIOS: PLYDCRB1.86B.0088.D09.16060117363. Cloudera* 5.4.2 with Snappy* Software vs. Intel® QuickAssist Technology hardware solution. Conducted by Intel Applications Integration Team.Claim is actual performance measurement. Intel® Xeon® processor E5-2699 v4 (56 cores enabled) 256 GB DDR4 1.6 TB NVMe SSD 1 Intel® C6xxx-based card (24x)10 Gbps CentOS* 6.7 w/ 2.6.32 kernel Cloudera* 5.4.2QAT driver 0.9.1 Snappy* 1.1.2 (popular, fast compression codec)One NameNode Eight DataNodes 10 Gbps network2- 24 Core Intel(r) Xeon Scalable Platform -SP @1.8GHz, Single (UP) Processor configuration. Intel(r) C627 PCH with crypto acceleration capability (in x16 mode) Neon City platform. DDR4 2400MHz RDIMMs 6x16GB(total96 GB), 6 Channels, 1 x Intel® CorporationRed Rock Canyon 100GbE Ethernet Switch in the x16 PCIe slot on Socket 0. 8 cache ways allocated for DDIO.

1 1

2

3


z

Gen 2 Intel® QuickAssist Technology with Intel® Key Protection Technology (KPT)

Intel® QuickAssist Technology

DRAM

Symmetric Wrapping Key (SWK)Private Key (PK) Wrapped Private Key (WPK)

Secure Channel

Intel® PTT

Intel® C627 Chipset with Intel® QAT

DRAM

LBG KPT

Unwrapping (Decrypt)

Without Intel® KPT With Intel® KPT

• Private key exposes in clear text• Key is not protected and unsafe• Subject to attacks

• Private key is wrapped (encrypted)• Key is protected and safe• NOT subject to attacks

1


sync mode

Synchronous: For high performance “single

stream” accelerator Simple program model

Asynchronous: For multi-engines hardware High throughput Complex control flow

Cryptographic hardware program model

IO

Engine

Computing Wait

Hardware

CPUWait

async mode

IO

Engine

Computing

Hardware

CPUSwap

Engine

Engine

Offload OffloadComplete Complete

Offload Complete


Extent

Synchronous call causes performance drop!

Ext4 encryption with hardware acceleratorExtent

Page


Submit BIO

bio_vec

• add the encrypted page to bio_vec

• encrypt the next page

vec

CryptoEngine

Sequential

all pages are encrypted

n

Page 2

Page 1

Page

Page

Page

vecvec

Callback

wakeup

0

200

400

600

800

1000

off s/w encryption qat-syncencryption

Throughput(MB/s)

HW is not fully utilized!

Intel® microprocessor. Processor: Intel® Xeon® E5-2699 v4 @ 2.20GHz @88 HT cores, Memory configuration:, DDR4–2400. Populated with 1 (16 GB) DIMM per channel, total of 4 DIMMs. 1X P3700 NVMe. Intel® QuickAssist Technology DH8950, in-tree driver in kernel 4.12

dd if=/dev/zero of=/mnt/ext4/testfile bs=1M count=10000 oflag=sync


Block size (128KB) is “big enough” for one offloading. ZIO pipeline drives compression, checksums, data

redundancy, and encryption, etc., with a pool of threads (taskq) to improve performance.

Hardware accelerator is fully utilized.

ZOL encryption with hardware accelerator

Intel® microprocessor. Processor: Intel® Xeon® E5-2699 v4 @ 2.20GHz @88 HT cores, Memory configuration:, DDR4–2400. Populated with 1 (16 GB) DIMM per channel, total of 4 DIMMs. 3X P3700 NVMe. Intel® QuickAssist Technology driver: QAT1.6 v2.6. Cent OS* 7.2 (Kernel 3.10) ZOL 0.7.0 with PR (https://github.com/zfsonlinux/zfs/pull/5769). FIO-2.1.2, sequential write, 16 threads per NVMe disk.ZOL encryption algorithm: AES-CCM-128

FIO Sequential Write to different number of NVMe drives

0

1000

2000

3000

4000

5000

1 x nvme 2 x nvme 3 x nvme

Throughput(MB/s)

off s/w qat

05

1015202530354045

1 x nvme 2 x nvme 3 x nvme

CPU utilization @88 HT cores (%)

off s/w qat

CPU utilization reduced

https://github.com/zfsonlinux/zfs/pull/5769


Offloading cost: Create request descriptor Interrupt handling, response

polling Hardware* offloading cost is

almost consistent: ~1600-1800 cycles. Choose right buffer size to

get best performance with reasonable offloading cost.

Software cost varies by buffer size.

Hardware offloading cost

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

0 2000 4000 6000 8000 10000 12000 14000 16000 18000

Cost

(CPU

Cyc

les)

Buffer Size(Bytes)

Intel® QAT (DH8950) offloading cost vs. Software cost (Intel® Xeon® E5-2699 v4 )

qat_aes_xts xts-aes-aesni

Intel® microprocessor. Processor: Intel® Xeon® E5-2699 v4 @ 2.20GHz @88 HT cores, Memory configuration:, DDR4–2400. Populated with 1 (16 GB) DIMM per channel, total of 4 DIMMs. * Intel® QuickAssist Technology DH8950, in-tree kernel driver.Benchmark code calls Kernel Crypto API (skcipher), compute average CPU time with 1000x API calls.Cycle count as measured at Kernel Crypto (skcipher) API for AES-XTS algorithm.


Page (4KB)

Page (4KB)

dm-crypt with hardware acceleration dm-crypt calls Kernel Crypto API for data

encryption, it can integrate with a hardware accelerator seamlessly.

asynchronous call: sending 8x requests for one 4KB page in-parallel, no wait.

But: offloading cost is high - Encrypting a 4KB page 8x

encryption requests. Encrypting 1MB data 2048x

encryption requests.

Page (4KB)Sector 512B

Submit BIO

iv

dm-crypt(for each page)

Sector 512B

Sector 512B

iv

iv

Kernel Crypto API

Accelerator

API call x8 Callback x8

Page (4KB)

Sector 512B

Sector 512B

Sector 512B

When 8 sectors all done

Submit

request x8 Callback x8

Driver


Page (4KB)

Page (4KB)

dm-crypt optimization Encrypt 8 sectors (sg_list) in a single call. Offloading cost is reduced greatly. Kernel Crypto API or accelerator hardware to

compute IV for each sector.

0100200300400500600700800900

w/o optimization with optimization

Throughput (MB/s)

Page (4KB)

Sector 512B

Submit BIO

dm-crypt(for each page)

Sector 512B

Sector 512B

Kernel Crypto API

Accelerator

API call x1 Callback x1

Page (4KB)

Sector 512B

Sector 512B

Sector 512B

Submit

request x1 Callback x1

Driver

sg_list

Intel® microprocessor. Processor: Intel® Xeon® E5-2699 v4 @ 2.20GHz @88 HT cores, Memory configuration:, DDR4–2400. Populated with 1 (16 GB) DIMM per channel, total of 4 DIMMs. 1X P3700 NVMe. Intel® QuickAssist Technology DH8950, in-tree driver in kernel 4.12

dd if=/dev/zero of=/dev/mapper/dm-c bs=1M count=10000 oflag=sync

Note: experimental result, patch is not in upstream kernel yet.

iviv

iv


Key Management


In-kernel key management and retention facility:http://man7.org/linux/man-pages/man7/keyrings.7.html

Key management – keyring

keyring

keyctl

Kernel sub-system

request key


kernel

setkey

user

http://man7.org/linux/man-pages/man7/keyrings.7.html


Key management - LUKSMaster

key

Password

slot#0

slot#7

Encrypted Data

Disk Format

PlaintextData

slot#2

Encryption

Linux Unified Key Setup, it is a specification (on-disk format) for disk encryption.

Implemented in cryptsetup utility + dm-crypt (kernel module) for disk encryption.

Master key to encrypt / decrypt data. Password to encrypt / unlock the master key. Support 8 key slots.

# cryptsetup luksDump /dev/nvme3n1LUKS header information for /dev/nvme3n1Version: 1Cipher name: aesCipher mode: xts-plain64Hash spec: sha1Payload offset: 4096MK bits: 512MK digest: ed 36 e3 09 00 72 ea ba 24 04 6e 11 7e 69 1a 87 5a d1 32 57MK salt: bc 41 51 8e e0 90 65 20 70 6d ef 2f 3f 80 06 a7…MK iterations: 48000UUID: 0f6c5194-b299-4f3e-9f54-17dc67a62f04Key Slot 0: ENABLED

Iterations: 192480Salt: 62 4b 4d 54 b1 d5 0c 44 c2 a9 a0 d3 03 36 59 78…Key material offset: 8AF stripes: 4000

Key Slot 1: DISABLEDKey Slot 2: DISABLEDKey Slot 3: DISABLEDKey Slot 4: DISABLEDKey Slot 5: DISABLEDKey Slot 6: DISABLEDKey Slot 7: DISABLED


If data encryption / decryption is done in host CPU, keys (FEK, master key, private key…) have to be exposed in memory Cold boot (memory dump) attack Heartbleed-like attack

Possible solution SGX Enclave Total Memory Encryption (TME) Debug-register-based or cache-

based key storage

Memory-based key management

CPU

Memory

Enc / DecLibrary


HSM = Hardware Security Module KEK = Key Encryption Key Session key (e.g., FEK) is generated by HSM

and wrapped by KEK, the wrapped-key can send back to host memory.

The wrapped-key is a “key handler” to request HSM service.

Key storage and data encryption / decryption are all kept inside device. Clear key material never leaves device.

HSM-based key management

HSM

CryptoHardware Engine

Root Key

CPU

Memory

Key Gen

Wrapped-key

CryptoLibrary

Key hierarchy tree KEK

Buffer

Enc / Dec

Key provision


File-system and full-disk encryption are important software technologies to implement data-at-rest encryption solution.

Data encryption at different layer has its own pros and cons. Data encryption software shall consider how to best utilize the

hardware crypto accelerator underneath in order to get best performance, e.g., buffer size, sync vs. async, offloading cost, etc.

Besides data encryption, secure-key protection is even more important for persistent data storage.

Hardware based key management is a possible solution to protect key leakage.

Summary


References http://ecryptfs.sourceforge.net/ecryptfs.pdf https://zfs.datto.com/slides/caputi.pdf https://technet.microsoft.com/library/bb457116.aspx#EJAA http://kernsec.org/files/lss2014/Halcrow_EXT4_Encryption.pdf www.intel.com/content/www/us/en/embedded/technology/quickassist/overview.html

http://www.linuxjournal.com/article/9400?page=0,0

https://zfs.datto.com/slides/caputi.pdf

https://technet.microsoft.com/library/bb457116.aspx#EJAA

http://kernsec.org/files/lss2014/Halcrow_EXT4_Encryption.pdf

http://www.intel.com/content/www/us/en/embedded/technology/quickassist/overview.html

file-system and block-layer encryption: theory, …...data at-rest encryption file-system encryption...

Documents