RISCV with Sanctum EnclavesVictor Costan, Ilia Lebedev, Srini Devadas

If computing remotely, what is the TCB?

CPU HW

Hypervisor

Process

OS

2

Today, privilege implies trust (1/3)

trusted computing base

Process Process

OS

VM boundaryProcess boundary

Priv

iledg

e

Sanctum decouples HW protection from trust!

CPU HW

Hypervisor

Process

OS

Process

150KLOC

12,000KLOC

??? LOC

Process

OS

3VM boundary

Today, privilege implies trust (2/3)

Process boundary

trusted computing base

Priv

iledg

e

If computing remotely, what is the TCB?

Side channels leak privacy

Sanctum uses hardware-assisted isolation

Strong privacy & integrity with low overhead

CPU HW

Hypervisor

Process

OS

Process Process

OS

Today, privilege implies trust (3/3)

4

Priv

iledg

e

VM boundaryProcess boundary

Remote Software Attestation (1 /2)

5

Trusted HW

SoftwareEcosystem

App

Remote user

Measures (hash)and Signs

Diffie Hellman

Trusted HW creates proof for remote user

6

Trusted HW

SoftwareEcosystem

App

Remote user

Diffie Hellman

Private data

Private result

Remote user decides whether or not to trust certificate

Remote Software Attestation (2 /2)

Measures (hash)and Signs

Prior work: TPM, TXT

7

Tamper-resistant HW

BIOS,OS,

Device drivers and all

App

12,000+KLOC

Remote user

Diffie Hellman

Private data

Private result

Trusted HW must keep its keys private!

Software ecosystem must not be vulnerable

Prior work included too much SW in their attestation

Trusted Platform Module

Trusted Execution Technology

Measures (hash)and Signs

Recent “Success”: Intel SGX

8

Intel Factory

Intel SGX CPU

Process

OS

ProcessRing 3

(user mode)

12,000KLOC

??? LOC

Ring 0(supervisor mode)

Enclave

Priv

iledg

e

Enclave ≔ immutable commitment of resources to one process.

Threat Model (Intel SGX)Protect privacy* and integrity** of an enclave against

a privileged SW adversary (OS/Hypervisor)

SGX prevents:

9

Directly reading/ tampering with:

- Config. registers- Enclave memory- Enclave process structures- DMA-capable devices

Some specific physical attacks:

- DRAM contents- Device secret

Physical access!

*Indirect observation?

**Can SGX keep its keys private?

Threat Model (Sanctum)

10

Directly read/ tamper with:

- Config. registers- Enclave memory- Enclave process structures- DMA-capable devices

Indirectly* observe private state via shared:

- Caches- Microarchitectural state- Data structures managed by the OS- Interrupts / Faults

No protection against physical access or fault injection

Protect privacy and integrity of an enclave against

a privileged SW adversary (OS/Hypervisor)

Sanctum prevents:

Sanctum’s Chain of Trust

11

Manufacturer

Sanctum HW

Security Monitor(SM)

Process

OS

ProcessUnprivileged

(user mode)

Privileged(supervisor mode)

Machine mode

Priv

iledg

e Certificate Authority

Strongly Isolated Enclave

12

- Formally verifiable open source SM

- Truly small TCB for attestation

(small SW and HW)

- Hardware-assisted isolation with strong guarantees

Sanctum’s ContributionsManufacturer

Sanctum HW

Security Monitor(SM)

Process

OS

Process

Certificate Authority

Strongly Isolated Enclave

Open source SW

Stronger security with low overhead+ 2% area

- 1-13% performance

Unprivileged(user mode)

Privileged(supervisor mode)

Machine mode

12,000KLOC

5 KLOC

Hard-wired

Priv

iledg

e

Sanctum modifies a RISCV Rocket Chip:

13

LLC Bank

Private Cache Private Cache

Core Core

DRAM

LLC Bank

TLB TLB

PTW PTW

Datapath Datapath

2MB LLC

16KB

4 cores

64b RISCV

Build hardware support for

small, trusted software to

enforce isolation guarantees

for unprivileged software.

Explicitly multiplex between private and OS-controlled state

Enclaves execute on private cores

14

LLC Bank

Private Cache Private Cache

Core Core

DRAM

LLC Bank

TLB TLB

PTW PTW

Datapath Datapath

- Private L1 caches- Private registers,- Private branch

target buffer- Private TLB

Security monitor cleans up private state when allocating cores to enclaves / OS

Isolated Physical Memory in Sanctum

15

“DRAM region” is a unit of isolated memory

Physical Address Space

Security Monitor OS / non-enclave processes Enclave 1 Enclave 2

DRAM regions are non-overlapping

OS Virtual Address Space An enclave’s Virtual Address Space

OS Page Tables Enclave private data

Enclave 1 Private Page Table

Kernel Memory

Only security monitor has access

Isolating in the LLC (1/8)

16

Set Index Line OffsetPhysical Address Tag6

Set associative (banked) cache

==

1115

====

64 bytes

Select sets with set index

Find way with matching tag

8 ways

512 sets,

4 banks

Select word from line

LLC

Isolating in the LLC (2/8)

17

Set Index Line OffsetPhysical Address Tag6

LLC sharing leaks privacy!

==

1115

====

64 bytes

LLC OS starves enclave at LLC!

Availability of this set leaks privacy

Isolating in the LLC (2/8)

18

Set Index Line OffsetPhysical Address Tag6

LLC sharing leaks privacy!

==

1115

====

64 bytes

LLCGive private LLC sets to enclaves!

Isolating in the LLC (4/8)

19

Physical Page Number

Page Offset

Page Offset

Virtual Address

Physical Address

Page tablesOS controls

4KB

1220

52 or fewer

Virtual Page Number

Virtual address translation

(TLB caches translations)PTW performs

translation

Isolating in the LLC (6/8)Physical Page Number Page Offset

12

Set Index Line OffsetTag61115

“DRAM Region Index”

Address translation

LLC

12-6=5bits controlled by the

OS”!

20

To isolate enclaves in LLC, allocate exclusively, at region granularity!

Isolating in the LLC (7/8)Physical Page Number Page Offset

12

Set Index Line OffsetTag61115

“DRAM Region Index”

DRAM

Each region is 1

4K page in size

Address translation

LLC

12-6=5bits!

Toy example: 3 DRAM region bits

21

32KBSmall problem : A DMA buffer

To isolate enclaves in LLC, allocate exclusively, at region granularity!

...

Isolating in the LLC (8/8)

22

Physical Page Number Page Offset

12

Set Index Line OffsetTag61115

“DRAM Region Index”

DRAM

Each region is contiguous, 32KB

Rotate PPN to make colors contiguous in DRAM

32KB

Now top PA bits determine DRAM region...

Toy example: 3 DRAM region bits

23

Page Table Walker

Cache & DRAM

Valid bit Page table entry

Requ

est

ResponseAddress

&Responsevalid bit

TLBHardware-assisted Isolation

Maintain an invariant:TLB entries are safe!

HW enforces invariants at page walk

Invariant checker

S.M. sanitizes mode switch

24

Page Table Walker

Cache & DRAM

PRBASE

==

Valid bit Page table entry

Requ

est

&Responsevalid bit

&PRMASK

TLBProtect SM memory from everyone

OS could rewrite S.M. code, do evil

fix by...

Never map VAddr to SM memory

ResponseAddress

Isolating Enclaves in Physical Memory

25

Page Table Walker

Cache & DRAM

DRBMAP

&

Region index to One Hot

Valid bit

Requ

est

ResponseAddress

&Responsevalid bit

Page table entry

TLB

...other invariantsOS could read/write Enclave memory

fix by...

Enforce DRAM Region permissions to at page walk

S.M. updates permissions when scheduling enclaves

Isolating enclave page tables

26

EVMASK &VAddr

EVBASE ==

Should this VA use enclave’s tables?

PTBR

DRBMAP

PARBASE

PARMASK

EPTBR EPARBASE

EDRBMAP EPARMASK

OS could spy on enclave’s page table entries

fix by...

Implement enclave-private page tables

Sanctum modifies RISCV Rocket Chip (1/3)

Enclave PTs

PTW invariant checker

27

LLC Bank

Private Cache Private Cache

Core Core

DRAM

LLC Bank

TLB TLB

Datapath Datapath

~500 Gates, ~700 FFsPer core

Add hardware at interfaces only!PTW PTW

Sanctum modifies RISCV Rocket Chip (2/3)

Enclave PTs

PTW invariant checker

LLC addr. Rotation

DMA whitelist

(hacks upon hacks)28

LLC Bank

Private Cache Private Cache

Core Core

DRAM

LLC Bank

TLB TLB

Datapath Datapath

600 Gates, 128 FFs

~50 Gates

Add hardware at interfaces only!PTW PTW

~6% overall performance overhead!

Sanctum modifies RISCV Rocket Chip (3/3)

Enclave PTs

PTW invariant checker

LLC addr. Rotation

DMA whitelist

Device secret

Boot ROM

29

LLC Bank

Private Cache Private Cache

Core Core

DRAMBoot ROM

DeviceSecret

LLC Bank

TLB TLB

Datapath Datapath

~2% area increase in total!

128 FFs

Add hardware at interfaces only!PTW PTW

Measuring Overheads: Experimental setup

30

RISCV Compiler

Platform headers

Spec‘06*

RISCV isa-sim

Performance counters for

Sanctum-related events

Sanctum LLC model

riscv-linux

>275 billion instructions simulated!

Post-process event log and counters

Estimated static costs for Monitor API calls.

$ and DRAM access costs modeled after real devices

this slide is intentionally left blank

31

32

Enclave overhead with a DRAM region allocation of 1/4 of LLC sets

Do not use LLC effectively

Perform lots of IO

Benefit from a large LLC

A Detailed Look at Sanctum’s Overheads (2/3)

33

Enclave overhead for various enclave sizes

A Detailed Look at Sanctum’s Overheads (3/3)

Trusted Manufacturer as CA

34

Remote Software Attestation

35

Sanctum’s Software Stack

36

Attestation in Sanctum

37

Physical Memory in Sacntum (1/2)

38

Physical Memory in Sacntum (1/2)

39