A Double-Edged Sword: Security Threats and Opportunities

in One-Sided Network Communication

Shin-Yeh Tsai Yiying Zhang

CPU

Traditional (Two-sided communication)

�2

Memory

User A

Server

CPU

Traditional (Two-sided communication)

�2

Memory

User A GET Key-50

Send→ Server

CPU

Traditional (Two-sided communication)

�2

Memory

User A GET Key-50Key-50

Send→ Server

CPU

Traditional (Two-sided communication)

�2

Memory

User A GET Key-50Key-50

←Reply Server

CPU

Traditional (Two-sided communication)

�2

Memory

User A GET Key-50

SET Key-100

Key-50

User B

Send→ Server

CPU

Traditional (Two-sided communication)

�2

Memory

User A GET Key-50

SET Key-100

Key-50

Key-100User B

Send→ Server

CPU

Traditional (Two-sided communication)

�2

Memory

User A GET Key-50

SET Key-100

Key-50

Key-100User B

←Reply Server

CPU

Traditional (Two-sided communication)

�2

Memory

User A GET Key-50

SET Key-100

Key-50

Key-100User B

One-sided communication

MemoryCPU

Server

CPU

Traditional (Two-sided communication)

�2

Memory

User A GET Key-50

SET Key-100

Key-50

Key-100User B

One-sided communication

MemoryCPU

Server

CPU

Traditional (Two-sided communication)

�2

User A

Memory

User A GET Key-50

SET Key-100

Key-50

Key-100User B

One-sided communication

MemoryCPU

Server

CPU

Traditional (Two-sided communication)

�2

GET Key-50User A

Memory

User A GET Key-50

SET Key-100

Key-50

Key-100User B

One-sided communication

MemoryCPU

Server

Key-50

CPU

Traditional (Two-sided communication)

�2

GET Key-50User A

User B

Memory

User A GET Key-50

SET Key-100

Key-50

Key-100User B

One-sided communication

MemoryCPU

Server

Key-50

CPU

Traditional (Two-sided communication)

�2

SET Key-100

GET Key-50User A

User B

Memory

User A GET Key-50

SET Key-100

Key-50

Key-100User B

One-sided communication

MemoryCPU

Server

Key-50

Key-100

�3

RDMA

Omni-Path

NVMeOF

GPUDirect

Gen-Z

�3

[ATC ’13]Pilaf

[NSDI ’14]FaRM

[SIGCOMM ’14]HERD

[SOSP ’15]DrTM

[SoCC ’17]APUS

[SOSP ’15]FaRM + Xact

[ASPLOS ’15]Mojim

[EuroSys ’16]DrTM+R

[VLDB ’16]RSI

[OSDI ’16]FaSST

[ATC ’16]Cell

[OSDI ’16]Wukong

[SoCC ’17]Hotpot

[ATC ’17]Octopus

[VLDB ’17]NAM-DB

[OSDI ’18]DRTM+H

[SOSP ’17]LITE [SOSP ’17]

KV-Direct

[FAST ’19]Orion

[SYSTOR '19]Storm

RDMA

Omni-Path

NVMeOF

GPUDirect

Gen-Z

�3

[ATC ’13]Pilaf

[NSDI ’14]FaRM

[SIGCOMM ’14]HERD

[SOSP ’15]DrTM

[SoCC ’17]APUS

[SOSP ’15]FaRM + Xact

[ASPLOS ’15]Mojim

[EuroSys ’16]DrTM+R

[VLDB ’16]RSI

[OSDI ’16]FaSST

[ATC ’16]Cell

[OSDI ’16]Wukong

[SoCC ’17]Hotpot

[ATC ’17]Octopus

[VLDB ’17]NAM-DB

[OSDI ’18]DRTM+H

[SOSP ’17]LITE [SOSP ’17]

KV-Direct

[FAST ’19]Orion

Performance[SYSTOR '19]

Storm

RDMA

Omni-Path

NVMeOF

GPUDirect

Gen-Z

�3

[ATC ’13]Pilaf

[NSDI ’14]FaRM

[SIGCOMM ’14]HERD

[SOSP ’15]DrTM

[SoCC ’17]APUS

[SOSP ’15]FaRM + Xact

[ASPLOS ’15]Mojim

[EuroSys ’16]DrTM+R

[VLDB ’16]RSI

[OSDI ’16]FaSST

[ATC ’16]Cell

[OSDI ’16]Wukong

[SoCC ’17]Hotpot

[ATC ’17]Octopus

[VLDB ’17]NAM-DB

[OSDI ’18]DRTM+H

[SOSP ’17]LITE [SOSP ’17]

KV-Direct

[FAST ’19]Orion

Performance[SYSTOR '19]

Storm

Scalability

RDMA

Omni-Path

NVMeOF

GPUDirect

Gen-Z

�3

[ATC ’13]Pilaf

[NSDI ’14]FaRM

[SIGCOMM ’14]HERD

[SOSP ’15]DrTM

[SoCC ’17]APUS

[SOSP ’15]FaRM + Xact

[ASPLOS ’15]Mojim

[EuroSys ’16]DrTM+R

[VLDB ’16]RSI

[OSDI ’16]FaSST

[ATC ’16]Cell

[OSDI ’16]Wukong

[SoCC ’17]Hotpot

[ATC ’17]Octopus

[VLDB ’17]NAM-DB

[OSDI ’18]DRTM+H

[SOSP ’17]LITE [SOSP ’17]

KV-Direct

[FAST ’19]Orion

Performance[SYSTOR '19]

Storm

ScalabilityUsability

RDMA

Omni-Path

NVMeOF

GPUDirect

Gen-Z

�3

[ATC ’13]Pilaf

[NSDI ’14]FaRM

[SIGCOMM ’14]HERD

[SOSP ’15]DrTM

[SoCC ’17]APUS

[SOSP ’15]FaRM + Xact

[ASPLOS ’15]Mojim

[EuroSys ’16]DrTM+R

[VLDB ’16]RSI

[OSDI ’16]FaSST

[ATC ’16]Cell

[OSDI ’16]Wukong

[SoCC ’17]Hotpot

[ATC ’17]Octopus

[VLDB ’17]NAM-DB

[OSDI ’18]DRTM+H

[SOSP ’17]LITE [SOSP ’17]

KV-Direct

[FAST ’19]Orion

Performance

What about Security?

[SYSTOR '19]Storm

ScalabilityUsability

RDMA

Omni-Path

NVMeOF

GPUDirect

Gen-Z

Outline• Introduction and Background

• Vulnerabilities in One-Sided Communication

• Vulnerabilities in One-Sided Hardware

• Opportunities in One-Sided Communication

• Conclusion

�4

Vulnerability 1: Lack of Accountability

�5

Memory

User ACPU

• WRITE accountability

Vulnerability 1: Lack of Accountability

�5

Memory

User ACPU

SET Key-50

• WRITE accountability

Vulnerability 1: Lack of Accountability

�5

Memory

User ACPU

SET Key-50

Server: Who SET the (corrupted) record?

• WRITE accountability

Vulnerability 1: Lack of Accountability

�5

Memory

User ACPU

SET Key-50

Server: Who SET the (corrupted) record?

• WRITE accountability

• READ accountability

Vulnerability 1: Lack of Accountability

�5

Memory

User ACPU

GET Key-100

SET Key-50

Server: Who SET the (corrupted) record?

User B

• WRITE accountability

• READ accountability

Vulnerability 1: Lack of Accountability

�5

Memory

User ACPU

GET Key-100

SET Key-50

Server: Who SET the (corrupted) record?

Server: Who GET the (corrupted) record?User B

• WRITE accountability

• READ accountability

Vulnerability 2: Denial of Service

�6

Attack hardwareMetadata1 Metadata2

• Hard to trace attackers

• Can overload NICs easily

Attacker

Vulnerability 2: Denial of Service

�6

Attack hardwareMetadata1 Metadata2

• Hard to trace attackers

• Can overload NICs easily

Attacker

Vulnerability 2: Denial of Service

�6

Attack hardwareMetadata1 Metadata2

• Hard to trace attackers

• Can overload NICs easily

Attacker

Vulnerability 2: Denial of Service

�6

Attack hardwareMetadata1 Metadata2

• Hard to trace attackers

• Can overload NICs easily

Discussion and Defense• Adding intermediate layer at the sender side

�7Library NIC CPU

MemoryClient

Client NIC CPU

Memory

Intermediate Layer

Discussion and Defense• Adding intermediate layer at the sender side

• Enhancing SmartNIC at the receiver side

�7

SmartNIC

Client NIC CPU

Memory

Client NIC CPU

MemorySoC

Library NIC CPU

MemoryClient

Client NIC CPU

Memory

Intermediate Layer

Outline• Introduction and Background

• Vulnerabilities in One-Sided Communication

• Vulnerabilities in One-Sided Hardware

• Opportunities in One-Sided Communication

• Conclusion

�8

One- and Two-Sided Hardware

�9

Memory

BerkeleySocket

CPU User

Kernel

Two-Sided

One- and Two-Sided Hardware

�9

Memory

BerkeleySocket

CPU User

Kernel

Two-Sided

One- and Two-Sided Hardware

�9

Memory

BerkeleySocket

CPU User

Kernel

Two-Sided

One- and Two-Sided Hardware

�9

1. Address mapping 2. Permission checking 3. Resource isolation

Memory

BerkeleySocket

CPU User

Kernel

Two-Sided

One- and Two-Sided Hardware

�9

1. Address mapping 2. Permission checking 3. Resource isolation

Memory

CPU UserKernel

One-sided CommunicationOne-Sided

Memory

BerkeleySocket

CPU User

Kernel

Two-Sided

One- and Two-Sided Hardware

�9

1. Address mapping 2. Permission checking 3. Resource isolation

Memory

CPU UserKernel

One-sided CommunicationOne-Sided

Memory

BerkeleySocket

CPU User

Kernel

Two-Sided

One- and Two-Sided Hardware

�9

1. Address mapping 2. Permission checking 3. Resource isolation

Memory

CPU UserKernel

One-sided CommunicationOne-Sided

Memory

BerkeleySocket

CPU User

Kernel

Two-Sided

Memory Region 1. rkey/lkey 2. Address

Vulnerability 3 - Predictable Hardware Managed Keys

�10

Virtual addr + rkey

Physical addr

PTE Translation

rkey

/lkey

Val

ue

0

1M

2M

3M

nth-MemoryRegion Registered0 1000 2000 3000 4000 5000

ConnectX-3ConnectX-4ConnectX-5

Vulnerability 3 - Predictable Hardware Managed Keys

�10

Virtual addr + rkey

Physical addr

PTE Translation

rkey

/lkey

Val

ue

0

1M

2M

3M

nth-MemoryRegion Registered0 1000 2000 3000 4000 5000

ConnectX-3ConnectX-4ConnectX-5

Vulnerability 3 - Predictable Hardware Managed Keys

�10

Virtual addr + rkey

Physical addr

PTE Translation

rkey

/lkey

Val

ue

0

1M

2M

3M

nth-MemoryRegion Registered0 1000 2000 3000 4000 5000

ConnectX-3ConnectX-4ConnectX-5

Vulnerability 3 - Predictable Hardware Managed Keys

�10

Virtual addr + rkey

Physical addr

PTE Translation

rkey

/lkey

Val

ue

0

1M

2M

3M

nth-MemoryRegion Registered0 1000 2000 3000 4000 5000

ConnectX-3ConnectX-4ConnectX-5

Vulnerability 4 - Side Channel in NICs

�11

ConnectX-5, 1KB READ request latency

Virtual addr + rkey

Physical addr

PTE Translation Pe

rcen

tile

0

10

20

30

Latency (us)0 1 2 3 4 5 6 7 8

HitMiss-PageTableEntriesMiss-MemoryRegionInfo

Vulnerability 4 - Side Channel in NICs

�11

ConnectX-5, 1KB READ request latency

Virtual addr + rkey

Physical addr

PTE Translation Pe

rcen

tile

0

10

20

30

Latency (us)0 1 2 3 4 5 6 7 8

HitMiss-PageTableEntriesMiss-MemoryRegionInfo

Vulnerability 4 - Side Channel in NICs

�11

ConnectX-5, 1KB READ request latency

Virtual addr + rkey

Physical addr

PTE Translation Pe

rcen

tile

0

10

20

30

Latency (us)0 1 2 3 4 5 6 7 8

HitMiss-PageTableEntriesMiss-MemoryRegionInfo

Side-Channel Attacks in RDMA (Pythia, USENIX Sec ‘19)

�12

Server Machine

RDMA Network

RNIC SRAM

Main Memory CPU

Client Machine

Attacker

Client Machine

Victim

QP MR PTE

PCIePTEMRData Ac

cess

Pro

babi

lity

(%)

0

20

40

60

80

100

Timeline (ms)0 20 40 60 80 100 120 140

VictimAttacker

Discussion and Defense• Generate memory registration keys cryptographically

�13

Sequential to Random

Discussion and Defense• Generate memory registration keys cryptographically

• Isolate on-board resources for different clients

�13

On-boardResource

Sequential to Random

Discussion and Defense• Generate memory registration keys cryptographically

• Isolate on-board resources for different clients

• Enhancing SmartNIC at the receiver side

�13

On-boardResource

SmartNIC

Client NIC CPU

Memory

Client NIC CPU

MemorySoC

Sequential to Random

Outline• Introduction and Background

• Vulnerabilities in One-Sided Communication

• Vulnerabilities in One-Sided Hardware

• Opportunities in One-Sided Communication

• Conclusion

�14

Opportunity of One-sided Communication

�15

ORAM Access Server

Opportunity of One-sided Communication

�15

ORAM Access Server

ORAM READ/WRITE

Opportunity of One-sided Communication

�15

ORAM Access Server

ORAM READ/WRITE

Opportunity of One-sided Communication

�15

ORAM Access Server

ORAM READ/WRITE

Opportunity of One-sided Communication

�15

ORAM Access Server

ORAM READ/WRITE

Opportunity of One-sided Communication

�15

ORAM Access Server

ORAM READ/WRITE

Opportunity of One-sided Communication

�15

ORAM Access Server

One-sided READ

ORAM READ/WRITE

Opportunity of One-sided Communication

�15

ORAM Access Server

One-sided READ

ORAM READ/WRITE

Opportunity of One-sided Communication

�16

ORAM Access Server

K% One-sided READ

(1-K)% ORAM READ100% ORAM WRITE

Opportunity of One-sided Communication

�17

ORAM Access Server

X% One-sided READ

(1-X)% ORAM READ100% ORAM WRITE

Thro

ughp

ut (K

OPS

)

0

10

20

30

K% of One-Sided READ Operations0 25 50 75 100

Conclusion• Security concerns of one-sided communication

• Tradeoffs between Performance and Security

• Hardware Vendor, Software Developers, and Datacenter

�18

Conclusion• Security concerns of one-sided communication

• Tradeoffs between Performance and Security

• Hardware Vendor, Software Developers, and Datacenter

�18

Thank you Questions?

wuklab.io