Implementing Remote Implementing Remote Procedure CallsProcedure Calls

ANDREW D. BIRRELL and BRUCE JAY ANDREW D. BIRRELL and BRUCE JAY NELSONNELSON

Presented byPresented byTony BockTony Bock

Remote Procedure Calls Bring Remote Procedure Calls Bring Threaded Model to IPCThreaded Model to IPC

Individual processes have their own Individual processes have their own address spaces providing a natural address spaces providing a natural protection boundary relative to other protection boundary relative to other processesprocesses

Previously, coordination between address Previously, coordination between address spaces required message passing and spaces required message passing and event-oriented styleevent-oriented style

RPC allows process-oriented, thread driven RPC allows process-oriented, thread driven programming model for IPC whether local programming model for IPC whether local or remoteor remote

Advantages of RPCAdvantages of RPC

SimpleSimple – Straightforward semantics – Straightforward semantics make it easier to build and maintain make it easier to build and maintain correct distributed programscorrect distributed programs

EfficientEfficient – Procedure call – Procedure call mechanism appears simple enough mechanism appears simple enough to enable rapid communicationto enable rapid communication

GeneralGeneral – Procedures are the – Procedures are the common way to communicate common way to communicate between portions of a programbetween portions of a program

ChallengesChallenges

Linking functions in a separate address Linking functions in a separate address spacespace

Discovering/specifying RPC targetDiscovering/specifying RPC target Calling/returning across processesCalling/returning across processes Passing arguments by referencePassing arguments by reference Unreliability of networksUnreliability of networks

• Retransmits must not lead to multiple callsRetransmits must not lead to multiple calls• Crashes can occur at either end at any timeCrashes can occur at either end at any time

Deliver High PerformanceDeliver High Performance

Linking: Use Stubs to Import/Export Linking: Use Stubs to Import/Export InterfacesInterfaces

Application is linking against functions that reside in a different address spaceApplication is linking against functions that reside in a different address space Server is returning results to separate address spaceServer is returning results to separate address space Server and client link to programmatically generated stubs of a common interface Server and client link to programmatically generated stubs of a common interface

module at compile timemodule at compile time RPC runtime manages communication between machines/processesRPC runtime manages communication between machines/processes User writes interface, client, and server code but doesn’t need to write any code for User writes interface, client, and server code but doesn’t need to write any code for

communication mechanismcommunication mechanism

Client Stubs(Import)

Server Stubs(Export)

Interface Modulebool Foo(int)int Bar(char)int Baz(int)

ApplicationFunctionLibrary

RPCRuntime

RPCRuntime

NETWORK

Binding – Matching Callers to Binding – Matching Callers to CalleesCallees

Stubs are only place holders, client still needs to find the serverStubs are only place holders, client still needs to find the server Servers register their exported interfaces with secure database serversServers register their exported interfaces with secure database servers

• Register time-based unique server IDRegister time-based unique server ID Clients can specify or select server from available list or bind statically by Clients can specify or select server from available list or bind statically by

network addressnetwork address• Get server’s ID and network addressGet server’s ID and network address

Clients include server’s ID and unique sequence number with each callClients include server’s ID and unique sequence number with each call

`

NameAddressInterfaceServer ID Interface? Name

AddressServer ID

InterfaceFunction Offset

Server IDSequence Number

Server ClientDatabase

Crossing Process BoundariesCrossing Process Boundaries

Caller and Callee have separate address spacesCaller and Callee have separate address spaces• Can’t dereference pointers directlyCan’t dereference pointers directly• Caller needs to call functions in separate processCaller needs to call functions in separate process• Callee needs to return to separate processCallee needs to return to separate process

RPC runtime layers on server & client coordinate communication RPC runtime layers on server & client coordinate communication across networkacross network• Client calls result in messages sent by RPC to server with index of Client calls result in messages sent by RPC to server with index of

desired function from client stubsdesired function from client stubs• Server RPC uses index to find desired function in server stubs and Server RPC uses index to find desired function in server stubs and

switches to serverswitches to server• Server “calls home” to ask client for values when dereferencingServer “calls home” to ask client for values when dereferencing

Client Stubs(Import)

Server Stubs(Export)

Interface Modulebool Foo(int)int Bar(char)int Baz(int)

ApplicationFunctionLibrary

RPCRuntime RPC

Runtime

Foo()

Index of Foo()

Foo()

Index of Foo()

*value?Caller Callee

ReliabilityReliability RPC must overcome network’s inherent RPC must overcome network’s inherent

unreliabilityunreliability• Network can drop any packetNetwork can drop any packet• Either machine can crash at any timeEither machine can crash at any time

Mustn’t crash others by so doingMustn’t crash others by so doing Need to discern crash vs. running slowlyNeed to discern crash vs. running slowly

Each application issues at most one RPC to Each application issues at most one RPC to a particular server/interface at timea particular server/interface at time• Each request includes a monotonically Each request includes a monotonically

increasing sequence number & the server IDincreasing sequence number & the server ID• Sequence ID includes “conversation ID”, a Sequence ID includes “conversation ID”, a

unique time-based client identifierunique time-based client identifier

ReliabilityReliability Server checks ID’s when receiving calls Server checks ID’s when receiving calls

and keeps track of current (highest) and keeps track of current (highest) sequence numbersequence number• If server ID doesn’t match, server has crashed If server ID doesn’t match, server has crashed

since client connected: dropsince client connected: drop• Keeps track of current (highest) sequence Keeps track of current (highest) sequence

numbernumber < current, this is a retransmit of old request: drop< current, this is a retransmit of old request: drop = current, retransmit of current request: issue ACK if = current, retransmit of current request: issue ACK if

requested, then droprequested, then drop > current, new request: client acknowledges receipt > current, new request: client acknowledges receipt

of RPC resultof RPC result• Conversation IDs ensure sequence numbers Conversation IDs ensure sequence numbers

don’t repeat if client restartsdon’t repeat if client restarts

Packet Protocol – Simple CasePacket Protocol – Simple Case

Existing protocols focused on one-way bulk data transfersExisting protocols focused on one-way bulk data transfers• Even substantial per-connection overhead is insignificant to data transmit timeEven substantial per-connection overhead is insignificant to data transmit time

RPC generates many small packets in each direction so it uses a minimal RPC generates many small packets in each direction so it uses a minimal protocolprotocol• RPC result implies acknowledgment of request – function has run exactly onceRPC result implies acknowledgment of request – function has run exactly once• Each RPC request implies acknowledgment of previous result – Each application Each RPC request implies acknowledgment of previous result – Each application

has at most one outstanding requesthas at most one outstanding request• Sender must resend until acknowledged, retransmits include request for ACKSender must resend until acknowledged, retransmits include request for ACK

Advantages of RPC protocolAdvantages of RPC protocol• Minimal per-connection setup and teardown costsMinimal per-connection setup and teardown costs• Minimal state when connection is idleMinimal state when connection is idle

Server just keeps last sequence numberServer just keeps last sequence number Client has a list of server addresses & IDsClient has a list of server addresses & IDs

• Minimize delay between RPC request and response – no handshaking phaseMinimize delay between RPC request and response – no handshaking phase• No idle time communication, i.e.: keep-alive packetsNo idle time communication, i.e.: keep-alive packets

`

Request

Result

Multi-Packet RequestsMulti-Packet Requests

Need to make sure all packets arrive in orderNeed to make sure all packets arrive in order• Ask for explicit ACK with each transmit except the Ask for explicit ACK with each transmit except the

lastlast• Retransmits still ask for explicit ACKRetransmits still ask for explicit ACK• Return result still implies receipt of last packetReturn result still implies receipt of last packet

`

REQ A

REQ A+2

REQ A+1

ACK

RESULT

ACK

Crashed or Slow?Crashed or Slow?

If server crashes, caller needs to move on If server crashes, caller needs to move on otherwise wait for slow serverotherwise wait for slow server

While waiting for response caller While waiting for response caller periodically pings serverperiodically pings server• If responses keep coming, server is just slow: If responses keep coming, server is just slow:

wait forever (just like local procedure call)wait forever (just like local procedure call)• If responses stop coming, server is crashed: If responses stop coming, server is crashed:

throw exceptionthrow exception

`

YouThere?

Yes.Exception:Comm_Failed

PerformancePerformance Both machines maintain multiple idle server Both machines maintain multiple idle server

processes ready to handle RPCsprocesses ready to handle RPCs• Reduces cost of process creationReduces cost of process creation• Process may be created/destroyed to adapt to trafficProcess may be created/destroyed to adapt to traffic

Each packet includes source and destination Each packet includes source and destination process IDprocess ID• When expecting a packet, a process registers with the When expecting a packet, a process registers with the

Ethernet handlerEthernet handler• When the handler receives an expected packet, it When the handler receives an expected packet, it

passes it directly to the waiting process with only one passes it directly to the waiting process with only one context switchcontext switch

• If no one’s waiting, packet passes to idle server process If no one’s waiting, packet passes to idle server process for decision makingfor decision making

SummarySummary

RPCs allow for procedure oriented RPCs allow for procedure oriented programming across process boundariesprogramming across process boundaries• Programmatically generated stubs abstract Programmatically generated stubs abstract

transition between processestransition between processes• RPC runtime translates calls to networkRPC runtime translates calls to network• User only writes application and server codeUser only writes application and server code

Lightweight protocol minimizes per-client Lightweight protocol minimizes per-client server load, handshaking, and idle state server load, handshaking, and idle state maintenancemaintenance

System of time-based ID’s along with active System of time-based ID’s along with active state pinging enhances reliability and aids in state pinging enhances reliability and aids in securing networksecuring network