dynamic verification of multicore communication applications in mcapi subodh sharma, uofu

1

Dynamic Verification of Multicore Communication Applications in MCAPI

Subodh Sharma, UofUGanesh Gopalakrishnan, UofU

Eric Mercer, BYU

1Supported by NSF CCF 0903408/0903491 and SRC 2009-TJ-1993/1994

HLDVT 2009

2

Concurrency Space in Multicore Era

S/W solution H/W solution

Cloud/HPC Computing MPI, PVM, Mapreduce, MSCS

Clusters, Vector machines, Supercomputers

Desktop computing OOSD, Scripting, Pthreads, TBB, CT, OpenMP

Desktop machines

Embedded Computing Lightweight counterparts of the above.

FPGAs, DoC, SoC, etc.

3

Concurrency Space in Multicore Era

Requires Standardization

Multicore Association (MCA) Effort

S/W solution H/W solution

Cloud/HPC Computing MPI, PVM, Mapreduce, MSCS

Clusters, Vector machines, Supercomputers

Server/Desktop computing

OOSD, Scripting, Pthreads, TBB, CT, OpenMP

Desktop machines

Embedded Computing Lightweight counterparts of the above.

FPGAs, DoC, SoC, etc.

4

What is MCAPI (Multicore Communication API)?

• An API specification from MCA (Multicore Association)

• To program embedded systems like mobile phones, PDAs, routers, servers, etc.

• Not restricted to SPMD (like MPI) or multi threaded style of programming.

http://www.linuxdevices.com/articles/AT9423084269.html





5

Taxonomy for MCA APIs

MCAPI – Communication API• Message based• Packet/Scalar Channel based

MRAPI – Resource Management API

• Semaphores, mutexes• Shared memory segment allocation, deallocation

MTAPI – Task Management API• Specification work yet to begin• Thread Pooling, Work Stealing queues e.g. CILK, TBB, TPL etc.Inter API

interactions

6

State of MCA Efforts?

• MCAPI standard is released.

• MCAPI reference implementation exists• A commercial implementation – Polycore, Inc.• In-house implementation on FPGA in progress (Utah).

• MRAPI standard work in progress.

• MTAPI efforts yet to begin.

7

State of MCA Efforts?

• Standardization efforts take a long time to mature

• Need for FV in MCA API development.

• Our effort: Early Engagement of FV in the MCA APIs space

• To promote early adoption of the API• To promote better programming practices• To help avoid early pit-falls – e.g. unspecified behaviors

8

Achievements so far!

• Formal Specification work (at BYU)• Permits symbolic analysis

• Murphi -> CVC• Query Oracle for scenario outcome calculation.• Platform testing

• Dynamic Verification of MCAPI user apps (at Utah)• MCC tool created.• Current capabilities of MCC

• Dynamic search of dependencies• Exerting a deterministic runtime control

• Algorithmic Advances• Symmetry reduction in MCAPI domain space.

9

Focus of the talk!

• Formal Specification work (at BYU)• Permits execution based symbolic analysis

• Murphi -> CVC• Query Oracle for scenario outcome calculation.• Platform testing

• Dynamic Verification of MCAPI user apps (at Utah)• MCC tool created.• Current capabilities of MCC

• Dynamic search of dependencies• Exerting a deterministic runtime control

• Algorithmic Advances• Symmetry reduction in MCAPI domain space.

10

Related Work

• CHESS tool from MSR• Uses preemption bounded search • Cannot be used for message passing programs

• ISP tool from Utah• Used for MPI verification• MCC differs in the way determinism is enforced at runtime.

• Inspect tool from Utah• Used for shared memory programs• Implements the classical DPOR algorithm.

• JPF• A model checker for Java bytecode.

11

MCAPI Preliminaries

• MCAPI Node – • Processes, Threads, OS instance, Processor core,

etc.

• MCAPI Endpoint –• Socket-like communication termination points• Identified by a tuple <node_id, port_id>

• MCAPI Channels • Point to point unidirectional FIFO connections

between a pair of endpoints.

• A single communication universe

12

MCAPI Messages

Node 1

Port 1 <1,1>

Port 2 <1,2>

Node 2

Port 1 <2,1>

Connectionless Message

To <2,1>

• Message received at FIFO receive queue

13

MCAPI Connection Oriented Communication

Node 1

Port 1 <1,1>

Port 2 <1,2>

Node 2

Port 1 <2,1>To <2,1>

• Sent over a connected channel• <1,1> can communicate on ONE channel only

Node 3

Port 1 <3,1>

To <3,1>ERROR!!

14

MCAPI API generic + message calls

Initialize/Finalize Sets and deletes the environment. Must be called by each communicating node.

Create/Delete endpoint API calls to manage creation/deletion of endpoints on the owner node

Get_endpoint A blocking call to get the address of a remote endpoint.

Send (send_endpoint, recv_endpoint .. )

Sends a data-payload from a sending endpoint to a receiving endpoint. API provides blocking and non blocking versions.

Recv (recv_endpoint, ..) Receives a data payload from the receiving endpoint. API provides blocking non-blocking versions.

Wait (request_handle, ..), Test(req_handle, ..)

Checks the completion of the request. Wait – Blocking; Test – Non blocking

15

Illustration of MCAPI C program

int main () {…for(t=0; t<NUM_THREADS; t++){ rc = pthread_create(&threads[t], NULL, run_thread, (void*)&thread_data_array[t]); } for (t = 0; t < NUM_THREADS; t++) { pthread_join(threads[t],NULL); }}void* run_thread (void *t) {… mcapi_initialize(tid,&version,&status); if (tid == 1) { recv_endpt = mcapi_create_endpoint (PORT_NUM, &status);

mcapi_msg_recv(recv_endpt,msg, BUFF_SIZE,&recv_size, &status);mcapi_msg_recv(recv_endpt,msg, BUFF_SIZE, &recv_size, &status);} else {send_endpt = mcapi_create_endpoint (PORT_NUM,&status);recv_endpt = mcapi_get_endpoint (1,PORT_NUM,&status);mcapi_msg_send(send_endpt,recv_endpt, msg,strlen(msg), 1,&status);} mcapi_finalize(&status);return NULL;}

Potential deadlock

16

MCC – MCAPI Checker

Workflow of MCC.

MCAPI C ProgramMCAPI C Program

instrumentation

Instrumented ProgramInstrumented Program

compile

MCAPI Library WrapperMCAPI Library Wrapper

Scheduler

Executable

thread 1

thread n

request/permit

request/

permit

17

Bug Classes in MCAPI User Applications

• Non determinism due to communication race.• The recv() call is passed with only receiving

endpoint as a parameter. • The receiving endpoint extracts message from the

FIFO receive queue.

T0 T1 T2

--- --- --- The send that matches the recv may decide the correctness !!

POTENTIAL BUG

S(e1,e2) S(e1,e2) R(e2)

R(e2)

18

Bug Classes in MCAPI User Applications

• Mismatched Calls

“get_endpoint” invoked for a non-existent endpoint -- ERROR

T0 T1

--- ---

R(1) Get_endpoint(2)

S(1,2)

Does not exist

19

Challenge: exponential interleavings

19

T0 T1 T2 T3 T4 T5

T0 T1 T2 T3 T4 T5

A B

Dependent MCAPI calls

Only these 2 are RELEVANT!!!

Possible Interleaving

20

Relevant interleaving exploration by MCC scheduler

T0 T1 T2

S (ep1,ep3) S(ep2,ep3) R(ep3)

R(ep3)

Intra happens-Before Edge

21

T0 T1 T2

S (ep1,ep3) S (ep2,ep3) R (ep3)

R (ep3)

Enabled Transitions: S(ep1, ep3), S(ep2, ep3), R (ep3)

Match Set: <S(ep1, ep3), R (ep3)> <S(ep2, ep3), R (ep3)>

Interleaving 1

Issue to the runtime

Match sets are computed at “decision points”

Set of matchingtransitions

No runnable thread


22

T0 T1 T2


R (ep3)

Enabled Transitions: S(ep2, ep3), R (ep3)

Match Set: <S(ep2, ep3), R (ep3)>

Interleaving 1



23

T0 T1 T2


R (ep3)

Enabled Transitions: S(ep1, ep3), S(ep2, ep3), R (ep3)


Interleaving 2



24

T0 T1 T2


R (ep3)



Interleaving 2



25

T0 T1 T2


R (ep3)



Interleaving 2

Issue to the runtimeError Caught


26

Enforcing a deterministic runtime match• For non blocking requests, there is an additional problem

• Runtime communication race even after a scheduler decides deterministically.

T0 T1 T2

IS (ep1,ep3) IS (ep2,ep3) IR (ep3)

IR (ep3)

Match-set 1

Match-set 2

27

T0 T1 T2


IR (ep3)

Match-set 1

Match-set 2

Not finished yet!!

RACE

Enforcing a deterministic runtime match

28

Solution 1 to enforce a deterministic match.

• Addition of Probing function to the MCAPI library• Probes an endpoint’s receive queue for a message• Returns TRUE if the message is found

• Scheduler issues the next match-set only when the probe function returns TRUE.


29

T0 T1 T2


IR (ep3)

Probing Solution

Receive Queue

matched


30

T0 T1 T2


IR (ep3)

Probing Solution

Receive Queue

matched

Runtime MCC Scheduler

Probes


31

T0 T1 T2


IR (ep3)

Probing Solution

T1 Receive Queue

Probes



32

T0 T1 T2


IR (ep3)

Probing Solution

T1T0 Receive Queue


33

Solution 2 to enforce a deterministic match.

• Scheduler induces a “test” call

• Scheduler spin-loops on the request handle until the

successful completion of the “test” call.

We opt Solution 2 in our work as it is non-intrusive.


34

T0 T1 T2


IR (ep3)

Enabled Transitions: Send(ep1, ep3), Send(ep2, ep3), Recv (ep3)

Match Set: <Send(ep2, ep3), Recv (ep3)> <Send(ep1, ep3), Recv (ep3)>

Dummy “wait” Solution

matched


35

T0 T1 T2


IR (ep3)




Test ()

36

T0 T1 T2


IR (ep3)




Test()

matched

37

Concluding remarks and future work

• MCC currently supports connectionless API constructs• Checks for safety assertion violations and

Deadlocks

• Steadily improve MCC over this year• Support for connection-oriented API calls, sanity

checks etc.

• Release MCC and assess experience of industrial app writers

38

www.cs.utah.edu/formal_verification

39

Happens-Before Relationship – Computed Dynamically

T0 T1 T2

IS (ep1,ep3, r1) IS (ep2,ep3, r3) IR (ep3, r4)

IR (ep3, r5)IS (ep1,ep4, r2)

R (ep3)W(r2)

40

t1: a[x] := …

t2: … := a[y] t1: a[x] := …

t2: … := a[y]

Equal ??

t2: … := a[y]

Review of POR

Dynamic POR

Dependencies based on concrete values

x=56, y=753 (e.g.) at runtime!

41

Happens-Before Relationship – Example

T0 T1 T2

IS (ep1,ep3, r1) IS (ep2,ep3, r3) IR (ep3, r4)

IR (ep4, r5)IS (ep1,ep4, r2)

R (ep3)W(r2)

42

OOO completion: Why construct Happens-Before relationship?

42

T0

T0

IS (1GB to T1)IS (1GB to T1)

IS (1 byte to T2)IS (1 byte to T2)

Must calls complete in PROGRAM ORDER?

NO!!!

dynamic verification of multicore communication applications in mcapi subodh sharma, uofu

Documents