![Page 1: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/1.jpg)
Verifying concurrent software using movers in CSPEC
Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai ZeldovichMIT CSAIL and *Microsoft
![Page 2: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/2.jpg)
Concurrent software is difficult to get right
!2
Programmer cannot reasonabout code in sequence…
![Page 3: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/3.jpg)
Concurrent software is difficult to get right
!3
instead, must consider many executions:Programmer cannot reasonabout code in sequence…
![Page 4: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/4.jpg)
Concurrent software is difficult to get right
!3
instead, must consider many executions:
…
Programmer cannot reasonabout code in sequence…
![Page 5: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/5.jpg)
Goal: verify concurrent software
!4
![Page 6: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/6.jpg)
Challenge for formal verification
• Proofs must also cover every execution
• Many approaches to managing this complexity• movers [Lipton, 1975]• rely-guarantee [1983]• RGSep [CONCUR 2007]• FCSL [PLDI 2015]• Iris [POPL 2017, LICS 2018, others]• many others
!5
![Page 7: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/7.jpg)
Challenge for formal verification
• Proofs must also cover every execution
• Many approaches to managing this complexity• movers [Lipton, 1975]• rely-guarantee [1983]• RGSep [CONCUR 2007]• FCSL [PLDI 2015]• Iris [POPL 2017, LICS 2018, others]• many others
• This work: our experience using movers
!5
![Page 8: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/8.jpg)
Movers: reduce concurrent executions to sequential ones
!6
time
B32A1
blue thread
green thread
1 2 3
A B
![Page 9: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/9.jpg)
Movers: reduce concurrent executions to sequential ones
!6
has the same effect as movers
B32A1
blue thread
green thread
1 2 3
A B
B32 A1
![Page 10: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/10.jpg)
Movers: reduce concurrent executions to sequential ones
!6
has the same effect as movers
sequential reasoning
B32A1
blue thread
green thread
1 2 3
A B
B32 A1
B32 A1
![Page 11: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/11.jpg)
Prior systems with mover reasoning
!7
CIVL [CAV ’15, CAV ’18] framework relies pen & paper proofs
IronFleet [SOSP ’15] only move network send/receive
![Page 12: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/12.jpg)
Contribution: CSPEC
• Framework for verifying concurrency in systems software
• general-purpose movers
• patterns to support mover reasoning
• machine checked in Coq to support extensibility
!8
![Page 13: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/13.jpg)
Contribution: CSPEC
• Framework for verifying concurrency in systems software
• general-purpose movers
• patterns to support mover reasoning
• machine checked in Coq to support extensibility
• Case studies using CSPEC
• Lock-free file-system concurrency
• Spinlock on top of x86-TSO (see paper)
!8
![Page 14: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/14.jpg)
Case study: mail server using file-system concurrency
!9
spool
file system
mbox
![Page 15: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/15.jpg)
Mail servers exploit file-system concurrency
!10
spool
file system
# accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup
unlink(“/spool/$TID”)
mbox
1 2 3
![Page 16: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/16.jpg)
Mail servers exploit file-system concurrency
!11
spool
file system
msg
mbox
1 2 3
# accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup
unlink(“/spool/$TID”)
![Page 17: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/17.jpg)
Spooling avoids reading partially-written messages
!12
spool
file system
mbox
1 2 3
$TID =10
# accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup
unlink(“/spool/$TID”)
![Page 18: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/18.jpg)
Spooling avoids reading partially-written messages
!12
spool
file system
10
mbox
1 2 3
$TID =10
# accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup
unlink(“/spool/$TID”)
10
![Page 19: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/19.jpg)
Threads use unique IDs to avoid conflicts
!13
spool
file system
10
msg
mbox
1 2 3
$TID =10 $TID =11
# accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup
unlink(“/spool/$TID”)
![Page 20: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/20.jpg)
Threads use unique IDs to avoid conflicts
!14
spool
file system
10
mbox
1 2 3
$TID =10 $TID =11
# accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup
unlink(“/spool/$TID”)
![Page 21: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/21.jpg)
Threads use unique IDs to avoid conflicts
!14
spool
file system
10 11
mbox
1 2 3
$TID =10 $TID =11
# accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup
unlink(“/spool/$TID”)
![Page 22: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/22.jpg)
Timestamps help generate unique message names
!15
spool
file system
mbox
1 2 3 410 11
link(/spool/11, /mbox/4)
# accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup
unlink(“/spool/$TID”)
![Page 23: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/23.jpg)
Timestamps help generate unique message names
!16
spool mbox
1 2 3
file system
410 11
link(/spool/10, /mbox/4)
EEXISTS ✗
# accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup
unlink(“/spool/$TID”)
![Page 24: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/24.jpg)
Timestamps help generate unique message names
!17
spool
file system
mbox
1 2 3 4 510 11
link(/spool/10, /mbox/5)
# accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup
unlink(“/spool/$TID”)
![Page 25: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/25.jpg)
Delivery concurrency does not use locks
!18
spool
file system
mbox
1 2 3 5410
# accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup
unlink(“/spool/$TID”)
![Page 26: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/26.jpg)
Delivery concurrency does not use locks
!19
spool
file system
mbox
1 2 3 4 5
# accept def deliver(msg): # spool create(“/spool/$TID”) write(“/spool/$TID”, msg) # store while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break # cleanup
unlink(“/spool/$TID”)
![Page 27: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/27.jpg)
Proving delivery correct in CSPEC
!20
file-system spec
delivery specification
implementation and proof
CSPEC CSPEC provides supporting definitionsand theorems
![Page 28: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/28.jpg)
Proof engineer reasons about file-system operations
!21
def deliver(msg): create(“/spool/$TID”, msg) while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break
unlink(“/spool/$TID”)
create( /sp/$TID, msg)
✓
link( /sp/$TID, /mbox/$t)
EEXISTS ✗
link( /sp/$TID, /mbox/$t)
✓
unlink( /sp/$TID)
✓
![Page 29: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/29.jpg)
Proof engineer reasons about file-system operations
!21
def deliver(msg): create(“/spool/$TID”, msg) while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break
unlink(“/spool/$TID”)
create( /sp/$TID, msg)
✓
link( /sp/$TID, /mbox/$t)
EEXISTS ✗
link( /sp/$TID, /mbox/$t)
✓
unlink( /sp/$TID)
✓
create(“/spool/$TID”) write(“/spool/$TID”, msg)
collapsed toone operation
![Page 30: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/30.jpg)
Proof engineer reasons about interleaving of file-system operations
!22
def deliver(msg): create(“/spool/$TID”, msg) while True: t = time.time() if link(“/spool/$TID”, “/mbox/$t”): break
unlink(“/spool/$TID”)
We assume file-system operations are atomic
create( /sp/$TID, msg)
✓
link( /sp/$TID, /mbox/$t)
EEXISTS ✗
link( /sp/$TID, /mbox/$t)
✓
unlink( /sp/$TID)
✓
create
✓link unlink
![Page 31: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/31.jpg)
Proving atomicity of delivery
!23
atomicity: concurrent deliveries appear to execute all at once (in some order)
create
✓link
✗
link unlinkcreate
✓link unlink
create
✓link unlink
delivercreate
✓link
✗
link unlinkdeliver
![Page 32: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/32.jpg)
Proving atomicity of delivery
!23
atomicity: concurrent deliveries appear to execute all at once (in some order)
create
✓link
✗
link unlinkcreate
✓link unlink
create
✓link unlink
delivercreate
✓link
✗
link unlinkdeliver
Step 1: developer identifies commit point
![Page 33: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/33.jpg)
Proving atomicity of delivery
!23
atomicity: concurrent deliveries appear to execute all at once (in some order)
create
✓link
✗
link unlinkcreate
✓link unlink
create
✓link unlink
delivercreate
✓link
✗
link unlinkdeliver
Step 1: developer identifies commit point
Step 2: prove operation occurs logically at commit point
![Page 34: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/34.jpg)
Example of movers for this execution
!24
create
✓link
✗
link unlinkcreate
✓link unlink
![Page 35: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/35.jpg)
Example of movers for this execution
!24
create
✓link
✗
link unlinkcreate
✓link unlink
create
✓link
✗
link unlinkcreate
✓link unlink
![Page 36: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/36.jpg)
Example of movers for this execution
!24
create
✓link
✗
link unlinkcreate
✓link unlink
create
✓link
✗
link unlinkcreate
✓link unlink
create
✓link
✗
link unlinkcreate
✓link unlink
![Page 37: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/37.jpg)
Right mover can be reordered after any green thread operation
!25
Ar
Ar
![Page 38: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/38.jpg)
Right mover can be reordered after any green thread operation
!25
Ar
Ar
left movers are the converse
![Page 39: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/39.jpg)
Movers need to consider only possible operations from other threads
!26
create( /sp/$TID, msg)
link( /sp/$TID, /mbox/$t)
EEXISTS ✗
link( /sp/$TID, /mbox/$t)
unlink( /sp/$TID)
is one of
Ar
Arfor all green operations ,
is a right mover ifA
left movers are the converse
![Page 40: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/40.jpg)
Example mover proof: failing link is a right mover
!27
Proof sketch (only case):link
link( /sp/$TID, /mbox/$t)
✓
link( /sp/$TID, /mbox/$t)
✓
link( /sp/$TID, /mbox/$t)
EEXISTS ✗
link( /sp/$TID, /mbox/$t)
EEXISTS ✗
![Page 41: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/41.jpg)
Example mover proof: failing link is a right mover
!27
Proof sketch (only case):link
link( /sp/$TID, /mbox/$t)
✓
link( /sp/$TID, /mbox/$t)
✓
$t $t≠
link( /sp/$TID, /mbox/$t)
EEXISTS ✗
link( /sp/$TID, /mbox/$t)
EEXISTS ✗
✗
link
✓link
(otherwise then is impossible)
![Page 42: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/42.jpg)
Example mover proof: failing link is a right mover
!27
Proof sketch (only case):link
link( /sp/$TID, /mbox/$t)
✓
link( /sp/$TID, /mbox/$t)
✓
$t $t≠
link( /sp/$TID, /mbox/$t)
EEXISTS ✗
link( /sp/$TID, /mbox/$t)
EEXISTS ✗
✗
link
✓link
(otherwise then is impossible)
link operations are independent⟹
![Page 43: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/43.jpg)
Failing link does not move left
!28
![Page 44: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/44.jpg)
Failing link does not move left
!28
link( /sp/$TID, /mbox/$t)
EEXISTS ✗
link( /sp/$TID, /mbox/$t)
✓
link( /sp/$TID, /mbox/$t)
✓
link( /sp/$TID, /mbox/$t)
EEXISTS ✗
if = $t $t
![Page 45: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/45.jpg)
Challenge: how to limit what other operations to consider in mover proofs?
!29
Delivery
File system
•deliver
•create(f, d) •link(f1, f2) •unlink(f) •rename(f1, f2)
mover proof?
![Page 46: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/46.jpg)
Challenge: how to limit what other operations to consider in mover proofs?
!29
Delivery
File system
•deliver
•create(f, d) •link(f1, f2) •unlink(f) •rename(f1, f2)
mover proof?
create( f1, d)
create( f2, d)
create( f1, d)
create( f2, d)
if filenames are identical
![Page 47: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/47.jpg)
Layers enable mover reasoning
!30
Delivery
File system
•deliver
•create(f, d) •link(f1, f2) •unlink(f) •rename(f1, f2)
Layers limit what operations are available use multiple layers to make operations movers⟹
![Page 48: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/48.jpg)
Layers enable mover reasoning
!31
Delivery
File system
Restricted file system
restrict arguments to include $TID
•create(/spool/$TID, d) •link(/spool/$TID, /mbox/$t) •unlink(/spool/$TID)
mover proof✓
Layers limit what operations are available use multiple layers to make operations movers⟹
![Page 49: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/49.jpg)
Layers enable mover reasoning
!31
Delivery
File system
Restricted file system
upper layers can only use restricted operations
•create(/spool/$TID, d) •link(/spool/$TID, /mbox/$t) •unlink(/spool/$TID)
mover proof✓
Layers limit what operations are available use multiple layers to make operations movers⟹
![Page 50: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/50.jpg)
Movers are a layer proof pattern
!32
mover pattern
Obligation for developer: movers for each implementation
layer 1
layer 2 A B C D
foo bar
![Page 51: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/51.jpg)
Movers are a layer proof pattern
!32
mover pattern
A B DC
B A C
def foo:
def bar:
Obligation for developer: movers for each implementation
layer 1
layer 2 A B C D
foo bar
![Page 52: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/52.jpg)
Movers are a layer proof pattern
!32
mover pattern
A B DC
B A C
def foo:
def bar:
Obligation for developer: movers for each implementation
CSPEC theorem: entire layer implementation is atomic
layer 1
layer 2 A B C D
foo bar
![Page 53: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/53.jpg)
CSPEC provides other patterns to support mover reasoning
• Abstraction / forward simulation
• Invariants
• Error state
• Protocols
• Retry loops
• Partitioning
!33
(see paper for details)
patternobligations proof connecting
layers
![Page 54: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/54.jpg)
Using CSPEC to verify CMAIL
!34
file-system spec
mail library spec
implementation layers
patterns
CMAIL (Coq)
Coq
CSPECauto generated
framework
![Page 55: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/55.jpg)
Using CSPEC to verify CMAIL
!34
file-system spec
mail library spec
implementation layers
patterns
CMAIL (Coq)
Coq
calls to file-system
SMTP + POP3
extracted implementation
CMAIL (Haskell)
Coq extraction
CSPECauto generated
framework
![Page 56: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/56.jpg)
Using CSPEC to verify CMAIL
!34
file-system spec
mail library spec
implementation layers
patterns
CMAIL (Coq)
Coq
calls to file-system
SMTP + POP3
extracted implementation
CMAIL (Haskell)
Coq extraction
GHC
CSPEC
executable
auto generated
framework
Linux
![Page 57: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/57.jpg)
What is proven vs. assumed correct?
!35
file-system spec
mail library spec
implementation layers
patterns
CMAIL (Coq)
Coq
calls to file-system
SMTP + POP3
extracted implementation
CMAIL (Haskell)
Coq extraction
GHC
CSPEC
executable
✓ok
Coq proof checker
Linux
proven
auto generated
assumed correct
![Page 58: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/58.jpg)
Concurrency inside CMAIL is proven
!36
file-system spec
mail library spec
implementation layers
patterns
CMAIL (Coq)
Coq
calls to file-system
SMTP + POP3
extracted implementation
CMAIL (Haskell)
Coq extraction
GHC
CSPEC
executable
✓ok
Coq proof checker
Linux
proven
auto generated
assumed correct
![Page 59: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/59.jpg)
Trust that the tools and OS are correct
!37
file-system spec
mail library spec
implementation layers
patterns
CMAIL (Coq)
Coq
calls to file-system
SMTP + POP3
extracted implementation
CMAIL (Haskell)
Coq extraction
GHC
CSPEC
executable
✓ok
Coq proof checker
Linux
proven
auto generated
assumed correct
![Page 60: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/60.jpg)
Mail server-specific assumptions
!38
file-system spec
mail library spec
implementation layers
patterns
CMAIL (Coq)
Coq
calls to file-system
SMTP + POP3
extracted implementation
CMAIL (Haskell)
Coq extraction
GHC
CSPEC
executable
✓ok
Coq proof checker
Linux
proven
auto generated
assumed correct
![Page 61: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/61.jpg)
Evaluation
• Can CMAIL exploit file-system concurrency for speedup?
• How much effort was verifying CMAIL?
• What is the benefit of CSPEC’s machine-checked proofs?
!39
![Page 62: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/62.jpg)
CMAIL achieves speedup with multiple cores
!40
kreq
/s
0
35
70
105
140
# cores1 2 3 4 5 6 7 8 9 10 11 12
CMAIL GoMail
![Page 63: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/63.jpg)
CMAIL was work but doable
!41
proof:code ratio
CMAIL 11.5x
CertiKOS 13.8x
IronFleet 7.7x
IronClad 4.8x
CompCert 4.6x
Took two authors 6 months
{concurrent
{sequential
![Page 64: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/64.jpg)
Machine-checked proofs give confidence in framework changes
!42
Three anecdotes of changes to CSPEC:
Machine-checked proofs ensure soundness of entire system
![Page 65: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/65.jpg)
Machine-checked proofs give confidence in framework changes
!42
• Implemented partitioning pattern to support multiple users
Three anecdotes of changes to CSPEC:
Machine-checked proofs ensure soundness of entire system
![Page 66: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/66.jpg)
Machine-checked proofs give confidence in framework changes
!42
• Implemented partitioning pattern to support multiple users
• Improved mover pattern for a CMAIL left mover proof
Three anecdotes of changes to CSPEC:
Machine-checked proofs ensure soundness of entire system
![Page 67: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/67.jpg)
Machine-checked proofs give confidence in framework changes
!42
• Implemented partitioning pattern to support multiple users
• Improved mover pattern for a CMAIL left mover proof
• Implemented error-state pattern for the x86-TSO lock proof
Three anecdotes of changes to CSPEC:
Machine-checked proofs ensure soundness of entire system
![Page 68: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/68.jpg)
CSPEC is a framework for verifying concurrency in systems software
• Layers and patterns (esp. movers) make proofs manageable
• Machine-checked framework supports adding new patterns
• Evaluated by verifying mail server and x86-TSO lock
github.com/mit-pdos/cspec
!43
![Page 69: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/69.jpg)
CSPEC is a framework for verifying concurrency in systems software
• Layers and patterns (esp. movers) make proofs manageable
• Machine-checked framework supports adding new patterns
• Evaluated by verifying mail server and x86-TSO lock
github.com/mit-pdos/cspec
!43
poster #1
![Page 70: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/70.jpg)
!44
Backup slides
CMAIL perf experimental setup
![Page 71: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/71.jpg)
Performance experiment setup for CMAIL
!45
in-memory file system
process
client
deliver +pickup
CMAIL
core 1
![Page 72: Verifying concurrent software using movers in CSPECosdi18-slides.pdf · Verifying concurrent software using movers in CSPEC Tej Chajed, Frans Kaashoek, Butler Lampson*, Nickolai Zeldovich](https://reader035.vdocuments.mx/reader035/viewer/2022071020/5fd4462bfbcecc5d28424302/html5/thumbnails/72.jpg)
Performance experiment setup for CMAIL
!45
…
in-memory file system
process
client
deliver +pickup
CMAIL
core 1
process
client
deliver +pickup
CMAIL
core 2
process
client
deliver +pickup
CMAIL
core 12