Introduction Multikernel Tornado Conclusion Discussion Outlook References

Multiprocessor Operating SystemsCS 6410: Advanced Systems

Kai Mast

Department of Computer ScienceCornell University

September 4, 2014

Kai Mast — Multiprocessor Operating Systems 1/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Let us recallMultiprocessor vs. Multicore

Figure: Multiprocessor [10] Figure: Multicore [10]

Kai Mast — Multiprocessor Operating Systems 2/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Let us recallMessage Passing vs. Shared Memory

Shared MemoryThreads/Processes access the same memory regionCommunication via changes in variablesOften easier to implement

Message PassingThreads/Processes don’t have shared memoryCommunication via messages/eventsEasier to distribute between different processorsMore robust than shared memory

Kai Mast — Multiprocessor Operating Systems 3/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Let us recallMiscellaneous

Cache CoherenceInter-Process CommunicationRemote-Procedure CallPreemptive vs. cooperative MultitaskingNon-uniform memory access (NUMA)

Kai Mast — Multiprocessor Operating Systems 4/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Current Systems are Diverse

Different Architectures (x86, ARM, ...)Different Scales (Desktop, Server, Embedded, Mobile ...)Different Processors (GPU, CPU, ASIC ...)Multiple Cores and/or Multiple ProcessorsMultiple Operating Systems on a System (Firmware,Microkernels ...)

Kai Mast — Multiprocessor Operating Systems 5/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

How about the Future?Moore’s Law

(Source: Wikimedia Commons)

Kai Mast — Multiprocessor Operating Systems 6/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

How about the Future?Single-Core doesn’t scale anymore

Figure: Possible power-consumption of a 10GHz chip [3]

Kai Mast — Multiprocessor Operating Systems 7/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

How about the Future?Rock’s Law

Manufacturing cost increases with amount ofsemiconductorsRock’s Law eventually collides with Moore’s LawOne solution: Higher production quantityAnother approach: Multiple mid-range processors insteadof one high-end processor

Kai Mast — Multiprocessor Operating Systems 8/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

How about the Future?But...

Multiprocessor Systems are reality today!Existing Operating System had to be adapted to supportmultiple coresApplications heavily rely on multi-threading (just think ofthe assignment...)

Kai Mast — Multiprocessor Operating Systems 9/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Interconnects are evolvingDirect Wiring does not scale

On-chip networks are more efficient in terms ofpower-consumption and area [2].

Kai Mast — Multiprocessor Operating Systems 10/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Interconnects are evolvingMany-Core Chips

Figure: 36-core Chip from MIT [4]

Kai Mast — Multiprocessor Operating Systems 11/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Are Operating Systems ready for this?In-Kernel Locking

n threads on n cores execute the following:1 f = open ( ” f i l e n a m e ” ) ;2

3 w h i l e ( t r u e ) {4 f 2 = dup ( f ) ;5 c l o s e ( f 2 ) ;6 }

Kai Mast — Multiprocessor Operating Systems 12/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Are Operating Systems ready for this?In-Kernel Locking

Figure: Decreasing performance with increasing amount of Coresin Linux [8]

Kai Mast — Multiprocessor Operating Systems 13/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Are Operating Systems ready for this?

OSes optimized for most common configuration(s)Evolutionary improvements towards scalabilitySome special applications are highly coupled to hardwareconfigurationCan we abstract from hardware and gain performance?

Kai Mast — Multiprocessor Operating Systems 14/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Multikernel and Tornado

Figure: Barrelfish/Mulitkernel[1] Figure: Tornado [6]

Kai Mast — Multiprocessor Operating Systems 15/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

The Multikernel OSThe Paper

”The Multikernel: A new OS architecture for scalablemulticore systems”

Presented on SOSP in 2009

Kai Mast — Multiprocessor Operating Systems 16/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

The Multikernel OSAuthor Info

Andrew BaumannWas post-doc at ETH ZurichNow at Microsoft ResearchSeveral Projects focused around OS design

Simon PeterWas post-doc at ETH ZurichNow at University of WashingtonCurrent Project: Arrakis[9] (a Barrelfish fork)

Kai Mast — Multiprocessor Operating Systems 17/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

The Multikernel OS

The OS itself is a distributed systemActually, multiple operating systemsExplicit communication between coresAbstract design to allow easier portabilityNote, that only the communication layer is abstracted

Kai Mast — Multiprocessor Operating Systems 18/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

BarrelfishWhat is it?

Multikernel OS is just a conceptBarrelfish is an example for an actual implementationClaims to have all the properties described before(scalable, modular, portable...)Let us evaluate and discuss later!

Kai Mast — Multiprocessor Operating Systems 19/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

BarrelfishOverview

Figure: Structure of Barrelfish [1]

Kai Mast — Multiprocessor Operating Systems 20/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

BarrelfishComponent Summary

Application(Possibly) distributed over several kernels

MonitorGeneric (same for all cores)But still single threaded

CPU driverArchitecture/Hardware specificSingle-threaded

Kai Mast — Multiprocessor Operating Systems 21/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

BarrelfishMemory

Memory is still a shared and global resourceLogic is handled by the monitor, not the CPU driverPages of memory a mapped to specific monitorsBut virtual/shared memory pages are also possible

Kai Mast — Multiprocessor Operating Systems 22/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

BarrelfishPerformance Evaluation

Figure: Latency of Unmapping a Memory Page [1]Kai Mast — Multiprocessor Operating Systems 23/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

BarrelfishPerformance Evaluation

Are the numbers meaningful?No complex applications were evaluatedOnly implemented on x86OS doesn’t support any advanced features yet

Kai Mast — Multiprocessor Operating Systems 24/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Is this an important paper?

ProsProposes a new type of Operating SystemsThe concept could represent a paradigm-shiftSuch an approach would make OSes ”future proof”

ConsNo complex benchmarks exist yetDoes not support systems that are distributed over thenetwork

Kai Mast — Multiprocessor Operating Systems 25/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Open Questions

Does it make sense to split monitor and CPU driverperformance-wise?What would be a good communication model forMultikernels?How to support systems without a global shared memory?

Kai Mast — Multiprocessor Operating Systems 26/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Other MultikernelsInvasive Computing

Figure: invasIC Architecture [7]

Kai Mast — Multiprocessor Operating Systems 27/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Tornado

”Tornado: Maximizing Locality and Concurrency in a SharedMemory Multiprocessor Operating System”

Presented on SOSP in 1999Evaluated mostly on NUMAchine at UofToronto

Kai Mast — Multiprocessor Operating Systems 28/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

TornadoAuthors

Ben GamsaFormer Ph.D. student at University of TorontoNow working at Altera (unrelated to his research)

Orran KriegerFormer VMware employeeWorking IBM T.J. Watson Research Center at the time ofpublicationNow leading the ”Center for Cloud Innovation” at BostonUniversity

Kai Mast — Multiprocessor Operating Systems 29/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

TornadoOverview

Core 1 Core 2

Application 1 Application 2

Clustered Object(s)

Server Object(s)

Kernel

Server Object(s)

Kernel

Kai Mast — Multiprocessor Operating Systems 30/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Clustered Objects

Figure: Tornado [6]

Same problem as before: Some resources need to besharedShared object can have more than one instance (orrepresentative)

Kai Mast — Multiprocessor Operating Systems 31/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Resolving Clustered Objects

User calls a function

Has Reference?Call Object Miss Handler

Object Unknown?

Call Global Miss Handler

Retrieve Reference Forward call to Rep

Yes

No

Yes

No

Kai Mast — Multiprocessor Operating Systems 32/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Resolving Clustered ObjectsMiss Handler

Figure: Miss Handling Table [6]

Kai Mast — Multiprocessor Operating Systems 33/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Garbage Collection

Clustered Object

Thread(s)Shared Memory

temporary referencepersistent reference

might use

Object must ensure that all references are gone beforeremovalFortunately, we know of all references because of the misshandler

Kai Mast — Multiprocessor Operating Systems 34/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Inter-Process Communication

IPC is a core component of any modern OSExecuting on local core is more effective (handoffscheduling)Cross-process call through local rep

Kai Mast — Multiprocessor Operating Systems 35/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Both Papers in Numbers

Tornado MultikernelAuthors 4 9

Year 1999 2009Citations 182 497

Why does Multikernel seem to have a higher impact?

Kai Mast — Multiprocessor Operating Systems 36/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Conclusion

SimilaritiesThreat OS as network of (almost) independent coresAs little globally shared data as possibleHowever, both assume global shared memory

DifferencesTornado hides more from the userBarrelfish is built more modularTargeting different hardware (10 years difference)

Kai Mast — Multiprocessor Operating Systems 37/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

Discussion

Is the support for virtual memory a good idea? Should amodern OS expect the applications to do messagepassing?Is a hardware-neutral operating system realistic?Even with modularity, can one OS (architecture) cover allpossible configurations? What about low-powerembedded systems?Are the approaches really future-proof? What aboutsystems that are distributed across the network?

Kai Mast — Multiprocessor Operating Systems 38/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

ExokernelsYet Another Approach

Figure: ”End-to-End” Design of an Exokernel [5]

Kai Mast — Multiprocessor Operating Systems 39/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

ExokernelsCorey

Figure: A Webserver powered by the Corey OS [8]

Kai Mast — Multiprocessor Operating Systems 40/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

ExokernelsArrakis

Figure: Design of Arrakis (a Barrelfish fork) [8]

Kai Mast — Multiprocessor Operating Systems 41/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

References 0Slides

”Multiprocessors/Multicores”CS 6410 (Fall 2013) by Yue Gao”Operating Systems in a Multicore World”CS 6410 (Fall 2012) by Colin Ponce

Kai Mast — Multiprocessor Operating Systems 42/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

References ILiterature

[1] Andrew Baumann et al. “The Multikernel: A New OSArchitecture for Scalable Multicore Systems”. In:Proceedings of the ACM SIGOPS 22Nd Symposium onOperating Systems Principles. SOSP ’09. Big Sky,Montana, USA: ACM, 2009, pp. 29–44. isbn:978-1-60558-752-3. doi: 10.1145/1629575.1629579.url: http://doi.acm.org/10.1145/1629575.1629579.

[2] Evgeny Bolotin et al. “Cost Considerations in Networkon Chip”. In: Integration-The VLSI Journal, specialissue on Network on Chip, Volume 38, Issue 38 (2004),pp. 105–128.

Kai Mast — Multiprocessor Operating Systems 43/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

References IILiterature

[3] Shekhar Borkar. “Thousand Core Chips: A TechnologyPerspective”. In: Proceedings of the 44th AnnualDesign Automation Conference. DAC ’07. San Diego,California: ACM, 2007, pp. 746–749. isbn:978-1-59593-627-1. doi: 10.1145/1278480.1278667.url: http://doi.acm.org/10.1145/1278480.1278667.

[4] B.K. Daya et al. “SCORPIO: A 36-core research chipdemonstrating snoopy coherence on a scalable meshNoC with in-network ordering”. In: ComputerArchitecture (ISCA), 2014 ACM/IEEE 41st InternationalSymposium on. 2014, pp. 25–36. doi:10.1109/ISCA.2014.6853232.

Kai Mast — Multiprocessor Operating Systems 44/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

References IIILiterature

[5] Dawson R. Engler, M. Frans Kaashoek, andJames O’toole. “Exokernel: An Operating SystemArchitecture for Application-Level ResourceManagement”. In: 1995, pp. 251–266.

[6] Benjamin Gamsa and Benjamin Gamsa. “Tornado:Maximizing Locality and Concurrency in aShared-Memory Multiprocessor Operating System”. In:In Proceedings of the 3rd Symposium on OperatingSystems Design and Implementation (OSDI. 1999,pp. 87–100.

Kai Mast — Multiprocessor Operating Systems 45/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

References IVLiterature

[7] Jan Heisswolf et al. “The Invasive Network on Chip - AMulti-Objective Many-Core CommunicationInfrastructure”. In: Architecture of Computing Systems(ARCS), 2014 27th International Conference on. 2014,pp. 1–8.

[8] Ong Mao et al. Corey: an operating system for manycores.

[9] Simon Peter and Thomas Anderson. “Arrakis: A Casefor the End of the Empire”. In: Presented as part of the14th Workshop on Hot Topics in Operating Systems.Santa Ana Pueblo, NM: USENIX, 2013. url:https://www.usenix.org/conference/hotos13/arrakis-case-end-empire.

Kai Mast — Multiprocessor Operating Systems 46/47

Introduction Multikernel Tornado Conclusion Discussion Outlook References

References VLiterature

[10] Understanding Parallel Hardware: Multiprocessors,Hyperthreading, Dual-Core, Multicore and FPGAs.url: http://www.ni.com/white-paper/6097/en/.

Kai Mast — Multiprocessor Operating Systems 47/47