toolchain independent distributed compilation
TRANSCRIPT
![Page 1: Toolchain Independent Distributed Compilation](https://reader036.vdocuments.mx/reader036/viewer/2022062503/587914ec1a28abf13a8b46c1/html5/thumbnails/1.jpg)
Toolchain Independent Distributed CompilationDietmar HauserHead of Console TechnologySproing Interactive Media GmbH
![Page 2: Toolchain Independent Distributed Compilation](https://reader036.vdocuments.mx/reader036/viewer/2022062503/587914ec1a28abf13a8b46c1/html5/thumbnails/2.jpg)
http://xkcd.com/303/
![Page 3: Toolchain Independent Distributed Compilation](https://reader036.vdocuments.mx/reader036/viewer/2022062503/587914ec1a28abf13a8b46c1/html5/thumbnails/3.jpg)
Is it possible?Input● 1 cpp/c source file● Many h/hpp header files→ “Compilation Unit” (“CU”)Output● 1 obj/o binary file● (Misc. helper files)
![Page 4: Toolchain Independent Distributed Compilation](https://reader036.vdocuments.mx/reader036/viewer/2022062503/587914ec1a28abf13a8b46c1/html5/thumbnails/4.jpg)
Research previous solutionsIncrediBuild● The de-facto standard● Easy setup, works well● Pretty, but pricey● Limited platforms● Coordinated load balancing
distcc● Free, but narrow focus● Needs homogenous setup● Two methods:
● Preprocess & distribute● Analyse & distribute
![Page 5: Toolchain Independent Distributed Compilation](https://reader036.vdocuments.mx/reader036/viewer/2022062503/587914ec1a28abf13a8b46c1/html5/thumbnails/5.jpg)
The PlanSend input files● Start with .cpp/.c● Find required .h/.inl/...● Precompiled Header (PCH)● Compiler executable(s)● Calculate hash for every file
Receive output files● Main .obj binary file● Misc .ti/.sbr/.d/... files● Log output● Cache inputs using hash● Cache output by
combining input hashes
![Page 6: Toolchain Independent Distributed Compilation](https://reader036.vdocuments.mx/reader036/viewer/2022062503/587914ec1a28abf13a8b46c1/html5/thumbnails/6.jpg)
Finding all input filesPreprocess Method● Run Preprocessor locally● Distribute Result● Easy to do● Less parallelisation● Input file cache not possible● No PCH support possible
Analyse Method● Analyse .cpp file ● Find all dependencies● Sounds simple, but is tricky● Slightly more data to send● Good cache behaviour● PCH support possible
![Page 7: Toolchain Independent Distributed Compilation](https://reader036.vdocuments.mx/reader036/viewer/2022062503/587914ec1a28abf13a8b46c1/html5/thumbnails/7.jpg)
Precompiled Header Files● Speeds up compilation● Contains “global” includes● Is included in every CU● Proprietary format● Often very big● Contains extra dependency
information● May not be deterministic
Source: http://www.ogre3d.org
![Page 8: Toolchain Independent Distributed Compilation](https://reader036.vdocuments.mx/reader036/viewer/2022062503/587914ec1a28abf13a8b46c1/html5/thumbnails/8.jpg)
Fun with preprocessor directives● Directory search order● <> vs. “” includes● Multi line includes● Case mismatches● Conditional includes● PP constants in includes● PP macros in includes
● Conservative approach● Find all possible dependencies● Reasonable overhead● Cache dependencies locally
● Still a world of pain● Trial & Error
![Page 9: Toolchain Independent Distributed Compilation](https://reader036.vdocuments.mx/reader036/viewer/2022062503/587914ec1a28abf13a8b46c1/html5/thumbnails/9.jpg)
Putting it all together● Collect all input files● Send them if needed (zipped)● Build directory structure in Temp● Cache & Compile● Collect all output files● Cache & Send back (zipped)● ???● Profit!
![Page 10: Toolchain Independent Distributed Compilation](https://reader036.vdocuments.mx/reader036/viewer/2022062503/587914ec1a28abf13a8b46c1/html5/thumbnails/10.jpg)
It kinda works...Little problems:● PCH files don't work● Long & deeply nested file names● Absolute paths● Some compilers need registry● Issues with parallel jobs● And some more...
Big problem:● Debug info stores absolute
paths to source files!
![Page 11: Toolchain Independent Distributed Compilation](https://reader036.vdocuments.mx/reader036/viewer/2022062503/587914ec1a28abf13a8b46c1/html5/thumbnails/11.jpg)
Sandboxie to the rescue● Virtual file system
● Recreate original paths● No concurrency issues● Simple clean up
● Virtual registry● Not free (~€10-25 per user)● Does not solve all problems● But it's good enough!
![Page 12: Toolchain Independent Distributed Compilation](https://reader036.vdocuments.mx/reader036/viewer/2022062503/587914ec1a28abf13a8b46c1/html5/thumbnails/12.jpg)
Miscellaneous titbits● “Screen Saver” mode● Automatic server updates● Output file cache (~ccache)● Data compression woes● 100 MBit/s vs. 1 Gbit/s● Local compilation server● Parallel local compilation● Parallel linking experiment
![Page 13: Toolchain Independent Distributed Compilation](https://reader036.vdocuments.mx/reader036/viewer/2022062503/587914ec1a28abf13a8b46c1/html5/thumbnails/13.jpg)
So, is it worth the hassle?● Measuring this is tricky
● Real projects● In a live environment
● 34 servers, ~17 available● Maximum speed up: ~17● Uncached: 0.6 – 6.68● Cached: 1.06 – 13.13
![Page 14: Toolchain Independent Distributed Compilation](https://reader036.vdocuments.mx/reader036/viewer/2022062503/587914ec1a28abf13a8b46c1/html5/thumbnails/14.jpg)
Sproing's Codebase (21 Projects)
![Page 15: Toolchain Independent Distributed Compilation](https://reader036.vdocuments.mx/reader036/viewer/2022062503/587914ec1a28abf13a8b46c1/html5/thumbnails/15.jpg)
Schlag den Raab 2 (1 Project)
![Page 16: Toolchain Independent Distributed Compilation](https://reader036.vdocuments.mx/reader036/viewer/2022062503/587914ec1a28abf13a8b46c1/html5/thumbnails/16.jpg)
3rd Party Codebase (64 Projects)
![Page 17: Toolchain Independent Distributed Compilation](https://reader036.vdocuments.mx/reader036/viewer/2022062503/587914ec1a28abf13a8b46c1/html5/thumbnails/17.jpg)
Conclusions
It's possible distribute compilation with any compiler
Speed up is highly dependent on the environment and use case
Speed up is almost always positive, often greatly
![Page 18: Toolchain Independent Distributed Compilation](https://reader036.vdocuments.mx/reader036/viewer/2022062503/587914ec1a28abf13a8b46c1/html5/thumbnails/18.jpg)
What's next?● Get other developers involved?● Leverage an external cloud?● Distribute other processes? (Asset conversion,...)● Find a better solution for PCH?● Improve or unify front end with LLVM & Clang?● Distributed linking?
![Page 19: Toolchain Independent Distributed Compilation](https://reader036.vdocuments.mx/reader036/viewer/2022062503/587914ec1a28abf13a8b46c1/html5/thumbnails/19.jpg)
Thank you for your attention!
@Rattenhirn
http://www.sproing.com
http://fb.me/sproing
Questions?