where is my compiler
TRANSCRIPT
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 1/46
Wheres My Compiler?Developer tools: past, present, and future
Jim Miller
Software Architect, Developer Frameworks
Microsoft Corporation
(with help from Carol Eidt, Phoenix Project, Microsoft Corporation)
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 2/46
2-Jan-12 Where's My Compiler? 2
Outline What Is A Compiler?What Is A Compiler?
A Brief History of Developer Tools My First Compiler
Compilers, compilers, everywhere
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 3/46
2-Jan-12 Where's My Compiler? 3
What Is A Compiler? A converter from one representation
(source code) to another (executablecode)
Preserves (most of) the meaning of thesource
One part of a modern tool chain usedto produce executable artifacts(applications)
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 4/46
2-Jan-12 Where's My Compiler? 4
A Compiler
Compiler
Source CodeDescribes desiredbehavior
ExecutableCode
Has desired behavior, but
May have different internalstruct ure
May execute in different (unobservable) order
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 5/46
2-Jan-12 Where's My Compiler? 5
Figures of Merit Code Quality: how efficient is the
generated code?
Speed and Space: these arent independent, but they arent the sameeither
Throughput: how fast is the codegenerated?
Footprint: how large is the compiler?
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 6/46
2-Jan-12 Where's My Compiler? 6
Outline What Is A Compiler?
A Brief History of DeveloperT
ools A Brief History of DeveloperT
ools My First Compiler
Compilers, compilers, everywhere
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 7/46
2-Jan-12 Where's My Compiler? 7
1950s: Just a Compiler, Please The compiler references a runtime, but the runtime is
supplied by the OS at a fixed location in memory FORTR AN runtime: input/output formatting
COBOL runtime: also search and sort
OS loader loads the compiler output into memory,transfers control
Address space is small (< 8K word), CPU is slow (<
1,000 instructions/sec.) Figure of merit: Code Quality
Compiler must optimize code for space
Compiler must optimize code for speed
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 8/46
2-Jan-12 Where's My Compiler? 8
Inside the Compiler (in concept)
Source Code
Front End
Back End
ExecutableCode
Compiler
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 9/46
2-Jan-12 Where's My Compiler? 9
Inside the Compiler (in concept)
Source Code
Back End
ExecutableCode
Compiler
Front End
Parse source code
Produce abstract syntax tree (AST)
Produce symbol table
Generate errors
Syntax errors
Type errors
Unbound references
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 10/46
2-Jan-12 Where's My Compiler? 10
Inside the Compiler (in concept)
Source Code
ExecutableCode
Compiler
Linearize parse tree
Code Analysis Basic block analysis Control- and data-flow graph analysis
Optimize (machine-independent) Redundant and dead code elimination Code restruct uring
Convert to executable code Register allocation Peephole optimization Branch prediction and tensioning
Back End
Front End
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 11/46
2-Jan-12 Where's My Compiler? 11
1960s: Linkers Programs are growing in size
Programs are built with libraries Libraries provide reusable code fragments
Virt ual memory systems are invented
Tool chain is in two stages Compile independent modules Combine the modules using a linker
Figure of merit: Code quality (speed)
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 12/46
2-Jan-12 Where's My Compiler? 12
Includes
externalreferences
Tools: Compiler + LinkerSource Code
Front End
Back End
Object Code
Compiler
Source Code
Front End
Back End
Object Code
Source Code
Front End
Back End
Object Code
Linker
Executable Code
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 13/46
2-Jan-12 Where's My Compiler? 13
1970s: Symbolic Debugger OS written in high-level language
Compilers provide sufficient code performance and
low-level access High-level languages provide large runtime
libraries in multiple units Static linker pulls only required units into a given
program image
Compiler exports symbol table for use bydebugger, not just internal to front - /back-end
Figure of merit: Code quality (speed)
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 14/46
2-Jan-12 Where's My Compiler? 14
Compiler, Linker, DebuggerSource Code
Front End
Back End
Object Code
Compiler
Source Code
Front End
Back End
Object Code
Source Code
Front End
Back End
Object Code
Linker
Running Program
Symbol table(s)
Debugger
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 15/46
2-Jan-12 Where's My Compiler? 15
1980s: Dynamic Loading,
Threading To improve OS performance, by reducing physical memory pressure, read/only
parts of libraries are shared between applications Loaded on first reference
OS loader fixes up references to shared libraries just like the static linkers
Not all libraries are loaded into the same virt ual address
Concurrency issues addressed in programming languages Locks, monitors, events, polling Order of operations visible across thread boundaries Memory model semantics become an issue Ada introduces rendez-vous, other languages have other constructs
Tool chain Compiler(s)
Linker
Loader Symbolic debugger
Figure of merit: Code quality (speed, but this is related to space)
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 16/46
2-Jan-12 Where's My Compiler? 16
OS Dynamic LoaderSource Code
Front End
Back End
Object Code
Compiler
Source Code
Front End
Back End
Object Code
Source Code
Front End
Back End
Object Code
Static Linker
OS Loader
Running Program
Symbol table(s)
Debugger
Includes fixups forshared code
Image FileImage File Image File
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 17/46
2-Jan-12 Where's My Compiler? 17
1990s: JITs and Managed
Runtimes Garbage Collection goes mainstream
Previously: LISP, APL, SmallTalk 1990s: Java, Jscript, C#, VB
Verification requires runtime to analyze code Verification is similar to front -end compiler work Can be done to native code, but much simpler with an intermediate
language
Just -in-time (JIT) compilation increases performance over pureinterpretation Typically by a factor of 5 to 15
Tool chain: split the compiler in two! Linearize the AST to create Intermediate Language (IL) Save symbol table as metadata Reorder the chain
Figures of merit: Throughput first, code quality second
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 18/46
2-Jan-12 Where's My Compiler? 18
OS Dynamic Loader (repeat)Source Code
Front End
Back End
Object Code
Compiler
Source Code
Front End
Back End
Object Code
Source Code
Front End
Back End
Object Code
Static Linker
OS Loader
Running Program
Symbol table(s)
Debugger
Includes fixups forshared code
Image FileImage File Image File
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 19/46
2-Jan-12 Where's My Compiler? 19
OS Dynamic Loader (repeat)Source Code
Front End
Back End
Object Code
Static Linker
OS Loader
Running Program
Debugger
Image File
Compiler
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 20/46
2-Jan-12 Where's My Compiler? 20
Managed Runtime
Running Program
Debugger
Source Code
Front End
Back End
Object Code
Static Linker
OS Loader
Image File
OS Loader
DynamicLinker
Image File
Back End
Runtime
Compiler
Compiler
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 21/46
2-Jan-12 Where's My Compiler? 21
Managed Runtime
Running Program
Debugger
Source Code
Front End
Back End
Object Code
Static Linker
OS Loader
Image File
OS Loader
DynamicLinker
Image File
Back End
Runtime
Compiler
Compiler
Metadata +IntermediateLanguage
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 22/46
2-Jan-12 Where's My Compiler? 22
2000s: Reflection-based
Computation Reflection: ability of a program to observe and possibly modify its
struct ure and behavior Compilers preserve meaning but runtime reflection makes more
information visible, so optimizations are more limited
Metadata (symbol table) or equivalent needed at runtime, not just compile/link time
Interactive Development Environments (IDEs) Intellisense Refactoring Interactive syntax analysis
Query Integration Builds expression trees (ASTs) at compile time Runtime operations to combine and manipulate them
Figures of merit: Compiler and JIT compiler: throughput Pre-JIT compiler: balance of throughput and code quality
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 23/46
2-Jan-12 Where's My Compiler? 23
Runtime Reflection
Running Program
Debugger
Source Code
Front End
OS Loader
DynamicLinker
Image File
Back End
Metadata +IntermediateLanguage
Metadata(symbol table)
Development Environment
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 24/46
2-Jan-12 Where's My Compiler? 24
Outline What Is A Compiler?
A Brief History of Developer Tools My First CompilerMy First Compiler
Compilers, compilers, everywhere
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 25/46
2-Jan-12 Where's My Compiler? 25
1970: Numbles Number puzzles for Nimble minds Column in Computers and Automation Numble verifier written by St uart Nelson Input language:
SEND
+ MORE
======
MONEY
Output :
a program to try all possible values for letterassignments to digits
Handled +, -, *, and = Hand coded in PDP-9 assembly language
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 26/46
2-Jan-12 Where's My Compiler? 26
Outline What Is A Compiler?
A Brief History of Developer Tools
My First Compiler Compilers, compilers, everywhereCompilers, compilers, everywhere
Free-standing compilers
Under the hood
Inside applications In the tool chain
Inside libraries
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 27/46
2-Jan-12 Where's My Compiler? 27
Special-Purpose Compilers Compile-to-hardware
Aspect -Oriented Programming (AOP) weaver
Parser finds new syntax to mark insertion points
Back-end inserts code snippets for different aspects
More generally: assembly rewriting
Work-flow and object design languages
Input may be text ual or graphic layouts
Output may be code or graphic designs
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 28/46
2-Jan-12 Where's My Compiler? 28
Mark-up Compilers XML schema (or DTD)
Output : parser
Output : deserializer
Web-services Description (WSDL) Output : proxy that parses input and dispatches
Output : code to convert data struct ure to XML (serializer)
XAML (Windows Presentation Framework) Ou
tput :
parser Output : executable code
XSL
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 29/46
2-Jan-12 Where's My Compiler? 29
Outline What Is A Compiler?
A Brief History of Developer Tools
My First Compiler Compilers, compilers, everywhereCompilers, compilers, everywhere
Free-standing compilers
Under the hood
Inside applications In the tool chain
Inside libraries
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 30/46
2-Jan-12 Where's My Compiler? 30
Modern Hardware: CPU Compile machine code to micro code
CPU Architect ure is the abstraction boundary RISC vs CISC is an old debate
x86 and x64 are CISC on the outside, RISC on the inside Part of the instruction cache
Engineering note: an icache miss now often means a pauseto compile in addition to a memory fetch!
Allows innovation in act ual hardware while still
running existing code Chips optimized for specific usage scenarios
Chips take advantage of materials science advances Chips take advantage of new internal architect ures (multi-
core)
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 31/46
2-Jan-12 Where's My Compiler? 31
Modern Hardware: Graphics Graphics memory isnt just for data Very sophisticated compilation steps
Parallel execution with CPU Adapts to changing hardware organization
Raster scan vs vector Resolution, speed, synchronization
Adapts to predominant u
sage pattern Animation 3D Shading
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 32/46
2-Jan-12 Where's My Compiler? 32
Outline What Is A Compiler?
A Brief History of Developer Tools
My First Compiler Compilers, compilers, everywhereCompilers, compilers, everywhere
Free-standing compilers
Under the hood
Inside applications In the tool chain
Inside libraries
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 33/46
2-Jan-12 Where's My Compiler? 33
Databases SQL is a f ull programming language
Compiled to intermediate form on client Intermediate form is passed to server for execution Server optimizes the intermediate form to produce an execution
plan Query optimization
Additional inputs include Size of tables Frequency of query types Indexing information
Outputs include
Executable code Temporary indexes Background indexing requests Updated frequency information
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 34/46
2-Jan-12 Where's My Compiler? 34
Hardware Emulators Object code translation at runtime
HP3000 to PA-RISC in 1983 Vax to Alpha in 1990s 32-bit programs on 64-bit hardware
Alternate hardware emulation Device emulators for everything from smart cards
to cell phones to iPod to pocket PCs
JIT compilation trades start -up time for highperformance execution Often, but not always, a good trade-off
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 35/46
2-Jan-12 Where's My Compiler? 35
Code Analysis Tools Analyzing API surface
Simple to do with front end ASTs
Remodularizing implementation Requires static and dynamic dependency analysis normal compiler back end work
Requires rebuilding the program, easily doneusing front end ASTs
Race detection Instrument code at compile time
Gather data as it runs under high stress
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 36/46
2-Jan-12 Where's My Compiler? 36
Tree Shakers Start with AST tree and appropriate
dependency graph
Pull AST nodes fo
und starting at a givengraph node, recursively
Convert resulting set of AST nodes toappropriate output format
Example uses: Subset library based on initial set of types Statically link subset of library for a given
application
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 37/46
2-Jan-12 Where's My Compiler? 37
Outline What Is A Compiler?
A Brief History of Developer Tools
My First Compiler Compilers, compilers, everywhereCompilers, compilers, everywhere
Free-standing compilers
Under the hood
Inside applications In the tool chain
Inside libraries
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 38/46
2-Jan-12 Where's My Compiler? 38
A Modern Interactive
Development Environment (IDE) Code editor
Knows the programming language, provides syntax support and context -sensitive name lookup
Project system
Tracks the public shape of components Tracks dependencies between components
Build system Orders clean-up, compile, and link operations
Debugger Allows inspection and modification of values at runtime Allows control operations (e.g., breakpoint, continue, restart)
Dynamic Support Allows program modification interwoven with execution (edit and
continue) Global interaction space (read-eval-print loop)
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 39/46
2-Jan-12 Where's My Compiler? 39
Compilers in the IDE (I) In the code editor
Incrementally parses the code as it is beingentered. Note: must deal with incorrect syntax
and partial programs. Suggests possible completions based on a symbol
table. Note: symbol table must include externalreferences maintained by the project system.
Refactoring operations require both syntactic and
semantic analysis. Note: refactoring requiresinformation maintained by the project system.
In the debugger Expression evaluation
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 40/46
2-Jan-12 Where's My Compiler? 40
Compilers In the IDE (II) Dynamic support
Edit -and-continue Requires a f ull, incremental compiler
For efficiency, it also requires the ability to compress theoutput as a diff between the original and the new code
Interactive workspace Like LISP, APL, SmallTalk, Python, etc.
Requires a compiler or
an interpreter -- really, a compiler front end to generate an ASTcombined with a tree walker to execute the tree.
The compiler must be capable of generating code that usescode and objects resident in the evaluation environment, whichgenerally means a reliance on reflection.
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 41/46
2-Jan-12 Where's My Compiler? 41
Compilers in the Linker The linker sees the whole program, so its better positioned to
do global analysis Solution: write a compiler
Input language is object file format (native code or IL) Output language is OS image file format
Optimizations: Aggressive in-lining across module boundaries Code motion across module boundaries Full type system analysis (treat leaf types as sealed)
Issues: These flow graphs are *big* The linker doesnt see the whole program (dynamic linking) Reflection and dynamic linking reduce permitted optimizations
Or require the ability to back out or recompute optimizations at runtime
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 42/46
2-Jan-12 Where's My Compiler? 42
Profile-Guided Optimization Idea: Instrument the program, run it with
typical loads, then re-optimize using thisprofiling data. (Similar to Hotspot)
Optimizations: Optimize only hot code fragments
So you can spend more time on them
Method and basic block reordering to increasecode density
Code reordering to optimize branch prediction andminimize long references
Cache locality optimizations for data and code
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 43/46
2-Jan-12 Where's My Compiler? 43
Outline What Is A Compiler?
A Brief History of Developer Tools
My First Compiler Compilers, compilers, everywhereCompilers, compilers, everywhere
Free-standing compilers
Under the hood
Inside applications In the tool chain
Inside libraries
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 44/46
2-Jan-12 Where's My Compiler? 44
For the Developer Regular expression parsing
Grammar is usually more powerf ul than
regular expressions
Serialization and Deserialization
Reflects on data type to be marshalled
Generates specialized code to convert tostream format (serialization) or parse intoin-memory format (deserialization)
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 45/46
2-Jan-12 Where's My Compiler? 45
For the Compiler Writer Parser-generators
lex yacc
AST tool kits Microsoft is investing in this area Provides integration into may aspects of the IDE
Executable file format tool kits Queensland University of Technology PERWAPI
Optimization tool kits Microsofts Phoenix project
8/3/2019 Where is My Compiler
http://slidepdf.com/reader/full/where-is-my-compiler 46/46
2-Jan-12 Where's My Compiler? 46
Questions?