bootstrapping meta-languages of language workbenches

Bootstrapping Meta-Languages of

Language WorkbenchesGabriël Konat

Joint work with: Sebastian Erdweg, Eelco Visser

2

Tools and environment to develop languages

3

Spoofax Language Workbench

4

Domain-specific meta-languages

Syntax Name binding Type system Dynamic semantics

5

Bootstrap the meta-languages

Syntax Name binding Type system Dynamic semantics

6

Syntax Name binding Type system Dynamic semanticsHigher-level implementation

7


Large-scale test case

8



Coverage

9



Coverage

Improvement dissemination

Bootstrapping Meta-Languages

11

SDFStratego

12

SDFStratego

context-free syntaxDef .Ent = <ent <Id> {<Decl*>}>Decl.Prop = <<Id>:<Type>>Type.Str = <String>

13

SDFStratego


PT-gen PT

14

SDFStratego


PT-gen PT

Parser ASTParseEnt("person",[ Prop("addr", Str())])

ent person { addr:String}

15

SDFStratego


PT-gen PT



SigSig-gen

Types

signaturesEnt : Id * List(Decl) -> DefProp : Id * Type -> DeclStr : Type

16

SDFStratego


PT-gen PT



SigSig-gen

Types


PP-gen

PP

17

SDFStratego


PT-gen PT



SigSig-gen

Types


PP-gen

PPtransformation rulestojava: Ent(e, ds) -> Class(…)tojava: Prop(p, t) -> Field(…)tojava: Str() -> Type(…)

18

SDFStratego


PT-gen PT



SigSig-gen

Types


PP-gen

PPtransformation rulestojava: Ent(e, ds) -> Class(…)tojava: Prop(p, t) -> Field(…)tojava: Str() -> Type(…)

ASTTransform

19

SDF

Sig-gen

PP-gen

Stratego

20

SDF Stratego (STR)Sig-gen

PP-gen

21

SDF STRSig-gen

PP-gen

22

SDF STRSig-gen

PP-gen

23

SDF STRSig-gen

PP-gen

24

SDF STRSig-gen

PP-gen

25

SDF STRSig-gen

PP-gen

26

SDF STR

Sig

SigSig-gen

PP-gen

27

SDF STR

Sig

Sig

PP

PP

Sig-gen

PP-gen

28

SDF STRSig-gen

PP-gen

Sig

Sig

PP

PP

Sig-gen : STR

PP-gen : STR

29

SDF STRSig-gen

PP-gen

Sig

Sig

PP

PP

Sig-gen : STR

PP-gen : STR

Sig : STR

Sig : STR

PP : STR

PP : STR

30

SDF STRSig-gen

PP-gen

Sig

Sig

PP

PP

Sig-gen : STR

PP-gen : STR

Sig : STR

Sig : STR

PP : STR

PP : STR

31

SDF STR

Multiple meta-languages

32

SDF STR

Self and mutual dependency

33

SDF STRSig-gen

PP-gen

Sig

Sig

PP

PP

Sig-gen : STR

PP-gen : STR

Sig : STR

Sig : STR

PP : STR

PP : STR

Generate files of each other

34

SDF STRSig-gen

PP-gen

Sig

Sig

PP

PP

Use files of each other

Sig-gen : STR

PP-gen : STR

Sig : STR

Sig : STR

PP : STR

PP : STR

35

SDF STRSig-gen

PP-gen

Sig

Sig

PP

PP

Complex to bootstrap

Sig-gen : STR

PP-gen : STR

Sig : STR

Sig : STR

PP : STR

PP : STR

36

SDF STR

Sig

Sig

PP

PP

Sig-gen

PP-gen

Sig-gen : STR

PP-gen : STRPP-gen* : STR

Sig : STR

Sig : STR

PP : STR

PP : STR

37

SDF STR

Sig

Sig

PP

PP

Sig-gen

PP-gen

Sig : STR

Sig : STR

PP : STR

PP : STR

Sig-gen : STR

PP-gen : STR PP : STRPP-gen* : STR

38

Bootstrap to find defects

SDF STR

Sig

Sig

PP

PP

Sig-gen

PP-gen

Sig : STR

Sig : STR

PP : STR

PP : STR

Sig-gen : STR

PP-gen : STR PP : STRPP-gen* : STR

39

SDF STRSig-genPP-gen PP

Baseline

40


Baseline

STR

PP

Iteration 1Fixpoint bootstrapping

41


Baseline

STR

PP

Iteration 1 SDF

Sig-genPP-gen*

Sig

String(s)->Op("S", [Str(s), Str(";")])

String(s)->Op("S", [Str(s)])

42


Baseline

STR

PP

Iteration 1 SDF

Sig-genPP-gen*

Sig

String(s)->Op("S", [Str(s), Str(";")])

String(s)->Op("S", [Str(s)])

Hidden defect

43


Baseline

STR

PP

Iteration 1 SDF

Sig-genPP-gen*

Sig

SDF STRSig

Sig-genPP-gen PP

Iteration 2

pp-Decl : Signature(t) -> [ H([ SOpt(HS(), "0") , S("signatures") , S(";") ]) , <pp-Sig> t]

44


Baseline

STR

PP

Iteration 1 SDF

Sig-genPP-gen*

Sig

SDF STRSig

Sig-genPP-gen PP

Iteration 2

SDF STRSig

Sig-genPP-gen PP

Iteration 3

pp-Decl : Signature(t) -> [ H([ SOpt(HS(), "0") , S("signatures") , S(";") ]) , <pp-Sig> t]

45


Baseline

STR

PP

Iteration 1 SDF

Sig-genPP-gen*

Sig

SDF STRSig

Sig-genPP-gen PP

Iteration 2

SDF STRSig

Sig-genPP-gen PP

Iteration 3

SDF Sig

Sig-genPP-gen

Iteration 4

module; signatures/sorts/Sorts-sigsignatures; :; String ->; SortSort :; Sort ->; SymbolPSort :; Sort *; List(;Symbol); ->; Symbol

Defect found

46


Baseline

STR

PP

Iteration 1 SDF

Sig-genPP-gen*

Sig

SDF STRSig

Sig-genPP-gen PP

Iteration 2

SDF STRSig

Sig-genPP-gen PP

Iteration 3

SDF Sig

Sig-genPP-gen

Iteration 4


Defect found

47


Baseline

STR

PP

Iteration 1 SDF

Sig-genPP-gen*

Sig

SDF STRSig

Sig-genPP-gen PP

Iteration 2

SDF STRSig

Sig-genPP-gen PP

Iteration 3

SDF Sig

Sig-genPP-gen

Iteration 4

modulesignatures SortPSort

47

Dynsem

NaBL2

TSNaBL

ESV

Stratego

SDF3

SDF2

48


Baseline

STR

PP

Iteration 1 SDF

Sig-genPP-gen*

Sig

SDF STRSig

Sig-genPP-gen PP

Iteration 2

SDF STRSig

Sig-genPP-gen PP

Iteration 3

SDF Sig

Sig-genPP-gen

Iteration 4


Fixpoint bootstrapping

49


Baseline

STR

PP

Iteration 1 SDF

Sig-genPP-gen*

Sig

SDF STRSig

Sig-genPP-gen PP

Iteration 2

SDF STRSig

Sig-genPP-gen PP

Iteration 3

SDF Sig

Sig-genPP-gen

Iteration 4



Binary comparison

50


Baseline

STR

PP

Iteration 1 SDF

Sig-genPP-gen*

Sig

SDF STRSig

Sig-genPP-gen PP

Iteration 2

SDF STRSig

Sig-genPP-gen PP

Iteration 3

SDF Sig

Sig-genPP-gen

Iteration 4



Binary comparison

Existing baseline

51


Baseline

STR

PP

Iteration 1 SDF

Sig-genPP-gen*

Sig

SDF STRSig

Sig-genPP-gen PP

Iteration 2

SDF STRSig

Sig-genPP-gen PP

Iteration 3

SDF Sig

Sig-genPP-gen

Iteration 4



Binary comparison

Existing baseline

Deterministic compilers

52


Baseline

STR

PP

Iteration 1 SDF

Sig-genPP-gen*

Sig

SDF STRSig

Sig-genPP-gen PP

Iteration 2

SDF STRSig

Sig-genPP-gen PP

Iteration 3

SDF Sig

Sig-genPP-gen

Iteration 4



Binary comparison

Existing baseline


Produce baselines

53


Baseline

STR

PP

Iteration 1 SDF

Sig-genPP-gen*

Sig

SDF STRSig

Sig-genPP-gen PP

Iteration 2

SDF STRSig

Sig-genPP-gen PP

Iteration 3

SDF Sig

Sig-genPP-gen

Iteration 4



Binary comparison

Existing baseline


Produce baselines

Roll back on defects

54


Baseline

STR

PP

Iteration 1 SDF

Sig-genPP-gen*

Sig

SDF STRSig

Sig-genPP-gen PP

Iteration 2

SDF STRSig

Sig-genPP-gen PP

Iteration 3

SDF Sig

Sig-genPP-gen

Iteration 4



Binary comparison

Existing baseline


Produce baselines

Roll back on defects

General

Systematic Meta-Language Bootstrapping

Model

56

SDF1SDF1

Language definition

Model

57

SDF1SDF1VersioningLanguage definition

Model

58

SDF1SDF1

Sig-gen : STR

Grammar : SDFSource artifacts

Model

59

SDF1SDF1

Sig-gen : STR

Grammar : SDF

SDF0 STR0 Language product

Model

60

SDF1SDF1

Sig-gen : STR

Grammar : SDF

SDF0 STR0 Language productBaseline: set of products

Model

61

SDF1SDF1

Sig-gen : STR

Grammar : SDF

SDF0 STR0

Sig : STR Export artifacts

Model

62

SDF1SDF1

Sig-gen : STR

Grammar : SDF

SDF0 STR0

Sig : STR

Sig-gen : SDF ⇒ STR Export generators

Model

63

SDF1SDF1

Sig-gen : STR

Grammar : SDF

SDF0 STR0

Sig : STR

Sig-gen : SDF ⇒ STR

Dependency

Model

64

SDF1SDF1

Sig-gen : STR

Grammar : SDF

SDF0 STR0

Sig : STR

Sig-gen : SDF ⇒ STR STR2Java :

STR ⇒ JAR

Compilation

SDF1

Sig-gen : STR

Grammar : SDF

SDF0 STR0

Sig : STR


STR ⇒ JAR

65

SDF1Compile

Compilation

SDF1

Sig-gen : STR

Grammar : SDF

SDF0 STR0

Sig : STR


STR ⇒ JAR

66

SDF1CompileGrammar : SDF

Compilation

SDF1

Sig-gen : STR

Grammar : SDF

SDF0 STR0

Sig : STR


STR ⇒ JAR

67


SDF0

Compilation

SDF1

Sig-gen : STR

Grammar : SDF

SDF0 STR0

Sig : STR


STR ⇒ JAR

68


SDF0


Compilation

SDF1

Sig-gen : STR

Grammar : SDF

SDF0 STR0

Sig : STR


STR ⇒ JAR

69


SDF0


Sig : STR

Compilation

SDF1

Sig-gen : STR

Grammar : SDF

SDF0 STR0

Sig : STR


STR ⇒ JAR

Sig : STR

SDF1Compile

70

Sig-gen : STR

Compilation

SDF1

Sig-gen : STR

Grammar : SDF

SDF0 STR0

Sig : STR


STR ⇒ JAR

Sig : STR

SDF1Compile

71

Sig-gen : STR

STR0

STR2Java : STR ⇒ JAR

Compilation

SDF1

Sig-gen : STR

Grammar : SDF

SDF0 STR0

Sig : STR


STR ⇒ JAR

Sig : STR

SDF1Compile

72

Sig-gen : STR

STR0


SDF0

Sig : STR

Compilation

SDF1

Sig-gen : STR

Grammar : SDF

SDF0 STR0

Sig : STR


STR ⇒ JAR

Sig : STR

SDF1Compile

73

Sig-gen : STR

STR0


SDF0

Sig : STR


Bootstrapping

Bootstrapping - inputs


76

STR0

SDF0*

Set of language definitions


77

STR0

SDF0*

STR0

SDF0

Baseline


78

STR0

SDF0*

STR0

SDF0

Baseline


79

STR0

SDF0*

STR0

SDF01

Version number

80

STR0

SDF0*

STR1

SDF1*

STR0

SDF0

Bootstrapping - first iteration

Apply version number


81

STR0

SDF0*

STR1

SDF1* SDF1

STR1STR0

SDF0

Compile


82

STR0

SDF0*

STR1

SDF1*

STR0SDF0

SDF1

STR1

Store baseline


83

STR0

SDF0

STR0

SDF0*

STR1

SDF1*

STR0SDF0

STR1

SDF1

Use compiled products


84

STR0

SDF0

STR0

SDF0*

STR1

SDF1*

STR0SDF0

STR1

SDF1

Use compiled products

Fixpoint Bootstrapping

85

STR0

SDF0

STR0

SDF0*

STR1

SDF1* SDF1

STR1

STR0SDF0

STR1

SDF1


86

STR0

SDF0

STR0

SDF0*

STR1

SDF1* SDF1

STR1

STR0SDF0

STR1

SDF1

STR0

SDF0

STR1

SDF1

STR1

SDF1

Binary comparison


87

STR0

SDF0

STR0

SDF0*

STR1

SDF1* SDF1

STR1

STR0SDF0

STR1

SDF1

STR0

SDF0

STR1

SDF1

STR1

SDF1

Binary comparison

STR1

SDF1

STR1

SDF1


88

STR0

SDF0

STR0

SDF0*

STR1

SDF1* SDF1

STR1

STR0SDF0

STR1

SDF1

STR0

SDF0

STR1

SDF1

STR1

SDF1

STR1

SDF1

STR1

SDF1

Binary comparison


89

STR0

SDF0

STR0

SDF0*

STR1

SDF1* SDF1

STR1

STR0SDF0

STR1

SDF1

STR0

SDF0

STR1

SDF1

STR1

SDF1

STR1

SDF1

STR1

SDF1

STR1SDF1


90

STR0

SDF0

STR0

SDF0*

STR1

SDF1* SDF1

STR0SDF0

STR1

SDF1

Found defects


91

STR0

SDF0*

STR1

SDF1*

STR0SDF0

Discard products


92

STR0

SDF0*

STR1

SDF1*

STR0

SDF0

Rollback


93

STR0

SDF0*

STR0

SDF0

Undo version changes


94

STR0

SDF0*

STR0

SDF0 SDF0*

Store error

STR0

SDF0

STR0

SDF0* SDF1

STR1

STR0SDF0

9595

General (language-parametric)

STR0

SDF0

STR0

SDF0* SDF1

STR1

STR0SDF0

9696

General (language-parametric)

Fully automated

97

-- Model for languages, language definitions with sources, and language products with artifacts and generators.

type Version = Int

type Lang = String

data LangID = LangID { name :: Lang, version :: Version }

data Artifact = Artifact { aname :: String, alang :: Lang, acontent :: String }

data LangDef = LangDef { dlang :: LangID, dsources :: [Artifact], dartDeps :: [LangID], dgenDeps :: [LangID] }

data Generator = Generator { gname :: String, gsource :: Lang, gtarget :: Lang, generate :: Artifact -> Artifact }

data LangProd = LangProd { plang :: LangID, partifacts :: [Artifact], pgenerators :: [Generator] }

type Baseline = [LangProd]

getProd :: LangID -> Baseline -> LangProd

getProd lang baseline = fromJust $ find (\prod -> lang == plang prod) baseline

-- Compile. Sort languages by generator source/target and run relevant generators against relevant artifacts.

compile :: LangDef -> Baseline -> LangProd

compile def baseline = createLangProd def (runGenerators sortedLangs generators inputs)

where inputs = dsources def ++ [ a | l <- dartDeps def, a <- partifacts (getProd l baseline) ]

generators = [ g | l <- dgenDeps def, g <- pgenerators (getProd l baseline) ]

sortedLangs = topsort [ l | LangID l _ <- dgenDeps def ] [ (gsource g,gtarget g) | g <- generators]

runGenerators :: [Lang] -> [Generator] -> [Artifact] -> [Artifact]

runGenerators [] gens inputs = inputs

runGenerators (lang:langs) gens inputs = runGenerators langs gens (inputs ++ runGeneratorsFor lang gens inputs)

runGeneratorsFor :: Lang -> [Generator] -> [Artifact] -> [Artifact]

runGeneratorsFor lang gens inputs = [ generate g a | g <- gens, a <- inputs, gsource g == lang, alang a == lang ]

createLangProd :: LangDef -> [Artifact] -> LangProd -- Implemented by the language workbench

-- Fixpoint bootstrap language definitions with a baseline. Update versions in the first iteration, then fixpoint.

bootstrap :: Version -> [LangDef] -> Baseline -> (Baseline, [LangDef])

bootstrap v defs baseline =

let firstBuild = [ compile (setVersion v def) baseline | def <- defs ] in

bootstrapFixpoint (prepareFixpoint v defs) firstBuild

bootstrapFixpoint :: [LangDef] -> Baseline -> (Baseline, [LangDef])

bootstrapFixpoint defs baseline =

let newBaseline = [ compile def baseline | def <- defs ] in

if baseline == newBaseline

then (newBaseline, defs)

else bootstrapFixpoint defs newBaseline

setVersion :: Version -> LangDef -> LangDef

setVersion v (LangDef (LangID l _) srcs gdeps adeps) = LangDef (LangID l v) srcs gdeps adeps

prepareFixpoint :: Version -> [LangDef] -> [LangDef]

prepareFixpoint v defs = [ prepareFixpointDef v bootstrappedLangs def | def <- defs ]

where bootstrappedLangs = [ l | LangDef (LangID l _) _ _ _ <- defs ]

prepareFixpointDef :: Version -> [Lang] -> LangDef -> LangDef

prepareFixpointDef v langs (LangDef (LangID l _) srcs adeps gdeps) =

LangDef (LangID l v) srcs [ updateDep v langs dep | dep <- adeps ] [ updateDep v langs dep | dep <- gdeps ]

updateDep :: Version -> [Lang] -> LangID -> LangID

updateDep v langs (LangID l vold) = if l �elem� langs then LangID l v else LangID l vold

Figure 2. Model for sound bootstrapping, with algorithms for compilation and fixpoint bootstrapping, encoded in Haskell.

3.3 Fixpoint Bootstrapping

We can now use compilation to define fixpoint bootstrapping.In general, there is no way to know how many bootstrappingiterations are required before it is safe to stop. Therefore, weiteratively bootstrap meta-languages until reaching a fixpoint.We define a general fixpoint bootstrapping algorithm usingthe model and compilation algorithm from above.

Function bootstrap takes the version of the new baseline,a list of meta-language definitions, and an existing baseline,and it produces a new baseline of the given version. The basicidea of the algorithm is to compile meta-language definitionsin iterations, until we reach a fixpoint. However, to avoidbuilding against the old baseline repeatedly, we have to updatethe versions of the language definitions in the first iteration.

6 2016/10/12

Source Code in our paper

55 lines of Haskell code

Model Compilation algorithm

Bootstrapping algorithm

Implementation

Implementation in Spoofax

99


100


101

Evaluation

103

Dynsem

NaBL2

TSNaBL

ESV

Stratego

SDF3

SDF2

Bootstrapped 8 of Spoofax’s

Meta-languagesDynsem

NaBL2

TSNaBL

ESV

Stratego

SDF3

SDF2

104

Dynsem

NaBL2

TSNaBL

ESV

Stratego

SDF3

SDF2


Meta-languages

Complex dependencies

105

Dynsem

NaBL2

TSNaBL

ESV

Stratego

SDF3

SDF2


Meta-languages


Results in paper and online

35

SDF STRSig-gen

PP-gen

Sig

Sig

PP

PP


Sig : STR

Sig : STR

PP : STR

PP : STR

Sig-gen : STR

PP-gen : STR

44


Baseline

STR

PP

Iteration 1 SDF

Sig-genPP-gen*

Sig

SDF STRSig

Sig-genPP-gen PP

Iteration 2

SDF STRSig

Sig-genPP-gen PP

Iteration 3

SDF Sig

Sig-genPP-gen

Iteration 4


Defect found

35

SDF STRSig-gen

PP-gen

Sig

Sig

PP

PP


Sig : STR

Sig : STR

PP : STR

PP : STR

Sig-gen : STR

PP-gen : STR

44


Baseline

STR

PP

Iteration 1 SDF

Sig-genPP-gen*

Sig

SDF STRSig

Sig-genPP-gen PP

Iteration 2

SDF STRSig

Sig-genPP-gen PP

Iteration 3

SDF Sig

Sig-genPP-gen

Iteration 4


Defect found

Compilation

SDF1

Sig-gen : STR

Grammar : SDF

SDF0 STR0

Sig : STR


STR ⇒ JAR

Sig : STR

SDF1Compile

71

Sig-gen : STR

STR0


SDF0

Sig : STR



86

STR0

SDF0

STR0

SDF0*

STR1

SDF1* SDF1

STR1

STR0SDF0

STR1

SDF1

STR0

SDF0

STR1

SDF1

STR1

SDF1

STR1

SDF1

STR1

SDF1

STR1SDF1

35

SDF STRSig-gen

PP-gen

Sig

Sig

PP

PP


Sig : STR

Sig : STR

PP : STR

PP : STR

Sig-gen : STR

PP-gen : STR

44


Baseline

STR

PP

Iteration 1 SDF

Sig-genPP-gen*

Sig

SDF STRSig

Sig-genPP-gen PP

Iteration 2

SDF STRSig

Sig-genPP-gen PP

Iteration 3

SDF Sig

Sig-genPP-gen

Iteration 4


Defect found

Compilation

SDF1

Sig-gen : STR

Grammar : SDF

SDF0 STR0

Sig : STR


STR ⇒ JAR

Sig : STR

SDF1Compile

71

Sig-gen : STR

STR0


SDF0

Sig : STR



86

STR0

SDF0

STR0

SDF0*

STR1

SDF1* SDF1

STR1

STR0SDF0

STR1

SDF1

STR0

SDF0

STR1

SDF1

STR1

SDF1

STR1

SDF1

STR1

SDF1

STR1SDF1Implementation in Spoofax

99 104

Dynsem

NaBL2

TSNaBL

ESV

Stratego

SDF3

SDF2


Meta-languages


7 changes

35

SDF STRSig-gen

PP-gen

Sig

Sig

PP

PP


Sig : STR

Sig : STR

PP : STR

PP : STR

Sig-gen : STR

PP-gen : STR

44


Baseline

STR

PP

Iteration 1 SDF

Sig-genPP-gen*

Sig

SDF STRSig

Sig-genPP-gen PP

Iteration 2

SDF STRSig

Sig-genPP-gen PP

Iteration 3

SDF Sig

Sig-genPP-gen

Iteration 4


Defect found

Compilation

SDF1

Sig-gen : STR

Grammar : SDF

SDF0 STR0

Sig : STR


STR ⇒ JAR

Sig : STR

SDF1Compile

71

Sig-gen : STR

STR0


SDF0

Sig : STR



86

STR0

SDF0

STR0

SDF0*

STR1

SDF1* SDF1

STR1

STR0SDF0

STR1

SDF1

STR0

SDF0

STR1

SDF1

STR1

SDF1

STR1

SDF1

STR1

SDF1

STR1SDF1Implementation in Spoofax

99 104

Dynsem

NaBL2

TSNaBL

ESV

Stratego

SDF3

SDF2


Meta-languages


7 changes

Systematic Meta-Language Bootstrapping

bootstrapping meta-languages of language workbenches

Science