gilad bracha ministry of truth embedding dsls in newspeak: newshell, ebnf and hopscotch

Gilad BrachaMinistry of Truth

Embedding DSLs in Newspeak:

NewShell, EBNF and Hopscotch

Three DSLs

Shell Scripting

Parsing

UI

Newspeak Syntax for Java

Programmers

Newspeak Syntax for C#

Programmers

Syntax: Unary Expressions

x foo

means

x.foo()

Binary Expressions

6 + 3 factorial

evaluates to 12.

In javanese, we’d write

6.+(3.factorial()) or, really

6.plus(3.factorial())

Keyword Expressions

Windows version: 7 + 0i .

is equivalent to all these:

Windows version: (7 + 0i).

Windows version: (7 + (0 i)).

Windows.version(7 + 0.i()) ;

Keyword Expressions

add: 1 to: Cobol

in javanese would be

add(1, Cobol);

Big advantage - no arity errors

Three DSLs

Shell Scripting

Parsing

UI

NewShell

class NewShell = (

ls value: ‘-l’.

ls - ‘lt’.

echo value: ‘Hello World’.

svn value: ‘ls’ value: ‘svn://myDir’.

(ls value findTokens: {Character cr}) collect: ls.

)()

Three DSLs

Shell Scripting

Parsing

UI

BNF

id = letter (letter | digit) *

Parser Combinators

BNF


Newspeak

id = letter, (letter | digit) star.

Parser Combinators

BNF


Newspeak


Javanese

id = letter().seq(letter().or(digit()).star());

Parser Combinators

How it Works


How it Works


letter

How it Works


letter

letter

How it Works


letter

digitletter

How it Works


letter

|

digitletter

How it Works


letter star

|

digitletter

How it Works


letter star

|

digitletter

,

Why is this Pretty?


Why is this Ugly?


Why is this Ugly?


vs.


vs.


Why is it Ugly?

A programming language is low level when its programs require attention to the irrelevant

- Alan Perlis

class ExampleGrammar1 = ExecutableGrammar ( |

digit = charBetween: ‘0’ and: ‘9’.

letter = (charBetween: ‘a’ and:’z’) | (charBetween: ‘A’ and:’Z’).


identifier = tokenFor: id.

hat = tokenFromChar: ‘ˆ’.

expression = identifier.

returnStatement = hat, expression.

| )()

A Complete Grammar

returnStatement = hat, expression wrapper:[:r : e | ReturnStatAST on: e ]

Building an AST

Building an AST

returnStatement = hat, expression

wrapper:[:r : e |

ReturnStatAST on: e

]wrapper: ,

expression

^


wrapper:[:r : e |

ReturnStatAST on: e

]

Grammar


wrapper:[:r : e |

ReturnStatAST on: e

]

Semantic Action

Superclass (pure grammar specification):


Subclass (AST builder):

returnStatement = (

super returnStatement

wrapper:[:r : e | ReturnStatAST on: e] )

Factor out Grammar

class ExampleParser1= ExampleGrammar1 () (

id = (

ˆsuper id

wrapper:[:fst :snd | fst asString, (String withAll: snd)]

)

identifier = (

ˆsuper identifier

wrapper:[:v | VariableAST new name: v token;

start: v start; end: v end].

)

r

Modular Parser

…

returnStatement = (

ˆsuper returnStatement

wrapper:[:r :e | ReturnStatAST new expr:e;

start: r start; end: e end].

)

)

Modular Parser

class ExampleGrammar2 = ExampleGrammar1 (

|

if = tokenFromSymbol:#if.

then = tokenFromSymbol:#then.

else = tokenFromSymbol:#else.

ifStatement = if, expression, then, statement, else, statement.

statement = ifStatement | returnStatement.

|

)()

Extending a Grammar

Mutual RecursionIfStmtreturnStat

|

if then else

stmt

Mutual Recursion

statement

ifStatement

letterid

identifierhat

expressionreturnStatement

if

thenelse

digit nilnilnilnilnilnilnil

nil

nilnilnil

nil

Mutual Recursion

statement

ifStatement

letterid

identifierhat


if

thenelse

digit a parsernilnilnilnilnilnil

nil

nilnilnil

nil

Mutual Recursion

statement

ifStatement

letterid

identifierhat


if

thenelse

digit a parsera parser

nilnilnilnilnil

nil

nilnilnil

nil

Mutual Recursion

statement

ifStatement

letterid

identifierhat


if

thenelse

digit a parsera parsera parser

nilnilnilnil

nil

nilnilnil

nil

Mutual Recursion

statement

ifStatement

letterid

identifierhat


if

thenelse

digit a parsera parsera parsera parser

nilnilnil

nil

nilnilnil

nil

Mutual Recursion

statement

ifStatement

letterid

identifierhat


if

thenelse

digit a parsera parsera parsera parsera parser

nilnil

nil

nilnilnil

nil

Mutual Recursion

statement

ifStatement

letterid

identifierhat


if

thenelse

digit a parsera parsera parsera parsera parsera parsera parser

nil

nilnilnil

nil

Mutual Recursion

statement

ifStatement

letterid

identifierhat


if

thenelse


a parser

nilnilnil

nil

Mutual Recursion

statement

ifStatement

letterid

identifierhat


if

thenelse


a parser

a parsernilnil

nil

Mutual Recursion

statement

ifStatement

letterid

identifierhat


if

thenelse


a parser

a parsera parser

nil

nil

Mutual Recursion

statement

ifStatement

letterid

identifierhat


if

thenelse


a parser

a parsera parserboom!

nil

Addressed with laziness in Haskell

Can be addressed with closures

ifStatement = if, [expression], [then], [statement],

[else], [statement].

statement = ifStatement | [returnStatement].

Mutual Recursion

Use Reflection instead of Laziness.

Framework initializes all slots to “stand-in” parsers which act as forward references

Being Reflective rather than Lazy

Mutual Recursion

statementifStatement

letterid

identifierhat


ifthenelse

digit fwd reffwd reffwd reffwd reffwd reffwd reffwd ref

fwd reffwd reffwd reffwd reffwd ref

Mutual Recursion


letterid

identifierhat


ifthenelse

digit a parserfwd reffwd reffwd reffwd reffwd reffwd ref


Mutual Recursion


letterid

identifierhat


ifthenelse

digit a parsera parserfwd reffwd reffwd reffwd reffwd ref


Mutual Recursion


letterid

identifierhat


ifthenelse

digit a parsera parsera parserfwd reffwd reffwd reffwd ref


Mutual Recursion


letterid

identifierhat


ifthenelse

digit a parsera parsera parsera parserfwd reffwd reffwd ref


Mutual Recursion


letterid

identifierhat


ifthenelse

digit a parsera parsera parsera parsera parserfwd reffwd ref


Mutual Recursion


letterid

identifierhat


ifthenelse



Mutual Recursion


letterid

identifierhat


ifthenelse


a parserfwd reffwd reffwd reffwd ref

Mutual Recursion


letterid

identifierhat


ifthenelse


a parsera parserfwd reffwd reffwd ref

Mutual Recursion


letterid

identifierhat


ifthenelse


a parsera parsera parserfwd reffwd ref

Mutual Recursion


letterid

identifierhat


ifthenelse


a parsera parsera parsera parserfwd ref

Mutual Recursion


letterid

identifierhat


ifthenelse


a parsera parsera parsera parsera parser


|

if then else

stmt

FwdRef

Being Reflective rather than Lazy

Use Reflection instead of Laziness.

Framework initializes all slots to “stand-in” parsers which act as forward references

When grammar is completely initialized, stand-ins are connected to originals and forward calls to them


|

if then else

stmt

FwdRef

The Mirror is Mightier Than the Sloth

Grammar1

Grammar2 Parser1

Parser2

Grammar1

Grammar2

Parser1

Parser2

class ExampleParser2 = ExampleParser1 mixin | >

ExampleGrammar2 ()

(

ifStatement = (

ˆsuper ifStatement

wrapper:[:ifKw :e :thenKw :s1 :elseKw :s2 | …

]

)

)

Composing Parsers

Shareable among several IDEs/compilers that require distinct ASTs

Shareable among other language tools: syntax colorizers, postprocessing tools

Grammar is shared executable

specification

Three DSLs

Shell Scripting

Parsing

UI

Hopscotch

A GUI application framework

Higher level of abstraction than widgets

Provides DSL for composing presentations out of pieces known as fragments

Fragments

Fragments of presentation

Atomic widgets correspond to leaf fragments

Aggregates represented via compound fragments

Composed via combinators

Fragment Combinators

blank, filler

button:action:, link:action:, image:, menuWithLabelsAndActions:

row:, column:, centered:, collapsed:expanded:

deferred:, draggable:subject:

Dispatching up the fragment hierarchy

Dispatching up the visual containment hierarchy makes sense

Old idea (at least since NewtonScript)

Realized using doesNotUnderstand:

Connections

Self & Smalltalk

ConnectionsParser Combinators.

Too much to survey; some highlights:

Parsec (Haskell parser combinator library)

Swierstra et al.

Sparsec (Scala parser combinator library)

PEGs (Ford, POPL04, ICFP)

Connections

Parser Combinators (continued)

Tedir (Plesner-Hansen’s masters thesis)

Ometa (OOPLA 07 DLS)

Conclusions

Newspeak supports DSLs via

Unconventional syntax

keywords

operators as methods

implicit receivers

Dynamic typing and Reflection

doesNotUnderstand:

Conclusions

Newspeak also supports DSLs via

Mixins, Modules, Hierarchy inheritance

IDE

Combinators are a powerful technique

Compositional

Useful in libraries and languages

Status Available at http://newpeaklanguage.org

Open source under Apache 2.0 license

Work in Progress

Expect some tweaks to syntax and semantics

Implementation still not complete - especially libraries

Credits

Peter Ahe

Vassili Bykov

Yaron Kashai

Bill Maddox

Eliot Miranda

This file is licensed under the Creative Commons Attribution ShareAlike 3.0 License. In short: you are free to share and make derivative works of the file under the conditions that you appropriately attribute it, and that you distribute it only under a license identical to this one. Official

license.

The Newspeak eye used in the bullets, slide background etc. was designed by Victoria Bracha and is used by permission.

gilad bracha ministry of truth embedding dsls in newspeak: newshell, ebnf and hopscotch

Documents