gilad bracha ministry of truth embedding dsls in newspeak: newshell, ebnf and hopscotch

Gilad BrachaMinistry of Truth

Embedding DSLs in Newspeak:

NewShell, EBNF and Hopscotch

Three DSLs

Shell Scripting

Parsing

Newspeak Syntax for Java

Programmers

Newspeak Syntax for Java

Programmers

Newspeak Syntax for C#

Programmers

Syntax: Unary Expressions

x.foo()

Syntax: Unary Expressions

x.foo()

Binary Expressions

6 + 3 factorial

evaluates to 12.

In javanese, we’d write

6.+(3.factorial()) or, really

6.plus(3.factorial())

Keyword Expressions

Windows version: 7 + 0i .

is equivalent to all these:

Windows version: (7 + 0i).

Windows version: (7 + (0 i)).

Windows.version(7 + 0.i()) ;

Keyword Expressions

add: 1 to: Cobol

in javanese would be

add(1, Cobol);

Big advantage - no arity errors

Three DSLs

Shell Scripting

Parsing

NewShell

class NewShell = (

ls value: ‘-l’.

ls - ‘lt’.

echo value: ‘Hello World’.

svn value: ‘ls’ value: ‘svn://myDir’.

(ls value findTokens: {Character cr}) collect: ls.

Three DSLs

Shell Scripting

Parsing

id = letter (letter | digit) *

Parser Combinators

Newspeak

id = letter, (letter | digit) star.

Parser Combinators

Newspeak

Javanese

id = letter().seq(letter().or(digit()).star());

Parser Combinators

How it Works

letter

How it Works

letter

How it Works

letter

How it Works

letter

How it Works

letter

How it Works

letter

How it Works

letter

How it Works

letter

How it Works

letter

digitletter

How it Works

letter

digitletter

How it Works

letter star

digitletter

How it Works

letter star

digitletter

Why is this Pretty?

Why is this Ugly?

Why is it Ugly?

A programming language is low level when its programs require attention to the irrelevant

- Alan Perlis

class ExampleGrammar1 = ExecutableGrammar ( |

digit = charBetween: ‘0’ and: ‘9’.

letter = (charBetween: ‘a’ and:’z’) | (charBetween: ‘A’ and:’Z’).

identifier = tokenFor: id.

hat = tokenFromChar: ‘ˆ’.

expression = identifier.

returnStatement = hat, expression.

A Complete Grammar

returnStatement = hat, expression wrapper:[:r : e | ReturnStatAST on: e ]

Building an AST

returnStatement = hat, expression

wrapper:[:r : e |

ReturnStatAST on: e

]wrapper: ,

expression

wrapper:[:r : e |

ReturnStatAST on: e

Grammar

wrapper:[:r : e |

ReturnStatAST on: e

Semantic Action

Superclass (pure grammar specification):

Subclass (AST builder):

returnStatement = (

super returnStatement

wrapper:[:r : e | ReturnStatAST on: e] )

Factor out Grammar

class ExampleParser1= ExampleGrammar1 () (

id = (

ˆsuper id

wrapper:[:fst :snd | fst asString, (String withAll: snd)]

identifier = (

ˆsuper identifier

wrapper:[:v | VariableAST new name: v token;

start: v start; end: v end].

Modular Parser

returnStatement = (

ˆsuper returnStatement

wrapper:[:r :e | ReturnStatAST new expr:e;

start: r start; end: e end].

Modular Parser

class ExampleGrammar2 = ExampleGrammar1 (

if = tokenFromSymbol:#if.

then = tokenFromSymbol:#then.

else = tokenFromSymbol:#else.

ifStatement = if, expression, then, statement, else, statement.

statement = ifStatement | returnStatement.

Extending a Grammar

Mutual RecursionIfStmtreturnStat

if then else

Mutual Recursion

statement

ifStatement

letterid

identifierhat

expressionreturnStatement

thenelse

digit nilnilnilnilnilnilnil

nilnilnil

Mutual Recursion

statement

ifStatement

letterid

identifierhat

thenelse

digit a parsernilnilnilnilnilnil

nilnilnil

Mutual Recursion

statement

ifStatement

letterid

identifierhat

thenelse

digit a parsera parser

nilnilnilnilnil

nilnilnil

Mutual Recursion

statement

ifStatement

letterid

identifierhat

thenelse

digit a parsera parsera parser

nilnilnilnil

nilnilnil

Mutual Recursion

statement

ifStatement

letterid

identifierhat

thenelse

digit a parsera parsera parsera parser

nilnilnil

Mutual Recursion

statement

ifStatement

letterid

identifierhat

thenelse

digit a parsera parsera parsera parsera parser

nilnil

nilnilnil

Mutual Recursion

statement

ifStatement

letterid

identifierhat

thenelse

digit a parsera parsera parsera parsera parsera parsera parser

nilnilnil

Mutual Recursion

statement

ifStatement

letterid

identifierhat

thenelse

a parser

nilnilnil

Mutual Recursion

statement

ifStatement

letterid

identifierhat

thenelse

a parser

a parsernilnil

Mutual Recursion

statement

ifStatement

letterid

identifierhat

thenelse

a parser

a parsera parser

Mutual Recursion

statement

ifStatement

letterid

identifierhat

thenelse

a parser

a parsera parserboom!

Addressed with laziness in Haskell

Can be addressed with closures

ifStatement = if, [expression], [then], [statement],

[else], [statement].

statement = ifStatement | [returnStatement].

Mutual Recursion

Addressed with laziness in Haskell

Can be addressed with closures

ifStatement = if, [expression], [then], [statement],

[else], [statement].

statement = ifStatement | [returnStatement].

Mutual Recursion

Use Reflection instead of Laziness.

Framework initializes all slots to “stand-in” parsers which act as forward references

Being Reflective rather than Lazy

Mutual Recursion

statementifStatement

letterid

identifierhat

ifthenelse

digit fwd reffwd reffwd reffwd reffwd reffwd reffwd ref

fwd reffwd reffwd reffwd reffwd ref

Mutual Recursion

letterid

identifierhat

ifthenelse

digit a parserfwd reffwd reffwd reffwd reffwd reffwd ref

Mutual Recursion

letterid

identifierhat

ifthenelse

digit a parsera parserfwd reffwd reffwd reffwd reffwd ref

Mutual Recursion

letterid

identifierhat

ifthenelse

digit a parsera parsera parserfwd reffwd reffwd reffwd ref

Mutual Recursion

letterid

identifierhat

ifthenelse

digit a parsera parsera parsera parserfwd reffwd reffwd ref

Mutual Recursion

letterid

identifierhat

ifthenelse

digit a parsera parsera parsera parsera parserfwd reffwd ref

Mutual Recursion

letterid

identifierhat

ifthenelse

Mutual Recursion

letterid

identifierhat

ifthenelse

a parserfwd reffwd reffwd reffwd ref

Mutual Recursion

letterid

identifierhat

ifthenelse

a parsera parserfwd reffwd reffwd ref

Mutual Recursion

letterid

identifierhat

ifthenelse

a parsera parsera parserfwd reffwd ref

Mutual Recursion

letterid

identifierhat

ifthenelse

a parsera parsera parsera parserfwd ref

Mutual Recursion

letterid

identifierhat

ifthenelse

a parsera parsera parsera parsera parser

if then else

FwdRef

Being Reflective rather than Lazy

Use Reflection instead of Laziness.

Framework initializes all slots to “stand-in” parsers which act as forward references

When grammar is completely initialized, stand-ins are connected to originals and forward calls to them

if then else

FwdRef

The Mirror is Mightier Than the Sloth

Grammar1

Grammar2 Parser1

Parser2

Grammar1

Grammar2 Parser1

Parser2

Grammar1

Grammar2

Parser1

Parser2

class ExampleParser2 = ExampleParser1 mixin | >

ExampleGrammar2 ()

ifStatement = (

ˆsuper ifStatement

wrapper:[:ifKw :e :thenKw :s1 :elseKw :s2 | …

Composing Parsers

Shareable among several IDEs/compilers that require distinct ASTs

Shareable among other language tools: syntax colorizers, postprocessing tools

Grammar is shared executable

specification

Three DSLs

Shell Scripting

Parsing

Hopscotch

A GUI application framework

Higher level of abstraction than widgets

Provides DSL for composing presentations out of pieces known as fragments

Fragments

Fragments of presentation

Atomic widgets correspond to leaf fragments

Aggregates represented via compound fragments

Composed via combinators

Fragment Combinators

blank, filler

button:action:, link:action:, image:, menuWithLabelsAndActions:

row:, column:, centered:, collapsed:expanded:

deferred:, draggable:subject:

Dispatching up the fragment hierarchy

Dispatching up the visual containment hierarchy makes sense

Old idea (at least since NewtonScript)

Realized using doesNotUnderstand:

Connections

Self & Smalltalk

ConnectionsParser Combinators.

Too much to survey; some highlights:

Parsec (Haskell parser combinator library)

Swierstra et al.

Sparsec (Scala parser combinator library)

PEGs (Ford, POPL04, ICFP)

Connections

Parser Combinators (continued)

Tedir (Plesner-Hansen’s masters thesis)

Ometa (OOPLA 07 DLS)

Conclusions

Newspeak supports DSLs via

Unconventional syntax

keywords

operators as methods

implicit receivers

Dynamic typing and Reflection

doesNotUnderstand:

Conclusions

Newspeak also supports DSLs via

Mixins, Modules, Hierarchy inheritance

Combinators are a powerful technique

Compositional

Useful in libraries and languages

Status Available at http://newpeaklanguage.org

Open source under Apache 2.0 license

Work in Progress

Expect some tweaks to syntax and semantics

Implementation still not complete - especially libraries

Credits

Peter Ahe

Vassili Bykov

Yaron Kashai

Bill Maddox

Eliot Miranda

This file is licensed under the Creative Commons Attribution ShareAlike 3.0 License. In short: you are free to share and make derivative works of the file under the conditions that you appropriately attribute it, and that you distribute it only under a license identical to this one. Official

license.

The Newspeak eye used in the bullets, slide background etc. was designed by Victoria Bracha and is used by permission.

gilad bracha ministry of truth embedding dsls in newspeak: newshell, ebnf and hopscotch

Documents

reverse-engineering ebnf grammars into mof metamodels...

ebnf: a notation for describing syntax n languages and...

ific newspeak

ebnf+ ebnf+ a hassle-free method for syntax - semantic...

ebnf: a notation for describing syntax

7. syntax: grammatiken, ebnf...kontextfreien grammatiken...

newspeak volume 15, issue 15, september 9, 1987 - … ·...

newspeak volume 10, issue 07, april 6, 1982

shana tova from bracha

colourline painting by bracha l. ettinger: the transject

newspeak and new media: contemporary orwell in interactive

a history of newspeak

newspeak volume 12, issue 08, april 10, 1984

newspeak volume 13, issue 01, january 29, 1985

newspeak programming language draft speci cation version...

newspeak volume 20, issue 1, january 21, 1992 · worcester...

newspeak volume 19, issue 7, february 26, 1991 · worcester...

newspeak dictionary

bracha right to be heard in rule making

actor-based concurrency in newspeak 4