domain specific language design

DSL DesignA conceptual frameworkfor building good DSLs

Markus Voelterindependent/itemis

voelter@acm.org

www.voelter.devoelterblog.blogspot.de

@markusvoelter+Markus Voelter

E.Visser@tudelft.nlhttp://eelcovisser.org/

based on materialfrom a paper written with Eelco Visser

Introduction

A DSL is a focussed, processable language for

describing a specific concern when building a system in a

specific domain. The abstractions and notations

used are natural/suitable for the stakeholders who specify that

particular concern.

more in GPLs more in DSLDomain Size large and complex smaller and well-defined

Designed by guru or committee a few engineers and domain experts

Language Size large small

Turing-completeness

almost always often not

User Community large, anonymous and widespread

small, accessible and local

In-language abstraction

sophisticated limited

Lifespan years to decades months to years (driven by context)

Evolution slow, often standardized fast-paced

Incompatible Changes

almost impossible feasible

LEGO Robot Control

Components

State Machines

Sensor Access

General Purpose

Domain Specific

Modular Language

composable moduleswith many optional,

Case Studies

Components

Example

Components

Example

RefrigeratorsRefrigerators

Example

Refrigerators

Example

Refrigerators

Example

Extended C

Example

Pension Plans

Example

Pension Plans

Example

WebDSL

Example

Terms &

Concepts

A schematic description of a system, theory, or phenomenon that accounts for its known or inferred properties and may be

used for further study of its characteristics

www.answers.com/topic/model

A representation of a set of components of a process, system,

or subject area, generally developed for understanding, analysis, improvement, and/or

replacement of the process

www.ichnet.org/glossary.htm

an abstraction or simplification of reality

ecosurvey.gmu.edu/glossary.htm

which ones?

what should we leave out?

… code generation… analysis and checking… platform independence… stakeholder integration… drives design of

language!

Model Purpose

… code generation… analysis and checking… platform independence… stakeholder integration

Components

Example

Refrigerators

Model Purpose

Refrigerators

Example

Extended C

Model Purpose

Extended C

Example

Model Purpose

PensionPlans

Example

Domain

body of knowledge

in the real worlddeductivetop down

existing software (family)

inductivebottom up

Domain

body of knowledge

in the real worlddeductivetop down

RefrigeratorsRefrigerators

Example

PenionPlans

Example

Domain

Componetns

Example

existing software (family)

inductivebottom up

Domain

Extended C

Example

A DSL LD for D is a language that is

specialized to en-coding PD programs.

more efficientsmaller

Programs are trees

element

Fragments are subtrees w/ root

fragment

fragment rootf

Parent-ChildRelation

Programs and Fragments

Programs are graphs, really.

reference

Programs are graphs, really.

Languages are sets of concepts

Programs and languages

Language: concept inheritance

Independence

Languagedoes not depend on any other language

Fragmentdoes not depend on any other fragment

Independence

Refrigerators

Example

HomogeneousFragmenteverything expressed with one language

HeterogeneousC

Statemachines

Testing

Heterogeneous

Extended C

Example

Domain Hierarchy

all programs

embeddedsoftware

automotiveavionics

Extended C

Example

Design Dimensions

expressivitycoveragesemanticsseparation of concerns

completenessparadigmsmodularityconcrete syntax

process

Expressivity

process

Shorter Programs

More Accessible Semantics

For a limitedDomain!

Domain Knowledge

encapsulated in language

Smaller Domain

More Specialized Language

Shorter Programs

The do-what-I-want

language

Single Program vs. Class/Domain

No Variability!

Domain Hierarchy

more specialized domainsmore specialized languages

Reification

==Language Definition

Transformation/Generation

?Overspecification!Requires Semantic Analysis!

!Declarative!Directly represents Semantics.

Linguistic Abstraction

Def: DSLA DSL is a language at D that provides linguistic abstractions for common patterns and idioms of a language at D-1 when used within the domain D.

Def: DSL cont’dA good DSL does not require the use of patterns and idioms to express semantically interesting concepts in D.Processing tools do not have to do “semantic recovery” on D programs.

Declarative!

Another Example

Extended C

Example

Another Example

Turing Complete!Requires Semantic Analysis!

Extended C

Example

Extended C

Example

Extended C

Example

In-LanguageAbstractionLibrariesClassesFrameworks

In-LanguageAbstractionUser-DefinableSimpler Language

AnalyzableBetter IDE Support

In-LanguageAbstractionUser-DefinableSimpler Language

AnalyzableBetter IDE Support

Special Treatment!

In-LanguageAbstraction

Std Lib

Refrigerators

Example

Coverage

process

Domain DL defined inductively by L

CL(L) == 1 (by definition)

not very interesting!

(the domain that can be expressed by L)

Def: Coverage

to what extend can a language L cover a domain D

Def: Coverage

why would CD(L) be != 1?

1) L is deficient

2) L is intended to cover only a subset of D,

corner cases may make Ltoo complex

Rest must be expressed in D-1

Def: Coverage

Coverage is full.You call always write C.

Extended C

Example

Def: Coverage

Only a particular style of web apps are suported.

Many more are conceivable.

WebDSL

Example

Def: Coverage

DSLs are continuously evolved so the relevant parts of the deductive domain aresupported.

PensionPlans

Example

Refrigerators

Example

Components

Example

Semantics &Execution

process

Static Semantics

Execution Semantics

Static Semantics

Execution Semantics

Static Semantics

ConstraintsType Systems

Unique State Names

Unreachable States

Dead End States

Extended C

Example

Unique State Names

Unreachable States

Dead End States

Easier to do on a declarative Level!

Extended C

Example

Unique State Names

Unreachable States

Dead End States

Easier to do on a declarative Level!

Thinking of all constraints is a coverage problem! Exten

Example

Assign fixed types

Derive TypesCalculate Common Types

Check Type Consistency

What does a type system do?

Intent +Check

Derive

More code

Better error messages

Better Performance

More convenient

More complex checkers

Harder to understand for users

Refrigerators

Example

Execution Semantics

What does it all mean?

Def: Semantics… via mapping to lower level

OB: Observable Behaviour (Test Cases)

TransformationInterpretation

Transformation

Extended C

Example

Transformation

Known Semantics!

Transformation

TransformationCorrect!?

Transformation

Tests (D)

Tests (D-1)

Run tests on both levels; all pass.Coverage Problem!

Refrigerators

Example

Transformation

PensionPlans

Example

Behavior

Refrigerators

Example

Multi-Stage

Modularization

Multi-Stage: Reuse

Reusing Later Stages

Optimizations!

Multi-Stage: Reuse

Extended C

Example

C Text

C (MPS tree)

State Machine

Components

Robot Control

Multi-Stage: Reuse

Extended C

Example

C Text

C (MPS tree)

State Machine

Components

Robot Control

SyntacticCorrectness, Headers

C Type System

ConsistencyModel Checking

Efficient Mappings

Multi-Stage: Reuse

Reusing Early Stages

Portability

Multi-Stage: Reuse

L0b Extended C

Example

JavaC#

Multi-Stage: Preprocess

Adding an optional, modular

emergencystop feature

Platform

PensionPlans

Example

Temporal DataVersioning

Database AccessTransactionsDistribution(JEE)

Platform

Refrigerators

Example

Data Collection

DeviceDrivers

A program at D0 thatacts on the structureof an input program at D>0

Interpretation

imperative step throughfunctional eval recursivelydeclarative ? solver ?

Interpretation

PensionPlans

Example

Refrigerators

Example

Behavior

Refrigerators

Example

Interpretation

An interpreter :-)

Interpretation

Transformation Interpretation

+ Code Inspection

+ Debugging

+ Performance & Optimization

+ Platform Con- formance

+ Turnaround Time

+ Runtime Change

TransformationInterpretation

Multiple Mappings… at the same time

Lx Ly Lz

Similar Semantics?

Multiple Mappings… at the same time

Lx Ly Lz

Similar Semantics?

T T Tall green!

PensionPlans

Example

Refrigerators

Example

Multiple Mappings… alternatively, selectably

Lx Ly Lz

Extend LD to include explicit data that determines transformation

Lx Ly Lz

Extend LD to include explicit data that determines transformation

Extended C

Example

Restricted Port leads to reduced overhead C

Lx Ly Lz

External Data: - Switches- Annotation Model

Lx Ly Lz

External Data: - Switches- Annotation Model

PensionPlans

Example

Switch Control Java vs. C

Lx Ly Lz

Heuristics: Analyze model to try to decide

Lx Ly Lz

TESTING!

Reduced Expressiveness

bad? maybe.

good? maybe!

Model CheckingSAT Solving

Exhaustive Search, Proof!

Unique State Names

Unreachable States

Dead End States

Guard Decidability

Exhaustive Search, Proof!

Reachability

Extended C

Example

Extended C

Example

Separation of

Concernsexpressivitycoveragesemanticsseparation of concerns

process

Several Concerns… in one domain

integrated intoone fragment

separated intoseveral fragments

Viewpoints

independent

dependent

Viewpoints

Refrigerators

Example

Hardware

Behavior

Viewpoints: Why?

Sufficiency

Different Stakeholders

Different Steps in Process – VCS unit!

Viewpoints

Refrigerators

Example

Hardware

Behaviour

Product Management

Thermo-dynamics-Experts

Viewpoints

sufficient?

contains allthe data for running a meaningfultransformation

independent

Viewpoints

sufficientHardware structuredocumentation

Refrigerators

Example

sufficientimplementationcode

Viewpoints: Why?

1:n Relationships

Viewpoints

Refrigerators

Example

Hardware

Behaviour

Viewpoints

Well-defined Dependencies

No Cycles!

Avoid Synchronization!(unless you use a projectional editor)

Viewpoints: Why?

LAnnotation

Progressive Refinement

Views on Programs

Pension Plans

Example

Completeness

process

Can you generate 100% of the code from the DSL program?

More generally: all of D-1

Semantics:

Introduce G (“generator”):complete

incomplete

Incomplete: What to do?

FD-1OB(FD) !=

Incomplete: What to do?

+ FD-1,

OB(FD) ==

Manually written code!

Manually written code?

Call “black box” code (foreign functions)

Embed LD-1 code in LD program

Refrigerators

Example

Embed C statementsin components, state machinesor decision tables

Use composition mechanisms of LD-1 (inheritance, patterns, aspects, …)

Components

ExampleGenerate base classeswith abstract methods;implement in subclass

Use protected regions (if you really have to…)

Use protected regions (if you really have to…) DON’T!

Incomplete: When?Good for technical DSLs: Devs write LD-1 code

Bad for business DSLs. Maybe use a LD-1 std lib that LD code can call into?

Components

Example

Refrigerators

Example

PensionPlans

Example

Extended C

Example

WebDSL

Example

Prevent Breaking Promises!

class B extens BBase {

public void doSomething() { Registry.get(„A“).someMethod(); }

Prevent Breaking Promises!

Dependency InjectionStatic analysis tools

Better:

Components

Example

Roundtripping

L’D-

………

Roundtripping – Don’t!

L’D-

………

Semantic Recovery!

FundamentalParadigms

process

Structure

Modularization, VisibilityNamespaces,public/privateimporting

Structure

Modularization, Visibility

divide & conquerreusestakeholder integration

Namespaces,public/privateimporting

Structure

Partitioning (Files)

VCS UnitUnit of sharingUnit of IP

!= logical modulesmay influence language design

Structure

Extended C

Example

Components

Example

Structure

Partitioning (Files)

change impactlink storagemodel organizationtool integration

Structure

Spec vs. Implementationplug in different Implsdifferent stakeholders

Structure

Spec vs. Impl.

Extended C

Example

Structure

SpecializationLiskov substitution Pleaving holes (“abstract”)

Structure

SpecializationLiskov substitution Pleaving holes (“abstract”)

variants (in space)evolution (over time)

Structure

Specialization

Pension Plans can inheritfrom other plans.

Rules can be abstract;

Plans with abstract rules are abstact

PensionPlans

Example

Structure

Superposition, Aspects

modularize cross-cuts

mergingoverlayAOP

Structure

Superposition, Aspects

Components

Example

Behavior

Not all DSLs specify behavior

Some just declare behavior

This section is not for those!

Behavior

Imperativesequence of statementschanges program state

write understand

analyze performance

simple simple - simple (step)

hard good

Behavior

Imperativesequence of statementschanges program state

Refrigerators

Example

Behavior

Functionalfunctions call other functions.no state. No aliasing.

write understand

analyze performance

simple - simple simple (tree)

good good -

Behavior

Functionalfunctions call other functions.no state. No aliasing.

PensionPlans

Example

Behavior

Functional

PensionPlans

Example

Behavior

Functionalpure expressions are a subset of functional(operators hard-wired)

guardspreconditionsderived attributes

Behavior

Declarativeonly facts and goals. no control flow.eval engine, solver (several)

write understand

analyze performance

simple simple - hard depends often bad

Behavior

Declarativeconcurrencyconstraint programmingsolvinglogic programming

Behavior

Declarativeonly facts and goals. no control flow.eval engine, solver (several)

WebDSL

Example

Behavior

Declarative

Extended C

Example

Behavior

Reactivereactions to events, more events are produced. Communication viaevents and channels/queues

write understand debug

analyze performance

simple simple/hard hard simple can be good

Behavior Reactive

Refrigerators

Example

Behavior

Data Flow

chained blocks consume continuous data that flows from block to block

analyze performance

simple - simple/hard hard simple can be good

Behavior

Data Flow

continuous, calc on changequantized, calc on new datatime triggered, calc every x

Behavior

Data Flow

Embedded ProgrammingEnterprise ETL & CEP

Behavior

State Based

states, transitions, guards, reactions

analyze performance

simple - simple/hard s/h simple +

can be good

event driven, timed

Behavior

State Based

Refrigerators

Example

Behavior

Combinationsdata flow uses functional, imperative or declarative lang inside block

Behavior

Combinationsstate machines use expressions in guards and often an imperative lang in actions

Behavior

Refrigerators

Example

Combinations

Behavior

Combinations

purely structural languages often use expressions to specify constraints

Extended C

Example

LanguageModularity

process

BehaviorLanguage Modularity, Composition and Reuse (LMR&C)

increase efficiency of DSL development

ReferencingReuseExtensionReuse

4 ways of composition:

distinguished regardingdependencies and fragmentstructure

4 ways of composition:

BehaviorDependencies:

do we have to know about the reuse when designing the languages?

BehaviorDependencies:

do we have to know about the reuse when designing the languages?

homogeneous vs. heterogeneous(„mixing languages“)

Fragment Structure:

BehaviorDependencies &Fragment Structure:

ReferencingReferencing

Referencing

Dependent

No containment

Referencing

Used in Viewpoints

Referencing

Fragment

Referencing

references

Refrigerators

Example

Referencing

Containment

Dependent

Extension

more specialized domainsmore specialized languages

ExtensionExtension

Extension

Good for bottom-up (inductive) domains, and for use by technical DSLs (people)

Extension

BehaviorDrawbacks

tightly bound to basepotentially hard to analyze the combined program

Extension

Extended C

Example

Extension

ExtensionExtension

Extended C

Example

No containment

Independent

BehaviorOften the referenced language is built expecting it will be reused.

Hooks may be added.

Containment

Independent

Embedding

Pension Plans

Example

Embedding

BehaviorEmbedding often uses Extension to extend the embedded language to adapt it to its new context.

Embedding

BehaviorExtension and Embedding requires modular concrete syntax

Challenges - Syntax

Many tools/formalisms cannot do that

BehaviorExtension: the type system of the base language must be designed to be extensible/ overridable

Challenges – Type Systems

BehaviorReuse and Embedding: Rules that affect the interplay can reside in the adapter language.

Challenges – Type Systems

Behavior

Referencing (I)

Challenges – Trafo & Gen

Two separate, dependent single-source transformations

Can be Reused

Written specifically for the combination

Referencing (II)

A single multi-sourcedtransformation

Referencing (III)

A preprocessing trafo that changes the referenced frag in a way specified by the

referencing frag

Extension

Transformation by assimiliation, i.e.generating code in the host lang

from code expr in the extension lang.

Extended C

Example

Extension

Reuse (I)

Reuse of existingtransformations for both fragments plus

generation of adapter code

Reuse (II)

composing transformations

Reuse (III)

generating separate artifacts plus a weaving specification

Embedding (I)

Assimilation (as with Extension)

a purely embeddable language may not come with a generator.

Embedding (II)

Adapter language can coordinate the transformations for the host and for the

emebedded languages.

ConcreteSyntax

process

UI for the language!Important for acceptance by users!

TextualSymbolicTabularGraphical

Reuse existing syntax of domain, if any!

Tools let you freely combine all kinds.

Default: Text

Editors simple to buildProductiveEasy to integrate w/ toolsEasy to evolve programs

Default: Text

Editors simple to buildProductiveEasy to integrate w/ toolsEasy to evolve programs

… then add other forms, if really necessary

Graphical in case…

Relationships

Flow and Depenency

Causality and Timing

Symbolic

Either Mathematical, or often highly

inspired by domain

Tables

Combinations

Process

process

Domain Analysis

Interview Experts

Structure their

Knowledge

Build the Language

Create Examples

Get Feedback

Iterate to goal

Documentation

Create example-based tutorials!

Domain Folks Programming?

Precision vs.Algorithmics!

Domain Folks Programming?

DU Codin

DU/DevPaired

Dev CodingDU Reviewing

DSL as a Product

Release PlanBug TrackerTesting!SupportDocumentation

Reviews

Reviews become easier --- less code, more domain-specific

The End.This material is part of my

upcoming (early 2013) book

www.voelter.de/dslbook

DSL Engineering with Language Workbenches

Stay in touch; it will be cheap or maybe even free :-)

@markusvoelter+Markus Voelter

domain specific language design

semantic analysis

programsextended c

exampleexampleextended

example web dsl

dsl contda good dsl

domain hierarchymore

dsla dsl

specific domain

Technology

unrealscript: a domain-specific language

an embedded domain specific language for general purpose

dsl (domain specific language) for maps mashups

domain-specific modeling and verification language edola

nhom2 domain specific language development

a domain specific language for configurable traceability...

a domain specific language for extracting models in software

a multi-layered domain-specific language for … for stencil...

feature oriented domain specific language for dependency...

domain specific language implementation via compile-time...

domain specific language embedding with c++ template

a domain-specific programming language for particle

write your test cases in a domain-specific language

domain-specific metamodeling, and language families ›...

mps-based domain-specific language for defining rtsj systems

domain specific languages in...

pypy's approach to construct domain-specific language...

a quantum chemistry domain-specific language...

domain specific language

language workbenches - the killer-app for domain specific...