markus völter voelter@acm voelter.de

i n g e n i e u r b ü r o f ü r s o f t w a r e t e c h n o l o g i e w w w . v o e l t e r . d e © 2 0 0 5 M a rk u s V ö l t e r. - 1 -

Using DSLs in Practice - Benefits and Challenges

Markus Vö[email protected]

Using DSLs in Practice

Benefits and Challengeswww.mdsd-buch.de



Markus Völter

[email protected]

www.voelter.de

• Independent Consultant

• Based out of Heidenheim, Germany

• Focus on

• Software Architecture

• Middleware

• Model-Driven SoftwareDevelopment

About me



C O N T E N T S

• Introduction

• Benefits

• Challenges and Experiences

• Process

• Technical

• Adoption

• More or less Open

• Summary



C O N T E N T S

• Introduction

• Benefits


• Process

• Technical

• Adoption


• Summary



Domain Driven Development

• Domain Driven Development is about making software development more domain-related as opposed to computing related. It is also about making software development in a certain domain more efficient.

Software TechnologyConcepts

Domain Concepts

Software TechnologyConcepts

Domain Concepts

mental workof developers



A DSL and its context

Model

DomainSpecific

Language

Metamodeltextual

graphical

Domain

Ontology

bounded area ofknowlege/interest

semantics

precise/executable

multiple

partial

viewpoint

subdomains

composable

Metametamodeltarget

softwarearchitecture

softwarearchitecture

transform

compile

interpret

multi-step

single-step

noroundtrip

knowledge

several

designexpertise



How DSLs are usually used

• Developer develops model(s)based on certain metamodel(s), expressed using a DSL.

• Using code generation templates, the model is transformed to executable code.

• Alternative: Interpretation

• Optionally, the generated code is merged with manually written code.

• One or more model-to-model transformation steps may precede code generation.

ModelModel

Model

TransformerTranformation

Rules

Model

TransformerCode

GenerationTemplates

Generated CodeManuallyWrittenCode

optional

Metamodel

Metamodel

op

tio

nal

, ca

n b

e re

pea

ted



DSLs in Industry

• Model-Driven Development (MDSD)pragmatic technology, process building blocks

• OMG’s Model-Driven Architecture (MDA)standardization effort, technology-focus, platform independence, m2m transformations

• Microsoft’s Software Factories (SF)framework for domain-specific IDE tooling, DSLs are part of this approach

• Generative Programming (GP)traditional small scale, technology focused

• Language-Oriented Programming (LOP)integrate DSLs into IDE with editors, debuggers, symbolic integration



Reasons for using DSLs

• You want to provide a way for your domain-experts to formally specify their knowledge, and to provide a way for your technology people to define how this is implemented (using model transformations).

• You might want to provide different implementations (i.e. more concrete models) for the same model, perhaps because you want to run it on different platforms (.NET, Java, CORBA).

• You may want to capture knowledge about the domain, the technology, and their mapping in a clear, uncluttered format.

• In general, you don’t want to bother with implementation details when specifying your functionality.



C O N T E N T S

• Introduction

• Benefits


• Process

• Technical

• Adoption


• Summary



MDSD Benefits I

• Models are free of implementation artifacts – they directly represent domain knowledge and are thus reusable.

• Implementations for various platforms can be generated in principle – the technology change problem is adressed to some extend.

• Technology freaks and domain experts can take care of „their business“ (transformations and models, respectively) and need to care of each other‘s problems only in a limited way.

• Domain experts can play a direct role in development since they can more easily understand models expressed with a DSL as opposed to implementation code.

Domain Experts play the central role they deserve!



MDSD Benefits II

• Development becomes more efficient since repetitive implementation code can be generated automatically.

• Architectural contraints/rules/patterns can more easily be enforced, since the they are embedded in the templates rather than just being documented (and ignored).

This is especially important in really large teams, often in the context of Product-Line Engineering and Software System Families.

• Transformer/Generator can address cross-cutting concerns (just like an aspect weaver)



Benefits for Software Quality

• DSLs, if they describe software structure/architecture requires an explicit, well-defined architecture. Defining an architecture this way improves the quality of the system (indpendent of whether it is generated or not).

• Transformations capture expert knowledge. The generated code reflects this expert knowledge uniformly.

• An DSL-based Archtitecture defines a strict programming model for the manually developed parts – again, uniformity and constrained-ness improves quality.

• Generator does not produce accidental errors – either things are always right or always wrong. This makes finding errors easier!



Benefits for Software Quality II

• In general, defining a DSL forces you to take care of many good things, which you‘d like to have in any application development project:

• Domain/Application Scoping

• Variability Management

• Well-Defined Software Architecture, Architecture Metamodelling

• Trying to build a DSL/generator for a domain/target architecture enables your understanding of the domain/target architecture. This in itself is a huge benefit.



DSL/Code Generation “Financial Benefits”

Level of Detail

Result ofAnalysis

virtual or realImplementation model Implementation

Info

rmat

ion

Gai

n

Start

GoalImplementation

Analysis

Design

Effort

• „Normal“ Implementation Effort



DSL/Code Generation “Financial Benefits” II

Level of Detail

Results ofAnalysis

virtual or realimplementation model Implementation

Info

rmat

ion

Gai

n

Start

GoalImplemen-

tation

Analy

sis

Effort

Mod

ellin

g

Automation

Savings based onthe use of a semantically

rich platform

Savingsbecause

of Generation

A

A' B

• Realistic DSL based Implementation Effort



DSL/Code Generation “Financial Benefits” III

Level of Detail

Results ofAnalysis

virtual or realimplementation model Implementation

Info

rmat

ion

G

ain

Start

Goal

Ana

lysi

s

Effort

Mod

ellin

g

Automation

Savings based onthe use of a semantically

rich platform

Savingsbecause

of generation

A

BA'

• Ideal DSL based Implementation Effort



C O N T E N T S

• Introduction

• Benefits


• Process

• Technical

• Adoption


• Summary



DSLs/Generation and Agility

• Agile Manifesto:We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: - Individuals and interactions over processes and tools - Working software over comprehensive documentation - Customer collaboration over contract negotiation - Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more.

• DSL-based Models are no „paperwork“, they are the code which is translated to executable code automatically

• Agility does not oppose tools in general – compilers are ok, model transformers are a kind of compiler



MDSD and Agility II

• Project automation (ant, cruisecontrol) is ok in „agile minds“, so is automation of the writing of repetitive code

• Automation of the development process makes responding to change easier and faster (single source principle).

• Changes in the model respond to changes in the functional requirements

• Changes in the templates/transformations can be used to evolve the architecture

• The customer on-site can be integrated better, if we have languages that are better related to domain concepts as opposed to 3GL code or the like.

• Pair programming between developer and domain expert is more realistic.



MDSD and Agility III

• Tests can still be written manually (even before generation), generators can help is building mocks or scenarios

• We have done Test-Driven, Model-Driven Development

• We do not recommend a waterfall that first builds metamodel/DSL/generators and then builds apps, rather, both are iteratively evolved in parallel.

• Domains Architectures are based on experience, not based on „big design upfront“

• Developers can do what they can do best:

• Some deal with applications and customer requirements,

• Others deal with technical architecture, platforms and generators

• So: There is no conflict between Agility and DSLs/Generation!



Teaming issues

• Using DSLs is not very different from “normal” programming – every developer can basically do it.

• Defining DSLs is, however, something completely different:

• Finding the „right“ abstractions, defining metamodels, keeping the various metalevels sorted, etc. is not everybody‘s business.

• Some of the tools to define metamodels, DSLs, generators and model-2-model transformations are not (yet) intuitively usable.

• Therefore I recommend to keep DSL/generator development to a limited group of people in your project.



Teaming issues II - Roles

• Not all of these roles are necessary in every project, of course

• However, as an example, there is a fundamental difference between those who understand the domain and its abstractions (left branch) compared to those who know how to best use some platform technology (right branch)

Customer

Domain Analyst

Domain Expert

Domain Architect

Language Designer

ReferenceImplementor

Prototype Developer

PlatformDeveloper

TransformationDeveloper



Iterative Dual Track Development

• Develop DSL/Generator and at least one application at the same time.

• Establish rapid feedback from application developers to domain architecture developers.

• Develop both aspects iteratively and incrementally. Use strict timeboxing.

• Infrastructure develops iteration n+1 whereas application developers use iteration n.

• Introduce new DSL/Generator releases only at the beginning of iterations.

Integrationand

Feedback

ApplicationDevelopment

(n)

InfrastructureDevelopment

(n+1)

feedback



Iterative Dual Track Development II - Roles

• Note that the „men“ inthe diagram on the right are roles, youcan easily have someof them be handledby the same person!

Application Engineering

ApplicationDeveloper

ApplicationArchitect

ProjectManager

Customer

RequirementsAnalyst

Tester

Domain Engineering

ProductManager

DomainAnalyst

DomainArchitect

Test Engineer

Domain Expert



Extract the Infrastructure

• Before starting ITERATIVE DUAL-TRACK DEVELOPMENT, Extract the transformations from manually developed application.

• Either, start by developing this prototype conventionally, then build up the DSL/Generator infrastructure based on this running application,

• Or extract the code from applications developed in the respective domain before doing MDSD (but only if the quality is sufficiently good!)

ManuallyDevelopedPrototype

Transformations

DSL(s)

Metamodel

Application Model

Platform(s)Application

Development

InfrastructureDevelopment



How do I come up with a good metamodel?

• Incrementally!

• Based on experience from previous projects, and by „mining“ domain experts.

• A very good idea is to start with a (typically) very well known domain: the target software architecture (platform) Architecture-Centric DSLs see below, Cascading



How do I come up with a good metamodel? II

• In order to continuously improve and validate the metamodel for a domain, it has to be exercised with domain experts as well as by the development team.

• In order to achieve this, it is a good idea to use it during discussions with stakeholders by formulating sentences using the concepts in the meta model.

• As soon as you find that you cannot express something using sentences based on the meta model,

• you have to reformulate the sentence

• the sentence’s statement is just wrong

• you have to update the meta model.

• (Based on Eric Evans’ Ubiquitous Language)



How do I come up with a good metamodel? III

• A component owns any number of ports.• Each port implements exactly one interface.• There are two kinds of ports: required ports and provided

ports. • A provided port provides the operations defined by its

interface.• A required port provides access to operations defined by

its interface.

Component

Portowns *

Interfaceimplements 1

Required Port Provided Port

providesoperationsdefined by

provides access to operations defined by

• Example:



C O N T E N T S

• Introduction

• Benefits


• Process

• Technical

• Adoption


• Summary



One DSL is not enough

GUI

Persistence

Processes

Acc

ou

nti

ng

Hu

man

R

eso

urc

s

CR

M

Partitions

Su

bd

om

ain

s

SYSTEM

• Most systems can be structured into various

• partitions: functional subsystems

• subdomains: technical aspects

• It is hardly possible to describe each of these with the same DSL.

• You will need to come up withseparate DSLs

• … that have to be „connectable“in order to build the complete system



One DSL is not enough II

• Structure your system into several technical subdomains such as persistence, GUI, deployment.

• Each subdomain should have its own meta model and specifically, its own suitable DSL.

• Define a small number of gateway metaclasses, i.e. meta model elements that occur in several meta models to help you join the different aspects together.

Technical Subdomain 1(e.g. Business logic)

Metamodel1

DSL 1

Technical Subdomain 2(e.g. Persistence)

Metamodel2

DSL 2

Technical Subdomain 3(e.g. GUI)

Metamodel3

DSL 3



One DSL is not enough III - Example

Form Layout

Entity Attribute*

KeyAttribute

TableColumnType

type

*

Mapping

*

Business Objects

Page Form

FormField

*

*

maps

Button

target

Pages, Forms and Workflow

representsrepresents

FormLayout

TabularLayout

SimpleLayout

[...]

[...]

Persistence



Rich Domain-Specific Platform

• Define a rich domain-specific application platform consisting of

• Libraries

• Frameworks

• base classes

• interpreters, etc.

• The transformations will “generate code” for this domain-specific application platform.

• As a consequence, the trans-formations become simpler.

• DSLs and Frameworks are two sides of the same coin

DomainPlatform

TechnicalPlatform/Middleware

Operating System

Programming Language

- Persistence- Transactions- Distribution- Scheduling- Hardware Access- ...

- Core Domain Classes (Entities, Value Types, ...)- Business Rules- Business Services- ...

Generated Applications



Rich Domain-Specific Platform II - Integration

• A) Generated code can call non-generated code contained in libraries

• B) A non-generated framework can call generated parts.

• C) Factories can be used to „plug-in“ the generated building blocks

• D) Generated classes can also subclass non-generated classes.

• E) The base class can also contain abstract methods that it calls, they are implemented by the generated subclasses(template method pattern)

a)

b)

c) d) e)

generated code non-generated code



Transformations as first class citizens

• Transformations (and Templates) are central assets in MDSD. You should treat them accordingly.

• Transformations should be versioned.

• Refactor transformations to keep them current and well organized.

• Modularize transformations, e.g. using object-oriented concepts such as encapsulation, polymorphism, inheritance, etc.



External Model Markings (AO-Modelling)

• In order to allow the transformation of a source model into a target model (or to generate code) it is sometimes necessary to provide “support” information that is specific to the target meta model.

• Example: Entity Bean vs. Type Manager

• Adding these to the source model “pollutes” the source model with concepts specific to the target model.

• MDA proposes to add “model markings”, but this currently supported well by only very few tools.

• Instead, we recommend keeping this information outside of the model (e.g. in an XML file); the transformation engine would use this auxiliary information when executing the transformations.



Testing through Model Verification

• In many cases it is possible to detect design errors already in the models. This step is called model verification.

• The most „extreme“ form is to interpret and simulate the whole model; this is however, not simple to achieve.

• However, it is easily possible to verify design constraints in the model before model transformation or code generation steps are done.

• So, a DSL definition includes checks („constraints“) that determine if a model is correct. •Note that those constraints report errors in a language that is

on the same abstraction level as the DSL – i.e. no low level compiler errors!



Testing through Model Verification II - Example

• Example Metamodel



Testing through Model Verification III – Example cont‘d

• Verifications in the metamodel (Implemented)

public class ECInterface extends generatorframework.meta.uml.Class {

public String CheckConstraints() { Checks.assertEmpty( this, Attribute(),

"must not have attributes." ); } // more …}

public class Component extends generatorframework.meta.Class {

public String CheckConstraints() { Checks.assertEmpty( this, Operation(),

"must not have attributes." ); Checks.assertEmpty( this, Generalization(), "must not have superclasses or subclasses." ); Checks.assertEmpty( this, Realization(), "must not implement any interface." ); Checks.assertUniqueNames( this, Port(), "a component's ports must have unique names." ); } // more …}



Generation vs. Interpretation I

• A model that can be used to generate code, can always also be interpreted.

• The usual tradeoffs apply: Performance, Changeability at runtime

• There are, however, additional things to consider:

• When generating code, an explicit intermediate step (i.e. the generated code) is available. Might make things more easily understandable, since the structure is often well known (although this is not always true

• Debugging is often easier, since you can debug „one metalevel down“ compared to an interpreter

• In many cases, the platform (J2EE, Embedded Osek, etc.) expects certain code structures. It is very hard to run an interpreter in these constraints.



Generation vs. Interpretation II

• From my experience a combination of both approaches makes sense in the following way:

• Code generation is used to generate structural artifacts (components, configuration files, documentation, etc.). Components are artifacts that provide interfaces, require other interfaces, etc.

• Different kinds of components use different metaphors for their implementation, such as• A rule engine component uses business rules to verify

business transactions• A state chart based component manages persistent, long

running processes• A mapping component is used to map data to legacy data

structures (legacy system integration)

• This implementation is often implemented as an interpreter.



Build an IDE

• DSL-based Development often includes a wealth of input, intermediate and output artifacts.

• Users (application developers) can easily be confused by these many artifacts.

• To improve acceptance of the approach, you should consider building an IDE for application developers which

• Hides unnecessary intermediate artifacts

• Provides editors/support for “programming” with the DSL

• Integrates all the validation, generation, build, etc.

• Interactively connects and synchronizes the various models used

• Eclipse, for example, is a good toolkit for building these kinds of IDEs.



Build an IDE: Example



C O N T E N T S

• Introduction

• Benefits


• Process

• Technical

• Adoption


• Summary



Adopting DSL-based Development

Traditional Software Development

Establish a disciplined, iterative requirements management process

Development of a reference

implementation

Extract the infra-structur

e

Define the softwareSupply chain

Mature, IterativeSoftware Development

Classify Software Inventory

Evolve the domain

architecture

DSL-based Application Development

Make sure processes are managed Standardising the infrastructure Architecture-Centric MDSD

Non-Managed Process DSL-based Development

Traditional Domain-EngineeringApplication-EngineeringKey

Development of an application

development platform



Cascading DSLs

• You would typically start with architecture-centric DSLs where the abstractions of the DSL correspond to the core concepts of the technical platform.

• This automates many aspects of the technical development/infrastructure;

• Results in a wide platform/infrastructure

• Many projects can be handled with the infrastructure

• In later phases, functional DSLs infrastructures will be built on this technical one, resulting in cascaded DSLs.



Cascading DSLs II

DSL Infrastructure

Input Models

Output Models

Basic Technical DSL Infrastructure

Code for Target Platform

Input Models

Functional Domain 1DSL Infrastructure

Domain 1 Model

Functional Domain 2DSL Infrastructure

Domain 2 Model

...

...

...

...

...

...



Cascading DSLs III – M2M Transformations

<configurations> <configuration name="addressStuff"> <deployment name="am" type="AddressManager"> <wire name="personDAO" target="personDAO"/> </deployment> <deployment name="personDAO" type="PersonDAO"/> </configuration> <configuration name="customerStuff"> <deployment name="cm" type="CustomerManager"> <wire name="addressStore" target=":addressStuff:am"/> </deployment> </configuration> <configuration name="test" includes="addressStuff, customerStuff"/></configurations>

<<component>>

AddressManager

<<interface>>AddressStore

addOrUpdateContact( p: Person) : voidaddAddress( p: Person, a: Address) : voidgetAddresses( p: Person ) : Address[]

<<entity>>

Person

name: StringfirstName: String

<<valuetype>>

Address

street: Stringzip: StringCity: String

0..n

<<component>>

CustomerManager

address-Store

<systems> <system name="production"> <node name="server" type="spring" configuration="addressStuff"/> <node name="client" type="eclipse" configuration="customerStuff"/> <system> <system name="test"> <node name="test" type="spring" configuration="test"/> <system></systems>

Type Model

Composition Model System Model

person

<<component>>

SomeComponent

<<generate>><<man-code>>

SomeCompo-nent.java

<<interface>>

SomeInterface

<<gen-code>>

Some-Interface.java

<<generate>>

<<gen-code>>

SomeComponentBase.java



Cascading DSLs IV – M2M Transformations II

<<gen-code>>

SomeEntity.java

<<entity>>

SomeEntity

<<generate>>

<<interface>>

SomeEntityDAO<<transform>>

<<generate>> <<gen-code>>

SomeEntity-DAO.java

<<component>>

SomeEntityDAO

<<transform>><<generate>> <<gen-code>>

SomeEntity-DAOBase

.java

<<gen-code>>

SomeEntity-DAO.java

<<generate>>



Levels of MDSD V – M2M Transformations III

openArchitectureWareModel(UML)

Model(XMI)

Parser

Model(Object Graph)

ModelTrans-former

Modified Model(Object Graph)

export

GeneratedCode

CodeGenerator

(may be repeated)

• This approach works well in practice, if

• you don‘t need to modify the intermediate transformation results manually

• You can run Unit Tests in the generator to ensure the model-2-model transformations are (and stay!) correct



Levels of MDSD VI – M2M Transformations IV

<<generate>>

<<gen-code>>

AProcess-Data.java

<<proc-component>>

AProcess

<<gen-code>>

AProcessBase.java

<<gen-code>>

AProcessProcBase.java

<<trigger-interface>>

AProcessInterface

*

1

sm AProcess

<<entity>>

AProcessData

<<

tran

sfor

m>

>

s1

s2

s3

<<generate>> <<generate>>operations...

attributes...

data

1

<<

tran

sfor

m>

>

guard operations... (abstract)action methods... (abstract)

<<man-code>>

AProcess.java



C O N T E N T S

• Introduction

• Benefits


• Process

• Technical

• Adoption


• Summary



More or Less Open Issues

• Composition of DSLs: There is not too much experience yet in compsing DSLs.

• It works more or less, if you build all of them with the same tool (all UML-based, all ASCII-based)

• Advanced IDE support for DSLs is limited, i.e. debuggers, context-aware editors, etc.

• As long as you work „plain ASCII“ or UML based things work reasonably well – but this is often not an alternative

• Integration with Versioning can be an issue, depending on the tool

• Model-2-Model Transformation languages are still not standardised (and maybe never will be, really).



Summary

• DSL-based software development can bring huge benefits to a development organization• Especially, if you work on product lines/software system

families

• To benefit, you have to make sure you adapt your processes and team structure accordingly

• You also need to make sure the „rest“ of your processes work, i.e. customer integration, requirements manangement – DSLs cannot solve all your problems :-)

• It is also important to select the right tools – the loudest marketing offensive does not necessarily lead to the best product.

• Finally, be prepared to „improvise“ wrt. to some aspects of your development process.



C O N T E N T S

• Introduction

• Benefits


• Process

• Technical

• Adoption


• SummaryTHE END.

THANK YOU!



Some advertisement

• For those, who happen to speak(or rather, read) german:

Völter, Stahl:

Modellgetriebene SoftwareentwicklungTechnik, Engineering, Management

dPunkt Verlag, 2005

www.mdsd-buch.de

• For all others:A translation (with Wiley) isunder way.

markus völter voelter@acm voelter.de

Documents