Transforming a 15 year old model-driven application from C++ to Java

Eric Jan Malotaux
Code Generation 2012, Cambridge, UK
2012-03-28

This presentation is tcase study about Transforming a 15 year old model-driven application from C++ to Java

Overview

Let me first give an overview of my presentation.

OVERVIEW

Who I Am

The Assignment

Renovating applications

The Strategy

Lessons Learned

Unsolved Problems

First I will introduce myself.

Then I will describe the assignment that this presentation is about

Since the assignment was essentially a renovation of an existing application, I will describe how renovation of old applications is done in general.

Next I will describe how we applied different strategies in succession as we learned on the job.

Finally I will try to draw some lessons from the experience and mention a number of problems that we did not yet find a good solution for.

WHO AM I

Eric Jan Malotaux

Software Architect with Ordina, ICT service provider in the Netherlands

Trained as a musician and musicologist

27 years experience in software development

What I learnedAutomated procedures (software factory) as important as programming skills

Communication and trust within a team even more important

Specialties anything that speeds up software developmentSoftware factories automating as much as possible

Agile development (extreme programming, Scrum)

Model-driven code generation

What I learned in those 27 years:- Automated procedures as important as programming skills or architecture- Automated build- Version management- Automated deployment- (Automatic unit testing)- Communication and trust within teamSpecialties:- Software factories 25+ years- used to be new but now more common- Model-driven code generation 10+ years- Agile Development- eXtreme Programming 10+ years- Scrum 3+ years

The Assignment

Transforming a 15 year old model-driven application from C++ to Java

What was our assignment?

THE ASSIGNMENT
THE APPLICATION

A very succesful mortgage application185 screens

100 users

Quarterly releases

Sizeable model: 200,000 lines (10Mb) of XMI

Using a proprietary model-driven code generator

Several interfaces to external systems: queues

Several native Windows DLL's for complicated calculations

A very succesful mortgage application- 185 screens (or parts of screens)- 100 users- regular quarterly releases- sizeable model: 200.000 lines, 10Mb of XMIUsing a proprietary model-driven C++ generator written in C++Several interfaces to external systems, queuesSeveral native Windows DLL's for complicated calculations

THE ASSIGNMENT
THE MODELER/GENERATOR

Old, but very complete

Model in relational tables (DB2)

Modeler written in MS-Access

C++ Class Tree to keep model in memory

Loader containing mapping (DB Class Tree) knowledge

Hand-written C++ framework, modeledPersistence support classes, including lazy loading

User Interface support classes

Action Language: COOL (Common Object Oriented Language)Methods on objects

Actions on UI elements, like buttons

Implicit metamodel (as far as known)

Implicit metamodel (as far as known)Model in relational tables (DB2)C++ Class Tree to load model in memory- analogous to the EMF generated Java APILoader containing the mapping knowledgeHand-written C++ framework- persistence support classes, including lazy loading- user interface support classesAction Language: COOL (Common Object Oriented Language), used everywhere:- methods of objects- actions on UI elements (buttons, fields)

THE ASSIGNMENT
WHY TRANSFORM?

Dependent on outdated C++ compiler and libraries

Scarce expertiseModel-driven techniques in general

The proprietary modeler/generator in particular

C++

Modeler/generator maintained, but not further development

Doesn't fit in Enterprise Architecture

Difficult to offshore

Want a mainstream Java web application

And no code generator anymore please, thank you very much.No dependency on proprietary tools

Current developers where quite happy, though

Scarce expertise- model-driven techniques in general- this particular modeler/generator in particular- C++Dependent on outdated C++ compiler and librariesModeler/generator maintained but not further developed: dead endDifficult to offshoreWanted: a regular Java webapplicationNo code generator- dependency on proprietary tool- Current developers where quite happy with the tool and productive

Renovating applications

RequirementsSpecificationsImplementationGoalsGoalsRequirementsSpecificationsImplementationAutomatic C++ to Java translationupdate

update

Legacy application

Renovated application

reverse engineeringreverse engineering

Start to establish goals for a new applicationCollect requirementsMake specificationImplementHowever: in many cases there is only source code, and if there is more documentation, it is often outdated.How to renovate:- automatic source translation? No idiomatic Java.- Better: reverse engineer specifications and re-implement- Even better (but more difficult): reverse engineer reqs.- End to Goals.

The Strategy

RequirementsFASTmodel(database)C++implementationGoalsMod4Jmogram(textual DSL)JavaimplementationC++ extractorXpandgeneratorLegacy application

Renovated application

C++ generatorC++ generatorFASTmodel(xmi)C++extractorXtend2generatorFASTmodel(xmi)C++extractorXtend2generatorBut in our case we had specification in the form of an executable (therefore correct) model.1) Write a new C++ generator to generate Java- Problem: large conceptual gap to bridge:- C++ to Java (fluency in both needed)- Different architecture- On top of model driven skills2) Use Mod4J as intermediate model- Smaller conceptual gap- Built-in Java architecture- Based on written reference architecture3) Migrate model to EMF for productivity - Better tools- Better software factory4) Mod4J incompatibly architecture- fat client layering- COOL language everywhere- lazy loading

Migrating the (meta)models

SQLFASTmetamodel(relational)FMAmodel(database)FMAuser data

expressed inexpressed in

AbstractSyntaxClass TreeC++Modelobjectgraphexpressed inexpressed inloadFMAexport(C++)

UMLFASTmetamodelexpressed inEA C++importEcoreFASTmetamodel(ecore)expressed inEA ecoreexportFMAmodel(xmi)expressed inFMAapplication

Exportgenerator(xtend2)

FMAgenerators(xtend2)

Old

M2

M1

M0

M3

New

Intermediate(Enterprise Architect)

FMAgenerators(C++)

Want to use EMF- Need an ecore metamodel- Derive from Class Tree- Via Enterprise Architect im/export- Generate an export program- Generate our new application- Loading time: 1,5 second- But initially 25 seconds (!)- Need a HashMap for id's

FASTmetamodelFMAmodel(database)FMAmodelLoad/extractFASTClass TreeEnterpriseArchitectFASTmetamodel(ecore)

FMAmodel(XMI)

EMFgenerator

FASTmodelJava APIFASTeditor

FMA(Java)

JNAadapters(Java)FASTframework(Java)External(MQ) classes(Java)maven/javac/build

FMA(JWS/jar)

FMAdata

FMAexample(Java)

External(MQ) classes(C++)FASTframework(C++)

Extractorgenerator(xtend2)

FMAgenerator(xtend2)

givengeneratedmanualtoolsLegenda

Lessons Learned

LESSONS LEARNED
The value of an explicit metamodel

Help in understanding the model

Generated export program

Generated tree editor

Generated model API

Basis for a new DSL

LESSONS LEARNED
Generator architecture

Consistent naming

Use an intermediate model

An intermediate model provides a natural place for names

LESSONS LEARNED
Incompatible source/target architecture

Layering: classic client server, modern three-tiered

Lazy loading assumed in the COOL Action Language

One Action Language for UI and Class Methods

Classic client server modern three tiered layeredLazy loading assumed in de Action LanguageOne Scope with UI scripts and Class Methods

LESSONS LEARNED
Know your tools

Know your toolsEMF is a great tool, once you obtain a copy of the EMF book

Xtext2, needed for COOL Parser, difficult to integrate

Xtend2, nice but still buggy

Know your tools- EMF is great, but you need the book- Xtext2 for the COOL Parser difficult to integrate- Xtend2, nice but buggy and slow

REMAINING UNSOLVED PROBLEMS

Eclipse is a hard to tame beast!Working combination of (versions of) plugins

Hard to get rid of red crosses

Continuous integrationMaven OSGi: two worlds

Eclipse artifacts (not) in public maven repositories

Eclipse is difficult- Working combination of (versions of) plugins- Hard to get rid of the red crossesContinuous integration- Maven OSGi: two worlds- Solve a similar problem- Different audiences: tool/application builders- But MDCD overlap the two- Eclipse artifacts not automatically in maven repositories.

REMAINING CHALLENGE
Maintaining the migrated model

Use the EMF-generated tree editoralready as good as, or better than, the Access application

Derive an Xtext grammar from the metamodelManual adjustments

Have a DSL almost for free

Familiar concepts: the metamodel (language) was not changed.

How to maintain the migrated model:- Use EMF tree editor- already quite good- Derive an Xtext grammar from the metamodel- beautify- DSL for free- The language is still the same- Familiar for the maintainers

www.ordina.nl

Click to edit the title text formatKlik om de stijl te bewerken

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline Level

Ninth Outline LevelKlik om de modelstijlen te bewerken

Click to edit the title text formatKlik om de stijl te bewerken

Click to edit the title text formatKlik om de stijl te bewerken

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline Level

Ninth Outline LevelKlik om de modelstijlen te bewerkenTweede niveauDerde niveauVierde niveauVijfde niveau