model driven software development - data model evolution
DESCRIPTION
A class on data model evolution in a model driven software development courseTRANSCRIPT
DataModel Evolution
Sander VermolenEelco Visser
This research is supported by NWO/JACQUARD project638.001.610, MoDSE: ModelDriven Software Evolution.
Data Models
User 1name bob real name Bob Johnsonemail [email protected]
Page 1title "The first page"isRedirect falsetext "Hello world"
Count page views
Version history
User 1name bob real name Bob Johnsonemail [email protected]
Page 1title "The first page"isRedirect falsetext "Hello world"
No page countNo revisions
Coupled Data Evolution
...
$dbh->bz_add_column('attachments', 'submitter_id', {TYPE => 'INT3', NOTNULL => 1}, 0);$dbh->bz_rename_column('bugs_activity', 'when', 'bug_when');_add_bug_vote_cache(); _update_product_name_definition(); _add_bug_keyword_cache();$dbh->bz_add_column('profiles', 'disabledtext', {TYPE => 'MEDIUMTEXT', NOTNULL => 1}, '');_populate_longdescs(); _update_bugs_activity_field_to_fieldid();
if (!$dbh->bz_column_info('bugs', 'lastdiffed')) {$dbh->bz_add_column('bugs', 'lastdiffed', {TYPE =>'DATETIME'});$dbh->do('UPDATE bugs SET lastdiffed = NOW()');
}
_add_unique_login_name_index_to_profiles(); $dbh->bz_add_column('profiles', 'mybugslink', {TYPE => 'BOOLEAN', NOTNULL => 1, DEFAULT => 'TRUE'});_update_component_user_fields_to_ids();$dbh->bz_add_column('bugs', 'everconfirmed', {TYPE => 'BOOLEAN', NOTNULL => 1}, 1);$dbh->bz_add_column('products', 'maxvotesperbug', {TYPE => 'INT2', NOTNULL => 1, DEFAULT => '10000'});$dbh->bz_add_column('products', 'votestoconfirm', {TYPE => 'INT2', NOTNULL => 1}, 0);_populate_milestones_table();$dbh->bz_alter_column('bugs', 'target_milestone', {TYPE => 'varchar(20)', NOTNULL => 1, DEFAULT => "'---'"});$dbh->bz_alter_column('milestones', 'value', {TYPE => 'varchar(20)', NOTNULL => 1});_add_products_defaultmilestone();
if (!$dbh->bz_index_info('cc', 'cc_bug_id_idx') || !$dbh->bz_index_info('cc', 'cc_bug_id_idx')->{TYPE}) {$dbh->bz_drop_index('cc', 'cc_bug_id_idx');$dbh->bz_add_index('cc', 'cc_bug_id_idx', {TYPE => 'UNIQUE', FIELDS => [qw(bug_id who)]});
}if (!$dbh->bz_index_info('keywords', 'keywords_bug_id_idx') || !$dbh->bz_index_info('keywords', 'keywords_bug_id_idx')->{TYPE}) {
$dbh->bz_drop_index('keywords', 'keywords_bug_id_idx');$dbh->bz_add_index('keywords', 'keywords_bug_id_idx', {TYPE => 'UNIQUE', FIELDS => [qw(bug_id
keywordid)]});}
...
Costly
High risk
Holds back the development process
Large infrequent development steps
Userid : integername : varcharrealName : varcharemail : tinytext
Pageid : integertitle : varcharauthor - User *isRedirect : booleancontent : text
set of
Userid : integer Uniquename : varcharrealName : varchar ?email : tinytext
Page : Mediumauthor - User min(1) max(8)content : textrefs : url * Indexed
abstract Mediumid : integer Uniquetitle : ANY_NAME
EvolvingData Models
SpecifyingData Model Evolution
Userid :: integername :: varcharrealName :: varcharemail :: tinytext
Pageid :: integertitle :: varcharauthor UserisRedirect :: booleancontent :: text
Userid :: integername :: varcharrealName :: varcharemail :: tinytext
Pageid :: integertitle :: varcharcounter :: bigintegerisRedirect :: booleanrevisions set of Revision
Revisionid :: integerpage Pagecomment :: tinyblobtimestamp :: timerevisionText :: textauthor User
Added type revisions
Added attribute revisions
Moved content to revision text
Added attribute counter
Revisionid :: integerpage Pagecomment :: tinyblobtimestamp :: timeauthor User
Pagerevisions set of Revision
RevisionrevisionText :: text
Pagecounter :: biginteger
What happened?
add or remove entity
change name of entity
change type of set
8 Basic Transformations
add or remove property
change name of property
change type of property
move property
1 Advanced Transformation
addRevision
id : integerpage - Pagecomment : tinyblobtimestamp : timeauthor - User
addcounter : biginteger
At / Entity Page / Property Titleadd counter : biginteger
At / Entity[Id=''Page''] / Property[Id=''Title'']add counter : biginteger
at // Property [Id = ../Id]add counter :: biginteger
RevisionrevisionText :: text
At / Entity Revision / Property timeStampmove page.content to revisionText :: text
at Entity Page / Property Titleadd counter :: biginteger
at Entity Revision / Property timeStampadd revisionText :: textfrom page.content
;
EvolvingData Models
8 basic transformations
1 advanced transformation
Language to specify transformations
Positioning sub language
Specify data model evolutions
DataMigration
Technical Domain
WebDSL
Stratego/XT
Stratego/XTGeneric Aterm
SQL Databases (MySQL)
Because
we really like program transformations
generally richer than regular data acessing languages (SQL)
data migration = model transformation
Program transformation for data migration
Userid :: integername :: varcharemail :: tinytext
Pageid :: integertitle :: varcharauthor User
User(id(1),name(“John”),email(“[email protected]”)
)
Page(id(2),title(“Hello World”),[author(1)]
)
User(id(1),name(“John”),email(“[email protected]”)
)
Page(id(2),title(“Hello World”),[author(1)]
)
<user><id>1</id><name>John</name><email>[email protected]</email>
</user>
<page><id>2</id><title>Hello World</title><authors>
<author>1</author></authors>
</page>
Userid :: integername :: varcharemail :: tinytext
Pageid :: integertitle :: varcharauthor User
User(id(1),name(“John”),.....
)
Page(id(2),title(“Hello World”),[author(1)]
)
Remove Attribute (1)
Signature:User := Id * Name * Email
Userid :: integername :: varcharemail :: tinytext
Pageid :: integertitle :: varcharauthor User
User(id(1),name(“John”),email(“[email protected]”)
)
Page(id(2),title(“Hello World”),[author(1)]
)
Remove Attribute (2)
Userid :: integername :: varcharemail :: tinytext
Pageid :: integertitle :: varcharauthor User
User( 0,[
id(1),name(“John”),email(“[email protected]”)
])
Page( 1,[
id(2),title(“Hello World”),author(0)
])
Generic Aterm (GTerm)
User( 0,[
id(1),name(“John”),email(“[email protected]”)
])
Page( 1,[
id(2),title(“Hello World”),author(0)
])
XMI
<user id='0'><id>1</id><name>John</name><email>[email protected]</email>
</user>
<page id='1'><id>2</id><title>Hello World</title><author>0</author>
</page>
GTerm Characteristics
Explicit references
No nesting
Implicit sets/lists
Storage...
GTerm Transformation
Gterm libraryObject creationModifying attributes
(add, remove, change, rename, ...)Object equivalenceObject traversals(Object graph traversals)
Data model libraryType examinationSuper/Sub type handlingAbstract type handling
GTerm Storage
Large quantities of data...
Storage engine:In memory list based ~10KIn memory hash table based ~500KIn database ~25M ...
User( 0,[
id(1),name(“John”),email(“[email protected]”)
])
Page( 1,[
id(2),title(“Hello World”),author(0)
])
GTerm Storage – In database (1)
0 User id 10 User name John0 User email [email protected] Page id 21 Page title Hello World1 Page author 0
0 User1 Page
GTerm Storage – In database
CREATE TABLE Attributes (id varchar(16),type varchar(30),name varchar(30),value text,INDEX USING HASH (id (5)),INDEX USING BTREE (v(10))
)
CREATE TABLE Objects (id varchar(16),type varchar(30)
)
GTerm Storage – In database (2)
GTerm Performance
Database indexes
Stratego memory usage
Parallel execution
GTerm Storage – Regular database
0 User id 10 User name John0 User email [email protected] Page id 21 Page title Hello World1 Page author 0
User:1 John [email protected]
Page:2 Hello World
PageUser:2 1
Old Database Generic Database
SQL Script
Data model
Migration (Stratego)
SQL Script
GTerm 2 SQL
New Database
ResearchrDBLP
<dblp><incollection mdate="20020103" key="books/acm/kim95/AnnevelinkACFHK95">
<author>Jurgen Annevelink</author><author>Rafiul Ahad</author><author>Amelia Carlson</author><author>Daniel H. Fishman</author><author>Michael L. Heytens</author><author>William Kent</author><title>Object SQL A Language for the Design and
Implementation of Object Databases.</title><pages>4268</pages><year>1995</year><booktitle>Modern Database Systems</booktitle><url>db/books/collections/kim95.html#AnnevelinkACFHK95</url><crossref>books/crc/KIM95</crossref>
</incollection>
....</dblp>
article {key : stringtitle : string *author : string *editor : string *booktitle : string *pages : string *address : string *journal : string *volume : string *number : string *publisher : string *crossref : string *series : string *school : string *chapter : string *month : string *year : string *url : string *note : string *mdate : string *cite : string *ee : string *cdrom : string *isbn : string *
}
book {key : stringtitle : string *author : string *editor : string *booktitle : string *pages : string *address : string *journal : string *volume : string *number : string *publisher : string *crossref : string *series : string *school : string *chapter : string *month : string *year : string *url : string *note : string *mdate : string *cite : string *ee : string *cdrom : string *isbn : string *
}
....
~ 800,000 Authors
~ 1,200,000 Publications
~ 14,000,000 Lines of XML
~ 16,000,000 Database records
DBLP Data
Publication {key : string Uniquetitle : stringauthors - Author + Indmonth : stringyear : stringdblpUrl : string ?doi : stringlinks - Link *abstract : stringnote : stringannote : stringmodified : datemodifiers - User *cites - Publication *ee : string ?isbn : stringissn : stringconflicts : bool
}
abstract PrintPublication - Publication {
pages : stringpublisher : string ?firstpage : intlastpage : int ?
}
Article - PrintPublication {journalname : stringvolumenumber: stringissuenumber : string
}
Alias {name : string Unique
}
AbstractAuthor {alias - Alias
}
Load objects into database
9 Stages of migration
Generate SQL
Migration Approach
BridgingMeta levels
article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?
}
Article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?
}
renameType(|oldType, newName)
?Transformation(path, Substitution(newName), _)
UPDATE At SET t=newName WHERE t = ...;
Proceedings - Collection {booktitle : string ?
}
Proceedings - Collection {conference : string ?
}
renameAtt(|oldName, newName, type, dmodel)
?Transformation(path, Substitution(newName), _)
UPDATE At SET n = newName WHERE n = oldNameAND t = <getSubTypeQuery(|dmodel)> type;
abstract Thesis - Publication {school : string
}
abstract Thesis - Publication {school : stringtype : string ?
}
...
?Transformation(path, Addition(Att(Name(attName), PrimType(_), annotations)),_);<getAttAnn(|"MinCard")> annotations; ?0
...
abstract Thesis - Publication {school : string
}
abstract Thesis - Publication {school : stringtype : string
}
<addDefaultAttributesToType(|type, <makeint> low, attName, attType, mmodel
)> model;
?Transformation(path, Addition(Att(Name(attName), PrimType(_), annotations)),_
);<getAttAnn(|"MinCard")> annotations; ?low
onType(addDefaultAttributes(|type, nr, attName, attType)
| type, mmodel)
1. find objects of type
2. divide into chunks
3. per chunk in parallel
Load objects in chunkper object
ssave object if changed
onType
Publication {key : string Uniquetitle : string ?authors : string + Indexedyear : string...
}
?Transformation(path, Substitution(DeclType(Name(newTypeName))), _)...Author alias mandatoryAuthor alias not unique
Publication {key : string Uniquetitle : string ?authors - Author + Indexedyear : string...
}
Author {alias : string
}
onType(for each author
create author objectset author attribute
)
Author {alias : string
}
?Transformation(path, Substitution(DeclType(Name(newTypeName))), _)...Alias name mandatoryAlias name unique
Author {alias : Alias
}
Alias {name : string Unique
onType(if alias exists then
set alias attribute to existing idelse
create alias objectset alias attribute to new id
)
IdentityPrimitive attribute addition (3)Complex attribute additionAttribute removalAttribute name changeAttribute move (2)Primitive type changeImplicit reference resolutionAttribute wrappingType additionType removalType name changeAbstract type handlingInverse annotation handling (2)Cardinality changes (2)
Supported transformations
Easy to define Hard to define
Easy to optimize No need to optimize
Expressive Limited expressiveness
Easy to abstract Abstraction near impossible
Performance OK Performance great
In memory vs. Database transformations
DetectingEvolution
article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?
}
Article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?
}
diff?
article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?
}
Article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?
}
Entity(Name(“article”),Name(“PrintPublication”),[
Att(Name(“reviewid”),PrimType(
Name(“string”)),[
MinCard(0),MaxCard(1)
]),...
])
Entity(Name(“Article”),Name(“PrintPublication”),[
Att(Name(“reviewid”),PrimType(
Name(“string”)),[
MinCard(0),MaxCard(1)
]),...
])
article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?
}
Article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?
}
Entity(Name(“article”),Name(“PrintPublication”),[
Att(Name(“reviewid”),PrimType(
Name(“string”)),[
MinCard(0),MaxCard(1)
]),...
])
Entity(Name(“Article”),Name(“PrintPublication”),[
Att(Name(“reviewid”),PrimType(
Name(“string”)),[
MinCard(0),MaxCard(1)
]),...
])
diff?
article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?
}
Article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?
}
Removed type article; Added type Article
Added type article; Removed type Article
Substituted article name with Article
What happened?
Article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string
}
Article - PrintPublication {reviewid : string ?rate : stringjournal : string ?volume : string ?nr : string ?
}
renamed rating; renamed number;changed rating cardinality; changed number cardinality
What happened?
Article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string
}
Article - PrintPublication {reviewid : string ?rate : stringjournal : string ?volume : string ?nr : string ?
}
renamed rating; renamed number;changed rating cardinality; changed number cardinality
swapped rating and number; renamed rating; renamed number
What happened?
Article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string
}
Article - PrintPublication {reviewid : string ?rate : stringjournal : string ?volume : string ?nr : string ?
}
renamed rating; renamed number;changed rating cardinality; changed number cardinality
swapped rating and number; renamed rating; renamed number
deleted Article; added Article
What happened?
Addition: 0.8 * relativeSize4
Removal: 0.5 * relativeSize4
Substitution 0.4 * relativeSize6
Weighing transformations
Custom weights:Type removalType substitution (0)Attribute substitution (0)
On both versions at the same time
Bound on weightIncreasing bound
Weight computation caching
Try them all!
HeterogeneousCoupled Evolution
Horizontal Generalization
Username :: varcharrealName :: varcharemail :: tinytext
Pagetitle :: varcharauthor UserisRedirect :: boolean
Entities
Properties
Types
Meta model / Grammar
Lists
More types
Inverse associations
Abstract types
Vertical Generalization
Heterogeneous Coupled Evolution
ofSoftware Languages
The ingredients
Software Language Definition Formalism
EvolvingSoftware Language
Software
DiverseEvolution
What did we generalize?
Why did we generalize?
A GenericArchitecture
Input
Coupled evolution scenario
Mapping from Mi to Mi+1
Output
Domain Specific Transformation Language
(DSTL)
Transformation Interpreter
Software Migration
Stratego
SDF SDF
Entity* > DataM ModelId "{" Prop* "}" > Entity EntityId "::" Type > Prop Prop"int" > Type Int"bool" > Type BoolId > Type "set of" Type > Type SetNAME > Id Id
Entity* > DataM Model
Lists
"add" Entity > LocalTransformation"remove" > LocalTransformation
[...]
NAME > Id Id
Lexicals
"substitute" NAME > LocalTransformation
''...''
"int" > Type Int"bool" > Type BoolId > Type "set of" Type > Type Set
Multiple productions
"substitute" Type > LocalTransformation
>*
"at" APath LocalTransformation > Transformation
Type checking
Generation of local transformation domainsAPath type derivation
Type checking
.../...
Transform
Constant
Larger grammars
Interpreter generation
Transformations library
Generic DSTL constructs
APath evaluation
A GenericArchitecture
Software Language Evolution
This research was supported by NWO/JACQUARD project638.001.610, MoDSE: ModelDriven Software Evolution.