[email protected] 2000 advanced oosa state of the art on software architecture declarative...

1December 2000 [email protected]

Advanced OOSAState of the Art on Software Architecture

Declarative Meta Programming

Session 5:

Lightweight Architectural Tools

Nantes, EMOOSE 2000–2001

Dr. Kim Mens, PROG, VUB

State of the Art on Software Architecture — Declarative Meta Programming Session 5, page 2

Course overview

Introduction Software Architecture Declarative Meta Programming Software Classification Lightweight Architectural Tools Automated Architectural Conformance Checking An Architecture-driven Software Development Tool Assignments


Session overview

Lightweight architectural tools Lightweight source-model extraction Software reflexion models Conceptual module querying Summary

Lightweight architectural tools Lightweight source-model extraction Software reflexion models Conceptual module querying Summary


Lightweight architectural tools

Lightweight source-model extraction automatically reverse engineer high-level source models from

implementation

Software reflexion models automatically check conformance of extracted source models

to desired high-level models

Conceptual module querying support reengineering allows software engineers to perform queries simultaneously

about both existing and desired source structure a conceptual module is a set of lines of source that are treated

as a logical unit


References Lecture “Connecting Task to Source” by Gail Murphy. Murphy, Notkin & Sullivan. Software Reflexion Models: Bridging the Gap

Between Source and High-Level Models. In Proceedings of the FSE’95 Symposium, pp. 18-28. ACM Press, 1995.

Murphy & Notkin. Reengineering with Reflexion Models: A Case Study. In IEEE Computer 30, 8, pp. 29-36, 1997.

Reflexion Model tool:

http://www.cs.ubc.ca/~murphy/software/rmtool/index.html Murphy & Notkin. Lightweight Lexical Source Model Extraction. ACM

Transactions on Software Engineering and Methodology, vol. 5, no. 3, July 1996, pp. 262-292.

Lightweight Source Model Extraction tool:

http://www.cs.ubc.ca/~murphy/software/lsme/index.html Baniassad & Murphy. Conceptual Module Querying for Software

Reengineering. In Proceedings of the 20th International Conference on Software Engineering, pp. 64-73. IEEE, 1998.

RECAP


Lightweight conformance checking techniques Lightweight source model extraction Software reflexion models Conceptual module querying Summary

Reference: Murphy & Notkin 1996. Lightweight Lexical Source Model Extraction.

ACM Transactions on Software Engineering and Methodology, vol. 5, no. 3, July 1996, pp. 262-292.

Session overview


Lightweight source-model extraction

Source-model extraction automatically extract high-level info from implementation

• call graphs, file dependencies, event interactions, cross-reference lists, program-dependence graphs, …

independent of implementation language• C, C++, CLOS, Eiffel, …

information can be extracted from• source code, macros, comments, …

Lightweight small and easy to write specifications of desired source model lexical specifications


Lexical approach

Scan only for constructs relevant to the source model for example: extracting a call graph

• only look for calls in the source code

• skip all data declarations, control constructs, …

Similar to lexically-based tools like ‘grep’ But additional support for

matching source constructs in context and across multiple lines

Alternative approach: parsing more precise but places syntactic constraints on the source code and building/modifying parsers is not trivial


Approximate information

Extracted information is approximate not all intended constructs may be extracted some unintended constructs may be extracted

Approach trades precision in extraction for increased efficiency

• more efficient than parsing but less precise

• false positives and missing information flexibility

• information can be extracted from code, macros, comments, … tolerance

• information can be extracted from non-compilable code, code that is not linked, …


Architecture of thesource-model extraction system

Source-modelspecification

SourceCode

SourceModelScannerScanner

Generate

AnalyserAnalyser

Generate

Intermediate

Representation


Source model specification

Pattern definitions: the patterns of interest in the source

code Action definitions:

actions to execute when a pattern is matched in the code

are used to generate a

Post-processing analysis operations: operations that combine local info

extracted from individual source code files into a global source code model

are used to generate an

Source-modelspecification

defines

ScannerScanner AnalyserAnalyser


Source model specification:pattern definitions Code patterns are described as regular expressions For example: extracting function names from C source

[<type>] <functionName> \( [ { <formalArg> }+ ] \)

[ { <type> <argDecl> ; }+ ] \{

optional type specification, followed by name of the function and left parenthesis, followed by optional list of formal arguments and right parenthesis, optional list of declarations of the formal argument types, and finally a curly brace to start the function body

This pattern does not capture all function definitions e.g. it misses type declarations of the form int * x; but can be refined to capture 99% of the cases


Source model specification:action definitions Attach action code to patterns to be executed when the

pattern is matched in the source code For example: print names of extracted functions

[<type>] <functionName>

@ write ( functionName ) @

\( [ { <formalArg> }+ ] \)

[ { <type> <argDecl> ; }+ ] \{


The generated tools

More info on the generated scanner and analyser, see Murphy & Notkin 1996

ScannerScanner

Generate

AnalyserAnalyser

Generate



Reference: Murphy, Notkin & Sullivan. Software Reflexion Models: Bridging the Gap

Between Source and High-Level Models. In Proceedings of the FSE’95 Symposium, pp. 18-28. ACM Press, 1995.

Murphy & Notkin. Reengineering with Reflexion Models: A Case Study. In IEEE Computer 30, 8, pp. 29-36, 1997.

Slides mainly based on lecture “Connecting Task to Source” by Gail Murphy.

Session overview


Software reflexion models

A typical estimation scenario Software reflexion model Software reflexion model technique Experiences



You are asked to provide, within 5 days, an estimate of the effort required to modify an implementation of a Unix operating system to page over a distributed network

include shareregresslib

sys

1947 files

860 files

NetBSD Kernel Source CodeSlid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y

A typical estimation scenario


"Calls"extracted fromNetBSD sourceand displayed

usingField [Reiss95]

Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y

Software visualization


Practical problems

Difficult to understand software structure directly from the (very large) source code

Visualisation tools suffer from essentially the same problem

Software reflexion models limit the scope by comparing a (small) desired diagram with

the source code do not work on the source code directly but on an extracted

source model


High-level model of a Unix virtual memory subsystem drawn by a domain expert

HardwareTranslation

VMPolicy

User

VirtualAddressMaint.

KernelFault Handler

Memory

Pager FileSystemModule

Call

Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y

High-level model


Software reflexion model

After comparing the high-level model with an (extracted) source model, a software reflexion model is generated

This is an annotated version of the high-level model convergences

• discovered interactions in the source that were expected by the high-level model

divergences• discovered interactions that were not expected

absences• interactions that were expected but were not found

Each of the lines is annotated with a number indicating the number of discovered interactions


Memory

HardwareTranslation

User

KernelFaultHandler

Pager


VMPolicy

FileSystem0

10

8

2

2

Convergence

Divergence

Absence

12 3

74

18

0 000

0

0

0

00

HardwareTranslation

VMPolicy

User


KernelFault Handler

Memory

Pager FileSystem

Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y



Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y

4Reflexion

Model

ExtractionTool

2Software System

Artefacts

SourceModel

Mapping

3

RMTools

Model1

Software reflexion model technique


Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y

1. State a high-level model

Syntactic

Multiple relations

“everyone has one or more”

CallData

Access

HardwareTranslation

VMPolicy

User


KernelFault Handler

Memory

Pager FileSystem


Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y

2. Extract a source model

ExtractionTool

... arp.c... vfsop.c

arpwhohasvget_internal

callscalls

... net.c

... rrip.csendrecv

rip_analyze

etc.

Use existing tools (e.g., cflow, Field, etc.) Lightweight lexical source model extractor (Murphy/Notkin) May contain multiple relations


Source Model Entities High-Level Model Entities

file=pager.c Pager file=vm_map.* VirtualAddressMaint. dir=vm func=active VMPolicy

Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y

3. State a declarative mapping

Name source model entities using: physical and logical software structure regular expressions

Many-to-many mapping


Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y

4. Investigate a reflexion model

KernelFault Handler

Pager

2

0

1

12966

Arc Values

lfs_truncate calls vnode_pager_setsize

etc.

Convergence

Divergence

AbsenceFileSystem


Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y

Iteration

Want to investigate the data relationships?

augment the source model

update the mapping:

var=queue.*active VMPolicy

recompute...

CallData

Access

HardwareTranslation

VMPolicy

User


KernelFault Handler

Memory

Pager FileSystem


Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y

Refined reflexion model

Memory

HardwareTranslation

User

KernelFaultHandler

Pager


VMPolicy

FileSystem0

10

8

2

2

Convergence

Divergence

Absence

12 3

7 4

18

0 000

0

0

0

00

0

0

3

966


Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y

Experience

compiler 4,000 lines of C++4,000 lines of Ada

restructuring tool 30,000 lines of CLOSSPIN 65,000 lines of Modula-3NetBSD 250,000 lines of Cindustrial audibles 6,000 lines of C++Excel >1,000,000 lines of C


Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y

Excel: experimental reengineering

A Microsoft engineer computed reflexion models several times a day for four weeks 120,000 calls and global variable references map file with over 1000 entries high-level model with 15 entities and 96 interactions 4 minutes to compute on a 486

Some lessons learned: map files evolved to be larger than expected scale places pressure on managing the information



Reference:

Elisa L. A. Baniassad & Gail. C. Murphy. Conceptual Module Querying for Software Reengineering. In Proceedings of the 20th International Conference on Software Engineering, pp. 64-73. IEEE, 1998.

Slides mainly based on lecture “Connecting Task to Source” by Gail Murphy.

Session overview


Conceptual module querying

Goal A typical reengineering scenario Different reengineering approaches Conceptual module approach Querying about conceptual modules Experiences with conceptual modules



Goal

Support software reengineering reengineering is more than source code analysis reengineering requires reasoning about source code…

• identification and extraction of new software components from existing source code

…but also about desired source structure• interactions of new components with remaining source code

• interactions among newly-formed components

Allow software engineer to simultaneously perform queries about the existing and desired source structure

A conceptual module is a set of lines of source that are treated as a logical unit


Conceptual modules are like software classifications

Both conceptual modules and software classifications

group a set of related source code entities

represent some logical or conceptual unit

may cross-cut the source code e.g., code lines included in a CM may

be split across existing procedure boundaries


A typical reengineering scenario

main() {for ( i=0; i< n files; i++ ) { char buf[8192]; F ILE *fp ; in t cc; fp=fopen( files[i], "r" ); tmp=tempname(); o fp=xtmpfopen(tmp);

Sort

Inpu tP ipe

O utpu tP ipe

so rt.c (1700 lines)

29 o the rC files

Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y


Reengineering scenario...main() for ( i=0; i< nfiles; i++ ) { char buf[8192]; FILE *fp; int cc;

fp=fopen( files[i], "r" ); tmp=tempname(); ofp=xtmpfopen(tmp);

sort( files, nfiles, ofp );

sort( char **files, int nfiles, FILE *ofp ) { fp=xfopen( files, "r" ); while ( fillbuf( &buf, fp ) ) { findlines( &buf, &lines )

if ( feof (fp) ) && !nfiles... tfp=ofp; else ++n_temp_files

mainprocedure:

InputPipe

sortprocedure:

Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y


Different reengineering approaches

Five different program understanding tools: xrefdb (part of Field)

• cross-reference database tool Unravel

• slicing tool Lackwit tool

• based on type-inferencing Software reflexion model Conceptual module tool

Task: create an input pipe component from the GNU sort program.


Cross-reference database

Identify variables of interest For each variable

where is the variable declared? where is the variable referenced?

Collate results Repeat

Result:

Using Field’s xrefdb, 126 lines were returned of which only 30% were relevant

0

20

40

60

80

100

120

140

sort

Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y


Slicermain() for ( i=0; i< nfiles; i++ ) { char buf[8192]; FILE *fp; int cc;





Compute backward slices on variables in pre-identified lines of code

Slice computation took several hours; only partial were viewed

Result:

Slices computed with Unravel were > 750 nodes in size and included many irrelevant procedures

Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y


Type inferencer

Compute which procedures affect the values of particular variables

The Lackwit tool produces graphs summarizing how values are transmitted through a program

Provides information on relevant procedures but not on relevant source code lines

Result:

Lackwit graphs are often large and are reported in terms of the existing structure

Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y



F ilte r

InputP ipe

O utputP ipe

5

2

180

0

4

C allsN o typeData

• Difficult to ascertain interface of the module

• No support for querying the source model

• Syntactic comparison

Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y


Conceptual module approach

Query

Source code

Source ModelExtraction Tool

SourceModel

Form a CMConceptual

ModuleTool

InterfaceAnalysis

QueryResults

Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y


Source model

Instead of reasoning directly about source code, the approach reasons about an extracted source model

Source model is directed towards a procedural language and consists of three relations: variable dependence relation

• uses and definitions of variables (described by line number) control transfer relation

• procedure calls (line number of call site and called procedure) procedure start relation

• source line number on which a procedure starts

Sourcecode

Source ModelExtraction Tool

SourceModel


Query language

Supports formation (definition) of conceptual modules Supports investigation of interactions involving

conceptual modules

Query

Form a CMConceptual

ModuleTool


Forming a conceptual module

Map lines of source code to a logical module

Two ways to map the code: by specifying line numbers

(individual, ranges, etc.) by specifying pieces of existing

logical structure (i.e., variablesor procedures)

• e.g., the fp variable

Each module has a name Formation can be iterative

main() for ( i=0; i< nfiles; i++ ) { char buf[8192]; FILE *fp; int cc;





For sort, we ended up including about 24 lines in the input pipe conceptual module.Sl

ide

take

n fro

m le

ctur

e “C

onne

ctin

g Ta

sk to

Sou

rce”

by

Gai

l Mur

phy


Conceptual modules are like virtual software classifications

Just like software classifications, conceptual modules can be specified either

extensionally, i.e. by explicit enumeration of the elements (line numbers) that belong to it

intentionally, by using a more semantic specification

This is similar to the distinction between ordinary and virtual software classifications


First define conceptual module Tool then scans the source model to determine the

lines of code with occurrences of input, local and output variables calls from CM to procedures in existing source

These lines are included as part of definition of the CM

Forming a conceptual module:interface analysis

ConceptualModule

Tool

SourceModel

InterfaceAnalysis


Picture of interactions between CM’s


Interface analysis rules

Input variable used on line inside module, but defined on line outside module

Output variable defined on line inside module, but used on line outside module

Local variable Two-phase analysis for local variables:

1. Use-def pairs: all uses & defs in module implies local variable.

2. Only uses or defs: consider identified input/output variables; promote to local if all uses and defs are inside module.

Control transfer call to procedure outside module


main() for ( i=0; i< nfiles; i++ ) { char buf[8192]; FILE *fp; int cc;





Input Variables: sortalloc, main.ofp, main.minus, …

Output Variables: main.mergeonly, sort.ofp, sortalloc, …

Local Variables: main.files, main.nfiles, sort.files

Control Transfers: xmalloc at sort.c 1796 fillbuf at sort.c 248

Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y

Interface analysis: an example

Local (interface) analysis is used to summarize how the module interacts directly with the existing code


Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y

Querying about conceptual modules

Once one or more conceptual modules are formed, the re-engineer typically needs to perform queries: How do conceptual modules relate to each other? How do conceptual modules relate to existing source?

Tool provides pre-coded queries and a programmable interface through which a user can code queries.

Query

Form a CMConceptual

ModuleTool


Conceptual module relationships

A

B

directdef

use

A

B

indirectdef

use

A B

overlapA

B

contains

Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y

4 types of useful queries


Conceptual module relationships

Simply determining the presence of a relationships between two modules is generally not sufficient to perform a reengineering task

Therefore, a programmable interface is provided which allows the engineer to access input, output and local variable names defs and uses of conceptual module variables lines of code spanned by a conceptual module calls made to and by (code in) a conceptual module relationship info between conceptual modules including

common line numbers, variables and calls


Programmable InterfaceSET common = new SET();// Get the use-def chains for all input and local variables of that module.

Module first = (Module)Module.ModuleTable.elementAt(0); common=DefUse.GetFullUseDefChain(first); for(int i=1; i<Module.ModuleTable.size(); i++) {

// Get the use-def chains for the next module

Module current = (Module)Module.ModuleTable.elementAt(i); SET curr_chain = DefUse.GetFullDefUseChain(current);

// Intersect the chains to determine common definition points

common = DefUse.INTERSECTION(common, curr_chain);}common.print();

Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y


Experiences with conceptual modules

Task System Size (loc)

Extract components GNU sort 5100

Restructure adventure 8000

C to C++ GNU plot 26,000

(compression) 10,000

Restructure(subset)

CUDD 47,000

SUIFSUIF

SUIFSUIF

SUIFSUIF

SUIFSUIF

xrefdbxrefdb

SUIFSUIF = tools built on SUIF provides use-def pairs

xrefdbxrefdb = Field’s xrefdb provides uses & defsSlid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y


Query Context and Form

Context is expressed in 2 parts: A good re-engineering tool should allow a software engineer to

express the context of a query being performed identify region of source over which to query

• conceptual module identifies region restrict the region for which results are reported

• interface analysis summarizes local results

Form includes both input and output: some tasks require queries over grouped items report results in terms of task

• results are reported in terms of target structure; can use conceptual module structure to query against source

Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y


Session overview



Partial and approximate techniques

Each of these characteristics can be an effective way to attack scale.

These characteristics can be combined to provide software engineers with a “smoother” means of managing source investigations.

Bottom line for most developments is that time is money.

Time

% P

rogra

m C

onsi

dere

d

approximate

conservative

Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y


Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y

Summary

Software Reflexion Model

“Definitely confirmed suspicions about the structure of Excel. Further, it allowed me to pinpoint the deviations. ... very easy to ignore stuff that is not interesting and thereby focus on the part of Excel I want to know more about.” Microsoft EngineerConceptual Module

“Not only did the tool verify the independent nature ofthe ZDD functionality and allow me to rip out all thatcode, but, the process of using your tool forced me toanalyze and understand the code in a way that I hadnot been doing and that ultimately it very quickly gave me the perspective I needed.” Yvonne Coady


Slid

e ta

ken

from

lect

ure

“Con

nect

ing

Task

to S

ourc

e” b

y G

ail M

urph

y

Summary

Demonstrated benefits of task-aware program understanding techniques current techniques are structurally task-aware

Demonstrated role for approximate information reflexion model technique makes engineer responsible conceptual modules takes some of responsibility

Goal is to get to “what-if” tools that would allow engineers to leverage, cost-effectively, connections between design and source


Session overview


Time for a break

[email protected] 2000 advanced oosa state of the art on software architecture declarative...

Documents

n lightweight conformance

software reengineering

n reflexion model tool

n murphy notkin

software engineering

n baniassad murphy

software engineers

references n lecture