software architecture evaluation methodologies
DESCRIPTION
Software Architecture Evaluation Methodologies. Presented By: Anthony Register. Software architecture (SWA) - PowerPoint PPT PresentationTRANSCRIPT
Software Architecture Software Architecture Evaluation MethodologiesEvaluation Methodologies
Presented By:Presented By:
Anthony RegisterAnthony Register
Software architecture (SWA)Software architecture (SWA)
- - “The fundamental organization of a system embodied in “The fundamental organization of a system embodied in its components, their relationships to each other, and its components, their relationships to each other, and to the environment, and the principles guiding its to the environment, and the principles guiding its design and evolution”design and evolution” as defined by IEEE as defined by IEEE
SWA Quality determined by quality SWA Quality determined by quality attributes such as:attributes such as:
FlexibilityFlexibility
UsabilityUsability
PerformancePerformance
MaintenanceMaintenance
Evaluation methods:Evaluation methods:
SAAMSAAM : : SScenario-Based cenario-Based AArchitecture rchitecture AAnalysis nalysis MMethodethod
ALMAALMA : : AArchitecture rchitecture LLevel evel MModifiability odifiability AAnalysisnalysis
ATAMATAM : : AArchitecture rchitecture TTrade-off rade-off AAnalysis nalysis MMethodethod
PASAPASA : : PPerformance erformance AAssessment of ssessment of SSoftware oftware AArchitecturerchitecture
AssumptionAssumption
- Several of the most common SWA - Several of the most common SWA evaluation methods are similar in the evaluation methods are similar in the techniques applied, but have evolved to techniques applied, but have evolved to serve different purposes or viewpoints serve different purposes or viewpoints for evaluating the quality of a specific for evaluating the quality of a specific SWA.SWA.
Scenario BasedScenario Based Documented in OO design and user interface Documented in OO design and user interface
as a technique to solicit requirementsas a technique to solicit requirements
Used for comparing SWA optionsUsed for comparing SWA options
Simulate system events or changesSimulate system events or changes
Determine impact on SWADetermine impact on SWA
Result in brief description of a functional Result in brief description of a functional requirement that eventually may be requirement that eventually may be implemented into the system at design time or implemented into the system at design time or in the futurein the future
Scenario BasedScenario Based
Benefits:Benefits:
Break down and clearly describe the Break down and clearly describe the architectural description and analysis architectural description and analysis points points
Catalyst for generating more questionsCatalyst for generating more questions
Types of ScenariosTypes of Scenarios
DirectDirect : requires no change to system : requires no change to system architecturearchitecture
IndirectIndirect : requires change to system : requires change to system architecturearchitecture
Scenario ExampleScenario Example
Scenario-Based Architecture Analysis Scenario-Based Architecture Analysis
5 steps:5 steps:1.1. Describe the candidate architectureDescribe the candidate architecture
2.2. Develop scenariosDevelop scenarios
3.3. Perform scenario evaluationsPerform scenario evaluations
4.4. Reveal scenario interactionReveal scenario interaction
5.5. Overall SWA evaluation Overall SWA evaluation
http://www.sei.cmu.edu/pub/documents/00.reports/pdf/00tr004.pdf#search=%22ATAM%20%22
SAAM SAAM (Steps 1 & 2)(Steps 1 & 2)
Describe the candidate architectureDescribe the candidate architecture Use a common syntactic and semantic Use a common syntactic and semantic
notation that all designers and stakeholders notation that all designers and stakeholders can easily understand. can easily understand.
Develop scenariosDevelop scenarios Direct and indirectDirect and indirect
SAAM SAAM (Steps 3 & 4)(Steps 3 & 4)
Perform scenario evaluationsPerform scenario evaluations Evaluate quality attributes Evaluate quality attributes (flexibility, usability, (flexibility, usability,
performance, maintenance, etc.)performance, maintenance, etc.)
Reveal scenario interactionReveal scenario interaction Indirect scenario affects need to be at a Indirect scenario affects need to be at a
minimum or the component may be a risk to minimum or the component may be a risk to the SWAthe SWA
Scenario analysis indicates that many indirect Scenario analysis indicates that many indirect scenarios affect the same component, then scenarios affect the same component, then too much architectural level coupling may be too much architectural level coupling may be indicatedindicated
SAAM SAAM (Step 5)(Step 5)
Overall SWA evaluation Overall SWA evaluation
Weighted ranking or scoring performed for Weighted ranking or scoring performed for the overall SWA evaluation. the overall SWA evaluation.
A scoring technique using an appropriate A scoring technique using an appropriate weighting analysis can be applied to track weighting analysis can be applied to track the affects of the indirect scenarios on the the affects of the indirect scenarios on the SWA for future decision makingSWA for future decision making
SAAMSAAM
Benefits:Benefits: Disadvantages:Disadvantages:
Detecting issues earlyDetecting issues early Lacking in tool supportLacking in tool support for managing for managing large amounts of data for system and large amounts of data for system and scenario descriptions scenario descriptions
SWA documentationSWA documentation through the through the documentation of scenarios, documentation of scenarios, outcomes, and affects on the system outcomes, and affects on the system architecture architecture
Lacks architectural metricsLacks architectural metrics for for assisting designers when making assisting designers when making decisions between alternative designs decisions between alternative designs
Improved understandingImproved understanding of possible of possible SWA issuesSWA issues
SAAM StrengthsSAAM Strengths Evaluate or compare future and existing Evaluate or compare future and existing
systemssystems, such as using the low cost method , such as using the low cost method (tabular calculations and comparisons) for (tabular calculations and comparisons) for measuring change when choosing between measuring change when choosing between SWA designsSWA designs
Evaluation result enables the designers Evaluation result enables the designers and users to focusand users to focus on the details of the SWA on the details of the SWA and not be distracted from the less important and not be distracted from the less important areas areas
Provides a guided approach for evaluationProvides a guided approach for evaluation and and provides for open dialogueprovides for open dialogue between the between the stakeholders of the system under evaluationstakeholders of the system under evaluation
Architecture Level Modifiability Analysis Architecture Level Modifiability Analysis (ALMA)(ALMA)
Focuses on modifiability of the Focuses on modifiability of the SWASWA
Risk assessmentsRisk assessments Distinguishes multiple analysis Distinguishes multiple analysis
goalsgoals Explicit assumptionsExplicit assumptions Provides repeatable techniques Provides repeatable techniques
for performing the steps for performing the steps Predicting future maintenance Predicting future maintenance
costs costs Evaluating the flexibility of a Evaluating the flexibility of a
system at an architectural levelsystem at an architectural level Main quality inputs are the Main quality inputs are the
SWA specification and quality SWA specification and quality requirements requirements
http://www.sei.cmu.edu/pub/documents/00.reports/pdf/00tr004.pdf#search=%22ATAM%20%22
ALMA Process StepsALMA Process Steps
5 Steps:5 Steps:1.1. Set the goal and determine the focus of the Set the goal and determine the focus of the
analysisanalysis
2.2. Create a description of the SWACreate a description of the SWA
3.3. Create scenarios from the functional Create scenarios from the functional requirementsrequirements
4.4. Evaluate the effects of the scenariosEvaluate the effects of the scenarios
5.5. Analyze the results of the independent Analyze the results of the independent scenario evaluationsscenario evaluations
ALMA ALMA (Steps 1 & 2)(Steps 1 & 2)
Goal SettingGoal Setting Maintenance cost prediction, risk assessment, Maintenance cost prediction, risk assessment,
and SWA selectionand SWA selection
Description of SWADescription of SWA SWA information in order to derive an SWA information in order to derive an
architectural description of the systemarchitectural description of the system Activity evaluates the decomposition of the Activity evaluates the decomposition of the
system components and evaluates the system components and evaluates the relationships between the componentsrelationships between the components
ALMA ALMA (Steps 3 & 4)(Steps 3 & 4)
Create scenarios from functional requirementsCreate scenarios from functional requirements Determine the impact of change to the SWA Determine the impact of change to the SWA Providing a metric for analysis performed in the Providing a metric for analysis performed in the
next step for evaluating the different scenariosnext step for evaluating the different scenarios
Evaluate effects of scenariosEvaluate effects of scenarios Collection of information to be used in SWA impact Collection of information to be used in SWA impact
analysisanalysis Identify components affected by the change, Identify components affected by the change,
determine what the effects were, and determine the determine what the effects were, and determine the ripple effects on relational componentsripple effects on relational components
ALMA ALMA (Step 5)(Step 5)
Analyze the results of the independent Analyze the results of the independent scenario evaluationsscenario evaluations InterpretationInterpretation of the results needs to align of the results needs to align
with the goal set forth for the evaluationwith the goal set forth for the evaluation
Goals:Goals: Maintenance cost prediction, the scenarios Maintenance cost prediction, the scenarios
should cover future events of the systemshould cover future events of the system Risk assessment, the scenarios should Risk assessment, the scenarios should
provide complex change events in order to provide complex change events in order to determine and interpret the effects of the determine and interpret the effects of the changeschanges
ALMAALMA
Benefits: Benefits: Identification of SWA risksIdentification of SWA risks Measurement of the amount of effort required for Measurement of the amount of effort required for
changeschanges Deciding between available SWA optionsDeciding between available SWA options Reduction in the number of scenarios and a Reduction in the number of scenarios and a
process that provides guidance as to when to process that provides guidance as to when to stop generating scenariosstop generating scenarios All change categories explicitly consideredAll change categories explicitly considered New change scenarios do not affect the classification New change scenarios do not affect the classification
structurestructure
Architecture Trade-off Analysis Method Architecture Trade-off Analysis Method (ATAM)(ATAM)
Evaluates a SWA for quality attribute target goals, but Evaluates a SWA for quality attribute target goals, but focuses more on the trade-offs between the quality focuses more on the trade-offs between the quality attributes. attributes.
http://www.sei.cmu.edu/architecture/ata_method.html
ATAMATAMMain inputs:Main inputs:
business goalsbusiness goals software specificationssoftware specifications SWA descriptionSWA description
Main Outputs:Main Outputs: list of scenario sensitivity pointslist of scenario sensitivity points trade-off pointstrade-off points risksrisks different approaches to the SWAdifferent approaches to the SWA utility treeutility tree quality attribute questions with the responsesquality attribute questions with the responses
ATAM Process StepsATAM Process Steps
The 4 phases are:The 4 phases are: PresentationPresentation Investigation and AnalysisInvestigation and Analysis TestingTesting
ReportingReporting
ATAM ATAM (Phase 1 & 2)(Phase 1 & 2)
PresentationPresentation Activities:Activities:
Presenting the ATAMPresenting the ATAM Presenting the business requirementsPresenting the business requirements Presenting the architecturePresenting the architecture
Investigation and analysisInvestigation and analysis Activities:Activities:
Identify the architectural approaches suggested Identify the architectural approaches suggested by the architect before they are analyzedby the architect before they are analyzed
Create an attribute utility treeCreate an attribute utility tree Analyze the architectural approachesAnalyze the architectural approaches
ATAM – Utility TreeATAM – Utility Tree
An analytical method that provides a top-down An analytical method that provides a top-down approach for decomposing the quality attributes as approach for decomposing the quality attributes as designated by the ATAM goals.designated by the ATAM goals.
http://www.sei.cmu.edu/architecture/ata_method.html
ATAM ATAM (Phase 3 & 4)(Phase 3 & 4)
TestingTesting ActivitiesActivities
Brainstorming scenariosBrainstorming scenarios Analyzing the architectural approachesAnalyzing the architectural approaches Utilizes the designated high priority scenarios to be used Utilizes the designated high priority scenarios to be used
for test casesfor test cases The goal is targeted to identify any hidden architectural The goal is targeted to identify any hidden architectural
approaches, risks, sensitivity points, and tradeoff pointsapproaches, risks, sensitivity points, and tradeoff points
ReportingReporting ActivitiesActivities
Present the resultsPresent the results Presented to the stakeholders in the form of a final Presented to the stakeholders in the form of a final
analysis reportanalysis report
ATAMATAM
Benefits:Benefits: The quality attributed requirements are clarifiedThe quality attributed requirements are clarified Provides for improved SWA documentation that Provides for improved SWA documentation that
can be used in the foundation of future SWA can be used in the foundation of future SWA decisionsdecisions
Promote communication between the Promote communication between the stakeholders, such as customers, architects, and stakeholders, such as customers, architects, and testers.testers.
Identify system risks early in the solution life-Identify system risks early in the solution life-cyclecycle
Performance Assessment of Software Architecture Performance Assessment of Software Architecture (PASA)(PASA)
Evaluates performance issues for SWA systemsEvaluates performance issues for SWA systems Goal of PASA utilizes performance based scenarios Goal of PASA utilizes performance based scenarios UniqueUnique : the documentation extracted comes from the : the documentation extracted comes from the
developers and source code (since can be performed developers and source code (since can be performed during development cycle)during development cycle)
Only includes interaction with the development team Only includes interaction with the development team which is different from the other techniqueswhich is different from the other techniques
Comparison of Scenario-Based Software Architecture Evaluation Methods
PASA ProcessPASA Process
Starts by setting goalsStarts by setting goals, identifying required , identifying required information, understanding stakeholder information, understanding stakeholder expectations, and describing the method of expectations, and describing the method of approachapproach
Selects key performance scenariosSelects key performance scenarios as elicited as elicited from the developersfrom the developers
Focuses on the architectural styleFocuses on the architectural style or patterns or patterns usedused
The output of the processThe output of the process is a presentation of is a presentation of the results.the results.
Common Goals and Activities of all Methods Common Goals and Activities of all Methods
Common GoalCommon Goal : : evaluate and predict the quality attributes as evaluate and predict the quality attributes as applied to a SWA evaluation analysisapplied to a SWA evaluation analysis
Common ActivitiesCommon Activities : : EvaluatingEvaluating Planning and preparationPlanning and preparation Explanation of SWA approachesExplanation of SWA approaches Elicitation of quality sensitive scenariosElicitation of quality sensitive scenarios Analysis of SWA optionsAnalysis of SWA options Interpretation and presentation of the evaluation Interpretation and presentation of the evaluation
results for final SWA decision makingresults for final SWA decision making
Methods OverallMethods Overall
SAAMSAAM attempts to identify the potential risks to attempts to identify the potential risks to SWA and assess the modifiability. SWA and assess the modifiability.
ALMAALMA attempts to predict the modifiability attempts to predict the modifiability based on risk assessment, support costs, and based on risk assessment, support costs, and SWA comparisons.SWA comparisons.
ATAMATAM analyzes the sensitivity and trade-off analyzes the sensitivity and trade-off points to determine what may prevent realizing points to determine what may prevent realizing the best combination of quality attributes for a the best combination of quality attributes for a given SWA.given SWA.
PASAPASA evaluates the performance risks. evaluates the performance risks.
DifferencesDifferences Participants:Participants:
SAAM and ATAM involve the architects, designers, and the SAAM and ATAM involve the architects, designers, and the end usersend users
ALMA only mostly includes the architect designerALMA only mostly includes the architect designer PASA only includes the developersPASA only includes the developers
SAAM and ATAM are the only twoSAAM and ATAM are the only two methodologies that are methodologies that are close to success when providing details as to the costs associated close to success when providing details as to the costs associated with a SWA evaluation or resource requirementswith a SWA evaluation or resource requirements
ATAM is one of the fewATAM is one of the few if not the only evaluation method that if not the only evaluation method that provides sufficient process steps. The other evaluation methods provides sufficient process steps. The other evaluation methods provide descriptions of the required activities, but do not provide provide descriptions of the required activities, but do not provide enough granular detail.enough granular detail.
All of the methods are influenced by non-technical All of the methods are influenced by non-technical issuesissues, such as stakeholder interests and political factors, but the , such as stakeholder interests and political factors, but the ATAM process is the only method that provides instructions for ATAM process is the only method that provides instructions for detailed guidelines and techniques to manage the social issuesdetailed guidelines and techniques to manage the social issues
Original AssumptionOriginal Assumption
The most common SWA The most common SWA evaluation techniques are similar in the evaluation techniques are similar in the methodology applied, but have evolved to methodology applied, but have evolved to serve different purposes or perspectives serve different purposes or perspectives for evaluating the quality of a specific for evaluating the quality of a specific SWA.SWA.
Final AssumptionFinal Assumption
Slightly reversedSlightly reversed in truth as it has been in truth as it has been determined that the methodologies applied are determined that the methodologies applied are different and unique in their approaches, but different and unique in their approaches, but they all serve the common goal for they all serve the common goal for determining the best SWA option for a given determining the best SWA option for a given set of business scenariosset of business scenarios
No single evaluation techniqueNo single evaluation technique provides a provides a clearly decisive result by specifying a single clearly decisive result by specifying a single SWA selection out of multiple available SWA SWA selection out of multiple available SWA optionsoptions
ConclusionConclusion
The various techniques of SAAM, ALMA, The various techniques of SAAM, ALMA, ATAM, and PASA use different approaches and steps ATAM, and PASA use different approaches and steps in the quest for selecting the best SWA solution. in the quest for selecting the best SWA solution. Ultimately the common goal or purpose of all of the Ultimately the common goal or purpose of all of the techniques is the selection of the SWA that provides techniques is the selection of the SWA that provides maximum usability and satisfies all of the quality maximum usability and satisfies all of the quality attributes as defined by the business goals.attributes as defined by the business goals.
Questions and DiscussionQuestions and Discussion