COTS Project Types CeBASE COTS research

Ye Yang, Jesal Bhuta, Dan Port

COTS Definition

• For the purposes of these classifications, we use the following definition of COTS

“A software product, developed by a third party (who controls its ongoing support and evolution), bought, licensed, or acquired for the purposes of integration into a larger system as an integral part, i.e. that will be delivered as part of the system to the customer of that system (i.e. not a tool), which might or might not allow modification at the source code level, but may include mechanisms for customization, and is bought and used by a significant number of systems developers.”

MBASE Milestone ElementsMilestone Element

Life Cycle Objectives (LCO) Life Cycle Architecture (LCA)

Definition of Operational Concept

Top-level system objectives and scope - Shared vision of expected initiatives and outcomes [COTS strategy for reuse and legacy elements] - System boundary [major COTS

component boundaries within system] - Environment parameters and assumptions

[CBS success factors] - Evolution parameters [major COTS

vendor product availability, upgrade cycles]

Operational concept - Operations and maintenance scenarios and parameters [COTS maintenance] - Organizational life-cycle responsibilities (stakeholders) [major COTS vendors]

Elaboration of system objectives and scope by increment

Elaboration of operational concept by increment [COTS operation and use summary, vendor support strategies, COTS adoption impact on organization]

System Prototype(s)

Exercise key usage scenarios [demonstrate key COTS capabilities and validate interfaces, scope use and value of major COTS products]

Resolve critical risks [COTS selection and evaluation plan]

Exercise range of usage scenarios [complete COTS evaluations, initial COTS configuration and training, initial COTS tailoring]

Resolve major outstanding risks [resolve COTS integration issues and level of use]

Definition of System Requirements

Top-level functions, interfaces, quality attribute levels, including:

- Growth vectors - Priorities - Legacy systems and environments - [Establish key COTS evaluation

attributes and screening parameters] Stakeholders’ concurrence on essentials

[mandated COTS packages, suppliers and vendors, identify negotiable and non-negotiable COTS requirements, COTS evaluation win-conditions and constraints]

Elaboration of functions, interfaces, quality attributes by increment

- Identification of TBDs (to-be-determined items) [mapping of COTS capabilities to requirements]

Stakeholders’ concurrence on their priority concerns [concurrence on COTS imposed requirements and 90% re-worked requirements tradeoffs]

MBASE Milestone Elements (2)Milestone Element

Life Cycle Objectives (LCO) Life Cycle Architecture (LCA)

Definition of System and Software Architecture

Top-level definition of at least one feasible architecture

- Physical and logical elements and relationships, including top-level domain factoring and identification of possible design patterns

- Choices of COTS and reusable software elements [mapping of critical system components to COTS products]

Identification of infeasible architecture options

Choice of architecture and elaboration by increment

- Physical and logical components, connectors, configurations, constraints

- COTS, reuse choices [CBS design (mapping of components and system factors to COTS), COTS package configurations]

- Domain-architecture and architectural style choices [design for COTS glue code and wrappers]

Architecture evolution parameters

Definition of Life-Cycle Plan

Identification of life-cycle stakeholders - Users, customers, developers,

maintainers, interpreters, general public, others [COTS suppliers and vendors]

Identification of life-cycle process model - Top-level stages, increments [COTS

deployment – prototyping, evaluation, schedule, risks impact of schedule and cost, stakeholders, license negotiations]

Top-level WWWWWHH* by stage [COTS product lifecycles (releases, delivery dates, costs, training needs)

Elaboration of WWWWWHH* for Initial Operational Capability (IOC)

- Partial elaboration, identification of key TBDs for later increments

- [COTS upgrades, patches] - [License managament (non-strandard

provisions)] - [COTS users, maintainer, and developer

skill set attributes] - [COTS training plan] - [COTS risks realized; integration,

cultural, economic]

Feasibility Rationale

Risk assessment and mitigation plans [COTS vendor issues – product availability, stability, costs]

Assurance of consistency among elements above

- Via analysis, measurement, prototyping, simulation, etc.

- Business case analysis for requirements, feasible architectures [buy versus build rationale, justification of CBS strategy, CBS tradeoffs]

Assurance of consistency among elements above [feasibility of COTS vendor relationships, COTS maintenance costs, organization readiness to implement COTS]

All major risks resolved or covered by risk management plan [completed license contracts, COTS product alternatives and marketplace issues, COTS organization culture issues, vendor commitments]

COTS Based Systems• A COTS Based System (CBS) is any system that

more than 50% of a systems functional requirements are implemented through use of COTS via glue code, tailoring, and assessment effort.– Glue code is the custom development needed to integrate

COTS packages within an application external to the packages themselves

– Tailoring effort is the activities relating to adapting COTS packages internally for use within an application

– Assessment effort is the suitability evaluation and analysis of the desired attributes of the COTS packages for an application

* The above definitions are compatible COCOTS definitions

CS577 CBS projects

0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

96-97 97-98 98-99 99-00 00-01 01-02

CBS Approaches

• There are a variety of approaches to developing COTS based systems (CBS). – Find and use COTS packages as is (“turnkey”)– Find and adapt COTS packages as needed (“adaptation”)– Use COTS as components in custom built application

(“integration”)

• These approaches differ considerably and driven primarily by the systems shared vision and desired characteristics & priorities (DC&P’s). – Effort distributed between

• Glue code• Assessment• Tailoring

CS577B COTS Effort Distribution

0%

20%

40%

60%

80%

100%

Team7

Team8

Team11

Team20

Team21

Team9

Team15

577ab2001-2002 Projects COTS effort data

Glue Coding

Tailoring

Assessment

0%

20%

40%

60%

80%

100%

Team 3 Team 6 Team 8 Team 14 Team 21

577ab 2000-2001 Project COTS Effort Data

Glue Coding

Tailoring

Assessment

Source: CS577 Project Effort Data2000-2002

CBS Types

• Roughly, a CBS project can be classified as either

• Assessment Intensive • Tailoring Intensive• Application Intensive

• These classifications provide an overview of the potential risks, attributes, and development effort priorities.

– Use for strategic and tactical development planning

Assessment Intensive

• An assessment intensive CBS project will have

– The DC&P’s covered without much modification (or tailoring) by a single COTS framework or through the simple integration of several.

– The primary effort is focused on identifying collection of candidate COTS products and evaluating (i.e. assessing) a feasible set of products whose existing capabilities directly address a desired operational concept.

Tailoring intensive CBS

• A tailoring intensive CBS project is where

– The DC&P’s are covered by a single (or very few) COTS framework(s).

– The primary effort is on adapting a COTS framework whose existing general capabilities can be customized (i.e. tailored) in a feasible way to satisfy the majority of a systems capabilities or operational scenarios.

Application Intensive

• The application intensive CBS project will have

– Some nontrivial DC&P’s covered by COTS, and some that must be covered by custom development.

– The primary effort is to identify COTS packages that can feasibly used as components to address subsets of the required system capabilities.

– Typically there are significant “buy versus build” decisions

• Critical factors are: available COTS products, schedule limitations, budget limitations, and desired current and future capabilities

Effects of CBS Types

• The CBS type effects many aspects of a project– MBASE development guidelines– Glue code, assessment, tailoring effort– Risk factors– Requirements flexibility– System evolution and maintenance – Cost factors

CBS project Type Guidelines

System shared vision Desired characteristics & priorities (DC&P’s)

System shared vision Desired characteristics & priorities (DC&P’s)

Relation of candidate COTS products to DC&P’s

Relation of candidate COTS products to DC&P’s

Standard MBASE Application Guidelines

Standard MBASE Application Guidelines

Application-Intensive CBS Guidelines

Application-Intensive CBS Guidelines

Tailoring-Intensive CBS Guidelines

Tailoring-Intensive CBS Guidelines

Assessment-Intensive CBS Guidelines

Assessment-Intensive CBS Guidelines

III III IV

CBS project Type Selection Matrix

CBS

Requirements Volatility

Developer Resources

Domain Adaptability

Growth Rate

Evolution Rate

Developer Autonomy

Support Stability

None VL VH M H VH VH H Application L H M H H H H Tailoring M M M-H L L H M Assessment H L H VL VL VL L

Evaluate with respect to business case and organization attributes

Assessment Intensive CBS

Ye Yang

Assessment Intensive CBS

• Assumption:

– COTS packages available to cover most of desired system capabilities identified according to system shared vision among stakeholders.

– A considerable number of COTS package combination needs to be assessed against established evaluation criteria.

 

Assessment Intensive CBS-Characteristics 1

• The primary effort is focused on identifying collection of candidate COTS products and evaluating a feasible set of products whose existing capabilities directly address a desired operational concept. – About 60-80% of COTS related effort is

assessment effort, and accounts for 14% of overall effort.

Assessment Intensive CBS- Characteristics 2

• Need high flexibility in requirements and design due to COTS uncertainty– COTS capabilities compose requirements– Requirements cannot be fixed (or “hard”)

before final COTS selection– We use Desired Characteristics & Priorities

(DC&P’s) to guide requirements gathering and flexibility

Assessment Intensive CBS- Characteristics 3

• System architecture is focused on a COTS package configuration model, since the class model, object model and interaction model are not available for most COTS products. – Example: Rework effort for CS577 2001-2002

team8’s SSAD

Assessment Intensive CBS- Characteristics 4

• A business case analysis for each COTS solution is highly recommended for the detailed COTS assessment – COCOTS Model for effort and schedule estimation

• Assessment, glue code, tailoring effort tradeoffs

– Return on Investment analysis • Value, risk, cost tradeoffs

– COTS specific risks and mitigation costs• vendor volatility, upgrade lifecycle, etc.

• Example: Dot-Project infeasibility in CS577A 2001 team 8

Identifying Assessment CBS

• You have an assessment intensive CBS when you have the previously stated characteristics.– High requirement volatility, high domain

adaptability, low developer resources…– Business case shows feasibility of focusing

effort on COTS assessment (rather than glue code, tailoring, or build from scratch)

Assessment Intensive CBS Project Guidelines

Assessment-Intensive CBS

Assessment-Intensive CBS

Assess COTS candidates with respects to DC&P’s

Assess COTS candidates with respects to DC&P’s

Develop glue code; Tailor COTS mix to DC&P’s

Develop glue code; Tailor COTS mix to DC&P’s

Integrate, test, transition IOC

Integrate, test, transition IOC

Evolve as tailored COTS mix

Evolve as tailored COTS mix

Follow Tailoring-Intensive COTS Guideline

Single COTS

Yes

No, COTS Mix Solution

Example

• Project profile– USC Collaborative Services System (team 8

CS577 2001-2002)• DC&P’S

– Project management– File management– Online Discussion Board– Group Calendar

Example- COTS Assessment Steps• Plan: set the evaluation effort scope and provide a baseline

for evaluation process. • COTS Identification: search for candidate COTS packages

based on the set of DC&P’s • Establish evaluation criteria based on functional,

performance, architectural, and financial characteristics of DC&P’s

• Conduct multiple cycle of evaluation to collect evaluation data against the defined evaluation criteria.

• Perform Hand-on Experiments on COTS to verify vendor’s claim

• Compare and contrast the evaluation data and make COTS final selection.

Major risks-1

• Many new non-technical activities introduced into software engineering process– Establishing evaluation criteria, – Planning and executing evaluation procedures,– Collecting and analyzing data from evaluation

process,– Making COTS recommendations, which

require training and experience.

Major Risks-2

• Within the COTS evaluation process:– You may miss some possible COTS candidates.

• Stay as broad as possible when doing the initial search for candidates.

– You might not include all key aspects for establishing the evaluation criteria set.

• Judgment of weight distributions require experienced and knowledgeable people

– Introducing new COTS candidates is likely and require re-planning or contingency planning.

• Limits on schedule and budget must be accounted for

Major Risks-3

• You may not get all the features of certain COTS products as advertised by its vendor.– Detailed assessment beyond literature review or

vendor provided documentation should be performed in the form of hands-on experiments.

Example - Comparison of results from initial assessment and detailed assessment

Evaluation Requirement COTS candidate 1-eProject

COTS candidate 2-

Blackboard

COTS candidate 3-

iPlanet collaborative package

Initial Detailed Initial Detailed Initial Detailed

Users can upload/modify/download/delete

authorized files ** ***** ** ** * None

Project management, including planning group tasks, timelines and tracking

** ***** ** *** None None

Message Board ** ***** ** **** None None

User Interface ** ***** ** ** ** *

Admin Interface ** ***** ** **** ** None

Group Calendar ** ***** ** ** ** *

Detailed analysis provides greater assurance of COTS characteristics with respect to vendor documentation (although at significant effort)

Major Risks-4

• The organization may not have the ability or willingness to accept the impact on its operation due to imposed COTS requirements. – Example:

• CS577 2001-2002 team 8: – COTS: Blackboard

– Required operational change: User access mode

Major Risks-5

• Difficulty in finding suitable time for key personal meetings to discuss COTS assessment may incur significant and unpredictable schedule delay– Based on analysis from student project weekly

effort reports, team interaction and client interaction are the top-2 activities for assessment intensive CBS projects.

Tailoring Intensive CBS

Jesal Bhuta

Tailoring Intensive CBS

• Assumption:– COTS packages have some sort of a

tailoring/development interface using which the COTS package can be adapted to the DC&P’s

– Little or no assessment is required (if required it is already complete) in selecting the COTS package

• CS 577 2001 – 2002 – Team 20 (577 MBASE in BORE)– Team 21 (Quality Management Through BORE)

Tailoring Intensive CBS Characteristic 1

• The primary effort is on tailoring the existing adaptable capabilities of the COTS package to satisfy the majority of a systems core capabilities or operational scenarios.

– CS 577 2001 – 2002 – Team 20 (577 MBASE in BORE)

– Team 21 (Quality Management Through BORE)

Tailoring Intensive CBS Characteristic 2

• Need moderate flexibility in requirements and design due to limitations in adaptability of COTS packages– Capabilities (and adaptability) of the COTS packages

may only partially satisfy the system requirements

– System requirements may need to be adapted to fit within COTS limitations

• Significant source of risk, focus on DC&P’s to manage requirements change

• Example: QA in BORE security levels requirement change

Tailoring Intensive CBS Characteristic 3

• The focus of the system architecture depends upon the type of the COTS package tailoring required in order to satisfy the DC&P’s– GUI Based Tailoring (ex. Spearmint)

• No focus on system architecture

– Parameter Based Tailoring (ex. Windows Media Player)

• Little or no focus on system architecture

– Programmable based Tailoring Interface (ex. Hyperwave)

• Moderate focus on system architecture

Tailoring Intensive CBS – Characteristic 4

• The Tailoring intensive CBS development usually require some amount of time for developer training to get acquainted to the COTS package.– A Learning Curve must be accounted for during

the system development process

Identifying Tailoring CBS

• You have an tailoring intensive CBS when you have the previously stated characteristics.– Business case shows feasibility of buying and

tailoring COTS packages to implement major system capabilities

Tailoring Intensive CBS Project Guide

Tailoring-Intensive CBS

Tailoring-Intensive CBS

Tailor COTS Framework to DC&P’s

Tailor COTS Framework to DC&P’s

Test, Transition,IOC

Test, Transition,IOC

Evolve as Tailoring-Intensive CBS

Evolve as Tailoring-Intensive CBS

Get acquainted with the COTS framework

Get acquainted with the COTS framework

Ensure that the core features to be used, function at an acceptable level

Ensure that the core features to be used, function at an acceptable level

Risk 1

• Version change may result in re-tailoring of COTS product– A constant feedback by the client to the vendor

about the features used and enhancements required may help in ensuring that the features used by the current CBS continue in the future versions of the project

• Ex. “The LOGO element specifies a graphic file to display as a logo. Not supported after Windows Media Player version 6.4.”

Risk 2

• Inadequate vendor support may result in significant project delays– Hiring consultants with experience in using

COTS product (Non-CS577 mitigation) – Ensuring an efficient Phone/Email support at

least during the development phase

Risk 3

• Faulty Vendor claims (COTS “surprises”) may result in Significant Delays

• (Ex) Oracle intermedia claims to search for all unicode characters however the search is not carried as specified

– Pre-testing of core features to be used before beginning of the actual development

Risk 4

• COTS package incompatibilities (integration clash)– In case of multiple COTS packages being used

to implement the DC&P’s there may exist incompatibilities between COTS solutions

• Pre-testing of compatibilities between packages to be used before beginning of the actual development

Risk 5

• Added complexity of un-used COTS capabilities– Adaptable and Tailoring capable COTS

packages usually come with additional complexity and cost that is incurred in the process of making the package tailorable

• Ex Microsoft Office Package

Assessment/Tailoring Intensive CBS Life Cycle and Mbase

Inception

Identify the Type of CBSNarrow the list of Vendors depending on available information

Elaboration

Develop and Test Functional Prototypes to ensure the functioning Core Capabilities

Collect Detailed Information and Evaluate the COTS vendors short-list

Construction 1

Obtain a shared Vision amongst stakeholders.Identify why to go CBS (window of opportunity, independence from evolutionary maintenance, …). Develop and collect Information on available COTS product in the market (RFI). Make a list of products which can satisfy the Core requirements and undergo a primary evaluation them

LCO LCA CCD IOC

Construction 2

Train the Developers to use the COTS product

Implement and test the Core requirements of the system

Implement the low priority requirements of the system

Support personnel must be trained to work with the COTS API Maintaining constant contact with the developer in terms of specifying evolution requirements

Identify the COTS product (s) that shall implement the system

Application Intensive CBS

Dan Port

Application Intensive CBS

• Assumptions:– COTS packages are available to satisfy

significantly valuable subsets of system capabilities or project limitations dictate use (e.g. schedule, skill, or complexity factors)

• Buy vs. build cost-benefit ROI

– Integration overhead is minimal with respect to custom development

– Low assessment and tailoring effort required

 

Assessment Intensive CBS-Characteristics 1

• The primary effort is focused on identifying system components and requirements that may be risky to custom implement and mapping them to a collection of candidate COTS products.– CS577 2001-2002 Team 15 DART’s use of

MySQL and Tomcat

Application Intensive CBS-Characteristics 2

• COTS expected to implement system requirements• Low requirements volatility and adaptability• Significant glue code effort expected• COTS packages used are not specialized to a

particular application domain (generic capabilities)• COTS components dependent on application to

function • Many possible COTS packages may apply• Many evolutionary requirements with respect to

custom components and anticipated COTS features

Identifying Application CBS

• You have an assessment intensive CBS when you have the previously stated characteristics.– Business case shows feasibility of buying

components to implement major system capabilities and focusing effort on COTS integration (glue code)

Application Intensive CBS Project Guidelines

Application-Intensive CBS

Application-Intensive CBS

Tailor COTSTailor COTS

Develop glue code, integrate COTS, custom components

Develop glue code, integrate COTS, custom components

Choose best mix of COTS, custom components

Choose best mix of COTS, custom components

Develop custom components

Develop custom components

Integrate, test, transition IOC

Integrate, test, transition IOC

Assess COTS candidates with respects to DC&P’s

Assess COTS candidates with respects to DC&P’s

Evolve as application-intensive CBS

Evolve as application-intensive CBS

Example

• Project profile– CS577

• Team 15 DART 2001-2002– Tomcat, MySQL

• Team 17 Web-mail 2000-2001– Tomcat

• Team 8 Fulltext Title Search 2000-2001– MySQL, Tomcat, Zope

• Team 9 Student/Staff Directories 2001-2002– Zope

Major risks-1

• Spending too much effort locating appropriate COTS products (too little value for cost)

• Overly optimistic expectation of COTS quality attributes

• COTS package incompatibilities (integration clash)• Added complexity of un-used COTS capabilities• Imposed black box testing of COTS components• Inadequate COTS assessments

Major risks-2

• Overly optimistic COTS package learning curve

• Compromising hard requirements to make use of particular COTS

• COTS “surprises”, capability shortfalls

• Application dependent on COTS vendors (legacy support, upgrades, bugs, etc.)

CBS Type Comparisons

Comparison of CBS project effort sources

0. 00%

5. 00%

10. 00%

15. 00%

20. 00%

25. 00%

1 2 3 4 5 6 7 8 9 10 11 12

Act i vi t i es

%

AssessmentI ntensi ve CBSs (2)Tai l or i ngI ntensi ve CBSs (3)Appl i cat i onI ntensi ve CBSs (2)

1 Team Interaction 7 Feasibility Rationale Document

2 Client Interaction 8 COTS Final Selection

3 COTS Initial Filtering 9 Control and Monitoring

4 Life Cycle Planning 10 COTS Tailoring

5 Project Web-site 11 Transition and Support Planning

6 Training and Preparation 12 Critical Component Implementation

Source: CS577 2001-2002 Project Effort Data

CBS project Type Comparisons CBS

Requirements Flexibility

Glue Code

Domain Specificity

Custom Development

Number of Components

COTS Requirements

Coverage

Evolution Requirements

Degree None Varies None Usually

highly domain

specific for mission

critical apps (exception is general product or

tools)

All Varies None Generally Very High

Application Low Significant Independent of domain

Development needed to implement

custom capabilities

Many possible

Covers effort intensive or risky subsets

High with respect to custom

components and

anticipated COTS features

Tailoring Change non-critical (esp.

UI) requirements

if needed

Some General to domain and

can be refined to specific

organization or mission

Little Few Mostly covered by adapting existing

capabilities

Limited to what is

tailorable

Assessment Change requirements

if needed

None to very little

Generic to domain or COTS is domain specific;

organization will adapt to

COTS

None One or Very Few

Existing capabilities

should satisfy as possible

Generally Very Low, but

included within

assessed products