pres 1117
Post on 20-May-2017
215 Views
Preview:
TRANSCRIPT
RCS1
Testing COTS-based Applications
Randall W. Rice, CSQA, CSTERice Consulting Solutions, LLC
Oklahoma City, OKwww.riceconsulting.com
STC 2003, Salt Lake City, UT
© 2003, Rice Consulting Solutions, LLC
RCS2
What is COTS?
• COTS stands for Commercial Off-the-shelf applications.
• Available from commercial vendors• Sometimes called “vendor” software
RCS3
Variations of COTS – COTS-based, GOTS
• COTS-based applications may be delivered by integrating two or more products.
• GOTS - Government Off-the-shelf applications– typically developed by the technical staff of the
government agency for which it is created. – It is sometimes developed by an external entity,
but with funding and specification from the agency.
RCS4
MOTS
• Modified or modifiable off-the-shelf, or military off-the-shelf
• Typically a COTS product whose source code can be modified.
• The product may be customized by the purchaser, by the vendor, or by another party to meet the requirements of the customer.
RCS5
MOTS• In the military context, MOTS refers to an off-the-
shelf product that is developed or customized by a commercial vendor to respond to specific military requirements.
• Because a MOTS product is adapted for a specific purpose, it can be purchased and used immediately.
• Since MOTS software specifications are written by external sources, government agencies are sometimes leery of these products, because they fear that future changes to the product will not be in their control.
RCS6
What is the Purpose of COTS?
• To reduce risks of internal software development
• To reduce the costs of internal software development
• To increase the speed and reliability of delivering applications
RCS7
What are the Challenges of COTS Applications?
• Selecting the right package• Dealing with vendor
licensing issues• Integrating with existing and
future applications• Testing from the external
perspective
RCS8
Key Point #1 – Buying the Right Product is Foundational.
No matter how much customization and testing is performed, if the COTS
product fails to meet your needs at the outset, the project will fail.
RCS9
What are the Challenges of Testing COTS Applications?
• Unknown structure• Unknown functional requirements• Requires an external, black-box
approach• Release schedules• Technology issues• Test tool issues
RCS10
What are the Risks of Implementing COTS Applications?
• Functional problems• Security issues• Compatibility issues• Integration and interoperability issues• Vendor issues• Procurement and licensing issues• Testing issues
RCS11
A COTS Testing Framework
Marketplace Qualified Adapted Assembled Evolving
Verify Needs &ResearchProducts
Verify Needs &ResearchProducts
Evaluate Products &Plan Tests
Evaluate Products &Plan Tests
Verify & Validate Functionality and Interfaces
Verify & Validate Functionality and Interfaces
Verify:•User needs•Technical needs•Mission needs
Research:•Candidate productsand vendors
Evaluate:•Products•Functional fit•Technical fit•Mission fit•Quality levels•Performance levels
Verify:•Integration plans•Test plans
Validate:•Implementation•Integration•Compatibility
Verify:•Test plans
Validate:•Integration•Interoperability•Middleware
Verify:•Test plans•Release and upgrade
plans
Validate:•Integration•Interoperability•Compatibility•Regression
Statement of Req’s.Evaluation CriteriaEval ScorecardsAcceptance Criteria
Graded ScorecardsTest objectivesTest Plans
Verified PlansTest Results
Verified PlansTest Results
Verified PlansRegression CasesTest Results
RCS12
Roles and Responsibilities for COTS Testing
• Vendors – to provide and support a quality product
• Customers – to select the right product and validate
• Senior Management – to guide the acquisition and testing efforts
• Users – to provide quality requirements and to participate in acceptance testing
RCS13
Roles and Responsibilities for COTS Testing
• In-house Developers – to assist in product integration
• Test Team Leadership – to plan tests and lead the testing process
• Testers – to understand tests and perform them correctly
• QA Analysts and Leaders – to gather data from the acquisition, integration and testing processes for process improvement
RCS14
Test Terminology• Verification
– All QC activities throughout the life cycle that ensure interim deliverables meet specific specifications.
• Validation– The “test phase” of the life cycle which
ensures that the end product (e.g..., software or system) meets user needs.
RCS15
Two Other Views of Quality• Producer View
– Did we build the system or application according to specifications?
– Does it work correctly?• Customer View
– Does the product work correctly for me?– Does the product meet my requirements?– Does it work in my environment?– Does it work with other things I need to use?– Is it easy to use?– Is it a good value?
RCS16
It’s possible for a product to meet producer specifications
and miss customer expectations.
RCS17
Key Point #2 – Testing COTS is a Black-box Process.
Requirements and specifications are great targets for testing, but for COTS products, testing becomes
more focused toward validating that the products meet operational or
mission needs.
RCS18
Specify(Verification)
Evaluate(Verification)
Acquire(Validate)
Implement(Verification)
How the COTS Testing Framework Fits Into the Overall COTS Lifecycle
RCS19
Specify(Verification)
Evaluate(Verification)
Acquire(Validate)
Implement(Verification)
Addresses:User needsMission needsTechnology fit- Integration- Interoperability- CompatibilityConstraints- Environment- User- Project
How the COTS Testing Framework Fits Into the Overall COTS Lifecycle
RCS20
Specify(Verification)
Evaluate(Verification)
Acquire(Validate)
Implement(Verification)
- Needs- Constraints- Acceptance Criteria
How the COTS Testing Framework Fits Into the Overall COTS Lifecycle
RCS21
Specify(Verification)
Evaluate(Verification)
Acquire(Validate)
Implement(Verification)
Addresses:Product FeaturesProject RisksProduct RisksProduct Trade-OffsVendor InformationTest Planning
- Needs- Constraints- Acceptance Criteria
How the COTS Testing Framework Fits Into the Overall COTS Lifecycle
RCS22
Specify(Verification)
Evaluate(Verification)
Acquire(Validate)
Implement(Verification)
- Needs- Constraints- Acceptance Criteria
- COTS Product- Test Strategy- Test Plan- Acceptance Criteria
How the COTS Testing Framework Fits Into the Overall COTS Lifecycle
RCS23
Specify(Verification)
Evaluate(Verification)
Acquire(Validate)
Implement(Verification)
Addresses:Product SelectionVendor ContractingIntegrationTesting (Functional)Testing (Behavioral)
- COTS Product- Test Strategy- Test Plan- Acceptance Criteria
How the COTS Testing Framework Fits Into the Overall COTS Lifecycle
- Needs- Constraints- Acceptance Criteria
RCS24
Specify(Verification)
Evaluate(Verification)
Acquire(Validate)
Implement(Verification)
- Needs- Constraints- Acceptance Criteria
- COTS Product- Test Strategy- Test Plan- Acceptance Criteria
- Tested COTS Product- Baseline- Test Evaluation
How the COTS Testing Framework Fits Into the Overall COTS Lifecycle
RCS25
Specify(Verification)
Evaluate(Verification)
Acquire(Validate)
Implement(Verification)
Addresses:DeploymentTrainingSupportPost-Implementation
Reviews
- Tested COTS Product- Baseline- Test Evaluation
How the COTS Testing Framework Fits Into the Overall COTS Lifecycle
- Needs- Constraints- Acceptance Criteria
- COTS Product- Test Strategy- Test Plan- Acceptance Criteria
- Post-ImplementationEvaluation
- Functional Baseline
RCS26
Specify(Verification)
Evaluate(Verification)
Acquire(Validate)
Implement(Verification)
- Needs- Constraints- Acceptance Criteria
- COTS Product- Test Strategy- Test Plan- Acceptance Criteria
- Tested COTS Product- Functional Baseline- Test Evaluation
- Post-ImplementationEvaluation
- Functional Baseline
How the COTS Testing Framework Fits Into the Overall COTS Lifecycle
RCS27
• The framework is:– Cyclical and repeatable
• New releases will have new or removed features• Test plans will change
– Evolving• Technical and user requirements will evolve.
– Integrated• Products must work with other products.• Test plans and environments must include multiple
applications and interfaces.– Dependent on multiple sources of involvement for
success• COTS testing is not just one team’s responsibility.
The Evolutionary Nature of the COTS Lifecycle and How it Impacts Testing
RCS28
Key Point #3 – Integration and Interoperability Testing Validates the Glue
that Holds Related Products Together.
• The vendor can’t test integration and operability for you.
• These kinds of tests are the customer’s responsibility.
• Failure to test integration and interoperability is to place the project at great risk of failure.
RCS29
Summary
Testing COTS-based applications is a
challenge, but with careful planning and
effective testing strategies tailored for
COTS, the project risks can be reduced.
RCS30
Bio - Randall W. Rice• Over 25 years experience in building and
testing information systems in a variety of industries and technical environments
• Certified Quality Analyst• Certified Software Test Engineer• Conference chair 1995 - 2002, QAI’s
annual software testing conference• Co-author with William E.Perry, Surviving
the Top Ten Challenges of Software Testing and forthcoming book, Testing Dirty Systems (2003).
RCS31
Contact Information
Randall W. Rice, CSQA, CSTERice Consulting Solutions, LLCP.O. Box 891284Oklahoma City, OK 73189Ph: 405-793-7449Fax: 405-793-7454Web site: www.riceconsulting.come-mail: rrice@riceconsulting.com
top related