early application of computer program systems integration ... · catalyst for early computer...
TRANSCRIPT
Copyright © 2004 Computer Sciences Corporation. All rights reserved. 3/16/2007 9:17:48 AM 1
NDIA National Test and Evaluation ConferenceMarch 12-15, 2007
Early Application of Computer Program Systems Integration, Test and Performance MeasurementTom SobieralskiComputer Sciences Corporation
3/16/2007 9:17:48 AM 2
Agenda
• Background
• Systems Integration, Test and Performance Measurement
• Summary
3/16/2007 9:17:48 AM 3
Background
3/16/2007 9:17:48 AM 4
What is Computer Program Systems Integration, Test and Performance Measurement?
• The ability to verify the interfaces, functions and measure performance of two or more computer programs on the target hardware suite and operating environment.
3/16/2007 9:17:48 AM 5
Catalyst for Early Computer Program Systems Integration, Test and Performance Measurement
• Introduction of COTS hardware• Conversion of legacy software to new languages• System complexity with multiple hardware and operating
environments
3/16/2007 9:17:48 AM 6
PB
System IntegrationTesting
Computer ProgramDefinition & Design
Computer ProgramImplementation
System Definition& Design
Computer ProgramTesting
Code
Unit Test
Element CPIntegration & Test
OperationalTests
System Demo &System Qual Test
System T&E,MEIT
EngineeringTest & Evaluation
Comp ProgramDetailed Design
ComputerProgram Design
CP PerformanceReqts Definition
OperationalReqts Definition
SystemReqts Definition
ElementReqts Definition
FB AB Development PhasesSystem Requirements Analysis/DesignSoftware Requirements AnalysisPreliminary DesignDetailed DesignCoding & CSU TestingCSC Integration & TestingCSCI TestingSystem Integration & Testing
DDR
IPR
CR
SDR
PDR
CDR
TPR
MRP
CPAP
PRIMEResponsibility
CSCResponsibility
HDR
Software Development Process
3/16/2007 9:17:48 AM 7
Multi-Processor Environments
• Legacy– AN/UYKs
• COTS Processors– Single Board Computers (SBCs)– Symetric Multi-Processors (SMPs)
AN/UYK-43
COTS ProcessorCabinet
OperatorConsole
UserDisplayConsole
3/16/2007 9:17:48 AM 8
Multi-Operating Environments• Legacy
– Aegis Tactical Executive System (ATES)• COTS
– Concurrent Powermax– Sun Solaris– Red Hawk Linux– LynxOS
3/16/2007 9:17:48 AM 9
Multi-Computer Programming Languages
• Legacy– Compiler Monitor System -2 (CMS-2)
• New– Ada– C/C++– Java
C/C++
3/16/2007 9:17:48 AM 10
Systems Integration, Test and Performance Measurement
3/16/2007 9:17:48 AM 11
Process Improvement
• Incremental Systems Integration, Test and Performance Measurement during the software development process
– Improve System Stability– Early Identification and Resolution of issues and defects– Significantly reduce Engineering Test & Evaluation failure rates– Processes and Procedures QA reviewed and approved
CSC DMEI DE CMMI Level 5
3/16/2007 9:17:48 AM 12
PB
System IntegrationTesting
Computer ProgramDefinition & Design
Computer ProgramImplementation
System Definition& Design
Computer ProgramTesting
Code
Unit Test
Element CPIntegration & Test
OperationalTests
System Demo &System Qual Test
System T&E,MEIT
EngineeringTest & Evaluation
Comp ProgramDetailed Design
ComputerProgram Design
CP PerformanceReqts Definition
OperationalReqts Definition
SystemReqts Definition
ElementReqts Definition
FB AB Development PhasesSystem Requirements Analysis/DesignSoftware Requirements AnalysisPreliminary DesignDetailed DesignCoding & CSU TestingCSC Integration & TestingCSCI TestingSystem Integration & Testing
DDR
IPR
CR
SDR
PDR
CDR
TPR
MRP
CPAP
PRIMEResponsibility
CSCResponsibility
Software Development Process
HDR
Systems Integration,Test and Performance Measurement
3/16/2007 9:17:48 AM 13
Systems Integration
• Validate and maintain operability of system hardware and operating environments
• Verify Computer Program Interfaces
Hardware, OE, Computer Program Interface issues resolved during the computer program development phase
3/16/2007 9:17:48 AM 14
Systems Integration continued• Integration issues tracking and reporting
– Integration Issue resolution prior to computer program delivery
0195271Totals
0163753Build 5
031518Build 4
BUILDS
ACTIVE ISSUES BEING
ADDRESSED
DEFECTS WHICH
BECAME TORs
ISSUES CLOSED
DURING BUILD
BUILD ISSUES FOUND
SYSTEMS INTEGRATION & TEST
3/16/2007 9:17:48 AM 15
Systems Test
• Development and Regression Functional Testing– Documentation
• Plans• Procedures
– Execution• Utilizing two or more computer programs
– Multiple hardware and operating environments
5% to 10% Improvement of Engineering Test & Evaluation
3/16/2007 9:17:48 AM 16
Systems Test continued• Test defect reporting and tracking
– Test defect resolution prior to computer program delivery
S y s t e m I n t e g r a t i o n & T e s t
0
2
4
6
8
10
12
14
C a p a b i l i t y
L O W
M E D I U M
H I G H
3/16/2007 9:17:48 AM 17
Systems Performance Measurement
• CPU and Memory Utilization• Thread and Response Timing• Measurement Tools
– Legacy• ATES Data Recording
– COTS• UNIX TOP and Kernel Trace• Concurrent Nightview• LynxOS Spyker
• Standard scenario– Function and information loading– Repeatable
3/16/2007 9:17:48 AM 18
Systems Performance Measurement continued• Performance Measurement Abnormalities
– CPU and Memory Utilization Increase– CPU Utilization Spikes– Memory Leaks– Slow Thread or Response
3/16/2007 9:17:48 AM 19
Systems Performance Measurement – CPU UtilizationU s e r / S y s e m C P U U t i l i z a t i o n
0 . 0 0
0 . 5 0
1 . 0 0
1 . 5 0
2 . 0 0
2 . 5 0
3 . 0 0
3 . 5 0
T i m e ( A u g - 2 9 - R u n )
S y s
U s e r
U s e r + S y s
3/16/2007 9:17:48 AM 20
Systems Performance Measurement – Memory UtilizationU s e r P r o c e s s M e m o r y
0
1 0
2 0
3 0
4 0
5 0
6 0
7 0
8 0
9 0
1 0 0
9 :4 6 : 3 2
9 :4 8 : 3 7
9 :5 0 : 4 2
9 :5 2 : 4 8
9 :5 4 : 5 3
9 :5 6 : 5 8
9 :5 9 : 0 31 0 : 0 1 :0 91 0 : 0 3 :1 4
1 0 : 0 5 :1 91 0 : 0 7 :2 4
1 0 : 0 9 :3 01 0 : 1 1 :3 5
T im e ( A u g - 2 9 - R u n )
Mem
ory
(Mem
) j a v a
t i m e s e r v i c e s
s y s te m c o n t r o l
3/16/2007 9:17:48 AM 21
Systems Performance Measurement – Memory LeakF r e e M e m o r y
1 7 0
1 8 0
1 9 0
2 0 0
2 1 0
2 2 0
2 3 0
2 4 0
2 5 0
2 6 0
2 7 0
9 :4 6 : 3 2
9 :4 8 : 3 7
9 :5 0 : 4 2
9 :5 2 : 4 8
9 :5 4 : 5 3
9 :5 6 : 5 8
9 :5 9 : 0 3
1 0 : 0 1 :0 9
1 0 : 0 3 :1 4
1 0 : 0 5 :1 9
1 0 : 0 7 :2 4
1 0 : 0 9 :3 0
1 0 : 1 1 :3 5
T im e ( A u g - 2 9 - R u n )
Mem
ory
(Meg
)
P 1
3/16/2007 9:17:48 AM 22
Systems Performance Measurement continued
• Performance Measurement issues resolved before computer program delivery
10% to 20% CPU Utilization Improvement
3/16/2007 9:17:48 AM 23
Summary
• Incremental Systems Integration, Test and Performance Measurement
• Integration and Test issue and defect resolution during the computer program development phase
• Improved Computer Systems Stability and Performance
Cost Effective by identifying and resolving systems issues and defects during the computer program development phase
3/16/2007 9:17:48 AM 24
Tom SobieralskiProject ManagerComputer Sciences Corporation304 West Route 38Moorestown, New Jersey 08057Voice: (856) 252-5052Email: [email protected]