presenter: david a. watt copyright protected material david a watt, 2010
TRANSCRIPT
DatabaseServer
LAN
PeripheralSystems
Client B
Client A
OtherComponents
ApplicationServer
Infrastructure Integration Testing:The Oft Ignored Testing Type
Copyright Protected MaterialDavid A Watt, 2010 2
Traditional Development MethodTraditional Development Method
Infrastructure Integration Testing:The Oft Ignored Testing Type
Copyright Protected MaterialDavid A Watt, 2010 3
Systems are Getting More ComplexSystems are Getting More Complex
From a Blog by Lori MacVittie on DevCentral
Load
Bal
ance
rs
Stor
age
Fabr
ic N
etw
ork
Infrastructure Integration Testing:The Oft Ignored Testing Type
Copyright Protected MaterialDavid A Watt, 2010 4
Enhanced Development MethodologiesEnhanced Development Methodologies
C
T
A
D
Spiral Model
V-ModelREQMTS
CODE
ACCEPTANCE
TEST
SYSTEM
TEST
INTEGRATION
TEST
UNIT TEST
DEVELOP
DESIGN
Development
Test
Infrastructure Integration Testing:The Oft Ignored Testing Type
Copyright Protected MaterialDavid A Watt, 2010 5
A B A B A B A BC C C
D DE
F
It 0 It 1 It 2 It 3 It 4Agile
(Agile Testing; Crispin, Gregory)
IT Systems for Mid to Large Sized CorporationsIT Systems for Mid to Large Sized Corporations
User PopulationUser Population
Training
EmplPortal
Procurement
Security MRP
End User
Support
BillingPoint
of Sale
HRFinanc
e
Infrastructure Integration Testing:The Oft Ignored Testing Type
Copyright Protected MaterialDavid A Watt, 2010 6
Common Infrastructure ElementsCommon Infrastructure ElementsDesktop / Laptop Configuration Variations
User Authentication and Digital Rights Management
Database Services
Web Services – Presentation, Delivery and Authentication
Load Balancing
Computer Hosting Platforms and Operating Systems (OS)
Storage Infrastructure
Network Infrastructure
Security Policies and Operations
People, Processes, and User Support
Operational Management and Oversight
Infrastructure Integration Testing:The Oft Ignored Testing Type
Copyright Protected MaterialDavid A Watt, 2010 7
Statement of the Problem:Statement of the Problem:Traditional system testing techniques teach and illustrate testing the system as a “complete” but stand-alone system.For some deliverable systems this is ok. But for IT solutions deployed into a medium to large scale corporate environment, this is naïve and insufficient.After all traditional system integration testing has been completed, another layer of testing is required.Testing in this layer verifies that the solution integrates with the rest of the IT infrastructure elements and is coined “Infrastructure Integration Testing”.
INFRASTRUCTURE
INTEGRATION
TEST
REQMTS
CODE
ACCEPTANCE
TEST
SYSTEM
TEST
INTEGRATION
TEST
UNIT TEST
DEVELOP
DESIGN
Development
Test
Infrastructure Integration Testing:The Oft Ignored Testing Type
Copyright Protected MaterialDavid A Watt, 2010 8
IT Infrastructure ElementsIT Infrastructure Elements
HVAC Facility Monitoring Operations PersonnelPower Floor Space Telecommunications
Gateways LAN / WAN VPN WirelessFirewalls Routers Switches IDS
DNS, DHCP Proxy Svrs VOIPExtranet Encryption Network Mgmt System QOS
Terminal Services / Citrix Virtual Machine Svcs SOA ArchitectureInternational Remote Access Systems
Windows Forest & AD LDAP Siteminder Svcs F5 Load BalancersFederated AD
Tiered Storage Solutions Message Store Disaster RecoveryStorage Management Systems
Disk Array Fabric Controllers On-line BackupStorage Area Net Network Attached Tape Silos
Operating Systems: Multi-vendor Resource Management File System: NFS, DFS, CIFS
Mainframe VM-Based Svr Open Sys PeripheralsMP Servers Intel Svrs Blade Svrs Backup Sys
Build Docs Help Desk, & TrainingStandard Operating Instrs Application & End User Performance
Configuration Mgmt Sys Patch / Upgrade Mgmt Sys Provisioning & Health MonSoftware Distribution Sys
Database Mgmt Systems Web Presentation ServicesWeb Delivery Services Development Svcs
E-Mail Svcs Document File & Share Incident MgmtBlackberry Office Communications Svcs Collaboration
Compliance OS SettingsSecurity Policy EPO PGP / PKI Enhanced Sec Vulnerability Mgmt
Business Reporting Svcs Department Applications Standard Products & ArchEnterprise Applications
Enterprise Applications Suites Data Access Rights MgmtApplication Authentication Application Security
VPN AccessDesktops Terminals Virus MgmtLaptops Mobile & Wireless Devices
PortalsOperating Systems Personal FirewallOffice Apps End User Apps Disability Support Tools
Encryption
FacilityInfrastr
ClientServices
OpsProcesses
Middleware
Infra Apps
Security
Authenti-cation Svcs
StorageInfrastr
ComputingInfrastr
NetworkInfrastr
ApplicationServices
Questions To Be AnsweredQuestions To Be AnsweredDo your solution’s parameters integrate with the parameters from other solutions? How does your solution impact performance of the rest of the environment?Does your solution comply with corporate security settings?Does your solution integrate with load balance, firewall, and gateway systems?Does your solution fit into production security certificate authority? Are there enough user certificates available? Is the version and configuration of your solution compatible with that of shared infrastructure? Are there adequate installation, operations, and support documents available?Has the help desk and operations support staff become acquainted with your solution?Does your solution require anything to be loaded on the client system? Is it compatible with all versions and configurations? Is your solution compatible with health and performance monitoring tools?Do the backup and disaster recovery requirements of your system integrate with the existing solutions?
Infrastructure Integration Testing:The Oft Ignored Testing Type
Copyright Protected MaterialDavid A Watt, 2010 14
Types of Testing to be PerformedTypes of Testing to be PerformedFunctional regression testing of your solution, infrastructure elements, and other solutionsCapacity, load, and performance testing of your solutionPerformance regression testing of the remaining infrastructure and other solutionsPerformance baseline testingFunctional testing of operational monitoring tools and processesRegression testing of your solution and others when changes are made to the infrastructure or other solutionsValidation testing of operational SOIs, training materials, help desk documentation, and other user support processesAd Hoc testing and use by support staffVulnerability scanning and remediationProblem isolation and remediationOn-going capacity modeling
Infrastructure Integration Testing:The Oft Ignored Testing Type
Copyright Protected MaterialDavid A Watt, 2010 15
The Challenge of “IIT” The Challenge of “IIT”
Infrastructure Integration Testing (IIT) may be a great idea but ……
Not having good answers to this question often leads to one of two deadly sins:
1. Perform Infrastructure Integration Testing in the production environment
OR
2. Test by putting a solution into production and letting the users find the
“bugs”
Q: How and where do you perform IIT?
Infrastructure Integration Testing:The Oft Ignored Testing Type
Copyright Protected MaterialDavid A Watt, 2010 16
Alternate ApproachAlternate Approach
Build an Enterprise Test Bed:
“An accumulation of separate and distinct test environments into an integrated and well managed test bed, which is isolated from the production environment and where functionality, interoperability, and performance testing can be performed on many solutions simultaneously”.
Infrastructure Integration Testing:The Oft Ignored Testing Type
Copyright Protected MaterialDavid A Watt, 2010 17
Network InfrastructureComputing Infrastructure
Enterprise Test BedEnterprise Test Bed
Storage InfrastructureAuthentication Services
Scale model of production solutionsPhysically and logically isolated from production environmentAccessible from anywhere on the corporate intranetReciprocal testing: Impact of
System Under Test against the rest of the test bedTest bed against System Under Test
Security & ComplianceInfrastructure Applications
MiddlewareOperational ProcessesApplication Services
Client Services
TestArticle
The Realities of an Enterprise Test BedThe Realities of an Enterprise Test Bed
Logistical
Political
Financial
Operations Management
Infrastructure Integration Testing:The Oft Ignored Testing Type
Copyright Protected MaterialDavid A Watt, 2010 19
Test Bed ManagementTest Bed ManagementExperience has shown that test beds must be managed using developed and enforced processes in order for the value of the investment to be maximized.
Consistency in following processes is the key to success.
Infrastructure Integration Testing:The Oft Ignored Testing Type
Copyright Protected MaterialDavid A Watt, 2010 2020
Network InfrastructureComputing Infrastructure
Agile and the Enterprise Test Bed Agile and the Enterprise Test Bed
Storage InfrastructureAuthentication ServicesSecurity & Compliance
Infrastructure ApplicationsMiddleware
Operational ProcessesApplication Services
Client Services
Regression Tests – on both sprint code as well as infrastructure
Results of the sprint iterations are tested and pronounced “Ready for Production use” in the ETB and held there until a “bundle” of changes are ready for insertion into production use. New code is then implemented into the production environment and enabled for use.(ETB is used as staging area)
ConclusionsConclusionsTreating new solutions as “stand alone” systems needs to change
For IT solutions, “Infrastructure Integration Testing” needs to be performed
IIT verifies that the new solution integrates successfully the rest of the computing environment
IIT testing involves such testing as compatibility, functional, performance, and process validation.
Existing test environments can be consolidated into a centrally managed Enterprise Test Bed.
Test Bed Management techniques must be employed.
Infrastructure Integration Testing:The Oft Ignored Testing Type
Copyright Protected MaterialDavid A Watt, 2010 22