dev-to-dev: the esri development process - qa and testing · about this talkabout this talk this...

Dev-to-Dev: The ESRI Development Process -QA and Testing

Krishna GummuluriKrishna Gummuluri

About this talkAbout this talkAbout this talkAbout this talk

This talk is:This talk is:•• An opportunity to share and discuss ESRI’s QA process An opportunity to share and discuss ESRI’s QA process

and testing strategyand testing strategyand testing strategyand testing strategy•• An opportunity to learn what ESRI testers do in order to An opportunity to learn what ESRI testers do in order to

assure high quality releasesassure high quality releasesff•• An opportunity to share specific examplesAn opportunity to share specific examples

This is not:This is not:This is not:This is not:•• A recommendation of any specific test tool and test A recommendation of any specific test tool and test

frameworkframework•• A forum to discuss specific defectsA forum to discuss specific defects

AgendaAgendaAgendaAgenda

•• QA processQA process•• Overview of test pointsOverview of test points•• Testing activitiesTesting activities•• Testing activitiesTesting activities•• Test design and automation strategyTest design and automation strategy•• Case studiesCase studies

–– Test designsTest designs–– Performance, durability and stability (PDS) testingPerformance, durability and stability (PDS) testing

•• Q&AQ&A•• Q&AQ&A

QA processQA processQA processQA process

•• Planning and envisioning phasePlanning and envisioning phase–– Test planningTest planning–– Test estimationTest estimation

•• Development phaseDevelopment phase–– Product teamProduct team–– Documentation teamDocumentation team–– I18N and L10N teamI18N and L10N team–– Holistic team (“Testing like the customer”)Holistic team (“Testing like the customer”)–– Release teamRelease team

•• Beta Phase Beta Phase –– Beta certificationBeta certification

QA process (contd )QA process (contd )QA process (contd.)QA process (contd.)

•• Release PhaseRelease Phase–– Release certification testingRelease certification testing–– Certification on multiple environmentsCertification on multiple environmentsC pC p–– Documentation and helpDocumentation and help

•• Quick Fix Engineering (QFE )Quick Fix Engineering (QFE )–– Hot Fix releaseHot Fix release–– Patch releasePatch releasePatch releasePatch release–– SP releaseSP release

ArcGIS functional test pointsArcGIS functional test pointsArcGIS functional test pointsArcGIS functional test points

M G S C MMAP

GLOB

SCEN

CATAL

REST

SOAP

DCOM

JS

FLEX

SL

MOBIL API

ArcGIS Desktop ArcGIS Server

IMS

Explorer

E E OG Services

TPM X LE

API

ArcGIS Desktop ArcGIS Server S

Arc S Core ArcObjects

Exte

GP

CAD

NetworkcObj

DK

Core ArcObjects nsions

Network

Track.

…

Schem.

ArcSDE

DBMS DBMSDBMS

Setups and licensing test pointsSetups and licensing test pointsSetups and licensing test pointsSetups and licensing test points

•• Over 70 SetupsOver 70 Setups

•• Environment (e g Platform Support)Environment (e g Platform Support)•• Environment (e.g. Platform Support)Environment (e.g. Platform Support)–– Several flavors of Windows (x2 for 32 & 64 bit support)Several flavors of Windows (x2 for 32 & 64 bit support)–– Solaris 9 & 10Solaris 9 & 10–– Linux Linux -- SUSE & Redhat, SUSE & Redhat, –– plus HP & IBM for SDEplus HP & IBM for SDE–– Application Servers (IIS, Apache)Application Servers (IIS, Apache)Application Servers (IIS, Apache)Application Servers (IIS, Apache)–– Multiple DBMS serversMultiple DBMS servers–– ……

•• Multiple levels of licensingMultiple levels of licensing

Testing activitiesTesting activitiesTesting activitiesTesting activities

Unit Test Generation

Test Design

ComponentPDS

Testing Test Execution

Defect Analysis

FunctionalHolistic

Defect Reporting

I18N/L10NReporting

Test designTest designTest designTest design

•• What to test?What to test?–– Anticipating customer needs and expectationsAnticipating customer needs and expectations–– Primary customer scenarios that will drive the testing effortPrimary customer scenarios that will drive the testing efforty gy g

•• What kind of tests to run? What kind of tests to run?

•• Which type of testing?Which type of testing?

•• What is the automation strategy?What is the automation strategy?

Test automation strategyTest automation strategyTest automation strategyTest automation strategy

•• To automate or not to automate?To automate or not to automate?

•• What to automate?What to automate?•• What to automate?What to automate?

•• What automation tool to use?What automation tool to use?

•• When to automate?When to automate?

Case StudyCase Study –– Test design sampleTest design sampleCase Study Case Study Test design sampleTest design sample

•• A sample test design MatrixA sample test design MatrixLayer_Type Composite_Type Vector Raster Layer_Props Layer_Spat_Ref Layer_DS DS_Location

GroupLayer CadastralFabric None None Relates World FGDB3 Local

FeatureLayer Topology V1 R1 Table_Joins Local SDE Remote

CompositeLayer GeometricNetworK V2 R2Definition‐Query PGDB

NetworkAnalyst V3 R3 HTML_Popup URL

NetworkDataset V4 R4 Time CAD

MosaicDataset V5 Labels Shapefile

Basemap Fields Coveragep g

Annotation None FileBasedRaster

IMSMapLayer All

MapServerLayer

WMSLayer

WCS

None

SchematicSchematic

Case StudyCase Study –– Test design sample (Test design sample (contdcontd))Case Study Case Study Test design sample (Test design sample (contdcontd))

•• Raster Data Definition (compressed formats, Raster Data Definition (compressed formats, uncompressed formats) uncompressed formats)

–– R1 R1 File based Raster File based Raster –– R2R2 GDB Based (SDE, FGDB, PGDB) GDB Based (SDE, FGDB, PGDB) –– R3 R3 RastercatalogRastercatalog, , GdbGdb RasterCatalogRasterCatalog

R4R4 Optimized Raster Tin TerrainOptimized Raster Tin Terrain–– R4 R4 Optimized Raster, Tin, TerrainOptimized Raster, Tin, Terrain•• Vector Data Definition Vector Data Definition

–– V1V1 Basic1 (L, Pt, Polygon), Basic1 (L, Pt, Polygon), NetCDFNetCDF–– V2 V2 Basic2 (Basic1 +Text + Dimension) + tool layers Basic2 (Basic1 +Text + Dimension) + tool layers –– V3 V3 Basic2 + Feature Dataset, Basic2 + Feature Dataset, DataDrivenDataDriven, holes , holes

V4V4 Basic2 + RC MultiBasic2 + RC Multi--patch Query Layerpatch Query Layer–– V4 V4 Basic2 + RC, MultiBasic2 + RC, Multi--patch, Query Layerpatch, Query Layer–– V5 V5 Basic2 + MultiBasic2 + Multi--part part

•• Compressed and uncompressed FGDBCompressed and uncompressed FGDB

Case StudyCase Study -- ArcGIS Server PDS testingArcGIS Server PDS testingCase Study Case Study -- ArcGIS Server PDS testing ArcGIS Server PDS testing

Goal is to identify defects and bottlenecks related toGoal is to identify defects and bottlenecks related toGoal is to identify defects and bottlenecks related to Goal is to identify defects and bottlenecks related to performance, durability and scalability of ArcGIS Serverperformance, durability and scalability of ArcGIS Server.

Product Team 1

Product Team 2

Product Team 3

• Define Requirements• Design tests• Run them locally• Identify and upload PDS testsIdentify and upload PDS tests

PDS• Establish PDS baselines • Maintain PDS test framework• Maintain hardwarePDS

Team• Maintain hardware• Run PDS tests regularly • Analyze results• Publish weekly regression results


•• Web testsWeb tests–– Check functionalityCheck functionality–– Establish unloaded transaction timeEstablish unloaded transaction time

•• Load TestsLoad Tests–– Step load testsStep load tests

CC–– Constant load testsConstant load tests•• Outcome of PDS testsOutcome of PDS tests

–– Weekly regression analysisWeekly regression analysisWeekly regression analysisWeekly regression analysis–– Baseline throughput and transaction timesBaseline throughput and transaction times–– Identify bottleneckIdentify bottleneck

Id tif l kId tif l k–– Identify leaksIdentify leaks–– Identify resource contentionsIdentify resource contentions


•• Sample test architectureSample test architecture

VS 2008 Network Load BalancerTeam Test

Repository+

Report Server

Repository+

Report Server

WS WS WS

Report ServerReport Server

PDS FrameworkPDS Framework SOM SOM

SOC SOC SOC SOC

ArcGIS ServerArcGIS ServerLoad Balanced ArcGIS ServerLoad Balanced ArcGIS Server


Sample PDS report:Sample PDS report:Sample PDS report:Sample PDS report:

AcknowledgementsAcknowledgementsAcknowledgementsAcknowledgements

•• Desktop and EngineDesktop and Engine•• GPGP•• ServerServer•• ServerServer•• GDBGDB•• ArcSDEArcSDE•• HolisticHolistic•• QAQA

L li iL li i•• LocalizationLocalization•• ReleaseRelease

Q&AQ&AQ&AQ&A

dev-to-dev: the esri development process - qa and testing · about this talkabout this talk this...

Documents