pnsqc 2013 administering qa for large...

All rights reserved. The information contained herein is subject to change without notice. ©2013, Cognizant Technology Solutions

Administering QA for Large Programs

Sreeram Gopalakrishnan PMP, CMST

Sr. Manager – Quality Engineering & Assurance

Cognizant Technology Solutions

PNSQC 2013

| ©2010, Cognizant Technology Solutions Confidential All rights reserved. The information contained herein is subject to change without notice. ©2013, Cognizant Technology Solutions 1

Agenda

o Large program characteristics

o Administering key QA facets

o Nuances & challenges

o Best practices & tools

o On nuances as against fundamentals

o On managerial aspects as against tool technicalities Focus


Large Programs – What Is Different?

Span & Integration Vendor Methodologies

Vendor Dynamics Executive Visibility

• Multi year, incremental capability releases

• Multiple business units impacted

• Discrete projects in early stages, integration

later

• Complexity and cross-impacts grow with

integration

• Different methodologies used by vendors pose

problems during integration

• Different contractual terms guiding vendor

behaviours

• Mutual dependencies between vendor teams

• Competitiveness impacts co-operation

• Challenge of building a badgeless team

• High executive visibility due to:

• Large budgets and spends

• Market expectations and stock price impacts


Organizing For Efficiency, Integration & Independence

Integration

Independence

Efficiency

Grouping functions for maximum

collaboration & productivity

To counter constant flux,

disjointed views,

communication gaps

For objective assessment of

quality & risks, and avoid bias in

reporting

• QA connect with business crucial – not just at operational level, but at tactical / strategic

• Program QA integration with production support / operations vital for feedback and

course corrections.

| ©2010, Cognizant Technology Solutions Confidential All rights reserved. The information contained herein is subject to change without notice. ©2013, Cognizant Technology Solutions

Managing Risks & Dependencies

4

o Get earliest visibility to changes, risks and dependencies

o Influence stakeholders positively to remedy risks and issues Objectives

Interdependency

Meetings

Integrated Weekly

Status Meetings

Risk & Issue

Trackers

Forum to discuss changes to the

solution or schedule

Analyze impacts across all projects or

application areas of the program

IT PMs, Solution Leads, Business

Leads, QA PMs attend

Integrated Program Plan is the prime

tool

Forum to discuss status with Program

Leadership

Program Leadership, IT PMs, Solution

Leads, QA PMs attend

Integrated Weekly Status Report is the

prime tool

Key tool to bring leadership attention

to risks and issues across the program


Illustration: Integrated Program Plan: GANTT View

5

Integrated Program Plan Version - IPP_1.0 Year

Quarter

PeriodProgram Milestones Week 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

Milestone DescriptionE

x

e

StatusStart

Date

End

date 30

-De

c

6-J

an

13

-Ja

n

20

-Ja

n

27

-Ja

n

3-F

eb

10

-Fe

b

17

-Fe

b

24

-Fe

b

3-M

ar

10

-Ma

r

17

-Ma

r

24

-Ma

r

31

-Ma

r

7-A

pr

14

-Ap

r

21

-Ap

r

28

-Ap

r

5-M

ay

12

-Ma

y

19

-Ma

y

26

-Ma

y

2-J

un

9-J

un

16

-Ju

n

23

-Ju

n

30

-Ju

n

Rel#1-March Design Y 6-Jan 18-Jan Rel#1 DESIGNRel#1-March Build (incl FUT) Y 20-Jan 22-Feb Rel#1 BUILDRel#1-March Peformance Execution Y 11-Mar 28-Mar Rel#1 PERFRel#1-March SIT & Regression Testing Y 17-Feb 28-Mar Rel#1 SIT/ REGRESSIONRel#1-March Training Y 4-Mar 15-Mar TRAININGRel#1-March UAT & Regression Testing Y 24-Mar 28-Mar Rel#1 UATRel#1-March Go Live & Warranty Y 29-Mar 26-Apr Rel#1 GO-LIVERel#1-April Design Y 14-Jan 29-Mar Rel#1- APR DESIGN (SAP/ BI)Rel#1-April Build Y 18-Feb 5-Apr Rel#1- APR BUILDRel#1-April FUT Y 1-Apr 9-Apr Rel#1- APR FUTRel#1-April SIT/ Regression Y 8-Apr 16-Apr Rel#1- APR SIT/ REGRESSIONRel#1-April Peformance Execution Y 15-Apr 19-Apr Rel#1- APR PERFRel#1-April UAT & Regression Testing Y 15-Apr 19-Apr Rel#1- APR UAT/ REGRel#1-April Go Live & Warranty Y 21-Apr 17-May Rel#1 - April Go-LIVERel#2-May Design Y 1-Feb 29-Mar Rel#2 - MAY DESIGNRel#2-May Build (incl FUT) Y 22-Feb 17-Apr Rel#2 BUILD/ FUTRel#2-May Performance Execution Y 29-Apr 17-May Rel#2 PERFRel#2-May SIT & Regression Testing Y 10-Apr 10-May Rel#2 SIT/ REGRESSIONRel#2-May Training Y 22-Apr 4-May TRAININGRel#2-May UAT & Regression Testing Y 12-May 17-May Rel#2 UATRel#2-May Go Live & Warranty Y 19-May 20-Jun Rel#2 GO LIVE

2013

1 2 3 4 5

Q1 Q2 Q3

6 7


Illustration: Integrating Program & QA Project Plans

6


Illustration: Integrating Program & QA Project Plans

7


Balance between Standardization & Flexibility

8

o Large programs have several test phases & test types

o Several of them non-standard & specific to the program

o Vendor methodology differences add to the issue

o Confusion results once program integration starts

Standardization of

Test Nomenclature

o Well defined processes with ETVX essential

o Knowing the fine line between process adherence and

excessive rigidity

o Defining tolerance limits and deviation approval mechanisms

Flexibility In

Governance of

Processes


Illustration: Definition of Test Phases & Types

9


Status Reporting

10

o Due to high executive visibility that large programs carry

o Challenges:

• Different details for different levels of audience

• Different timings of individual reports & co-ordination challenge

• Team or vendor biases clouding accuracy

Speed

Accuracy

Consistency

Lack of Bias


Illustration: Status Reporting Cadence

11

Mon Fri Tue Wed Thu Fri

QA WSR

created

QA WSR

review with

QA

Leadership

QA WSR

consolidated

into Program

WSR

+

delta updates

Program WSR

review with

Program Mgr

Next

week’s

QA WSR

created

Month-

end

QA Monthly

Operational

Report

published

Daily Status Reports


Illustration: Program-Level Weekly Status Report

12


Illustration: QA Weekly Status Report

13


Environment Planning & Management

14

• Multiple releases in testing phase at the same

time, hence right code versions to be ensured

• Collaboration between Environment Mgmt and

Release Mgmt functions to be well defined:

o RM accountable & responsible (OR)

o RM accountable, EM responsible

• Environments shared across multiple releases

within a program, or with other projects

• Code version / data conflicts on shared

environments

• Environment reservation calendar as a tool to

manage conflicts

• Synchronized data across applications in a

landscape for SIT / UAT is a real challenge

• Defining the right data to fulfill test scenarios is

the critical activity

o Data lead staffed with the best resource

o Integrated data planning sessions

• Data management tools to automate

mechanical tasks

• Not factoring in smoke test in test schedules is

an oft repeated mistake

• Smoke test of application environments

separate from smoke test of integration

• Different layers of the solution owned by

different teams makes fixes time consuming

• War-room for smoke test is a successful option

Integration with Release Mgmt Environment Reservation

Data Synchronization & Refresh Smoke Test

Single-most biggest dependency for QA is generally the readiness of test environments


Illustration: Environment Reservation Calendar

15


Illustration: Data Refresh Plan

16


Defect Management

17

o Triage during SIT

o Defining defect turnaround SLAs

o Usage of Severity and Priority Challenges

Triage during SIT Defect Turnaround

SLAs

Usage of Severity &

Priority

Large number of teams makes triage

unwieldy and inefficient

Focused triage as a best practice

o Application / business area-wise

triage meetings

o Dedicated defect managers

Fix turnaround impacts QA’s execution

productivity – thus a vital factor

SLAs may have contractual

connotations to dev vendors, hence

Turnaround “guidelines” coupled with

defect aging KPI’s can be an effective

method

Standard definitions

o Severity – business criticality

o Priority – urgency to tester

Confusion over which of the two

should drive defect resolution.

Recommendation:

o Priority – in early stages

o Severity – in later stages


Regression Approach

18

Regression testing is testing done to ensure

code has not regressed due to changes

Scope of

Regression

Ownership of

Regression

Methodology of

Regression

Coverage of functionality of previous

releases versus functionality of current

release

Impact-based (focused) regression

versus broad-based critical scenario

regression

Dedicated regression team

o Greater focus & higher cost

o Knowledge transition challenge

SIT team doing regression

o Cost efficient but lesser focus

o Knowledge transition challenge

Number of regression cycles planned

Overlap with SIT / Performance / UAT

cycles

Coverage in each cycle

o Core (critical) scenarios

o Impact or defect based scenarios

strategy to be made specific w.r.t coverage is a balancing act


Illustration: Regression Methodology

19


Integrating Performance Testing

20

o Performance does not get attention till it comes in the critical path

o Early integration of Performance Testing is an issue

o One-Face QA model as a best practice

• QA manager as a SPoC for all testing tracks – SIT, Regression,

Performance, UAT – regardless of separate ownership by vendors /

teams

Integrating

Performance


Application & End-to-End Performance Requirements

21

o Perception of performance being secondary to functional

o Rudimentary requirements gathering processes

o Lack of historical data on transactions / user volumes

o Solution architects / business resort to trail-and-error on SLAs

o E2E performance requirements need collaboration across multiple IT

teams

o Performance Requirements Workshop as a best practice

• Soln Architect, Business and Dev leads present

• Systematic, framework driven approach

• Production performance monitoring to provide supporting data

Performance

Requirements Challenges


Questions ? [email protected]


Thank You

pnsqc 2013 administering qa for large...

Documents