tulkita test training series - chicago quality assurance

© Tulkita Technologies Inc., 2013. All Rights Reserved.

Tulkita Test Training Series Measuring QA Value

© Tulkita Technologies Inc., 2013. All Rights Reserved. 2

Introducing your Instructor

Meet Jamie Campbell: •  Management Consultant – IT Strategy, Governance & Process Op<miza<on •  Test Leader – launched and managed 4000+ person, global tes<ng prac<ce •  Test Strategist – designed and op<mized over 120 test prac<ces, globally •  Test Architect – assessments, roadmaps, opera<ons, governance & metrics •  IT Sr. Leader – VP of Informa<on Technology, VP of Shared Services •  Founder, Tulkita Technologies Inc. – Consul<ng, Solu<ons & Training

Relevant Special3es: •  QA Prac<ce Setup and Fine-‐Tuning •  Process Ra<onaliza<on and Op<miza<on •  IT Metrics & Repor<ng •  QA Efficiency and Op<miza<on •  QA Calibra<on & Organiza<onal Alignment •  Communica<on Coach •  Lean IT Prac<<oner


Today’s Agenda

1.   Demys3fying your Metrics (Lecture) •  The Problem •  Approach •  Building a viable Measurement Framework •  Automa<ng your Metrics

2.   Building your Measurement Program

•  Iden<fy your Stakeholders •  Iden<fy Stakeholder Ques<ons •  Iden<fy Answers •  Iden<fy Measures/ Reports

3.   Stakeholder Communica3on •  Communica<on Essen<als 101 •  Establishing a fail-‐safe Communica<on Plan •  Managing and ‘Bea<ng’ your Stakeholders


Key Thoughts: •  Test Professionals rarely have the right metrics to quan<fy/ qualify value •  Metric collec<on and repor<ng are typically onerous ac<vi<es •  QA Metrics generally focus only on test execu<on •  Test Management Tools only have ~40% of the metrics you require •  TCoE Measurement should focus on strategic informa<on as well as tac<cal •  Only ~10% of all reports generated by a QA Lead are ever reviewed

Focus Areas: •  Choosing the right metrics •  Learn how to properly design & build a Test Measurement Framework •  Establish repor<ng that is meaningful to ALL stakeholders •  Use metrics to not only manage tes<ng, but act as a barometer reading for IT •  Show your value

4

Objective Why is this subject important?


Defining Metrics

Defect Removal Effectiveness

# Test Case/ Cycle

# of Critical Defects

ROI

Pass/ Fail Rates

Defect Aging

Productivity

Schedule Variance

Cost Variance

Do you know all of these metrics? Are they all test metrics?

5


Guiding Principles: •  Metrics can represent a point of <me •  Metrics can represent change over <me •  Metrics can be simple •  Metrics help to make decisions today •  Metrics help to make decisions for tomorrow •  Metrics are cri<cal to come, while meaningless to others

Starting Point You require a broader perspective of QA metrics.

Changing your mindset and understanding the power of metrics will propel your career forward, guaranteed.

6


Approach Use an iterative approach for identifying, designing and implementing metrics.

Iden3fy Stakeholders

Stakeholder Ques3ons

Stakeholder Answers

Metris Iden3fica3on

Measurement & Report Design

Test Prac3ce Measurement Framework & Dashboard

Collec3on & Repor3ng Infrastructure

Data Sources (defects, plans, 3me tracking)

7


1. Identify Stakeholders Who are the stakeholders that you are required to communicate with? Find out.

Stakeholders Questions Answers

CIO

VP - Quality

Director

Test Manager

Development Manager

Business Owner

Audit

Test Lead

Test Analyst

Sourcing Partner

Etc., Etc., Etc.

Do you know how to report to all of these individuals? What they want? What they need?

8


2. Stakeholder Q&A What questions are being asked by your stakeholders? Conduct Q&A workshops.

This step is cri>cally important! Doesn’t it make sense to prepare metrics that answer your stakeholder ques>ons?

9


3. Metrics Identification Select Metrics that address & answer your stakeholder questions.

10

Customer & IT Stakeholder Metrics

Test Practice Metrics

Project Metrics

Test Operations

Test Optimization

Test Planning

Test Quality

Test Status

Metrics Collection & Reporting

Data Source Systems

Stakeholder Requirements/Questions

to be Answered

Data Harvesting & Metric Generation

Metrics Definition


4. Measurement & Report Design Choose/ Build metrics that best align to your stakeholder requirements.

11

Test Opera3ons: •  Requirements Vola3lity •  Test Down3me •  Defects – Assigned To •  Cost of Poor Quality •  Cost of Quality

•  Defects – Time to Close (Aging) •  Defects – Arrival Time •  Defects – Closure Trend •  Schedule Performance Index (SPI) •  Cost per Test Case

Sample Test Prac3ce Measurements

•  Defects – Root Cause Trend •  Tes3ng Schedule Variance •  Test Program – Project Status •  Cost Performance Index (CPI) •  Cost of Produc3on Defects

Test Op3miza3on: •  Test Resource Produc3vity •  Test Automa3on Coverage •  Defects – Removal Eff.

•  Test Script Exec. Time •  Test Produc3vity Improvement

•  Test Regression Scope •  Test Effec3veness

Test Planning: •  Rqmts Coverage •  Test Case Complexity

•  Requirements Traceability •  Test Case Cri3cality

•  Requirements Review & Approval •  Test Execu3on Priori3za3on

Test Quality: •  Test Case Review Status •  Test Pass/ Fail Rate •  Defects -‐ # of Defects/ Scripts •  Defects – # Found in Prod.

•  Defects -‐ # of Open •  Defects – Priority/ Severity •  Defects -‐ # of Defects to Applica3on Size •  Defects -‐ # Post Go-‐Live

•  Defects Re-‐Opened •  Defects Ready for Retest •  Defects – Density •  Stage Containment Effec3veness

Test Status: •  Rqmts Status •  Test Case Build Trend

•  Test Exec. Schedule Trending •  Project Risk Analysis

•  Test Case & Script Sign-‐Off Status •  Test Data Prepara3on Effec3veness


5. Data Harvesting & Collection Measurement & Analysis is achievable only if you have the supporting data!

12


Executive Reporting

Customer Reporting

Project Reporting

Test Operations Test Optimization

Test Planning

Test Quality Met

rics

Rep

ortin

g

Automated Collection & Aggregation of Source Data

Col

lect

ion

Test Mgmt Tool

Time Tracking

Rqmts Management

Project Planning Other

Metric A For each metric, you must identify

the source systems/ data

required to calculate the

metric Test Status


6. Automated Reporting If you are not careful, test metrics can consume your bandwidth & resources.

13


Data Source Systems

Field Defini3ons

Tool Configura3on

Input Standards

Automated Data Quality Repor3ng

Tool Integra3on

Enterprise Metrics Collec3on Tool

Real-‐Time API to Source Systems

Metric Back-‐Up/ Archiving

Automated Extrac3on/ ETL Capability

Metrics Genera3on

Metric Categoriza3on

Metric Calcula3on – by Stakeholder, Category

Point of Time & Trending Analysis

Traceability Measurement

Repor3ng & Analy3cs

Customized Repor3ng

Automated Popula3on & Distribu3on

Excel/ Powerpoint Repor3ng

Scorecard Repor3ng – iden3fy risks/ issue areas

Remember, you are only as good as the data being entered – “Garbage In, Garbage out”…

Tulkita OPTIMEASURETM


7. Effective Reporting Work with your stakeholders to build the right reports, ‘canned’ reports seldom work.

14


Develop a report catalogue to help you review and select the right reports with your stakeholders


Root Cause Analysis: •  Iden<fy sources of defects •  Report to management & sr. leadership •  One of the most important sources of intelligence in IT •  Proac<vely review and analyze remedia<on steps with IT stakeholders •  Configure your test tool to consistently capture sources

Trending: •  Trending is simply showing metrics over pre-‐set <me intervals •  Allows stakeholders to see how measurements are changing •  90% of metrics displayed to IT Sr. Leadership should focus on trends •  Always determine ‘point of <me’ vs. ‘trend over <me’

Root Cause & Trending Arguably the most important metrics you can capture.

15


Business and Customer: •  Are you delivering a quality product? •  Are you mee<ng my <melines? •  Did you understand my requirements? •  How much is your QA cos<ng me?

IT Sr. Leadership: •  Are you mee<ng your budget? •  Are you working to schedule? •  How have you tried to cut costs? •  How have you improved efficiency? •  What are the causes of delays & poor quality? •  Is quality improving? •  What is your status?

Showing Your Value Determine what value means to each of your stakeholders.

16

Test Practice

Customer IT Operations Test Teams


Your Checklist: •  There is no one right answer •  Metrics are interpre<ve – everyone sees them differently •  Senior Leadership need metrics – you cannot avoid this •  Test Management Tools only have a frac<on of what you need •  Know what your stakeholders want answered •  Metrics oeen fail due to poor communica<on

In Prac3ce:

•  Define a metrics improvement plan & approach •  Conduct stakeholder workshops •  Create a Q&A Inventory •  Setup metrics catalogue & reports •  Develop suppor<ng infrastructure & opera<onalize

Takeaways Key success factors & implementing in practice.

17


Concluding Thoughts Hours, Days, Months are spent producing ineffective/ irrelevant metrics... Be smart!

18

•  It is es<mated that over 65% of an organiza<on’s standard test metrics are never reviewed or analyzed

•  Only 12% of Fortune 500 CIO’s believe their tes<ng organiza<ons are providing them with relevant informa<on

•  On average, it takes 3 – 4 hours of manual effort to prepare a report that is reviewed in 5 minutes or less


Tulkita Test Training Series Communication Essentials


Key Thoughts: •  Test Professionals do not spend the necessary <me communica<ng •  Stakeholders are oeen missed or worse, not even iden<fied •  Knowing your stakeholders is cri<cal to your success •  3 out of 5 projects FAIL due to stakeholder misalignment •  Selling and showing your value are becoming increasingly important •  The value and successes of tes<ng oeen go unno<ced

Focus Areas: •  Iden<fying your Stakeholders •  Establishing a Fail-‐Safe Communica<on Plan •  Managing and ‘Bea<ng’ your Stakeholders

20

Objective Why is this subject important?


The Language of Tes3ng: •  Defects •  Pass Rates •  Test Execu<on Status •  Test Planning Effec<veness

The Language of Business: •  Cost Effec<veness •  Total Cost of Ownership •  Return on Investment •  Cost of Poor Quality •  Efficiency and Produc<vity

Translation The majority of your stakeholders will have a much different ‘view of the world’ then you do.

21

Translate your QA data and ‘language’ into relevant informa>on that can be understood and consumed by each of your stakeholders.


Success Criteria:

•  Iden<fy ALL of your stakeholders from IT and the Business •  Develop a Communica<on Plan •  Conduct Monthly/ Quarterly Checkpoints & Opera<onal Reviews •  Achieve ongoing alignment and ac<vely manage •  Monitor Industry Trends & Benchmarks •  Complete Customer Sa<sfac<on Surveys •  Ac<ve Listening and Speaking •  Communicate in 3s?

Successful Communication ~70% of your job as a Test Professional should be spent communicating.

22

Know what ALL of your stakeholders want, and how they want to be communicated to.


Communication Approach Understand the essential steps required to connect and be understood by your stakeholders.

23

Communica>on becomes more important the higher in the organiza>on you go. OJen Sr. Leadership do not understand what you do on a day-‐to-‐day basis.

Stakeholder Acceptance Curve

Time

Deg

ree

of S

uppo

rt fo

r Cha

nge

CONTACTAWARENESS

UNDERSTANDING

ACCEPTANCE

BUY-IN

COMMITMENT

Commitment Threshold

AcceptanceThreshold


Basic Elements: •  Stakeholder Inventory •  Stakeholder Requirements (ques<ons) •  Communica<on Channels/ Types •  Frequency •  Suppor<ng Data & Metrics •  Status & History

The Approach: •  Iden<fy your stakeholders •  Setup mee<ngs with your stakeholders to determine what they want •  Determine channels to support communica<ons (i.e. mee<ngs, email) •  How oeen do your stakeholders want to be engaged? •  Setup suppor<ng reports, status & templates for team to follow •  Collect necessary data to support communica<ons

Communication Plan A Communication Plan is an integral component of your test practice. Build one!

24


Communication Plan A Communication Plan is an integral component of your test practice. Build one!

25

In Prac3ce: •  Everyone on your team should have individual communica<on plans •  ‘Quality vs. Quan<ty’ is the golden rule for building plans •  Setup frequent touch points with ‘difficult’ stakeholders •  Plans should be setup & implemented during Test Planning – do not wait! •  Use Communica<on Plans to script the rollout of new ideas, concepts •  Focus on change management ac<vi<es •  Monitor communica<on success & con<nually modify/ expand


Why this Topic? •  Many Test Professionals get ‘beaten’ by colleagues or management •  Stakeholders oeen are misaligned with what is happening in tes<ng •  Stakeholders do not understand why there are delays/ scope creep •  This happens in every organiza<on, it will never go away

Solu3on •  Iden<fy your top stakeholders that you want to ‘beat’ •  Document how/ when they ‘amack’ and for what reason •  Ar<culate proac<ve ac<vi<es to address these ‘amacks’ & assemble in plan •  Change your communica<on style with these individuals •  Provide informa<on that will circumvent/ ‘beat’ these individuals •  Repeat and con<nue these steps for 2 months (it takes <me), by the third month you now have to deal with someone else! J

Controlling Your Stakeholders Controlling, or ‘Beating’ your stakeholders should become your favorite activity.

26


Your Checklist:

•  Iden<fy all stakeholders in tes<ng, IT and the business •  Develop a simple, clear, Easy-‐to-‐use Communica<on Plan •  Avoid ‘silo’ thinking – iden<fy who you are impac<ng & when •  Con<nuously review and modify your Communica<on Plan •  Be proac<ve, plan ahead and understand top issues •  Achieve stakeholder buy-‐in of Communica<on Plan

Takeaways Key success factors & implementing in practice.

27


A Visual to Sum it Up You are the integrator and hub of the IT organization. Are you up to the challenge?

YOU

tulkita test training series - chicago quality assurance

Documents