tim koomen - testing package solutions: business as usual? - eurostar 2010

1

Testing package solutions: business as usual?

Tim Koomen EuroSTAR 2010

Seite 2

Start

Package solutions: business as usual?

Testing

QA

End

Test / QA staff

Wanted: package solution!

Business processes

Support processes

IT-policy

Standards

Other applications

3 Testing package solutions: business as usual?

Architecture

Infrastructure

Found: package solution

Testing package solutions: business as usual?4

Business processes

Support processes

IT-policy

Standards

Other applications

Architecture

Infrastructure

Implemented: customized package


ParametersStandardsoftware

Custom

made

Business processes

Support processes

IT-policy

Standards

Other applications

Architecture

Infrastructure

Life-cycle

Package selection

Implementation

Exploitation & maintenance


Changes in

system

managementAcceptance

There are no risks, because ...

The sales guy tells me this package is perfect for ourorganisation ...

The package supplier already found all defects ...

My implementation partner takes care of everything ...

We use the package as is ...

The design of the modified business processes is clear to the (super)users and implementation partner ...

The package should work on our infrastructure ...

The package is very user-friendly, so little training is required ...

If there are any problems, we just quickly change a few settings ...

Once in production, we can very easily bring out newreleases ...


Or are there, because...

Does the package functionality cover most of our requirements?

Has the package been parameterised correctly?

Is any custom made software built? Does it work correctly?

Does the package (still) work with business processes, existing systems, operational procedures, infrastructure, etc.?

Have the authorisations been defined properly?

Can the user work with the changed business processes and with the package?

Is the performance of the package sufficient?

Can the old data be converted correctly?

Can system managers install and operate the package?


Some news items ...

[July 2010] IBM says Queensland Health SAP failure is not its faultBut the start date for the project expanded from 8 to 26 months and when it finally went live on March 14, a litany of major problems ensued. Thousands of health staffers were incorrectly paid and the cost for the project blew out from an initial $6.19 million to $64.5m

[June 2010] In a important case, Marin County, California filed a complaint against Deloitte Consulting for its role in an over-budget SAP implementation. [.. ] still not working four years after it initially went live […] damages of at least $30 million

[February, 2009] Another High-Profile SAP Failure: State Of California Yet another black eye for German software giant SAP (SAP). Now the State of California -- already in dire financial straits -- is giving up on its SAP implementation after sinking $25 million into the project and seeing nothing out of it.

[March, 2008] Waste Management Inc.'s lawsuit against SAP for the "complete failure" of a $100 million software implementation could bruise the credibility of SAP's vertical-market strategy. Waste Management claims SAP duped it into purchasing untested software that wasn't ready to handle the complexities of the U.S. waste hauling market. ”


And not only SAP ...

[August 2008] “Prone to Failure: Why CRM and Billing Systems Implementations Are High Risk”


(Dr. Raul Katz, Columbia Business

School, presents findings from

research on why certain telecom

implementations are doomed from the

start)

Siebel ......


Risk control

Package solutions, both in implementation and maintenance, differ from custom made software

Myth: “packages are low risk solutions”

Underestimation of many (other) risks

“Changing software is peanuts compared to changing people”


Seite 13

Start


Testing

QA

End

Test / QA staff

From Wikipedia ...

“SAP implementation ...”

Testing is very important before going live with any system. Before going live with a SAP system, it is vital to do many different kinds of testing, since there is often a large, complex infrastructure of hardware and software involved. Both requirements as well as quality parameters are to be tested. Important types of testing are:

Functional testing: to test using functional use cases, i.e. a set of conditions or variables under which a tester will determine if a certain business process works

Integration testing

Regression testing

All tests should be preceded by creating solid test plans.”


Steps risks-driven testing


Product risk

Risk

Frequency of Use Chance of fault

Chance of failure

Damage

*

*


Determining participants

Risk

Frequency of Use Chance of fault

Damage

*

*• (End) users • (Functional and

system) managers• Line managers • Project manager • Client

Chance of failure


• Process (re)designers• Architects• Package consultants• Developers• DBA• QA• Test manager• Project manager

PRA: from business to IT

Business IT

test goalscharacteristics / test goal

object parts / characteristics

risk indication per object part / characteristic

packageunder

testbusiness processes, change requests or risks (to be covered) ,...

suitability,functionality,performance,usability,security, …

package modules(funct.),work processes (suitability)online, batch (perf.)user screens (usability),application, infra (security)

Chance of failure

High Medium Low

Damage High A B B

Medium B B C

Low C C C

Testing package solutions: business as usual?

Quality attributes

More (+) or less (-) risk?

+ Suitability (=fit in organisation / business

processes)

+ Functionality (custom-made)

+ Security

+ Operability

+ Infrastructure

+ Performance

- Functionality (package)

- User friendliness, reusability, flexibility, ...


Risks regarding “as is” x “customized”


High

Low

Package as-is Amount of custom-made changes

Test strategy Master Test Plan

Characteristic/ object part

PRA-RiskClass

Review Devel-oper.T

Sys. T / Funct. T

User Acc.T /Final Int.T

ProdAcc.T

suitability

- module 1 Medium

- module 2 High

- total integrated Medium

security

- infrastructure Medium

- application Medium

performance

- online Medium I

- batch Low I S

functionality High

manageability Low

Test strategyPRA24 Testing package solutions: business as usual?

Test types

Some examples:

Proof-of-concept testing (selection process)

Testing individual business processes

Testing integration between processes / package

Interface tests

End-to-end (business or final integration) testing

Data migration testing

(Scalable) Regression testing

Stress testing


... with a structured test process


TMap is a registered trademark of Sogeti BV

Example: TMap life-cycle and activities

Control

Planning

Preparation Specification Execution Completion

Setup and maintain infrastructure

Collect test basisTestability review

AssignmentPRA

Test strategy…

Consolidate plan

Specify testsDefine starting points

Specify intake test object

ManagementMonitoringReportingAdjusting

Intake/pretestPrepare starting points

(Re)testCheck & assess

Preserve TestwareEvaluate process


SpecifyingRealising

Specifying intakeIntake

MaintainConserve

Test basis

Process design (blue print)

Change requests

Specifications for custom-madesoftware

Release notes (from package vendor)

Informal:

Knowledge of users

Existing processes

Existing software systems


Seite 31

Start


Testing

QA

End

Test / QA staff

Possible (QA) measures

Product Process

Project

procedures:

configuration /

change / release /

defect management

monitoring tasks & responsibilities

project structure

communication plan

standards & templates

reviews &

inspections

workshops

testing

proof of concept

acceptance

criteria


Seite 34

Start


Testing

QA

End

Test / QA staff

Staffing QA and testing

Involved disciplines:

Package consultants

Business users

Infrastructure / system management

Test/QA professionals!

Required roles:

Test manager

Testers

Test support

QA manager


Discussion

• More important: test/QA or package expertise?

• Independent?

Seite 36

Package solutions have different risks:- Business risk instead of IT risk- Organizational complexity- Low risk awareness

Demands risk-based and transparent QA and Testing

(Independent) test/QA expertise and method is required

Conclusion

tak!

Finally ...


spørgsmål,vraag,question, pregunta,Frage, interrogação, domandafrågal

E. [email protected]

M. +31 (0)6 34139260

I. www.timkoomen.nl

Copyright Tim Koomen Testmanagement en -advies

tim koomen - testing package solutions: business as usual? - eurostar 2010

Technology