delivery center testfactory. lenkungsausschuss offshore...

Delivery Center Testfactory.Lenkungsausschuss Offshore Programm.

======!"§==Systems=

Enabling Offshore Test withModel-based TestingMarch 07, 2007; Test Factory, Andreas Braun

March 07, 2007; Dr. Andreas Braun

Contacts at Test Factory, T-Systems

Andreas BraunAndreas BraunAndreas BraunAndreas BraunProject Manager & Team Lead for MBT ResearchEmail: [email protected]: +49 89 1011-3461

HansHansHansHans----Peter GoorenPeter GoorenPeter GoorenPeter GoorenProject ManagerEmail: [email protected]: +49 211 6211-3370


History of Software Testing

‘‘‘‘80s/ early 80s/ early 80s/ early 80s/ early ‘‘‘‘90s: 90s: 90s: 90s: Test management and improvement of software qualityTest management and improvement of software qualityTest management and improvement of software qualityTest management and improvement of software quality

ApproachApproachApproachApproach: independent test team, test processes & methods

� more stable software systems, better software quality

� expensive, costly, no common approach

Late Late Late Late ’’’’90s: 90s: 90s: 90s: reduce cost, improve competitivenessreduce cost, improve competitivenessreduce cost, improve competitivenessreduce cost, improve competitiveness

ApproachApproachApproachApproach: IT-Offshoring

� competitive, effective, qualified workforce

� communication hurdles and breakdown, difficult knowledge transition

‘‘‘‘00: 00: 00: 00: reduce effort, systemize workreduce effort, systemize workreduce effort, systemize workreduce effort, systemize work

ApproachApproachApproachApproach: testing based on models

� Formal, complete content, improved communication

Ongoing research: Ongoing research: Ongoing research: Ongoing research: Enable offshore test, industrialize testEnable offshore test, industrialize testEnable offshore test, industrialize testEnable offshore test, industrialize test

HypothesisHypothesisHypothesisHypothesis: Model-based Testing helps to solve both issues

TTTT----SystemsSystemsSystemsSystemsTest FactoryTest FactoryTest FactoryTest Factory


Some Terms and Definitions

� Outsourcing� Near shore� Offshore� Near shoring vs. off shoring

� Captive vs. independent� Export or domestic market� Globalization

� Test, modeling, Model-based Testing


Enabling Offshore Test with Model-based TestingAgenda

Part 1: Offshoring Governance and ManagementPart 1: Offshoring Governance and ManagementPart 1: Offshoring Governance and ManagementPart 1: Offshoring Governance and Management� Introduction, Vision, Strategy� Governance and Management Approach

Part 2: ApproachPart 2: ApproachPart 2: ApproachPart 2: Approach� Masterplan, Iterative Approach� MBT Pilots, Goals and Measures

Part 3: ModelPart 3: ModelPart 3: ModelPart 3: Model----based Testing based Testing based Testing based Testing � The Model-based Testing approach revisited� Modeling-based Testing in CRM-T� An example from CRM-T

Bachelor/ MasterBachelor/ MasterBachelor/ MasterBachelor/ Master’’’’s/ Diploma Thesess/ Diploma Thesess/ Diploma Thesess/ Diploma ThesesPart 4: OutlookPart 4: OutlookPart 4: OutlookPart 4: Outlook

� Offshore Test Process� Organization and Roles� Offshore Suitability Metrics


Part 1---Offshoring Governance and ManagementAgenda

� Introduction to Offshoring and Outsourcing� Reminder: Modeling in Testing & the Offshoring Test Framework� Offshore Capability and Related OSM Level� Legacy Situation and Goals


Job Migration: Is Offshoring a Threat to the Local Job Situation?US IT Jobs 1999/2003 (BLS)

VarianceVarianceVarianceVariance20032003200320031999199919991999Role Role Role Role

2922

70

257

156

245

97

486

481

293

410

403

+8,71%2688Total

+16,67%60Hardware engineers

-8,54%281Computer systems managers ********

+59,18%98Network & data communications analysts

+19,51%205Network and systems admin

-3,96%101Database administrators ********

+13,55%428Computer systems analysts

+3,89%463Computer support

+40,19%209Software Engineering systems

+41,87%289Software engineering applications

-23,82%529Programmers *********


Job Migration: Is Prevention of Offshoring a Threat?A recent (near shoring) example from the automotive industries.

� Schefenacker produces almost 30% of rear-view mirrors in the automotive industry worldwide

� Schefenacker has no direct end-customer access

� Hence, Schefenacker is a supplier and lives on outsourcing

� Schefenacker utilized its market position to increase pressure on manufacturers

Problem SituationProblem SituationProblem SituationProblem Situation

� Production cost to high

� No window of opportunities for movement (no price hedging, resource cost increases, market pressure)

� They missed the entry point into outsourcing/ near shoring due to management mistakes and due to labor union pressure

� As a consequence and due to time pressure, mistakes in outsourcing (Slovenia) led to additional loss

� Schefenacker is now in for insolvency, covenant breach not reached

TodayTodayTodayToday

� As a consequence to the power struggle, Schefenacker later lost such customers

� Consequence: to be taken over by financial investors and hedge fonds (Permira)

� Loss of production at DaimlerChrysler

Lessons learntLessons learntLessons learntLessons learnt

� Evading offshoring not promising

� Market pressure will increasingly reduce window of opportunities

� Cost

� Time

Consequences:Consequences:Consequences:Consequences:

� Offshoring inevitable

� Offshoring must be productive and chargeable

� Early definition needed, experience needed

Our Vision and Strategy:Our Vision and Strategy:Our Vision and Strategy:Our Vision and Strategy:

� No staff replacement, but:

� free local resources for new projects

� Reduce cost of work/ cost of production

� Retain control

� No pure offshoring (“Fire & Forget”)

� Leverage mix of skills & cost

� Automation first!

Source: FAZ, October 31st, 2006Source: FAZ, October 31st, 2006Source: FAZ, October 31st, 2006Source: FAZ, October 31st, 2006

News from Friday, 16.02.2007 Schefenacker: Financial restructuring plan drawn up Senior lenders and the main shareholder of automotive supplier Schefenacker (Schwaikheim / Germany; www.schefenacker.com), Alfred Schefenacker, have agreed on a financial restructuring plan that foresees nearly 75% of the mirror specialist passing into the hands of British hedge funds. The company’s current debt of EUR 450m will be replaced by a new financial package that includes credit facilities worth EUR 305m, along with fresh capital of EUR 55m. The lighting business, accounting for 22% of total sales, is to be divested. �


Reminder: Modeling in Testing & the Offshoring Framework The Test Factory Offshoring Framework consists of five layers defining major activities of Offshore Management and Operations.

OffshoreGovernance

Offshore Management

Offshore Test Process

Tools & Methods

Operative ManagementProgram Management

Strategic Management

Project Management

Project Execution

VisionStrategy


Offshore Capability and Related OSM Level.To improve the offshore capability, the SIT aims at moving towards a blended workforce model (“Cross-border Collaborative Test”) with shared responsibilities.

Offs

hore Cap

ability/

Offs

hore Cap

ability/

Offs

hore Cap

ability/

Offs

hore Cap

ability/

Com

plex

ity of S

ervice

Delivery

Com

plex

ity of S

ervice

Delivery

Com

plex

ity of S

ervice

Delivery

Com

plex

ity of S

ervice

Delivery

ResponsibilityResponsibilityResponsibilityResponsibility RemoteLocal

Low

High

Onsite Onsite Onsite Onsite StaffStaffStaffStaff

Onsite Onsite Onsite Onsite ProjectProjectProjectProject

Onsite Onsite Onsite Onsite ––––Offshore Offshore Offshore Offshore projectprojectprojectproject

Pure Pure Pure Pure OffshoreOffshoreOffshoreOffshore

OffshoreOffshoreOffshoreOffshoreTest FactoryTest FactoryTest FactoryTest Factory

CrossCrossCrossCross----border border border border Collaborative Collaborative Collaborative Collaborative

TestTestTestTest

•Traditional approachTraditional approachTraditional approachTraditional approach•Staff augmentationStaff augmentationStaff augmentationStaff augmentation•Performance Performance Performance Performance

management: management: management: management: individual feedbackindividual feedbackindividual feedbackindividual feedback

•Offshore team works Offshore team works Offshore team works Offshore team works onononon----sitesitesitesite

•WellWellWellWell----defined scopedefined scopedefined scopedefined scope

•Team takes Team takes Team takes Team takes ownershipownershipownershipownership

•Client servicing, Client servicing, Client servicing, Client servicing, project management project management project management project management onshoreonshoreonshoreonshore

•ApplicationApplicationApplicationApplicationtesting offshoretesting offshoretesting offshoretesting offshore

•Better flexibility, Better flexibility, Better flexibility, Better flexibility, scalability, stabilityscalability, stabilityscalability, stabilityscalability, stability

�Rarely adoptedRarely adoptedRarely adoptedRarely adopted

�No onNo onNo onNo on----site presencesite presencesite presencesite presence

�High riskHigh riskHigh riskHigh risk

•Vendor setsVendor setsVendor setsVendor sets----up center up center up center up center offshore dedicated to offshore dedicated to offshore dedicated to offshore dedicated to client needsclient needsclient needsclient needs

• In return, client In return, client In return, client In return, client commits to buy a commits to buy a commits to buy a commits to buy a minimum amount of minimum amount of minimum amount of minimum amount of services over an services over an services over an services over an extended timeextended timeextended timeextended time

•Support complements Support complements Support complements Support complements clientclientclientclient‘‘‘‘s internal s internal s internal s internal standards and standards and standards and standards and proceduresproceduresproceduresprocedures

•Complementary Complementary Complementary Complementary resources from resources from resources from resources from multiples countries multiples countries multiples countries multiples countries pooled on a common pooled on a common pooled on a common pooled on a common delivery platformdelivery platformdelivery platformdelivery platform

•HighlyHighlyHighlyHighly----structured structured structured structured partnerships in the partnerships in the partnerships in the partnerships in the different countriesdifferent countriesdifferent countriesdifferent countries

• Leverage capabilities Leverage capabilities Leverage capabilities Leverage capabilities of each center to of each center to of each center to of each center to create a portfolio of create a portfolio of create a portfolio of create a portfolio of highhighhighhigh----quality, costquality, costquality, costquality, cost----effective solutionseffective solutionseffective solutionseffective solutions

•Performance Performance Performance Performance management: management: management: management: SLAsSLAsSLAsSLAs

Level 0 Level 1 Level 2 Level 3 Level 4 Level 5

Legacy Situation and GoalsThe Legacy Offshoring-Model is a mixture between Provision of Services and Assignment of Work Packages (Craft).

� Integration of offshore partners hampered

� “Local heroes”

Impact on Current OffshoringImpact on Current OffshoringImpact on Current OffshoringImpact on Current Offshoring

Standardized Notation and Tools, Standardized Notation and Tools, Standardized Notation and Tools, Standardized Notation and Tools, Formal Methods Formal Methods Formal Methods Formal Methods ��

� Industrial processes �UML, U2TP, TTCN-3�Model-based Testing�Artifact-based Onshore-Offshore Communication

�

Difficult transition over organizational boundaries and project contexts

�

Tools, Methods, and ProcessTools, Methods, and ProcessTools, Methods, and ProcessTools, Methods, and Process� Lack of formal methods and

notation� No standardized methods� Craftsmanship

GoalGoalGoalGoalResulting ProblemsResulting ProblemsResulting ProblemsResulting ProblemsAsAsAsAs----is Situationis Situationis Situationis Situation

Knowledge Sharing and Reuse Knowledge Sharing and Reuse Knowledge Sharing and Reuse Knowledge Sharing and Reuse ��Models encapsulate domain knowledge

�Model repositories enable reuse

�Costly training upfront�Many inquiries�Much overhead due to coaching�Low productivity

�

Person-dependent expertiseDifficult knowledge sharing and transfer

�

Tacit KnowledgeTacit KnowledgeTacit KnowledgeTacit Knowledge� Architecture� Organization� Business processes� Sources for information,

stakeholders� No up to date documentation

Defined communication paths Defined communication paths Defined communication paths Defined communication paths ��Onsite and Offshore-coordinators

� Local stakeholders and experts on both sides

� Language encapsulation

�Overhead of cost and time�Burden on local experts

�Translations are error proneMisunderstandings

�

Language BarriersLanguage BarriersLanguage BarriersLanguage Barriers� German documentation/ GUI� Jargon, Buzzwords� Translation to English� No artifact sharing

Leverage Balanced Teams Leverage Balanced Teams Leverage Balanced Teams Leverage Balanced Teams ��Blended workforce model� Increased responsibility�Critical mass

�Charging by the hour�No responsibility transfer to offshore partners

�No workload balancing, no resource sharing

�

No work packaging (entities)Difficult responsibility transfer and integration of offshore ressourcesReactive workstyle

�

Monolithic Project TeamMonolithic Project TeamMonolithic Project TeamMonolithic Project Team� Tight coherence and coupling of

team members� Strong intra-team bindings� Weak extra-team bindings� Tight informal communication� No artifact sharing


Any Questions?


Part 3---Update on Model-based Testing Agenda

� Model-Based Testing (MBT)� Definition� Activities

� MBT in CRM-T� Activities and Artifacts� Process and Iterations� Example out of CRM-T� Terminology� Results� Evaluation

� Summary & Conclusion


The Model-Based Testing Theme Song © 1999, Harry Robinson

(to the tune of “I'd Like to Teach the World to Sing”)

Pete Seeger once said that any good movement needs a theme song, so (with apologies to all) here is my submission for the model-based testing theme song, complete with harmony section:

I’d like to teach the world to model Increase code quality I’d like to find bugs in the spec And keep my weekends free (“… see my family…” )

I’d like to teach the world to model And teach monkeys to test So they can find the bugs at night While I’m home getting rest ( “…that sounds good to me…” )

I’d like to teach the world to model And teach monkeys to think So they could find bugs early on And not just on the brink (“ … of delivery… “)

I’d like to teach the world to model For perfect quality I’d like to automate their tests And ship stuff defect-free ( “ … that’s how it should be …” )


Update on Model-based Testing in CRM-T Definition & Benefits

� MBT is software testing in which test cases are derived in whole or in part from a model that describes some (usually functional) aspects of the system under test (SUT). [Wikipedia]

� MBT main activities:� Build Model

− Different Modeling Methods (e.g. Finite state machines, UML Diagrams)

� Test Generation� Test Execution

� Adopt MBT in CRM-T by:� Tailoring MBT activities according to our needs� Keeping it compatible with traditional test systems


Model-based ApproachProject phases and corresponding toolset.

Model-based Approach

Mapping the test model to TTCNTTCNTTCNTTCN----3333

Test Process

UMLUMLUMLUML Models used in:Requirements, Analysis; Modelling of domain knowledge (func., techn.); Test case generation

U2TPU2TPU2TPU2TP for test case specification(structure and behavior)

UMLUMLUMLUML for depicting the process flow

Analysis

Manual Execution

Automation

Evaluation

Design

Execution


Test ModelTest ModelTest ModelTest Model

Test StructureTest StructureTest StructureTest Structure

Test ControlTest ControlTest ControlTest Control

DeploymentDeploymentDeploymentDeploymentDiagramDiagramDiagramDiagram Test Case BehaviorTest Case BehaviorTest Case BehaviorTest Case Behavior

Test ConfigurationTest ConfigurationTest ConfigurationTest Configuration

ClassClassClassClassDiagramDiagramDiagramDiagram

Test PackageTest PackageTest PackageTest Package

InteractionInteractionInteractionInteractionDiagramDiagramDiagramDiagram

Modeling Categories: Specification vs. Test Model.Functional requirements are depicted in the UML. They build up the basis for test modeling with MBT.

System ModelSystem ModelSystem ModelSystem Model

Use CaseUse CaseUse CaseUse CaseDiagramDiagramDiagramDiagram

ActivityActivityActivityActivityDiagramDiagramDiagramDiagram

PlainPlainPlainPlainTextTextTextText

DescriptionDescriptionDescriptionDescription

Component Component Component Component /Deployment/Deployment/Deployment/Deployment

DiagramDiagramDiagramDiagram

InteractionInteractionInteractionInteractionDiagramDiagramDiagramDiagram


Advantages of the Methodology and Techniques selected.

� Models � Reduce tacit knowledge � knowledge is embedded in artifacts� Abstraction copes with complexity � improves clarity and comprehensibility� Improve communication� Easy to modify and maintain updates

� Model-based Testing� Combines testing with (semi-) formal methods� Allows usage early in the software life cycle � Higher test coverage, better results � the formal is approach less error-

prone

� Toolset� Same syntax for UML and U2TP

− Consistent specification− One single modeling tool− Reduced training and coaching effort

� Systematic mapping of U2TP to TTCN-3


Test Assignment

Deliver Test Basis

Estimation, Planning

Analyze Test Basis

System Modeling****

Test Case Generation

Test ModelingAllocate Test Data

Automated Test Execution

Log Results

Test Report

Quality AssuranceAuditCustomer Report

Manual Test Execution

Assignm

ent

Assignm

ent

Assignm

ent

Assignm

ent

Ana

lysis

Ana

lysis

Ana

lysis

Ana

lysis

Des

ign

Des

ign

Des

ign

Des

ign

Exec

ution

Exec

ution

Exec

ution

Exec

ution

Test Closu

reTe

st Closu

reTe

st Closu

reTe

st Closu

re

Defect?Defect?Defect?Defect?NoNoNoNo

YesYesYesYes

Model-based Testing Process: Overview.

Defect Management Proc.


Assignm

ent

Assignm

ent

Assignm

ent

Assignm

ent

Ana

lysis

Ana

lysis

Ana

lysis

Ana

lysis

**** If no models are delivered in the test basis only.If no models are delivered in the test basis only.If no models are delivered in the test basis only.If no models are delivered in the test basis only.

Test Assignment

Deliver Test Basis


Analyze Test Basis

System Modeling****

Model-based Testing Process: System Modeling.

CreateUse Case Diagrams

CreateDeployment Diagrams

CreateActivity Diagrams

System ModelingSystem ModelingSystem ModelingSystem Modeling



Assignm

ent

Assignm

ent

Assignm

ent

Assignm

ent

Ana

lysis

Ana

lysis

Ana

lysis

Ana

lysis

Test Assignment

Deliver Test Basis


Analyze Test Basis

System Modeling**** IdentifyValid Paths

Verify Path Coverage and Completeness

GenerateTest Cases

Combine with additional paths/

merge paths

Test Case GenerationTest Case GenerationTest Case GenerationTest Case Generation

Complete?Complete?Complete?Complete?

NoNoNoNoYesYesYesYes

Model-based Testing Process: Test Case Generation.

Identify TestObjectives


Assignm

ent

Assignm

ent

Assignm

ent

Assignm

ent

Ana

lysis

Ana

lysis

Ana

lysis

Ana

lysis

Des

ign

Des

ign

Des

ign

Des

ign

CreateTest Suites

ModelTest Configuration

ModelTest Architecture

ModelTest Cases

Test ModelingTest ModelingTest ModelingTest Modeling

Model

Test Control

Test Assignment

Deliver Test Basis


Analyze Test Basis

System Modeling****

Model-based Testing Process: Test Modeling.




ManualTest Execution

Assignm

ent

Assignm

ent

Assignm

ent

Assignm

ent

Ana

lysis

Ana

lysis

Ana

lysis

Ana

lysis

Des

ign

Des

ign

Des

ign

Des

ign

Exec

ution

Exec

ution

Exec

ution

Exec

ution

ModelExecution Flow of Events

PrepareTest Data

Create Test SuiteDocument

ManuelleManuelleManuelleManuelle TestdurchfTestdurchfTestdurchfTestdurchfüüüührunghrunghrunghrung

ExecuteTest Suite

Test Assignment

Deliver Test Basis


Analyze Test Basis

System Modeling****



Model-based Testing Process: Manual Test Execution.


Testfallgenerierung

Test ModelingAllocateTest Data


Assignm

ent

Assignm

ent

Assignm

ent

Assignm

ent

Ana

lysis

Ana

lysis

Ana

lysis

Ana

lysis

Des

ign

Des

ign

Des

ign

Des

ign

Exec

ution

Exec

ution

Exec

ution

Exec

ution

TTCN-3 Mapping

Create Templates for Test Data

Install/ AdaptSUT

Automated Test ExecutionAutomated Test ExecutionAutomated Test ExecutionAutomated Test Execution

ExecuteTest Suite

Test Assignment

Deliver Test Basis


Analyze Test Basis

System Modeling****

Model-based Testing Process: Automated Test Execution.


Test Assignment

Deliver Test Basis


Analyze Test Basis

System Modeling****




Log Results

Test Report

Quality AssuranceAuditCustomer Report

Manual Test Execution

Assignm

ent

Assignm

ent

Assignm

ent

Assignm

ent

Ana

lysis

Ana

lysis

Ana

lysis

Ana

lysis

Des

ign

Des

ign

Des

ign

Des

ign

Exec

ution

Exec

ution

Exec

ution

Exec

ution

Test Closu

reTe

st Closu

reTe

st Closu

reTe

st Closu

re

Defect?Defect?Defect?Defect?NoNoNoNo

YesYesYesYes

Model-based Testing Process: Overview.

Defect Management Proc.


Abstract

Test Suite

Update on Model-based Testing in CRM-TTerminology

� Test Suite: A high level concept grouping together a collection of test cases related to what they are intended to test.

� Scope varies from organization to organization.� Abstract and Executable (Testfallkette)

� Abstract:� Abstract view on prerequisites (communication, time and data)

� Executable (“Testfallkette”): A counterpart to the abstract test suite� Ready to be executed.� Concrete prerequisite states or steps and system configuration.

AbstractAbstractAbstractAbstract

Test SuiteTest SuiteTest SuiteTest Suite

ExecutableExecutableExecutableExecutable

T ran

sla te Te

s ts

T ran

sla te Te

s ts

T ran

sla te Te

s ts

T ran

sla te Te

s ts

Models

Test Cases

INPUT EXPECTED

OUTPUT

ExecutableTest Suite

(“Testfallkette”)


Any Questions?


Bachelor/ Master’s/ Diploma Theses@ T-Systems Test Factory

1. Design and Implementation of an Import/Export Filter for U2TP1. Design and Implementation of an Import/Export Filter for U2TP1. Design and Implementation of an Import/Export Filter for U2TP1. Design and Implementation of an Import/Export Filter for U2TP Test Suites in a Test Suites in a Test Suites in a Test Suites in a test management system.test management system.test management system.test management system.� Learning of the test process and quality features (ISO-Standards),

U2TP and UML 2.0� Build a Meta-model for the UML Testing Profile for SparxX Enterprise

Architect (EA).� Definition of a Taxonomy for Test Specification.� Mapping of SparxX EA XMI outputs in a testing data-model (Test Work

Bench, TWB).The goal is to design and implement an executable Prototype. Keywords/know-how: UML, U2TP, UML Profile, XMI, Java APIs, SQL DB.



2. Definition of 2. Definition of 2. Definition of 2. Definition of modelingmodelingmodelingmodeling guidelines and processes for Modelguidelines and processes for Modelguidelines and processes for Modelguidelines and processes for Model----based Testing based Testing based Testing based Testing (MBT).(MBT).(MBT).(MBT).

� Learning of the test processes and the quality features (ISO Standards), UML and MBT.

� Analyse the existing MBT processes of Testfactory T-Systems� Learning of cross project modeling examples� Derivation of Modeling guidelines� Identification of description possibilities for modeling guidelines

Goal is the definition of a framework for Model-based Testing.



3. Evaluation of 3. Evaluation of 3. Evaluation of 3. Evaluation of ModelingModelingModelingModeling and Specification methods for different Test and Specification methods for different Test and Specification methods for different Test and Specification methods for different Test characteristics and requirements.characteristics and requirements.characteristics and requirements.characteristics and requirements.

� Learning of the test processes and the quality features (ISO Standards)� Analysis of specifications of different test types and quality features.� Implementation of case studies and evaluation of concrete examples.� Evaluation of the possibility of automated test generation.� Scientific assessment of results.� Formulation of recommendations and suggestions.

Goal is the development of an overview of modeling guidelines for different test types and requirements.



4. From Use Cases to Activity Diagrams:4. From Use Cases to Activity Diagrams:4. From Use Cases to Activity Diagrams:4. From Use Cases to Activity Diagrams:� Convert textual-use case specifications into activity diagrams� Creation of templates for textual-use cases� Annotating use cases with stereotypes in such a way that the test

cases are clearly delimited and then having the tool automatically parse those annotated documents to convert them into XML metadata interchange (XMI) files for later input into Sysiphus[1].

Goal is the development of a plugin in Sysiphus which allows test designers to semiautomatically convert their existing textual-use case specifications into activity diagrams.[1] Sysiphus: http://sysiphus.in.tum.de


Part 4---Outlook Agenda

� Phases in the Offshore Project� Organizational Model

� Organigram� Roles & Expectations in the Project Domain� Roles & Expectations in the Offshore Domain


Phases in the Offshore ProjectThe Offshoring (Sub-) Project is organized accordingly to the Rational Unified Process (RUP)

ElaborationTest architecture� Spec. cycles� Int./ ext. reviews� Updates (CRs)� Final spec (TWB)

TransitionTest Closure

� Auditing� Test reporting� Test closure� Process

improvement

Baseline Test End Criteria

InceptionPlanning & Setup� Statement of work� Analysis� Estimation� Review� Kick-off

ConstructionTest control� Test execution� Steering� Defect management � Reporting

Draft !

Testanalyse

und -design

Testrealisierung

und -durchführung

Testauswertung

und -bericht

Testabschluss

Testplanung

und -steuerung

Testaktivitäten

Organizational ModelThe Offshoring and Project Domain are orthogonal and fulfill specific, defined assignments.

Draft !

Team Lead

Team Member

SPOCOnsite

Coordinator

Functional Lead

Assignment Owner

Status to: Functional Control by: Discipl. Reporting to:

Onsite: Offshore: (Dotted lines denote lose coupling)

Offshore D

omain

Project D

omain

Offshore ProgramManager

Rules:� Onshore-Offshore

communication is strictly artifact-based.

� Fixed resource allocation

Project Manager

Project Assignment:� Fulfill client expectations� Face to the client� Achieve project goals� Meet budget, deadlines� Delivery quality work� Define offshore work

packages/ assignments� Meet project’s offshore

capability

Offshore Assignment:� Meet offshoring strategy� Optimize chargeability� Improve method� Deliver quality work� Meet deadlines

Offshore Manager

OffshoreLiaison Manager

Program

D

omain

Scope, Q

ualityT

ime, B

udget

Program Manager

AD

SF

TF


CostCostCostCost�Non- profitable�Low profit/ break even�Clear profit margin

�Profit Margin (Lucrative)

�Core (Onsite)Core (Onsite)Core (Onsite)Core (Onsite)�Non-core (Offshore)

�Mission criticalMission criticalMission criticalMission critical�Non-critical

�High, i.e. dedicated individualsHigh, i.e. dedicated individualsHigh, i.e. dedicated individualsHigh, i.e. dedicated individuals�Medium, i.e. intra-company�Low, i.e. external

Value Value Value Value (starts with worst)

-200 to +80

�Confidentiality

StrategicStrategicStrategicStrategic(VETO)(VETO)(VETO)(VETO) �Criticality

�Core business

ScoringScoringScoringScoringCriteriaCriteriaCriteriaCriteriaCategoryCategoryCategoryCategory

Criteria Catalogue: Strategic Dimension Draft !

�Bad�Normal�Good

�Software quality/ defect analysis

�Floating�Normal�Stable

�Scope stability

�Pure Application�E2E

�Business process/ application test

�Bad/ instable�Limited availability/ stability�Full performing/ stable

�Infrastructure performance and stability

�Not available�Procurement�Available

�Availability of technical resourcesTechnology/Technology/Technology/Technology/AdministrationAdministrationAdministrationAdministration

-130 to +130

�Short, <3 months�Mid, 3 to 6 months �Long, > 6 months

�Duration

AssignmentAssignmentAssignmentAssignment

�Adhoc�Planned

�Forerun

�Undefined�Vague�Defined

�Scope Definition

�Small, 1,2 FTEs�Medium, 3,4 FTEs�Large, 5+ FTEs

�Size

�Process section (partial)�Full process (end to end)

�(Test) Process Coverage

�Native (German)�English

�Deliverable Language

�Native (German)�English

�High�Medium�Low


�Complexity

Work PackageWork PackageWork PackageWork Package

�Spec. & Doc Language

ScoringScoringScoringScoringCriteriaCriteriaCriteriaCriteriaCategory Category Category Category

Criteria Catalogue: Operations Dimension Draft !


-100 to +100

�Offshoring transparent�Offshoring basically accepted�Offshoring supported

�Expectations(Type/ Form/ Quality of Deliverables)

StakeholdersStakeholdersStakeholdersStakeholders�High interaction�Medium interaction�Low interaction

�Communication Interfaces

�Not available�Partly available�Available

�Availability of skilled resources

ResourcesResourcesResourcesResources

�Long, expensive�Brief, cheap

�Training upfront

�Tacit�Well defined�Not needed

�Tacit�Well defined�Formal Specifications

�Tacit�Well defined�Formal Specifications


�Contextual/ Cultural

�Application

�Business Process

CriteriaCriteriaCriteriaCriteria

KnowledgeKnowledgeKnowledgeKnowledge

ScoringScoringScoringScoringCategory Category Category Category

Criteria Catalogue: People Dimension Draft !


Offshore Assessment (Calculation Sheet) SIT Summary

Offshore Assessment

Assignment name: Testdata (reguläre Anforderungen)

EKI Archivierung, EMK, KAN

CRM_Sat ohne Archivierung

DMS INDIV KVS OM: neue Produkte ohne Business-Kritikalität

OM neu (Prozessänderungen)

TIMB

By: allDate: 23.11.2006 23.11.2006

Review by:

Strategic Dimension: 5 -10 0 -95 0 5 5 5 -95 5Operations Dimension: 30 -10 0 -5 -15 -15 0 25 -5 -35

People Dimension: -5 -20 -30 -30 -20 -15 -5 0 -20 -20

Total Score: 30 -40 -30 -130 -35 -25 0 30 -120 -50


Any Questions?


Samples

MBTMBTMBTMBT

ExampleExampleExampleExample


MBT Example CRM-T:Offshore Specification Process Flow

Use Case DiagramActivity DiagramUse Case Description

Testbase

AnalysisAbstract Test Suite

Initial Models

Final Models

Design

Review


MBT Example CRM-T: Test Basis 1/4Use-Case Diagram: 26U01 Agent in CTI und CRM-T anmelden


MBT Example CRM-T: Test Basis 2/4Use-Case Description: 26U01 Agent in CTI und CRMT anmelden

Fehlermeldung derWinset-Applikation

Anmeldevorgang imStartup-Tool

Siebelstart mit CTI ?

nein

ja

Anmeldung an ACD

ACD Anmeldungerfolgreich? nein

ja

Winset-Applikationstarten

"Login" Siebel ServerBusiness Service starten

Ist ACD Telefonein Teleset ?

ja

nein

Fehlermeldung derWinset-Applikation

Anmeldevorgang imStartup-Tool


nein

ja

Anmeldung an ACD

ACD Anmeldungerfolgreich? nein

ja

Winset-Applikationstarten

"Login" Siebel ServerBusiness Service starten


ja

nein

MBT Example CRM-T: Test Basis 3/4Activity Diagram: Use-Case 26U01 Agent in CTIund CRMT anmelden

Siebel wird mit CTIgestartet

Siebel wird ohne CTIgestartet


nein ja

Rückmeldung anStartup-Tool

Fehlermeldung

Zuweisung inCRM-C

erfolgreich?nein

"Login" Siebel ServerBusiness Service

Teleset / Konfigurations-einträge löschen

Ist Anwenderein CTI Agent?

nein

jaFehlermeldung


nein

ja

ggf. Teleset/Konfigurationseinträge anlegenbzw. aktualisieren

ja

Siebel wird mit CTIgestartet



Agent über CTI anACD anmelden


MBT Example CRM-T: Test Basis 4/4Use-Case: 26U01 Agent in CTI und CRMT anmeldenStandard and Alternative Workflows

Work Package 26 - Requirement Traceability MatrixSr No. Use Case

NumberUse case name Test Case

NumberTest Case Name Description Priority Flow of Events Validation

1 26U01 Agent Login in CTI and CRMT TC001 26B01_CRMT_26U01_LoginWithTeleset Start Siebel with CTI, ACD Telephone is Teleset.If user is CTI agent update the configuration entries. And since ACD telephone is teleset Login to ACD via CTI.

1 Precondition: 1) The telephone type is teleset2) User is a CTI Agent

Steps:1) Login to the startup tool2) Start Siebel with CTI3) If applicable, create or update teleset/configuration entries

Status = Not Available

Agent logs into ACD via CTI

Siebel Started with CTI

2 26U01 Agent Login in CTI and CRMT TC002 26B01_CRMT_26U01_LoginWithWinset Start Siebel with CTI, ACD Telephone is not teleset so starts Winset application and assign configuration entries.If allocation is succesful and ACD is not teleset then gives feedback to Start Up tool as Siebel started with CTI.

1 Precondition: 1) The telephone type is winset2) User is a CTI Agent

Steps:1) Login to the startup tool2) Start Siebel with CTI3) Login to ACD4) Start Winset5) If applicable, create or update teleset/configuration entries


Siebel Started with CTI

3 26U01 Agent Login in CTI and CRMT TC003 26B01_CRMT_26U01_LoginWithoutCTI Login to Siebel without CTI 3 Precondition: 1) The telephone type is teleset2) User is a CTI Agent

Steps:1) Login to the startup tool2) Start Siebel without CTI3)If necessary delete teleset/ configuration entries4)Feedback to startup tool as Siebel started without CTI


MBT Example CRM-T: Test Case GenerationAbstract Test Suites

s d B C 1 _ C R M -T_ L o g i n _ W i th W i n s e t

« T e st C o m p o n e n t »

U S E R


S ta r tu p To o l

« S U T »

C R M -T


C R M -C


C TI m i d d l e w a r e

« V a l i d a t i o n A c t i o n »

P a s s( : : )

se l e c t (si e b e l st a rt w i t h C T I )

N o : = i s A C D t e l e p h o n e t e l e se t

A C D L o g i n

Y e s: = A C d L o g i n S u c c e ssfu l l

S ta rt a p p l i c a t i o n (W i n se t )

S i e b e l L o g i n

S t a rt w i t h C T I

S t a rt

C T I st a rt e d

l o a d c o m p o n e n t s

y e s:= C T I a g e n t

C re a t e C o n f i g u ra t i o n e n t ry

A ssi g n C o n f i g u ra t i o n

A ssi g n m e n t su c c e ssf u l

N o : = i s A C D T e l e se t

R e g i st e r A g e n t to A C D

R e g i st ra t i o n su c c e ssf u l

C T I st a rt e d

C R M -T w i t h C T I st a rt e d(S t a t e = n o t re a d y )

se tS t a t e (S t a t e = R e a d y )

MBT Example CRM-T: Test ModelingTest Case 001: 26B01_CRMT_26U01_LoginWithWinset


sd BC1_CRM-T_ Login_WithoutCTI

«TestComponent»

User

«SUT»

CRMT

«TestComponent»

Startup Tool

«ValidationAction»

Pass(::)

Login

Yes:= Login Successful

Start without CTI

Login Without CTI

Delete Configuration entries

Started Without CTI

Siebel is started withoutCTI(State=not available)

setState(State=Ready)

MBT Example CRM-T: Test ModelingTest Case 002: 26B01_CRMT_26U01_LoginWithoutCTI

s d BC1 _ CRM -T_ Log in_ WithTe le s e t

« T e stCo m p o n e n t»

US E R


S ta rtup Too l

« S UT »

S ie be l CRM -T


CRM -C


CTI M iddle w a re

« V a l i d a ti o n A cti o n »

P a s s(: :)

se l e c t(si e b e l sta rt w i th CT I)

ye s:= i s A CD te l e p h o n e te l e se t

S ie b e l L o g i n

S ta rt wi thCT I

S ta rt

CT I sta rte d

L o a d Co m p o n e n ts

Y e s:= CT I a g e n t

Cre a te Co n fi g u ra ti o n e n try

A ssi g n Co n fi g u ra ti o n

A ssig n m e n t S u cce ssfu l l

Y e s:= i sA CDT e l e se t

Re g i ste r a g e n t to A CD

Re g istra t i o n S u cce ssfu l l

CT I S ta rte d

CRM -T w i th CT Ista rte d (S ta te = n o ta va i la b le )

se tS ta te (sta te =re a d y)

MBT Example CRM-T: Test ModelingTest Case 003: 26B01_CRMT_26U01_LoginWithTeleset


Backup

(Backup)(Backup)(Backup)(Backup)

Part 2Part 2Part 2Part 2


Part 2---Case Studies, Pilots, and CRM-TAgenda

� Goals and Measures � Masterplan� Pilot’s Approach� Pilot Results

� ETC� üVBT� TS India

� Summary & Lessons Learnt� Open Issues� Next Steps


Model-Based Testing Pilots: Goals and Measures The MBT Pilots evaluate the measures suggested in the Testfactory Offshoring Framework in real-world projects.

� Prepare a decision memo (Go, No Go, Go But) by end of October

�Gain insight:Gain insight:Gain insight:Gain insight:� Feasibility of MBT� Increase chargeability� Share knowledge

� Enable knowledge transfer� Transfer knowledge from heads/ experts into models� Training and coaching� Involve Indian teams (communication, newsletter, …)

�Prevent tacit knowledge and dismantle information barriers:Prevent tacit knowledge and dismantle information barriers:Prevent tacit knowledge and dismantle information barriers:Prevent tacit knowledge and dismantle information barriers:� Knowledge transfer� Artifact-based communication� Creation of knowledge repositories

� Open and standardized Methods� Modeling tools…� UML, U2TP, TTCN-3

�Common methods and tools:Common methods and tools:Common methods and tools:Common methods and tools:� Best-practice methodology� Standardized and industrial tools & methods

� Evaluate and Improve our current Offshore Test process� Encapsulate Test Assignments with necessary

knowledge for execution� Increase responsibility in India

�Transition to industrialized processes:Transition to industrialized processes:Transition to industrialized processes:Transition to industrialized processes:� Prepare for service level agreements (SLAs)� Allow for workload balancing� Re-use

Measures in the MBT Pilots:Measures in the MBT Pilots:Measures in the MBT Pilots:Measures in the MBT Pilots:Offshore Governance Goals:Offshore Governance Goals:Offshore Governance Goals:Offshore Governance Goals:


Masterplan: Iterations to Develop and Evaluate the Offshoring FrameworkThe Offshoring Framework Initiative started in December 2005 and improves the method and management approach in several iterations. To date, two iterations have been completed, the third iteration (CRM-T) has just been started and implements many recent improvements.

122005

012006

022006

052006

042006

032006

062006

072006

102006

112006

082006

092006

122006

012007

022007

032007

042007

Offshoring GovernanceKöhne, Geimer, (Schütte), Schmitt, Braun

TU-München

Diploma ThesisSakout, Braun

2 Case Studies ÜVBT:Sakout, Digambar

6 Pilots ÜVBT, ETC:Pawlyta, Vaughan, Unland, Morawetz, Kolarik, Meyer-Holtkamp, Klaus, Poller, Staudenmaier.

3 Work Packages CRM-T:CTI, KKM/ BMM, (Testdata)

Offshoring CoordinationKolarik, Paul

CoachingAl-Jadiri Research, Support

Al-Jadiri

????

Improvements:•Template, Taxonomy•Handbook•Training TU-München•TTCN-3 dropped

MBT TrainingProf. Brügge, Al-Jadiri

Improvements:•TWB integration (first attempt as attachments)•Improved taxonomy (test suite preconditions)•Onsite/ offshore coordinators•Test modelling in India, Test architecture in GER•Definition and Tracking of Work Packages

IEEE-Presentation, RioBraun, Brügge, Sakout

Offshore-Phase:Execution, Training


ChargeabilityChargeabilityChargeabilityChargeabilityup to KW40up to KW40up to KW40up to KW40

Model-Based Testing Pilots ÜVBT+ETCWe select six subprojects to evaluate different alternatives of MBT in two large-scale testing projects.

ÜÜÜÜVBT: DMSVBT: DMSVBT: DMSVBT: DMS

AnalysisAnalysisAnalysisAnalysis

Spec.Spec.Spec.Spec.DesignDesignDesignDesign

TestTestTestTestDesignDesignDesignDesign

ExecutionExecutionExecutionExecution

ÜÜÜÜVBT: Customer VBT: Customer VBT: Customer VBT: Customer MgmntMgmntMgmntMgmnt....

ÜÜÜÜVBT: OM NewVBT: OM NewVBT: OM NewVBT: OM New ÜÜÜÜVBT: VBT: VBT: VBT: iKDBiKDBiKDBiKDB

MUCMUCMUCMUC

MUCMUCMUCMUC

MUCMUCMUCMUC MUCMUCMUCMUC MUCMUCMUCMUC

IndiaIndiaIndiaIndia

IndiaIndiaIndiaIndiaTranslationTranslationTranslationTranslation

PlainPlainPlainPlain----Test Spec.Test Spec.Test Spec.Test Spec.ExecutionExecutionExecutionExecution

MUCMUCMUCMUC

MUCMUCMUCMUC


MUCMUCMUCMUC



Germany

India

ETC:ETC:ETC:ETC:KONZ+KONZ+KONZ+KONZ+ÜÜÜÜWS WS WS WS

MUCMUCMUCMUC

MUCMUCMUCMUC


MUCMUCMUCMUC

ETC: ETC: ETC: ETC: CRMCRMCRMCRM

MUCMUCMUCMUC

MUCMUCMUCMUC


MUCMUCMUCMUC

Henning Pavel Katharina Andreas Rosemary Dirk

100%

100% 100%

100% 100%

100%

MUCMUCMUCMUCIndiaIndiaIndiaIndia

Deepak**Deepak**Deepak**Deepak** 70%70%70%70%Bhakti **Bhakti **Bhakti **Bhakti ** 75%75%75%75%

Saylee**Saylee**Saylee**Saylee** 95% 95% 95% 95% Awadhoota*Awadhoota*Awadhoota*Awadhoota* 90% 90% 90% 90%

Teststart in Indien in ETCTeststart in Indien in ETCTeststart in Indien in ETCTeststart in Indien in ETCplanmplanmplanmplanmäßäßäßäßig ab KW37ig ab KW37ig ab KW37ig ab KW37

**** Awadhoota Awadhoota Awadhoota Awadhoota worksworksworksworks OnshoreOnshoreOnshoreOnshore******** BlockingsBlockingsBlockingsBlockings werewerewerewere usedusedusedused forforforfor testdatatestdatatestdatatestdata generationgenerationgenerationgeneration....


MBT Pilots Results in ETCThe application of MBT in ETC was successful and led to a high recognition both onsite and offshore. However, due to the project character of ETC, MBT is not applicable as the single method without changing major boundary conditions (client access, project language).

Indian colleagues report the feeling of higher productivity and less overhead.

+ higher productivity

The models are easy to understand and execute

+ easy to understand and execute

MBT highly approved by Indian collegues+ high approval of MBT

Due to the limited scope in ETC, no re-use applicable.Existing test cases need to be translated to UML.

Translation of specification to models takes approx. 1 day/ test case. ETC has > 1000 test cases in the current release.

-- Effort for modeling

English specification is welcomed in India, but basically two languages remain in ETC.

The ETC project has no direct client access. The boundary conditions are set and not easily to change.

Client requires German textual specifications and reporting for review purposes.

-- UML and English not client acceptable for ETC

Artifacts are transferable, tacit knowledge reduced, packaging.

Information is structured and formalized, less inquiries and misunderstandings.

+ Communication with offshore partner is improved

More defects found early in the specification stage

Better structure, complete description, and test data definition enforced by the process and templates.

+ Specification improved


MBT Pilots Results in üVBTThe application of MBT was successful and productive in a real-world project. However, due to the different character of the selected subprojects, the results varied. In summary, MBT is applicable in future ÜVBT projects and will likely continue to improve productivity, chargeability as well as the commitment of onsite and offshore staff.

+ test execution and coaching effort is reduced

Defects are found in earlier stages of testing, the test basis is analyzed more precisely.

+ the quality of the specification is clearly improved

-- make MBT a best practice process

Subsequent versions of connected models generate version conflicts.

-- versioning and maintenance of models open

Needs to be addressed and supported by external advisors (TUM)

Project-dependent guidelines for level of detail are missing.

-- missing or incomplete modeling guidelines

In progress-- Inadequate Tools integration (TWB)

+ well-defined roles & responsibilites

Both onsite and offshore, long-term perspectives.+ improved motivation

+ improved communication

Divide and conquer+ Decoupling of offshore resources

Easy tracking of results, forecast+ Test Suites become work packages

Knowledge repository in models in India created.+ knowledge sharing

For instance, CRM-C models reused in CRM-T, OM models reused.

+ re-use of models and knowledge


MBT Pilots Results in IndiaThe survey included 14 participants. Six indicators are considered for the evaluation and are displayed below. Overall results of each indicator reflect positive impression. However, there are training and coaching needs.

Learning & Implementation Productivity

General Approval

Motivation

Communication EfficiencyQuality of Testing

• Method and Tools simple to learn• For some, Modeling required extraeffort

• More knowledge leads to quicker execution

• work packages allows parallel working

• Modeling makes Testing more interesting and innovative

• Method experienced high acceptance• Rated as highly promising

• Clear communication channels• Less time required to solve an issue• Error localization quicker

• More detailed specifications• More detailed reporting possible


Summary of Results and Lessons LearntThe MBT approach is practical in real-world projects and instantly improves various aspects of both onshore and offshore testing.

Industrialized ProcessProcessProcessProcess Improved specifications Improved test design and architecture Reuse

KnowledgeKnowledgeKnowledgeKnowledge Sharing Encapsulated domain knowledge Prevents tacit knowledge

Organization & PeoplePeoplePeoplePeople Allows for improved organizational forms Increased satisfaction and commitment Reduces overhead


Open IssuesThe MBT approach is practical in real-world projects and instantly improves various aspects of both onshore and offshore testing.

� Depending on the project type, the impact and benefit varies� Offshore Suitability Catalogue!

� Still several open issues and lose ends to be further investigated� Project-specific tailoring and guidelines are missing

� Next steps, continuous improvement� The process is afar from being a “Best Practice”


Any Questions?

delivery center testfactory. lenkungsausschuss offshore...

Documents