testing an olympic challengemanagers of test activities and defects by application, sport, intel,...

Testing – An Olympic Challenge

Magnus Alvarsson, Atos Origin,

Canada

Europe’s Premier Software Testing Event

World Forum Convention Centre, The Hague, Netherlands

WWW.QUALTECHCONFERENCES.COM

“The Future of Software Testing”

Atos, Atos and fish symbol, Atos Origin and fish symbol, Atos Consulting, and the fish itself are registered trademarks of Atos Origin SA. September 2006

© 2006 Atos Origin

Testing an Olympic Challenge

EuroSTAR, The Hague, 12 November 2008

Magnus Alvarsson

3

The Olympic Challenge

» There is no second chance» Proven solutions» Reliable solutions» Meet the Users’

expectations

» World Wide Visibility

» Large IT Deployment

» Multiple partners, suppliers and

technology

4

Agenda

» Introduction

» The Olympic Challenge

» Framework

» Project Timeline

» Organization

» Testing

» Strategy

» Testing environments

» Process

5

Objective

» Ensure stable Games systems that:» Conform to functional requirements» Satisfy or exceed agreed service levels» Support the business of the customers

AND:

» Ensure the integration of:» The numerous and varied Games

systems » The partners, people and processes

supporting these systems

Such that ….

» The systems and business processes complement each other to ensure a seamless delivery of service to our customers

6

The Olympic Challenge - Scope

» Atos Origin Systems- 9 Core Games Management Systems (CGS)- 9 Info Diffusion Systems (IDS)- The Olympic Family Intranet in 2 or 3 languages- A large number of interfaces between these systems

» On Venue Results- One On Venue Result System per sport (15/36)- One Timing and Scoring System per Sport and/or Venue- One Scoreboard per Venue- Minimum of 1 TV-Graphics solution per sport

» External Interfaces- On Venue Results- 8 World News Press Agencies- 20 Internet Data Feeds clients (Rights Holding Broadcasters, Media, IF’s and

NOC’s)- Weather- Transportation- Biography and Background Provider(s)- Background Check

7

Integration of Partners, People, Processes

» Organizing Committee

» Timing and Scoring / On Venue Results Provider

» Hardware providers

» Network

» Servers

» Desk tops

» Cameras

» Score boards / Video boards

» Mobile devices

» Software Providers

» Databases

» Operating Systems

» Office suites

» Etc

The Olympic Challenge – Scope (Cont’d)

8

» Operations» Software Distribution Solution» Call Management System» Monitoring» etc

» External Stakeholders» International Olympic Committee» International Sporting Federations» Rights-holding Broadcasters and News Agencies» OCOG Functional Areas» etc

The Olympic Challenge – Scope (cont’d)

9

The Olympic Challenge - Scope

INTEGRATION

People, Partners

& Processes

• Integration of

sponsors/partners

• Customers (VANOC,

NBC, WNPAs etc)

• Atos TS, Ops, MEP

• Roles & Resp.

• Policies &

Procedures

• Communication

• Project / Risk / Issue

Management

• Omega – On-Venue

Results Systems

• TV Graphics :

Olympic Broadcast

Services

• Scoreboard

• RCMP, Immigration

• Data providers

Non-Atos

Systems

Core Games

Management Systems

• Sport Entries &

Qualification

• Accreditation

• Workforce

• Medical

• Arrivals &Departures

• Transportation

• Commentator

Information

System

• INFO 2010

• Internet Data Feed

• Results Data Feed

to WNPAs

• Print Distribution

System

• Central Repository

System

Information

Diffusion Systems

Testing / Integration Test Lab

10

Project Timeline

2006

Design

Planning

Business Process

Reviews

GAP Process

Functional

Requirements

Interface Definition

Software development

Lab build

Staff build-up

Software deliveries

Test Tools Acquisition

Test Case preparation

Build

Connectivity Tests

Functional Tests

Sport/Test Events

User Accesptance

System Tests

Multi Sport Tests

Test

Technical Rehearsals

Games TOC operational

Applications in production

Venue Deployment

Maximum staff numbers

Operate

Feb 2010Olympic Games

2010

40

100

200

1,200Resources

11

The Challenges

» Challenges» Applications from suppliers worldwide

» Synchronizing deliveries and compatible code

» Management and control of:

- defects

- testing environments

- the numerous & varied Integration Test Lab activities

12

The Answers

» The answers» A purpose-built Test Lab (500 m2; 5000 ft2)

» A dedicated Testing team (80+ people including partners)

» A comprehensive multi-faceted test program

» Strict test procedures and defect tracking

» > 200,000+ hours of testing; > 9,000+ test cases: happy v unhappy path

» Support from Technical Services and Operations

» Communication & Coordination

Testing: the single most important success factor!

13

Objective of Integration testing

Ensure that:

» the Games systems work together seamlessly

» Fulfill the required functionality and service levels

To successfully manage and support the Olympic and Paraolympic

Games.

14

To achieve the objective…

» Test functionality of different systems

» Availability test

» Verify operating procedures of the systems

» Test systems under extreme situations

» Focus the tests on data flowing end-to-end

through the systems.

» Simulate games

15

Integration Test Program

» Pre-Delivery Testing of:

» Supplier Development

» Supplier “Connectivity”

» Supplier Technical

» Integration Testing of:

» Delivery Acceptance

» Stand-alone

» Interface

» End-to-End

» Regression System 1

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Entire System

Application

1

Application

2

Application

3

System 1

Application

1

Application

2

Application

3

System 1

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Supplier

Development

Testing

Supplier

Connectivity

Testing

Delivery

Acceptance

Standalone

End-to-End

System Test

Interface

IOC Homologation,

CIS & VANOC UAT

Test Events, Multi-

Sport Tests, Tech

Rehearsals,

Software Freeze

16


Integration Testing of:

» Delivery Acceptance

» Stand-alone

» Interface

» End-to-End

» Regression

System 1

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Entire System

Application

1

Application

2

Application

3

System 1

Application

1

Application

2

Application

3

System 1

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Supplier

Development

Testing

Supplier

Connectivity

Testing

Delivery

Acceptance

Standalone

End-to-End

System Test

Interface

IOC Homologation,

CIS & VANOC UAT

Test Events, Multi-

Sport Tests, Tech

Rehearsals,

Software Freeze

17


» System Testing (3 cycles

of > 20 days each per

application)

» Load

» Stress

» Fault Tolerance

» etc

» Acceptance Testing

» Test Cases supplied by

the IOC and the Sports

Federations or the end

users

System 1

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Entire System

Application

1

Application

2

Application

3

System 1

Application

1

Application

2

Application

3

System 1

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Supplier

Development

Testing

Supplier

Connectivity

Testing

Delivery

Acceptance

Standalone

End-to-End

System Test

Interface

IOC Homologation,

CIS & VANOC UAT

Test Events, Multi-

Sport Tests, Tech

Rehearsals,

Software Freeze

18

System 1

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Entire System

Application

1

Application

2

Application

3

System 1

Application

1

Application

2

Application

3

System 1

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Supplier

Development

Testing

Supplier

Connectivity

Testing

Delivery

Acceptance

Standalone

End-to-End

System Test

Interface

IOC Homologation,

CIS & VANOC UAT

Test Events, Multi-

Sport Tests, Tech

Rehearsals,

Software Freeze


» Test Events

» Every Sport

» Elite level competition

» Real Games conditions

» 1 year before

» Multi-sport Tests

» Three cycles

» Subset of Games schedule each

time

» Overall system performance and

functionality

» 2 months period

19

System 1

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Entire System

Application

1

Application

2

Application

3

System 1

Application

1

Application

2

Application

3

System 1

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Application

1

Application

2

Application

3

Supplier

Development

Testing

Supplier

Connectivity

Testing

Delivery

Acceptance

Standalone

End-to-End

System Test

Interface

IOC Homologation,

CIS & VANOC UAT

Test Events, Multi-

Sport Tests, Tech

Rehearsals,

Software Freeze


» Technical Rehearsal (TR1 & TR2)

» Two cycles

» Subset of Games schedule

each time

» Operational readiness of

systems and people

» 10 days period

20

Technical Rehearsal main objectives

Technical rehearsals primary focus is to:

» Practice and evaluate:

» operational policies and procedures

applied during Games

» communication processes

» Establish relationships

» Verify roles and responsibilities

» Test staff utilizing systems/applications

in a Games time environment

Beijing example

35 Sports

39 Venues

500 Scenarios

2000 Staff

21

Test Organization

Integrated team made up with the following groups:

» Integration Test Board - Atos Origin / OCOG / Omega

Oversees, Highest Point of Escalation

» Integration Test Manager and Test Management Team - Atos Origin

Definer & Coordinator of testing program & lab

» Test Group Leaders - Atos Origin / OCOG

Test plan managers & delivery managers, escalation point for Test Team Leaders

» Test Team Leaders - Atos Origin / OCOG

Managers of test activities and defects by Application, Sport, Intel, Unix, Security

» Testers - Atos Origin / OCOG

Perform tests, log and track defects

22

Test Organization (cont’d)

A set of integration testing plans, policies and procedures ensuring quality and

consistency in the testing process, including:

» Integration Testing Strategy

» Integration Test Plan

» Defect Tracking Process

» Software Delivery Procedure

» Change Management Process

» Test plans for each sport/system

» etc

23

Test Lab Characteristics

» Production like environment

» Tools used for testing activities» Test Management tool (Quality Center)

- Test Cases and Scenarios

- Defect Tracking

» Test automation tool (QuickTest Professional)

- Test scripts

» Load and Performance testing tools

(Load Runner / Win Runner)

» Staff» 35 Atos Origin» 30 VANOC» 6 Omega» 10 Others

24

» Physical Infrastructure» 300 PCs» 100 laptops» 50 printers» 32 switches» 450 servers

(30 ITL, 420 TER)

Customer Engagement

Test Lab Characteristics

25

Test Cell

26

Criteria for Environment Strategy

» Meet objectives stated in the Integration Test

Strategy

» Integration and operation activities can be

smoothly executed.

» Minimize the total number of environments

» Consider the needs for conducting several

continuous tests in parallel without delays.

» Use experience gained from previous

Olympic Games.

27

Environments

» All applications are installed on all environments.

» Three environments must be production like

» Three environments must have the Games-time configuration

» The rest environments will meet the minimum requirements of the Integration team

» Four environments will be available during the Games,

Date

Environment J F M A M J J A S O N D J F M A M J J A S O N D J F M

PDC

E2E

T1

T2

SDC

PAR

PPP

MEP BCN

Paralympic Testing

Para PreProduction

Testing

Testing Test Events System / MSTs TRs Games

Testing

Testing

2008 2009 2010

Systems Test / DR DR

Homologation PreProduction

28

Integration Control

Keys to Integration Testing success for big scale projects:

1. Correlation with Project Plan

2. Defect Tracking Process

3. Control of Software Deliveries

4. Change Management Process

5. Clear Testing Strategy

6. Partner management/Stakeholder agreement

7. Support Tools

8. Common policies & procedures

To minimise risk, the overall IT

system and applications must be

exhaustively tested

Atos, Atos and fish symbol, Atos Origin and fish symbol, Atos Consulting, and the fish itself are registered trademarks of Atos Origin SA. September 2006

© 2006 Atos Origin

Questions and Answers

Atos Origin Major Events

Av. Diagonal, 210-218

08018 Barcelona (Spain)

www.atosorigin.com

Magnus Alvarsson

+1 778 328 3636

[email protected]

testing an olympic challengemanagers of test activities and defects by application, sport, intel,...

Documents