qualisystems-brief testshell

QualiSystems Proprietary & Confidential 1

TestShell Introduction

End-to-End

Test Automation

December 2010

Sta

nd

ard

s o

f Exce

llen

ce

Amy Ho

9001:2000 90003:2004

• Global Software Company

• Santa Clara, USA

• London, UK

• Munich, Germany

• Tel-Aviv, Israel

• APAC

• Singapore , APAC HQ (Q1 2011)

• Shanghai, China

• Taipei, Taiwan

• Tokyo, Japan (Q1 2011)

• APAC Channels, Australia, Korea, India

• Flag product – TestShellTM Test-Automation Framework

• “Customer First” Culture

• Honorary president Aryeh Finegold, Founder & President of Mercury (now HP)

QualiSystems – Company Overview



Customers Snapshot

We help our customers

Improve Product Quality

Shorten Time to Market

Reduce Expenditures

http://www.kepri.re.kr/index.php

http://www.mobixell.com/

http://www.mrv.com/

http://ceoworld.biz/ceo/wp-content/uploads/2009/05/telstra_logo.jpg

http://images.google.co.uk/imgres?imgurl=http://www.moldex3d.com/jla/en/images/stories/Success/Samsung/samsung-logo.jpg&imgrefurl=http://www.moldex3d.com/jla/en/index.php?option=com_content&task=blogcategory&id=93&Itemid=141&usg=__SRVpPm7bk-3lcdyOdhisxMp3zQ4=&h=350&w=1048&sz=18&hl=en&start=2&um=1&itbs=1&tbnid=rKGYtoQxBCy1CM:&tbnh=50&tbnw=150&prev=/images?q=samsung+logo&um=1&hl=en&sa=N&tbs=isch:1

http://images.google.co.uk/imgres?imgurl=http://www.streaming21.com/ximg/logo_alphanetworks.gif&imgrefurl=http://www.streaming21.com/streaming21.cfm?pageid=39&usg=__vXYcZ0Ci5mhKL_j51AHyoFMTSH0=&h=55&w=177&sz=3&hl=en&start=1&um=1&itbs=1&tbnid=qWEwRIRnG20HLM:&tbnh=31&tbnw=101&prev=/images?q=alpha+networks+logo&um=1&hl=en&sa=N&tbs=isch:1

• Significantly reduce Time-To-Market

• Full regression cycle 16 days → 1 night at RadVision

• Single test cycle 2 Months → 2 weeks at SanDisk

• Significantly reduce Operational Expenditures

• Resources involved in test creation down by 50% at Deutsche Telekom

• Single test cycle effort 5 Day → 6 Hours at Australian Service Provider

• Managing and Improving Quality Control

• Daily Build and regression at Ceragon Networks

• Customer service centers at Crow

• Saving X10 on RMA at Syneron

Proven ROI within the first year of use


Test Topology

TestShell by QualiSystems

7 QualiSystems Proprietary & Confidential

Create tests

Configure DUTs

Control traffic

Topology setup

Resource allocation

Execute tests

Debug root-cause

Collect results

Schedule execution

Coordinate resources

Report & analyze

Results, trends & stats

Central

Management

• Setup the lab topology into the resource manager:

• Define all lab resources

• Define all the lab physical connections

• Activate topologies from tests:

• Locking and reservation resources when tests run

• Performing actions on the resources based on topology

• Integration with Layer1 Switch to manage the physical

connections (optional)

• The Result: multiple users can load, create, lock and

reserve diverse multi-vendor resources when running tests

Topology Setup – Concept

Topology Setup

Tests Creation

Tests Scheduling

Tests Execution

Result Analysis

• Simplified Topology Creation – Graphical Diagram

• Guarantee resource locking and availability for the entire testing cycle

• Create multiple topology documents activated with a single click

• Publish resource availability across the testing team

• Track utilization records to maximize resource usage

• Verify resource availability prior to execution

• Automatically inherit all resource parameters directly to the test by

resource selection

TestShell Topology Setup - Benefits

Tests Creation



Create tests

Configure DUTs

Control traffic

Topology setup

Resource allocation

Execute tests

Debug root-cause

Collect results

Schedule execution


Report & analyze


Central

Management

Tests Creation –

Studio Topology

Setup Tests

Creation Tests

Scheduling Tests

Execution Result

Analysis

Manage and control

Routers, Switches,

Network elements

QualiSystems Proprietary & Confidential

12

TestShell Studio

Online

Dashboards

and Reporting

Control B/OSS–

Web, Java, Win app

Control Traffic

Generators , T&M

http://www.google.co.il/imgres?imgurl=http://www.cisco.com/en/US/prod/routers/ps5854/prod_large_photo0900aecd8017290a.jpg&imgrefurl=http://www.cisco.com/en/US/products/hw/routers/products_category_buyers_guide.html&usg=__S5ldgsD7_AXeNU1xdCs67JrGH6Q=&h=480&w=600&sz=68&hl=iw&start=16&um=1&itbs=1&tbnid=N00iXsviVhCJ4M:&tbnh=108&tbnw=135&prev=/images?q=Routers&um=1&hl=iw&tbs=isch:1

http://img.alibaba.com/photo/274946955/Agilent_E4407B_STD_ESA_E_Standard_Spectrum_Analyzer.jpg

Automated Test Script Starting -------------------------------

Ping 192.168.1.56 Done

Configure DUT Service Done

Configure IXIA Traffic Done

Start IXIA Traffic Done

Load IXIA result file ‘C:\IXIA\Logs\administrator\results.csv’ Done

Calculate Traffic Loss… IXIA packets sent = 10,000 IXIA Result = 144 packets

Check Against Criteria 112 < SPEC && 10,000 > SPEC

Save log file to path: C:\Automated Tests\Project Folder\Test_14_01_2010 Test Pass

TestShell Studio Code Free test creation

History: Intensive

Programming Test Scenario flow Today: Simple Drag & Drop Test

Creation using TestShell Studio

Effort Effort

• Requires Programming skills

• Hard to maintain

• Difficult to modify for future

tests and regressions

Tests automation - Sample Scenario Code Free Test Creation as easy as PowerPoint

14

TestShell Studio

Scenario

flow

• Messages, Instructions • Alerts, Remarks • Watchers, Graphs • etc.

Dialogs

• Variables • Cases, Loops, Delays • Parallelization, events • etc.

Logic Modules

• Traffic generators • Network analyzers • Spectrum, scope • Generators, etc.

T&M Modules

• Symbian • Windows Mobile • iPhone, Blackberry • Android

Mobile Modules

• Mail & SMS, File Engine • GUI Automation, Attributes • Requirements

Management, QTP ,LR , Etc..

Utilities

• Telnet, SSH, TCP, UDP • SNMP (& Traps), FTP • Traffic Generators, Switches • etc.

Networking

• Web GUI • Web Services • etc.

Web Modules

• Exe, DLLs, CLI, VIs • Script-Engines • TCL, Perl, Python, VB, Java • MatLab, OCR

Scripts & Software

TestShell System Interfaces to Software and hardware compnents (Any many more..)

• Drag & Drop Test Creator – non programmers can join the automation

• Reusable Building Blocks

• Record and Replay Actions

• Integration with both Traffic Generators and T&M

• Supported interfaces:

• Network Protocols (Telnet, SSH, Serial, TCP, UDP, SNMP, etc)

• GUI – Web, Java, Win app

• Logic and Dialog

• Integration with external files (configuration and results)

• Resource Asset Management & Deployment

• Documentation

• Scripts

• Executables

• Dlls

Tests Creation - Benefits

Tests Scheduling




Create tests

Configure DUTs

Control traffic

Topology setup

Resource allocation

Execute tests

Debug root-cause

Collect results

Schedule execution


Report & analyze


Central

Management

• Automatic scheduling according to time, station and priority

• Test scheduling across single or multiple stations

• Advanced "Outlook-like" interface

• Virtual station locking

• Fully customizable execution rules

• Automatic real-time execution conflict handling

Tests Scheduling –

Scheduler Topology

Setup Tests

Creation Tests

Scheduling Tests

Execution Result

Analysis

Tests Execution




Create tests

Configure DUTs

Control traffic

Topology setup

Resource allocation

Execute tests

Debug root-cause

Collect results

Schedule execution


Report & analyze


Central

Management

• Plan the execution

• Select Test or Regression

• Select Unit Under Test and Topology

• Record Admin information

• Serial, Version, Batch etc… (Dynamic)

• Run

• Complete Run

• Step by Step

• Automatic troubleshooting and action execution

• Real-time data tracking

• Standardize data collection

• Data transfer guarantee

Tests Execution –

Runner Topology

Setup Tests

Creation Tests

Scheduling Tests

Execution Result

Analysis

Results Analysis




Create tests

Configure DUTs

Control traffic

Topology setup

Resource allocation

Execute tests

Debug root-cause

Collect results

Schedule execution


Report & analyze


Central

Management

• View Real time tests results, statistics and trends

• From individual test report to aggregative results across labs

• Web-based application accessible at any time from any

location

• Built-in Customizable Report Generator Wizard

• Real-time Events and Actions Engine

Results Analysis –

InSight Topology

Setup Tests

Creation Tests

Scheduling Tests

Execution Result

Analysis

Test Analysis - InSight Online customizable display

The analayzed data is

presented via online

Dasboards:

• Providing real time analysis

• Reviewed any time anywhere

www

The dashboards are

completely customizable

according to user

preferences, supporting

multiple analysis & display

options, including OLAP

Deployment


Deployment

Mercury Confidential & Proprietary

Successful Deployment

Install, Configure and Integrate with NBN systems

Apply Methodology and Best Practices

Right team • Human Resources • Training

People Process Product

Product Deployment- Sample Project Plan

Mercury Confidential & Proprietary

Product- Deployment Architecture Samples


Integrations Architecture - Samples

TestShell Components


TestShell Components


Test Creation

TestShell Studio

Test Execution

TestShell Runner

TestShell Scheduler

TestShell Resource Manager

Business Intelligence

TestShell Insight

TestShell Server


Thank you

www.qualisystems.com

qualisystems-brief testshell

Documents

test topology testshell

lab topology

test creation

topology integration

resource availability

tests debug rootcause

resource usage

lab resources