starwest - test automation as a service - track
DESCRIPTION
We’ve all heard the claims that cloud computing will, without any up-front investment, provide instant scalability, flexibility, and availability for testing-on-demand. But how well does this work in practice? Is it really the perfect environment to create powerful testing solutions? If so, why would your businesses invest in creating its own custom test automation frameworks to support your solution lifecycle management, when there’s a solution to meet everyone’s needs? Based on his experience, describing recent successes and the pitfalls to be avoided, Jonathon Wright explores these questions, and the pros and cons of using the “Testing as a Service” model. Sharing real world examples, he shows how you can transform the automation services in your organization and how you can take your company on the journey into the cloud regardless of your automation maturity. Influence and contribute to the future direction in the “Testing as a Service” communityTRANSCRIPT
Test Automation as a Service
Jonathon Wright ([email protected]) www.linkedin.com/in/automation
STARWest (Thursday @ 1:30 PM, 4th October 2012)
© TestingAsAService.com
© BJSS Limited 2011 Commercial in Confidence 2
My starting point
Twitter @Jonathon_Wright with how many slides in this presentation?
First person to guess or be the closest!
GOTTAWANNAWIN
Quote - David Dubovsky, Boca Raton, 1999
The current economic climate is making companies review their approach to IT even more closely
“Existing technologies of virtualisation, business process modelling, cloud based test automation tools and rapid and easy internet access
allow for the development of approaches that allow companies to order testing as a service and pay only for what they use (PAYU).”
• Why (Benefits)
• What (Cloud Automation)
• When (Timeline)
• Scope (Features)
• Transition (AMMi)
• Dependencies (Cloud Type)
• Implementation (Agnostic)
• Universal Test Language
• Maintenance (Asset Loaders)
• Cloud Virtualisation
• Continuous Build & Integration
• Event Driven Automation
Funnel Virtualisation
Dashboards (Real-time)
Three Decades in Automation
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
2005 - 2006
QTP 8.x
2004 - 2005
QTP 7.x
2006 - 2008
QTP 9.x
2008 - 2010
QTP 10.x
2010 - 2011
QTP 11
1998 - 2001
Modularity driven testing
2001 - 2005
Data driven testing
2005 - 2008
Keyword Driven
2008 - 2011
Hybrid Keyword/Data Driven
Today
1994 - 2001
1999 - 2001
WinRunner 6.x
2001 - 2004
WinRunner 7.x
1998 - 1999
WR x.x
1994 - 1999
Why? – Bit of Background
Recent Explosion in Automation Tools
No longer one Tool to rule them all?
Automation is STILL prone to failure?
Cost of Failed Implementations 2004 2012 (Estimated)
Industry: Test Automation (net worth) $1 billion $6 billion
Automation Projects (failure cost) $½ billion $3 billion
60% 32%
8%
Failure
Working
ROI
92% FAIL TO MEET TARGET ROI
40%
*Ovum study projects the testing tool market to be worth 56 billion US dollars by 2013 growing at an annual rate of 9.5%
Automation is STILL an expensive business?
Manual (Prep + Execution)
Manual (Pure Execution)
Manual'mation
Framework
TAaaS
Effo
rt
Iterations
“The implementation of a self-maintaining test asset loader/scraper that automatically runs after each build and then self-validates against the universal testing language (DSL) and object models built within the
cloud promote a more maintainable and portable solution.”
Introducing Legacy Cloud (First Generation)
Private Combination
Public
SUT SUT SUT
Community Clouds
Public Clouds
Private Clouds
TAaaS TAaaS
“... the prediction that 60% of the average enterprise will have 60% of its applications in the cloud.”
- Karl Flinders, Computer Weekly (2011)
“IBM are working with other cloud suppliers to establish the standards for cloud service provider compliance.”
- Murat Gunenc, IBM Cloud Representative @ Rhodes W1 (2011)
Testing in the Cloud
New Challenges & Opportunities ? and/or
A match made in heaven?
New Challenges
What happens when these services FAIL?
Alternative Service Provider?
Unique Services
(Such as GPS)
Syn
c
Systemic Failure
(DNS or Network)
New Opportunities
“Don’t assume the ecosystems (community or commercial) and other fulfilment systems to be tested in
anything but isolation.” - Julie Gardiner (2012)
Community Clouds already tested?
Consumer Freedom
“Enable sophisticated modelling of the components of cloud deployments, including servers, storage, databases and data.
For the consumption of cloud resources” - PlanForCloud.com (2012)
Instant scalability
Flexible Test Execution
Flexible Test Assets Environments on-demand
Geo-based execution
Instant Accessibility
99.9% Availability (100% Web)
CompareTesting.com
Instant Portability
Best of Breed (BoB) tools on demand
No upfront investment
Pay As You Use (PAYU)
Technology
Process
People
Where are we going today?
Development (.NET 4.0)
AutomationDatabase
Automation Portal (TaaaS)
Application Under Test
2003/2007/2010
Lab Management
Windows
Service
ObjectHandler.dll
DataHandler.dll
EventHandler.dll
Application Programming Interface
Unified Reporting Model
DashboardReportingTest Management Tool Team Foundation Server
Application Lifecycle Management
CodedUI
“Pure ‘Testing as a Service’ Model - Born in the cloud (solution developed specifically with the cloud in mind).”
Automation: Born in the cloud
Taking the journey ‘to the cloud’
Customer Systems
Intranet
Site
Test Suite CMMI/ISO 9000
Rule StoreSoftware Build
Process
Desktop
Application Web Site
Web services PersistenceVirtual lab
The Cloud (System Infrastructure)
Customer Automation Suite
Business Level
Keyword Definition
United Reporting
Credentials
Flexible Cost
Planning
Support
Test Data
DashboardTest Asset
Management
Automation Lifecycle Management
(Test Automation as a Service Model)
Monitoring/
Operational management
Configuration
Reporting
Billing
Customer Take on Customer support
Development
DeploymentMaintenance
How to: Consume cloud services
Test Automation as a Service (TaaaS)
Cloud Web Endpoint
Test Scheduler Service
End Users
QA Person
Run a test«uses»
Schedule a future
test
«uses»
TaaaS Portal
Silverlight
Dashboard
Test Runner Environment - A Virtual Machine
created by the Framework VM Dispenser
TestRunner
Test Data
Set
Technology
AdapterTest Adapter
Platform
Adapter
Client System Test Repository
Business
Level
Keywords
(BPM/BPT)
Test Data
SetsResults
Object
Repository
View real time
test status
«uses»
Business User
«uses»
View historic test
results
«uses»
Dashboard Service
Test Assets Repository
(MTM / Quality Centre)
TD-API
Adapter
Results Service
Test Management
Unified Reporting (Dashboard)Check Development
Progress
Framework
Results
Adapter
Silverlight
Dashboard
Developer
View Live Defect
Build Client System
Build Automation
(Team City / TFS)
Framework
Object
Adapter
Asset Scanner Service
© BJSS Limited 2011 Commercial in Confidence 17
“Supporting natural language with context sensitive validation.” (ATDD, BDD & TDD Compatible)
Technology: Test Designer
Creating Cloud Tests any device, anytime and anywhere
© BJSS Limited 2011 Commercial in Confidence 18 “Supports the full three tier application deployment.”
Technology: Test Lab Designer
Creating Cloud Environments featuring VM Dispenser Technology
© BJSS Limited 2011 Commercial in Confidence 19
*iPhone5 (iOS6.0 not yet fully tested)
Technology: Test Cloud Runners
“Monitor concurrent execution across a instantly scalable number of cloud clients.”
Executing Cloud Tests across a number of platforms
© BJSS Limited 2011 Commercial in Confidence 20
Technology: Agnostic Solutions
“Platform, technology, client, browser, version, language agonistic solutions supporting cross browser testing
through only a few clicks in the lab designer.”
• The test definition is not implementation specific – Test execution is performed by cloud based technology adapters which
act as interpreters for the implementation agnostic language;
Test Run Generate
Environment Under Test (EUT)
Select Platform Adapter
Select Test
Adapter
Select Technology
Adapter
• Application Server • WebServices
• Database Server • Schema • Data
• End-Client Machine
• Windows 2012 • Windows 8 • Windows Hyper-V • Windows 7 • Windows 2008 R2
• Performance • Load
• Functional • Component • Integration • Unit Level • UI Level
• Coded-UI (Win32) • Selenium (ASP.net) • Ranorex (Flex 3/4)
• 1.2.0 (Flex 2) • 2.3.3 (Flex 3) • 3.0.2 (Flex 4)
• Automation.UI (WPF) • White (SWT(Java))
© BJSS Limited 2011 Commercial in Confidence 21
Technology: Event Driven Automation
• Automated testing based on trigger events (passive & active): – Third party system events such as user story or defect status workflow change
‒ Continuous build and integration or source code gated check-in
Pas
sive
A
ctiv
e
Technology
Process
People
Where are we going next?
© BJSS Limited 2011 Commercial in Confidence 23
Example Automation Projects Automation Level Investment in Effort Business Benefit
Description AMMi Gen Implementation Planning Execution Accel. ROI
Client Specific Implementation - Hybrid Approach 4 5 24 63 17 6.9 112%
Client Specific Implementation - BDD Approach 3 4 30 83 22 5.2 85%
Client Specific Implementation - Traditional 1 1 120 250 67 1.7 26%
Testing As A Service Model – Manual’mation 3 1 28 5 25 4.6 200%
Testing As A Service Model - Hybrid Approach 4 5 12 31 19 6.1 186%
TaaaS - Cloud Based Implementation 5 5 5.6 25 7 16.0 306%
Process: Automation Maturity Model Index (AMMi)
“Sometimes half the battle is just starting companies on their automation journey by taking them one step closer to becoming
ready for automation.”
© BJSS Limited 2011 Commercial in Confidence 24
Solution Lifecycle Management
Process: TestingAsAService.com - Model
TEST AUTOMATION AS A SERVICE
© BJSS Limited 2011 Commercial in Confidence 25
Process: Business Process Modelling v2.2
“Business process modelling allows non-subject matter experts (SMEs) to build and perform manual and automated tests
without any previous domain expertise.”
© BJSS Limited 2011 Commercial in Confidence 26
A1
Login.Process
Request.VM
Access.VM
Manage.VM
Logout.Process
A3
A2
C1C3
D2C2D1
C4
E2D4 E1
E3 E4
D3
Design complex tests scenarios from day one ‘first day’ testing
(Release 1.0 = A1>C3>E1>D4>E3>C4)
Process: Business Process Scenarios
“Business Process Scenarios map dynamically to an instance of the Solution Under Test (SUT).”
B1
Login.Process
Request.VM
Access.VM
Manage.VM
Logout.Process
B3
B2
C1C3
D2C2D1
C4
E2
D4 E1
E3 E4
D3
Accept.TermsA1
(Release 2.0 = A1>B1>C3>E1>D4>E3>C4)
© BJSS Limited 2011 Commercial in Confidence 27
Process: Dynamic Test Assets
“Test type agnostic – Pluggable test types so a smoke, regression, integration & performance can be reused against multiple platforms.”
Emerging Test Assets
Application / Component
(e.g. Virtualisation Platform)
Version (e.g. Build,
Release)
Solution (e.g. Time to Market
Improvement)
Client (e.g. Xerox,
Thomson Reuters)
Work Stream (e.g. Finance,
Marketing)
Language (e.g. en-GB
or en-US)
Location (Geo) (e.g. UK,
France, Asia)
Val
idat
e t
est
s
(aga
inst
mo
men
t in
tim
e)
Valid
ate te
sts (again
st new
bu
ilds via asset lo
ader)
© BJSS Limited 2011 Commercial in Confidence 28
“Fault tolerance is built in to report and continue on different levels of fuzzy matching. Combined with the non-technology specific
test definition language, investment in ‘non-fragile’ test assets is much more durable.”
Enter password here ……
Please enter your password to logon
logon logoff
Enter password here ……
Please enter your password to login
login logoff
Test assets are defined by the self-maintaining test asset loader that uses fuzzy logic matching to identify UI objects
Process: Object Recognition (via Fuzzy Logic Matching)
1. XPATH Check 2. Regular Expressions 3. Descriptive Programming 4. Fuzzy Logic 5. Artificial Intelligence 6. Social Intelligence
n/a
Businesses should focus on business, NOT building automation frameworks
2nd Gen
(Data)
3rd Gen
(Keyword)
5st Gen
????(Scriptless)????
4th Gen
(Hybrid)
1st Gen
(Modular)
Gen Zero
(Record)
Next Generations Collaborating Together in Community Testing Cloud?
© BJSS Limited 2011 Commercial in Confidence 30
Testing as a Service (TaaS) Dashboard (Realtime)
Risk Dashboard(Presentation Layer)
Benefit Analysis(Presentation Layer)
Story Board(Presentation Layer)
Business Risk
Project Risk
Technical Risk
Business Domain Language
Test Domain Language
Business Stories
Business Workflow
Business Rules
Business Process Design (BPD)(Visio Design Documentation)
Non-Functional Requirements (NFR)
(Dynamic Data Source (XML))
Detailed Session Definition (DSD)
(Dynamic Data Source (XAML))
Detailed Profile Definition (DPD)
(Dynamic Data Source (XAML))
Detailed Test Definition (DTD)
(Dynamic Data Source (XAML))
Business Components & Interfaces
(Dynamic Data Source (XAML))
Business Process Modelling (BPM)
(BPMNv2)
Business Process Scenarios (BPS)
(eXensible Process Definition Language)
Business Process Tests (BPT)
(Dynamic Data Source (XAML))
Business/Test Transactions Layer
Solution Under Test (SUT) Layer
Dynamic Data Adapters Dynamic Test Adapters
Business Questions
“Integration into current Solution Lifecycle Management (SLM) to include real-time reporting dashboards powered by WCF based reporting frameworks.”
Actionable Insight (Ambassadors of Data)
Funnel Virtualisation
Insight
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data Data
Data
Data
Data Data
Technology
Process
People
The Final Frontier
Testers are testers, not programmers Developers are developers, not testers
“Many of the best tester’s are technically minded and can program but a tester’s real skill is applying testing knowledge to generate reusable test assets.”
People: Cross-functional teams
Julie the ScrumMaster
© BJSS Limited 2011 Commercial in Confidence 33
People: Social Enterprise Ready D
ash
bo
ard
s
Funnel Virtualisation
Do
cum
enta
tio
n
Traditional TaaS
Living Meta Data “Master the Data” Inditex
“Encourage simplicity” D’O
© BJSS Limited 2011 Commercial in Confidence 34
Team
Wo
rkin
g
Remote collaboration
Act
ivit
y St
ream
s
Real-time information
People: Social Enterprise Ready
“Collaborate inventively” McLaren
“Embrace cloud workforces” Mosaic
People: Where are the testers?
Time Zone 1 Time Zone 2 Time Zone 3
Community Tester - Platform: Hybrid Community Cloud (e.g. Facebook)
Beta Tester - Platform: Public Cloud (e.g. beta.bbc.co.uk)
Alpha Tester - Platform: Public/Private Cloud (e.g. alpha.bbc.co.uk)
Cloud Tester - Platform: Private Cloud (e.g. vpn.internal.bbc.co.uk)
Internal Tester - Platform: Private Cloud (e.g. internal.bbc.co.uk)
Testing Command Center (Domain Specific)
• Supporting Core Testing Activities (Environment Monitoring/Management)
•Defect Monitoring & Risk Assessment (Social Enterprise Activity Feeds)
•Testing Management (Test Assets & Resource Enablement)
Solution Command Center (Solution Specific)
• End Client Dashboard/Reporting (KPIs)
• Solution Management (Programme Level)
•Service Provider Management (SLAs/Who does what when)
Automation should be a Global Marketplace
Smaller, possibly remote firms or individuals can now
have easy PAYU access to world class expertise.
© BJSS Limited 2011 Commercial in Confidence 37
Relevant Clear traceability of the business value of Automation through the visualisation of the tests via Business Process Modelling (BPMNv2.2 compliant)
Effective Self-validating test assets achieved using natural language with context sensitive validation against business and testing rules, workflows and data
Maintainable Self-maintaining test asset loader/scraper
Efficient Reports on SUT health including ratings such as percentage availability since build/release, reported errors over time and traffic to error ratio
Manageable Unified platform which non-domain experts can use a natural language to represent business processes and user story acceptance criteria
Portable Technology agonistic - Platform, client/component, browser, version & language Test type agnostic – smoke, regression, integration & performance
Reliable Fault tolerance is built in to report and continue on different levels of fuzzy matching combined with the non-technology specific test definition language
Diagnosable Actionable defects provided by environment under test (EUT) live pause-playback supported by dynamic data adapters (DDA) for accelerated defect investigation and resolution
Test Automation as a Service - Summary
Source: The Big Picture of Test Automation: Test Trustworthiness – Alan Page, Microsoft (2012)
© BJSS Limited 2011 Commercial in Confidence 38
Testing Automation as a Service: Final Thoughts
“The best aspects of these proven test approaches
demonstrate how they have evolved over the past decade,
and this echoes some of the changes towards more lean and
agile business methodologies. They are in a constant state of
evolution – just as the underpinning technology evolves over
time.”
My section in the ‘Experiences in Test Automation’ book
(Dorothy Graham & Mark Fewster) 2012.
Further Information
User: michaeljackson
Password: moonwalk
Hands on Demo
(Hyper-Test.com)
Jonathon Lee Wright ([email protected]) @jonathon_wright (twitter) | www.TaaaS.net (blog)
www.automation.org.uk | automation.sharepoint.com www.linkedin.com/in/automation