59 percent - info.perfectomobile.com · a recent research report conducted by ... and test...

Perfecto | How To Scale (Up) Your Test Automation Practices And Achieve Better App Quality 2

As organizations strive to be more agile without compromising on quality, DevOps is the end goal. As a matter of fact, 59 percent of North American and European enterprise developers are practicing Agile and 13 percent of organizations are implementing DevOps practices to drive faster and greater innovation for their end users.

More than ever, this conveys the key requirements to automate both testing activities as well as the entire release pipeline. Keeping that in mind, with the recent world quality report by Sogeti, the findings show that only 16 percent of the test activities are being automated.

In this ebook Perfecto identifies 5 key challenges that explain the low rate of test automation in Mobile, Web and IOT, and recommends best practices to address these challenges. Our recommendations include a strategy to define what tests should be automated and how many of the tests should be API-level compared to interface-based (UI). It also includes the importance of utilizing new tools and technologies like machine learning (ML) and artificial intelligence (AI) which injects intelligence into the process of maintaining, authoring and managing the test automation execution.

Fig. 1: Test automation challenges (Source: World Quality report by Sogeti, 2017-2018)

Challenge with the test data and environment availability

Lack of skilled and experienced test automation resources

Challenges with service virtualization

We have difficulties with integrating test automationinto a DevOps process

We have difficulties to integrate the differentautomation tools together

We don’t have the right automation tools

We have difficulties to automate because of use ofmultiple development lifecycles

We don’t have the right automation testing process or method

Current automation solution does not support mobile testing

42% 40%

42%

41% 35%

40%32%

39% 41%

39% 37%

38%

38%

35% 34%

2017

2016

Challenges in Achieving Desired Levels of Test Automation

Introduction

Perfecto | How To Scale (Up) Your Test Automation Practices And Achieve Better App Quality 3Perfecto | How To Scale (Up) Your Test Automation Practices And Achieve Better App Quality 3

A recent research report conducted by Perfecto, shows that for DevOps and Continuous Testing teams, time is the most challenging issue—trying to keep up with shorter release cycles, growing scope of tests for each iteration and increasing the scale of supported platforms.

Specifically, organizations are finding it hard to accomplish the following objectives:

1. Automate new tests that cover new functionality into existing cycles.

2. Drive continuous innovation with test suite growth, making the overall execution cycle lengthy. This creates a situation where there are more new tests to maintain in less time.

3. Automate testing and maintenance as new devices, desktop browsers, operating systems, and features are released almost on a monthly basis into the market. This requires additional investment in automation test maintenance so organizations can keep up with existing schedules.

Tighter release schedules leave less time to automate and test

5 Key Challenges & Recommended Practices for Digital Test Automation

“It is important to remove as much friction from your teams as possible to enable

automated testing. If it is allowing them to utilize a different open source tool

or focus on integration testing instead of unit testing, that is okay. The important

thing is that teams are performing automated testing. You can work through best

practices of testing with teams once you have them comfortable with the cadence of

automated testing vs. stopping them before they start due to onerous restrictions.”

Bryan OsterkampLead Technical Architect, Dev and Ops Infastructure Architecture Division, USAA


Recommended PracticesTo combat these challenges around tighter release schedules and automation, it requires changes to methodologies:

1. Consider adopting acceptance test-driven development / behavior-driven development (ATDD/BDD) practices that can enhance developer efficiency by having the test automation code written prior to the code itself.

2. Build an efficient and single continuous integration process that is joint to both Dev and Test teams and eliminate “shadow CI” processes. Such change will assure that the entire team gets a single view of the quality per each code commit, rather than dealing with multiple views that are not in sync.

3. Consider adjusting the test automation pyramid in a way that ensures both high-code coverage and time efficiency. Shifting more tests to the unit and API test buckets reduces the overall execution time and test flakiness that is often associated with UI-based testing.

4. Prioritize testing against target platforms through a risk-based approach, analytics and market data, to assure sufficient coverage in a shorter validation cycle.

5. Leverage “faster test frameworks” since few test automation frameworks can provide a faster feedback loop back by design. Frameworks like Espresso and XCUITest in the mobile testing landscape, and Puppeteer, JSDOM, PhantomJS in the cross-browser testing landscape provide faster setup time and test execution speed. Espresso and XCUITest will not need the full environment setup but Appium will. For the cross-browser testing, the above-mentioned solutions can test in a headless-browser mode and speed up the overall execution time, the environment setup (grid) time and more.

Fig. 2: The Test Automation Pyramid (Source: Accelerate Development with Automated Testing (Forrester, March 2017))

GUI Layer TestsSp

eed

Slowest

Fastest

Num

ber o

f Tes

ts

Fewest

Most

API or Service Layer Tests

Unit or Component Tests

Specialized Tests: Performance, Reliability, Security

Exploratory orManual Tests

Tighter release schedules leave less time to automate and test



Organizations are not following best practices of test automation and treating test code as normal code. This poses a challenge with automation in the digital space. In general, it is error-prone, however, with adding the complexity of the digital landscape, and mobile devices and desktop web platform instabilities, it further complicates the overall stability of the test automation suites.

Organizations are challenged with the following:1. Test suite flakiness is an ongoing affliction for test automation engineers. This is carried from

one release to the other, often neglected and expanding into a larger pain point for the organization. Flakiness of tests involves use of invalid and unstable object identifiers wrong definition of the test objectives and others.

2. As part of the testing stability as well as reliability, organizations can’t clearly identify key performance indicators (KPI) for their entire test suite data. i.e. – how many tests are constantly failing, how long it takes to complete an automation cycle within or outside of the CI job execution.

3. Failure to define proper pre-requisites that facilitate stable automation – ensuring that the platforms are in ready state prior to launching the test automation often result in false negatives, incomplete tests and delays. Having proper environment automation capabilities for all the digital platforms under test is hard to build and maintain.

4. Building test automation requires the same discipline as coding, the lack of source code management, reviews, optimization and maintenance results in un-optimized test code.

Test automation stability and reliability lacks best practices


“Another challenge with Test Data is the ‘it’s not my problem’ issue. In other

words, I need the ‘test data’ team to provide me with data. The more accurate

(and scalable) way is to ensure that any components that are built provide APIs

to create that data as well as tear it down.”Bryan OsterkampLead Technical Architect, Dev and Ops Infastructure Architecture Division, USAA


Recommended PracticesTo achieve the goal of minimized test automation flakiness, teams need strong discipline throughout the authoring and test automation suite maintenance life-cycle. There are several practices to consider to reach this goal and see greater value from the tests and have a continuous testing engine that actually works.

1. Author your test automation by following industry recommended practices:

a. Develop a page object model (POM) and make sure it includes the most unique object identifiers. Refer to an open source tool that helps validate the correctness of the object ID or use other validation tools.

b. Automate the tests that are most error-prone and repetitive, leaving out complex ones that are executed only in specific occasions.

c. Continuously maintain and validate the relevancy of the test code.

d. Consider breaking the tests into sub-test suites in an intelligent and context-based manner.

Remember: Tests aren’t flaky;

the test engineering practices

are causing the tests to be flaky.

Continuously monitor your tests and

isolate flaky tests.

Fig. 3: XPATH validation example on an iOS native app Object Tree (Source: Perfecto)



Fig. 4: Sketch Notes from SeleniumConf 2017 (source: KatJaSays Site)


2. Gain control over the test automation suites by remaining vigilant with overall quality and sustainability of the tests. Leverage various techniques and define a set of KPI’s around test stability:

a. # of failed tests cut by root cause (environment, test issue, real defect)

b. # length of each test execution run

3. Use source control tools to manage the test code, apply code and maintenance practices as with application code. Test code, just like normal code, should receive the same level of reviews, maintenance and various versions. This will ensure that there are proper reviews, backups, revision histories and other controlling elements to help debug issues faster, manage test suite growth, and eliminate and exclude tests no longer needed.

3. Build a stable lab environment that can assure high stability and continuous availability for test executions. When the lab setup is unstable, every test execution is a candidate for disaster. Make sure that the foundation for test automation is well built, available and can be easily launched as pre-condition.

5. Per each execution, assure that the target platforms are in ready-state mode. Prior to starting a new test execution on a mobile device or a desktop browser VM, make sure that the required platform under test was well setup with whatever preconditions that are required by the test – App installation, settings of the platform, application screen from where to start the test (i.e. Home Screen), and others.


Fig 5: Continuous Integration Dashboard Tracker (Source: Perfecto)



While having a stable test automation suite that covers app functionality is very important, establishing a fully optimized suite that can fit into short cycles is more important. Due to the growing size of test automation suites, organizations are finding it hard to control their test automation data.

The lack of intelligence as part of test automation slows down the entire quality of activities. The ability to have a learning mechanism with actionable data that can either reduce the test automation suite as well as execution time or identify critical issues with the tests and others is crucial.

There is a growing trend in the test automation landscape that involves growth in ML and AI engines, that can efficiently drill down into test data and help with test automation productivity and optimization.

When breaking down the lack of intelligence in test automation, the following common issues are:

• Teams can’t execute the “right” tests, while eliminating obsolete ones and duplicates, or tests that don’t add value to the overall testing activity. In this context, test execution management and the action of moving to advanced Agile and DevOps practices becomes a considerable challenge.

• Optimizing automation at scale that includes parallel execution, retries and resource management. It is a constant challenge to scale automation across platforms while dependent on data-driven resources and more.

• Efficient test data management is not well embedded into test framework. Activities such as creating users, accounts, setting up test data before the test runs and then tearing it down are not in the framework.

Intelligent Test Execution Management isn’t optimized enough to guide teams through the DevOps pipeline testing activities


Am I even working

on the right thing?

Fig. 6: Intelligent Test History Dashboard View Showing Problematic Tests (Source: Perfecto)

Click to view full size image


Intelligent Test Execution Management isn’t optimized enough to guide teams through the DevOps pipeline testing activities


Recommended PracticesThe challenge of data is growing as the product and test matures. Teams should consider adopting new tools that inject ML capabilities into test automation suites. Based on the recent research by Gartner (Hype Cycle for Emerging Technologies, 2017 – Fig. 7), tools are still evolving around ML. In the interim, organizations can leverage existing tagging mechanisms to easily drill down into test data post-execution, and leverage in-house analytics to determine test suite efficiency, platforms usage and optimize accordingly.

In addition, teams that are using BDD and/or ATDD practices, are well positioned to determine the level of use cases being automated, covered and that adhere to the customer requirement.

Fig 7: Gartner’s Hype Cycle for Emerging Technologies, as of July 2017 (Source: Gartner)

Hype Cyc le fo r Emerg ing Technolog ies , 2017

TIME

EXPE

CTAT

IONS

Plateau will be reached in:

InnovationTrigger

Peak of InflatedExpectations

Trough ofDisillusionment

Slope ofEnlightenment

Plateau ofProductivity

less than 2 years2 to 5 years5 to 10 yearsmore than 10 years

Smart Dust

4D PrintingArtificial General Intelligence

5G

Neuromorphic HardwareDeep Reinforcement Learning

Quantum ComputingDigital Twin

Human Augmentation

Serverless PaaS

Smart Workspace

Volumetric Displays

Conversational User Interfaces Brain-Computer Interface

Augmented Data Discovery Commercial UAVs (Drones)

IoT Platform

Edge ComputingSmart Robots

Virtual Assistants

BlockchainCognitive Computing

Connected Home

Cognitive Expert Advisors

Software-Defined Security

Augmented Reality

Virtual Reality

Enterprise Taxonomy and Ontology Management

Nanotube ElectronicsAutonomous Vehicles

Machine LearningDeep Learning


Organizations are challenged to maximize productivity using open source tools. When trying to build an efficient enterprise-ready test automation suite that can support advanced features, provide rich reports and leverage intelligent optimization engines, enterprises are finding limited open source tools. Things like environment automation, debugging capabilities and the ability to scale test automation are challenges of open source.

Another challenge is choosing the right tools; from commercial and service virtualization tools, cloud services, and others that integrate with the selected open source tools. In mobile test automation specifically, open source frameworks struggle to keep up with new technologies like image injection, face ID and fingerprint authentication, as well as chatbots. This forces an increase in manual test case execution.

Recommended PracticesWhile open source is on the critical path toward digital transformation and velocity by removing technological dependencies that exist with commercial tools, it is not sufficient to assure continuous quality.

The right path toward matured Agile practices lie in the combination of open source and commercial tools, especially when adding value around enterprise features, enhanced debugging capabilities and better authoring of tests.

By combining the right open source test automation framework with an enterprise commercial tool provider, it can allow execution of automation at scale in the cloud. Organizations can offload colossal challenges and risks from their engineers and be more successful in producing quality activities.


Evolving and maintaining test sets and maximizing productivity

NEW OPEN SOURCE

CLOUD-BASEDTOOLS

DEBUGGINGTOOLS

LATEST

TECHNOLOGIES

RELIABLE

FRAMEWORK

Fig. 8: Matching commercial and open-source tools throughout the DevOps Lifecycle (Source: Perfecto)

Click to view full size image


Every organization consists of various personas each with their own specific skills and strengths. When the tool stack isn’t tailored to individual team members’ skills, the overall quality of product health is impacted. Tool selection is important and needs to meet the testing practice that the team has adopted, whether it’s BDD, TDD, Agile or other.

What is happening in the digital test automation landscape is a shift toward modern open source test frameworks that requires enhanced coding skills from traditional test automation engineers. These requirements are not always accounted for in the overall project planning, therefore they introduce obstacles and delays the test automation completion. Having the ability to quickly switch from one framework to the other based on project need is becoming a demand that test automation engineers need to adopt.

Lastly, the role of quality assurance (QA) or the center of excellence (COE) body is fading away slowly, making room for a more Agile feature team that does the end-to-end product delivery from planning, through coding, testing and deployment. This evolution also results in re-alignment around tools, integrated development environments (IDE’s) and continuous integration (CI) servers. In many organizations this move is a great challenge that requires time and depends on a maturity curve.

Recommended PracticesTo scale automation and mature toward efficient DevOps, the entire organization needs to be in sync around practices, tools and people’s skill sets of the various products that are developed.

Succeeding in that objective, relies on the following:

1. Strategize your tool stack selection around the organizational skills and the systems development life cycle (SDLC) practices. If you practice BDD or ATDD - or just Agile - make sure that the tools your Dev and Test practitioners are using match that practice, and integrate into the overall CI/CD tools (i.e. Jira, Slack, Docker and IDEs)

2. Embrace open source and look ahead into what’s coming around automation and DevOps. Machine Learning/AI are potential productivity tools that can enhance quality across the pipeline, and should be considered tools to incorporate into the automation suite.

Test automation tool stack isn’t in sync with organizational capabilities causing testing inefficiencies


Fig. 9: Infrastructure as a freeway (source: USAA and Perfecto Webinar)

TOOL

CAPABILITIESTOOL

TOOL


Achieving continuous quality and maturing toward DevOps requires organizational alignment across tools, people and processes. A key for success in moving towards continuous testing and DevOps is to stabilize your test automation code – start small and grow the test suite while trying to shift left. These recommended best practices alongside the below visual can help guide your organization with testing decisions, reducing manual testing and increasing efficiency.

Fig. 10: Alignment of DevOps Principles, Practices and Challenges (Source: TechBeacon)

Inform Resolve

Principles

Automate Where Appropriate

Eliminate Risk Early

Eliminate Waste

Practices

Automated Testing

Automated Deployments

StandardizedDevOps Ecosystem

Value Stream Management

Challenges

#2 - Manual Testing

#21 - Manual Processes

#5 - Manual Deployments

#10 - Manual Release

#9 - No Integrated Tools Architecture

#16 - Traceability

#25 - Waste in Existing Processes

The Bottom Line

“Increasingly test automation and CI practices are merging into one discipline. Organizations should find the right ways to adopt existing test automation scripts and fit them into the CI executions. Such objective can be fulfilled by monitoring the stability of the test scripts over time and picking the most valuable and stable cases, that can provide the highest value back to developers. The next step in the maturity of CI/CD, would be to scale the CI quality suite and add more of these stable and high value test code to maximize efficiency and increase fast feedback velocity.”

Joe Colantonio

Joe Colantonio is currently a test automation architect for a large Fortune 100 company. He is also the founder of JoeColantonio.com, TestTalks, and Guild Conferences a blog, podcast and online conference platform dedicated to helping people and organizations succeed with creating automation awesomeness.


About Perfecto

Perfecto’s Continuous Quality Lab enables DevOps teams to accelerate

development, achieve continuous testing and monitoring and drive fast feedback

through actionable analytics for web, mobile & IoT apps. More than 3,000

customers rely on Perfecto’s cloud based Continuous Quality Lab as their digital

apps test environment and for authoring test automation executed on real

browsers, smart phones and devices under real end-user conditions.

For more information about Perfecto, visit perfectomobile.com,

join our community, or follow us on Twitter at @PerfectoMobile.


About the Author

Eran Kinsbruner A mobile development leader and veteran, Eran Kinsbruner has been in the dev & test industry since 1999. The creator and author of the quarterly Digital Test Coverage Index (Factors Magazine), and co-inventor of the test exclusion automated mechanism for mobile J2ME testing at Sun Microsystems, Eran serves as a prominent resource for every step of the app development cycle. As an influential blogger and speaker at global conferences like

StarEast, Eurostar Automation Guild and QAI Quest, Eran’s expertise has proven outstandingly valuable to developers and testers at organizations large and small. At a community level, Eran founded Meetups in both Boston and Israel to empower, educate, and bring together local dev/test experts, and launched a LinkedIn Group with over 5,400 mobile developers and testers from all over the world. Eran is the author of a book titled The Digital Quality Handbook, the book will offer intel on integrating quality into every step of the development cycle to help dev & test teams meet rising consumer’s standards for mobile and web applications. Currently, Eran is the Mobile Technical Evangelist at Perfecto, the leading cloud-based web and mobile quality lab, and was formerly the CTO for Mobile Testing at Matrix, and managed technical teams at Qulicke & Soffa, Sun Microsystems, General Electric and NeuStar.

59 percent - info.perfectomobile.com · a recent research report conducted by ... and test...

Documents