implement combinatorial test patterns for better mobile and iot testing

T10TrackSession4/21/20162:00PM

"ImplementCombinatorialTestPatternsforBetterMobileandIoT

Testing"

Presented by:

Jon Hagar

Grand Software Testing

Broughttoyouby:

340CorporateWay,Suite300,OrangePark,FL32073888-268-8770·904-278-0524·[email protected]·www.techwell.com

Jon Hagar Grand Software Testing

Jon Hagar is a systems software engineer and testing consultant, supporting software product integrity and verification and validation (V&V), with a specialization in mobile and embedded software system testing. For more than thirty years, Jon has worked in software testing and engineering projects. He authored Software Test Attacks to Break Mobile and Embedded Devices; consults, presents, teaches, and writes regularly in many forums on software testing and V&V; and is lead editor/author on committees including OMG UTP model-based test standard, IEEE 1012 V&V plans, and ISO/IEEE/IEC 29119 software test standard. Contact Jon at [email protected].

Implement Combinatorial Test Patterns for Better Mobile and IoT Testing

Jon D. Hagar, Consultant, Grand Software Testing [email protected]

1 Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"

• Scary stories • It only takes a few minutes of using an App before users like or hate it

• Worse than that. . .

– Many users will post a poor social media review of the app or device – You may be on the nightly news (bad press is not good) – A question I get a lot, “how do we deal with fragmentation?”

• So You want to be – Part of the billions of devices

» You want to be GREAT

2

The Mobile-IoT Opportunity

Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"

What is on the Horizon for Testing Hype Cycle for Emerging Technologies, 2014

Reference: Gartner report 2014 Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"

What We Will Cover

• Introduction and definitions

• Problem example

• A combinatorial test attack pattern

• Some Combinatorial (CT) Tools – Demo

• Wrap up


Basic Definitions • Test – the act of conducting experiments on something to determine

the quality (ies) and provide information – Many methods, techniques, approaches, levels, context

– Considerations: input, environment, output, instrumentation

• Quality (ies) – Value to someone (that they will pay for) – Functional

– Non-functional

– It “works”

– Does no harm

• Are there (critical) bugs?


The Mobile-IoT Embedded Space

6

Embedded

IoT

Mobile-Smart Personal Computers

Big Iron

Many Options Huge Numbers of Devices (billions)

Numbers of Devices (millions)


• Embedded – Software contained in “specialized” hardware… • Mobile and handheld devices—small, held in the hand, connected to

communication networks, including – Cell and smart phones – apps – Tablets – Medical devices

• IoT – Internet of Things are traditional devices with software and comms added • Mobile, Handheld, IoT typically have:

– Many of the problems of classic embedded systems – The power of PCs/IT – More user interfaces than classic embedded systems – Fast and frequent updates

• Devices are “evolving” with more power, resources, apps, etc. • Mobile and IoT are (currently) the “hot” area of computers/software

You know what they are. . . Right? Embedded, IoT, Mobile and Handheld?

Test Brakes What’s this?


• Embedded – Software contained in “specialized” hardware…

– Minimal networking-communication

PLUS

• Mobile and handheld smart devices—small, held in the hand, highly connected (web, cloud, servers,….)

• IoT – Internet of Things are “traditional” and new devices with software and communication added

What is an IoT device?

Test Brakes


IoT – 2.5 Main Segments (and a name a few categories)

9


Must be Interoperable Across segments

Industrial Value 2x Consumer

Industrial 4.0 Middle Consumer

Home - Security - Monitoring & control - Infotainment Human - Health - Fitness - Info

Vehicles - driverless - monitoring - Infotainment Office - Security - Energy - Worker info Medical - Health monitor/control - Records Retail - Ordering - Checkout - Advertize

Cities/States/Nation - Health - Safety - Info - Control and Monitor Transportation - Vehicles - Navigation - Logistics Worksite/Factories - Ops - Control - Info

Defining Software Capabilities

• James Whittaker defines 4 fundamental capabilities that all software possesses 1. Software accepts inputs from its environment

2. Software produces output and transmits it to its environment

3. Software stores data internally in one or more data structures

4. Software performs computations using input or stored data

• To this, we expand and refine based on an mobile-IoT context: – Within time

– Using specialized hardware (as sub of items 1 and 2 above) and control

– Security and privacy

– Different development lifecycle constraints


Attack-based Testing Patterns What is an attack?

• A pattern (of testing) based on a common mode of failure seen over and over – Some see this as a negative, when it is really a positive – Attacks seek the “bugs” that may be in the software – May include or use classic test techniques and test concepts

• Lee Copeland’s book on test design • Many other good books

• A Pattern (more than a process) which must be modified for the context at hand to do the testing

• Testers learn mental attack patterns when working over the years in a specific domain


Example Attacks (from “Software Test Attacks to Break Mobile and Embedded Devices”)

• Attack 1: Static Code Analysis • Attack 2: Finding White–Box Data Computation Bugs • Attack 3: White–Box Structural Logic Flow Coverage • Attack 4: Finding Hardware–System Unhandled Uses in

Software • Attack 5: Hw-Sw and Sw-Hw signal Interface Bugs • Attack 6: Long Duration Control Attack Runs • Attack 7: Breaking Software Logic and/or Control Laws • Attack 8: Forcing the Unusual Bug Cases • Attack 9 Breaking Software with Hardware and System

Operations • 9.1 Sub–Attack: Breaking Battery Power • Attack 10: Finding Bugs in Hardware–Software

Communications • Attack 11: Breaking Software Error Recovery • Attack 12: Interface and Integration Testing • 12.1 Sub–Attack: Configuration Integration Evaluation • Attack 13: Finding Problems in Software–System Fault

Tolerance • Attack 14: Breaking Digital Software Communications • Attack 15: Finding Bugs in the Data • Attack 16: Bugs in System–Software Computation • Attack 17: Using Simulation and Stimulation to Drive

Software Attacks • Attack 18: Bugs in Timing Interrupts and Priority Inversion • Attack 19: Finding Time Related Bugs

• Attack 20: Time Related Scenarios, Stories and Tours

• Attack 21: Performance Testing Introduction • Attack 22: Finding Supporting (User) Documentation

Problems • Sub–Attack 22.1: Confirming Install–ability • Attack 23: Finding Missing or Wrong Alarms • Attack 24: Finding Bugs in Help Files • Attack 25: Finding Bugs in Apps • Attack 26: Testing Mobile and Embedded Games • Attack 27: Attacking App–Cloud Dependencies • Attack 28 Penetration Attack Test • Attack 28.1 Penetration Sub–Attacks: Authentication —

Password Attack • Attack 28.2 Sub–Attack Fuzz Test • Attack 29: Information Theft—Stealing Device Data

• Attack 29.1 Sub Attack –Identity Social Engineering

• Attack 30: Spoofing Attacks • Attack 30.1 Location and/or User Profile Spoof Sub–Attack • Attack 30.2 GPS Spoof Sub–Attack • Attack 31: Attacking Viruses on the Run in Factories or

PLCs • Attack 32: Using Combinatorial Tests • Attack 33: Attacking Functional Bugs


In Mobile-IoT Many Example Combinations: Standards, Interfaces, Protocols, Platforms, Software, and Data Patterns

13

Network-Comm


Many of these Combinations Will Need Testing

Exercise: How should we test these? (How do you do it now?)

14

How many tests are needed? Coverage of combinations? How do we find errors?


Combinatorial Testing (CT) Math Offers Solutions

• CT has long history of Usage

• CT uses many tools

• CT is still underused

• CT has some cool possibilities

• CT should be one of the attack techniques used

• Find out how CT can help your testing


Math-based Testing

Testing is a sampling problem: How can Math aide testing?

• Test systematically the numbers of devices, configurations, networks, etc.

• Use sampling in environments and quality control

• Use sampling of data from the input domain space

• Use Big Data Analytics to feed testing


Pattern Attack 32: Combinatorial Tests

17

• When to apply this attack? – There are numerous related variables and variable values which

interact

– Validation Analysis Upfront

– Testing throughout the life cycle and in Maintenance Mode

• What faults make this attack successful? – Untested configuration combinations

– Data “bugs”

• Who conducts this attack? – Tester, analyst

• Where is this attack conducted? – Tool running in the lab or field

• How to determine if the attack exposes failures? – A test fails to meet success criteria

– Hard crash - NIST Data


Attack 32: Combinatorial Test Patterns

18

• How to conduct this attack – basic pattern

– Identify combinatorial situation

– Identify combinatorial tool

– Identify variables

– Identify values

– Identify constraints on values

– Enter variables and values into tool with constraints

– Exercise resulting combinations in usage scenario tests or automated tests

– Look for failures

– Repeat and refine as needed


• Android

Example Usage: Numbers of data choices, devices and configurations

19

• Hardware

• IoT devices

• Data

• Routers

• IoT Home Protocols

How many Tests? 10 x 2 x 13 x 6 x 6 x 7 = 65,520 tests!


Using the ACTS Combinatorial Tool: Example

20

Parameters:

Andriod AppPlatform [Device 1, Device 2, Device 3, Device 4, Device 6, Device 7, Device 8, Device 9, Device 10]

IoTProtocolHome [true, false]

IoT Devices

[Refrig, Stove, mircrowave, TV, front door, Garage door, Home gaurd, Stereo, Temp Control, Lights, Drapes, Water Heater, window openers]

Routers [0, 1, 2, 3, 4, 5]

Comm providers [Cell1, Broadband, cable, Cell 2, Space based, Vendor godzilla]

Data [1, 0, -1, 99999, -99999, 100, -200]

Test Case# Andriod AppPlatform IoTsHome IoTDevices Routers Comm providers Data

0 Device 1 false Refrig 1 Broadband 0 1 Device 2 true Refrig 2 cable -1 2 Device 3 false Refrig 3 Cell 2 99999

3 Device 4 true Refrig 4 Space based -99999

4 Device 6 false Refrig 5 Vendor godzilla 100

5 Device 7 true Refrig 0 Cell1 -200

119 Tests


Other Statistical Tools to Consider

General Technique Concept

Tool Examples (Note 1)

Examples of where technique can be used

Specific sub- technique examples

Combinatorial Testing

ACT [4], Hexawise[5] rdExpert [6] PICT[7]

Medical, Automotive, Aerospace, Information Tech, avionics, controls, User interfaces

Pairwise, orthogonal arrays, 3-way, and up to 6 way pairing are now available

Design of Experiments (DOE)

DOE ProXL[8] DOE++ [9] JMP [10]

Hardware, systems, and software testing where there are "unknowns" needing to be evaluated

Taguchi [12] DOE

Random Testing

Random number generator feature used from most systems or languages

Chip makers, manufacturing quality control in hardware selection

Testing with randomly generated numbers includes: fuzzing and use in model-based simulations

Statistical Sampling SAS [10]

Most sciences, engineering experiments, hardware testing, and manufacturing

Numerous statistical methods are included with most statistical tools

Software Black box Domain Testing

Mostly used in manual test design, though some tools are now coming available [11]

All environments and types of software tests. These are “classic” test techniques, but still underused

Equivalence Class, Boundary Value Analysis, decision tables (Note 2)


Many Variables and Choices

22


And the ACTS tool in real-time (be on the high wire)

Tool Demo

Link To ACTS Tools


Expanding Combinatorial Testing

• For Dev-Ops – Sampling user data

– Model-based testing

– Advanced data selection

• Support domain testing

• Do test without an Oracle – NIST Study

– Combine with automation

– Run 4-to-6 way combo’s

– Look for Major Crashes


Summary

Common Mobile-IoT Problem

• Data selection

• Dealing with numbers of configurations – Hardware, Software, Protocol,

etc.

• Testing within time and budget

Overlooked Solutions

• Data analysis with sampling – Classic testing

• Combinatorial Testing with tools – Test Automation (not a

requirement)

• Reduce combinations to fit within budget and schedule


•••••••

•••

Copyright 2016 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”

Book List

• “Software Test Attacks to Break Mobile and Embedded Devices” Jon D. Hagar, 2013

• “How to Break Software” James Whittaker, 2003

– And his other “How To Break…” books • “A Practitioner’s Guide to Software Test Design” Copeland, 2004

• “Introduction to Combinatorial Testing” D. Richard Kuhn Raghu N. Kacker Yu Lei , 2013

•

Copyright 2016 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”