Protecting the irreplaceable | f-secure.com

Olli-Pekka Puolitaival

• Lead quality engineer test automation

• 5 yrs test automation

• 4 yrs research

• Idea of model-based testing

• Story of OSMO tester

• How we test

• Experiences & tips

Index

MBT is new way to think

• Traditional way to think

• People do testing

• Regression automation reduce peoples work time

• New way to think

• Smart people want to do testing in smart way

• Computers are fast and hard worker, let them also try “stupid” things

5

6

7

8

9

10

11

12

13

Traditional MBT tools

• Hard to take in use

• Closed

• Limited extending

• Advanced algorithms

• Expensive

Osmo tester

• Easy start

• Flexible

• Easy to extend

• Simple algorithms

• Free (open source)

14

Why we made osmo?

Osmo tester idea

public class ExampleModel{

int i = 0;

@Guard(“increase") public guardfunction() { return true; //Return true if this transition is allowed to execute } @Transition(“example") public void transitionfunction() { System.out.println(i++); //This code is executed when transition is selected }

… public static void main(String[] args) { OSMOTester tester = new OSMOTester(new ExampleModel()); tester.generate(); }

}

15

“Model is pure java code, Which makes possible to use Any java library”

Many additional features

• Support online and offline MBT

• Requirements tagging

• Coverage calculations

• Many test design algorithms, also manual drive

• Coverage, time and length based stopping criteria

• Self written adapters to the SUT

• Custom test output formats

• Easy to integrate with Jenkins

• Model reuse and combining

• Probability numbers support

• Available: http://code.google.com/p/osmo/

16

Easy to start and custom for your needs

Tips

• Modelling

• Good model is high level, modular and cover core functionality

• Start from small and continue iteratively

• Adaptation

• Reuse code from test automation or development if possible

• Running

• Run tests always automatically

• Fixed run length for verifying and ”unlimited” for finding some new

• Reports

• Report what you need to know, visual is mostly better

• OSMO is not good if you need very sophisticated algorithms, but most people don’t need those

17

The domain is content cloud

Cli

ent

smo

ke

Our continuous integration

19

Backend components

Backend components

Clients

Backend components

Clients

Client Integration

test environment

Clients

Un

itte

st

Bu

ild

ing

Backend integration

Test Environment

Smo

ke

test

Reg

ress

ion

te

st

Pe

rfo

rman

ce

tes

t

Performance

Test Environment

Cli

ent

reg

ress

ion

Chain of Jenkins jobs runs everything

Jenkins

Android phone

Android client

CAN Solution Backend

Current test setup

Selenium webdriver

Firefox

Test status radiator

Web adapter

Nativedriver client

Nativedriver server

Model

OSMO

Android adapter

“run once per hour”

What kind of model?

• Idea of this testing

• Do actions and verify that other clients data is same

• Model includes actions which can be seen in other client like

• Upload a file

• Remove a file

• Rename a file

• Check quota

• Check that file exist

• Check that all files exist as supposed

21

Example test run

22

Osmo test engine Android Client Web page

Login

Upload file1.jpg

Login

Check file1.jpg

Upload file2.jpg

Remove file1.jpg Check that only file2.jpg exist

Logout

Logout

What are changing in test runs

• New file set each time

• New test accounts each time

• Random amount of uploaded files

• Randomly removed files

• Randomly renamed files

• Random checks

23

Experiences

• Useful testing because

• Tells well the status of core functionality

• Is very close the new user use case

• Test generation is easy to do with OSMO

• Modeling is easy and support well iterative development

• Test parameterization is easy

• The challenges

• Adapters are not easy to do and maintain

• I keep my testing enough simple, then I don’t need too much maintenance

• Automating test run was challenging in first time

• Developers was helping me

24

Next steps

• Look for new adapter for android

• Extend testing on other platforms

• Extend the power

• Cover many environments

• Many machines for testing

• Run continuously several kind of test sets:

• Short run for radiator

• “infinite run” for finding weakest point

25

My MBT cooking receipt

1. Find out the core functionality

2. Make a simple model of that

3. Make adaptation for test running

4. Automate test runs

5. Run model continuously against SUT

6. Put the results visible in your room for seeing the “temperature”

7. Keep the radiator GREEN

26