A Safety-Critical Java Technology Compatibility Kit

Hans SøndergaardStephan Korsholm

VIA University College, Horsens, Denmark&

Anders P. RavnAalborg University

JTRES 2014October 2014

2

Test Suite – why?

• Java Community Process (JCP) The formal process for developing or revising Java technology Specifications.

• Final ReleaseThe final stage in a JSR when the Specification, RI, and TCK have been completed.

• Technology Compatibility Kit (TCK)The suite of tests, tools, and documentation that allows an organization to determine if its implementation is compliant with the Specification.

3

Ingredients in a TCK

• TestsTest cases and test datasets

• Tools for test execution Execute the tests and generate results

• Test report documents Classification of results

4

Test Case Development

Checks conformance with a specification Must be implementation independent:

- data representations must be hidden Black-box testing using public entities only

Must check specification of classes Invariants and Pre- and Post-conditions? Intuition or formalization in development ?

! JML (Java Modeling Language)

5

Background

• Anders P. Ravn and Hans Søndergaard. A Test Suite for Safety-Critical Java using JML. 2013.

• Yoonsik Cheon and Gary T. Leavens. A Simple and Practical Approach to Unit Testing: The JML and JUnit Way. 2003.

• A. Sarcar and Y. Cheon. A new Eclipse-based JML compiler built using AST merging. 2010.

• JML4c http://www.cs.utep.edu/cheon/download/jml4c/index.php

• L. Zhao, D. Tang, and J. Vitek. A Technology Compatibility Kit for Safety Critical Java. 2003.

6

Specification of Invariants

Class: public abstract class HighResolutionTime

Specification:” A time object in normalized form represents negative time if both components

are nonzero and negative, or one is nonzero and negative and the other is zero.”

Invariant:

public invariant (this.getMilliSeconds() >= 0 && (0 <= this.getNanoSeconds() && this.getNanoSeconds() < 1000000)) || (this.getMilliSeconds() <= 0 && (-1000000 < this.getNanoSeconds() && this.getNanoSeconds() <= 0));

7

Pre/Post-conditions Normal behaviourClass: public abstract class HighResolutionTimeSpecification:public void set(HighResolutionTime time) “Change the value represented by this to that of the given time. … Throws ClassCastException if the time parameter is not of the same class as this. Throws IllegalArgumentException if the time parameter is null. ...”

public normal_behaviour requires time != null && this.getClass() == time.getClass();

ensures this.getMilliseconds() == time.getMilliseconds(); ensures this.getNanoseconds() == time.getNanoseconds(); ensures this.getClock() == time.getClock();

8

ExceptionsClass: public abstract class HighResolutionTimeSpecification:public void set(HighResolutionTime time) “Change the value represented by this to that of the given time. … Throws ClassCastException if the time parameter is not of the same class as this. Throws IllegalArgumentException if the time parameter is null. ...”

also public exceptional_behaviour requires time == null; signals (IllegalArgumentException) true; public exceptional_behaviour requires time != null && this.getClass() != time.getClass(); signals (ClassCastException) true;

JML annotations compiled and executed• Compilation of JML annotations– Uses a JML compiler– Compiled to Java bytecode– Translated to runtime assertion checks

• Execution of the extended Java bytecode– Constructor:

Runtime check of PreconditionConstructor executionRuntime check of Postcondition and Invariant

– Method:Runtime check of Invariant and PreconditionMethod executionRuntime check of Postcondition and Invariant

• Violation of runtime checks– Throws a JMLAssertionError

Test CaseUsing class TestCase from JUnit

public class TestAbsoluteTime extends TestCase { ... public void test(int i) { AbsoluteTime abs; switch (i) { ... // AbsoluteTime(long millis, int nanos) case 32: new AbsoluteTime(0,0); break; case 33: new AbsoluteTime(0,1000001); break; ... // wrap around value case 37: abs = new AbsoluteTime(Long.MAX_VALUE,1000001);

assert abs.getNanoseconds() == -999999; break; ...

Test Cases - exceptions // AbsoluteTime(AbsoluteTime time)

case 39: abs = new AbsoluteTime(); new AbsoluteTime(abs); break; case 40: abs = null; try{ new AbsoluteTime(abs); assert false; } catch (IllegalArgumentException e){};

break; ... case 91: ... default: break;}

public static final int testCount = 91;

12

Beyond Unit Testing Class: public abstract class ClockSpecification:

public abstract RelativeTime getResolution(); “Gets the resolution of the clock defined as the nominal interval between ticks. “ public behaviour requires true;

ensures \result != null; ensures (\result.getMilliSeconds() > 0 || \result.getMilliSeconds() == 0 && \result.getNanoSeconds() > 0);

Is unit testing sufficient for a Clock?

13

Specification of clock test method

Specification: behaviour requires true; ensures !failure; ensures (\forall int i; 0 < i && i < SIZE; sample[i-1].compareTo(sample[i]) < 0); // time is moving forward ensures (\forall int i; 0 < i && i < SIZE; (sample[i].subtract(sample[i-1])).compareTo(c.getResolution()) >= 0 ); // distance between two samples >= resolution

Use JML to specify test methods!

case 14: clockTest(clk); break;

14

Implementation of the clock test methodboolean failure; AbsoluteTime[] sample; RelativeTime resolution;

behaviour ... // JML specification here

void clockTest(Clock c) { resolution = c.getResolution(); sample = new AbsoluteTime[SIZE]; failure = false;

sample[0] = c.getTime(sample[0]); for (int i = 1; i < SIZE; i++){ int j = 0; do { sample[i] = c.getTime(sample[i]); j++; } while (sample[i].subtract(sample[i-1]). compareTo (resolution) < 0 && j < MAXLOOPS ); if (j == MAXLOOPS) { failure = true; break;} }

Tools for test execution • Requirements– Executable on different VMs, • e.g. VMs for resource constrained platforms• HVM has been extended with reflection

• JML compilers– jmlc works with Java 1.4 source files– jml4c works with Java 1.5, including generics

• built on the Eclipse Java compiler

• JUnit– a subset of junit-3.8.2 is used

• package framework

– class TestResult has been extended• add JML error to a list of errors 15

TestSuite assembly and executionpublic class AllTests { public static final TestResult result = new TestResult(); public static void main (String[] args) { TestSuite suite = new TestSuite();

suite.addTest(test_AbsoluteTime); suite.addTest(test_RelativeTime); ... suite.run(result); ...

// TestCase objects defined here (next slide) }

16

TestCase object

public class AllTests { public static final TestResult result = new TestResult(); ... public static TestCase test_AbsoluteTime = new TestAbsoluteTime(”AbsoluteTime”) { public void runTest () { try { for (int i = 1; i <= TestAbsoluteTime.testCount; i++) test(i); } catch (JMLAssertionError e) {result.addJMLError(this, e);} catch (Throwable e) { result.addError(this, e); } } }; }

17

Test report• No errors

Test cases: 2Test errors: 0JML errors: 0

• If e.g. an JML errorTest number is 67,By method AbsoluteTime.addRegarding specifications at File "./src/javax/realtime/AbsoluteTime.java", line 216, character 15With values nanos: 1000001 millis: 0

Where are we?javax.realtime

47 classes and interfaces24 finished15 are Empty

Where are we?javax.safetycritical

35 classes and interfaces12 (nearly) finished 3 are Empty

21

Conclusion• Tests

JML is an efficient vehicle for generating test conditions and may even improve the informal specifications.

• Concrete Test casesGuided by specifications and also additional cases based on “intuition”.

• Tools for test executionThe tools involved are able to run on a resource constrained VM (HVM).

• Test report documentsA simple test report.

Conformance tests of other SCJ implementations

• Extract the JML specifications from our SCJ implementation (simple)

• Merge this extract with another SCJ implementation (more complicated)

• Use the test cases (no change)• Compile classes with JML annotations into Java

bytecode, using jml4c (? depends of the VM)• Run the test suites, using (part of) JUnit (simple).

22