workshop test engineering and test management€¦ · robert rotter, product manager ... test...

V0.1 | 2019-03-11

PREEvision Userday 2019

Test Engineering and Management with PREEvision

2

Micha Dannenhauer

Senior Field Application Engineer

TE/TM, Functional Safety

At Vector since 2017

Robert Rotter,

Product Manager and Usability Engineer for PREEvision

RQM, CHM, TE/TM, Usability

At Vector since 2004

Short introduction round (10 min)

Name, company, background

Test Engineering use cases, actual or planed

Knowledge, experience in that field

Expectations for the workshop

Goals and Focus of the workshop (50min)

Introduction in the test process in PREEvision

Flyover the test phases, the corresponding tasks and editors

Last but not least: discussion, questions and answers

What is the focus and what are the expectations?

TE/TM

3

TE/TM

Test Process in PREEvision

Test Specification

Implementation

Planning

Execution

Analysis

Additional material

Agenda

4

How PREEvision E/E helps

Integration of the involved use cases

Single source of truth

Traceability:

Requirements linked with Test Specifications, Test Implementations and Test Results

Advanced traces between model data test

data

One tool to manage tests along the development lifecycle:

Manual test process completely in PREEvision

Management and analysis of automated test with flexible interfaces for tool chain integration for different test platforms

Typical Challenge in E/E Development

Growing complexity

Inconsistencies are really dangerous

Time to market is still very important

Motivation for TE/TM in PREEvision


5

Classic Test Engineering and Management in E/E Development


Test Engineering

and ManagementGap in time

6

Data Management, Collaboration, Platform Management, Safety, Change Management

Requirements and Test Engineering in PREEvision


Requirements Engineering and Management

Test Engineering and Management

FunctionalArchitectureDesign

SOADesign

SoftwareDesign

PhysicalArchitectureEvaluation

PhysicalArchitectureDesign

PhysicalSystemDesign

COMDesign

WH SeriesDesign

WHArchitectureDesign Specified Artifact under Test

Subject under Test

REQ-X Mapping

7

Data Management, Collaboration, Platform Management, Safety, Change Management

Overview Requirements and Test Engineering Processes


Requirements Engineering and Management

Test Engineering and Management

Specification of Tests

Design & Implementationof Tests

Execution & Documentationof Tests

Evaluation & Analysisof Test Results

Planningof Tests

Template Definition

Agreementwith Supplier

System Specification

Component Specification

8

Test Engineering Process and involved Roles





Evaluation & Analysis of Test Results

Planningof Tests

Requirements

Requirements Engineer

Test Specification

TestEngineer

Library

Reuse

SpecificationHeading

ItemItem Step

Derive

Test Implementation

Manual test sequence

Test script

TestEngineer

Synthesis

Create

Test Planning & Tasks

TestManager

Plan

Execute

Test Execution

Verdict

ar

Verdict

ar

TestEngineer

Test

Test Analysis

ReportReport

Report

TestManager

Analyze

9

Test Engineering: Import & Export Interfaces







Requirements Test Analysis

ReportReport

Report

Planningof Tests

Execute


TestManager

TestManager

Test Implementation


Test script

TestEngineer

Test Execution

Verdict

ar

Verdict

ar

Test Specification

TestEngineer

Library

Reuse

TestEngineer


ItemItem Step

AnalyzeDerive

Synthesis

Create PlanTest

Test Execution Tool

Vector CANoe3rd party

Parse for Verdicts andRequirements / Test Specification Associations

Test Design & Implementation Tool

Vector vTESTstudio3rd party

SynchronizeTest Scripts

Traceability Information

Requirements Source

RIF, ReqIFXLS

Import & Update

TestItemsSource

ReqIF

10


Test Engineering: Data Model

Test Case

Test Step 111

Test Step 112

init run shutdown

Test Step Run 113

Test Case Run

Test Step Run 111

Test Step Run 112

Test Execution Increment

Test Task

Test Run

Test Data Package

Test Heading 1

Test Specification

Requirements Package

Req. D

Test Heading

Requirements

TestSpec

Test Implementation

TestExecution

Req. A

Test Item Step 111

Test Item Step 112

Test Item Step 113

Manual Test Sequence

Test Step 113

Test Item 11Req. B

Req. C

Requirements can be part of other Product Lines than the Test Project.

Test Project

11

TE/TM


Test Specification

Implementation

Planning

Execution

Analysis

Additional material

Agenda

12

Tasks and Artifacts

Test Specification

Describe the Test Specification (test ideas)

Test Items and Test Item Steps

Map Requirements or E/E artifacts ( Subject under Test)

Define structure

Test Headings and

Test Information

Build up a Library

reusable Test Items,

reusable Test Item Steps

Conditions ( Usage relation)

Parameters ( Usage relation)

Requirements


Test Specification

TestEngineer

Library

Reuse


ItemItem Step

Derive

Requirements Source

RIF, ReqIFXLS

Import & Update

TestItemsSource

ReqIF

13

Test Item Steps

Provide meaningful details that can be used for implementations

Different roles: initiation, run, shutdown

Test Specification Library contains:

Test Conditions

Describe typical base preconditions

Related to relevant Test Items

Test Parameters

Applicable to Condition and Test Items

Related to relevant Test Items and Conditions

Test Item Steps

Steps and Items to be reused in “operative” Test

Reuse in specific roles

Test Specifications

Container

optional link to “Specified artifact under Test”

Test Heading & Test Information

Structure and Documentation

Importable with ReqIF

Test Items “Idea of a test” to be implemented

Link to requirements and other artifacts

Link to test implementation

Manual / Scripts

For different platforms

Attributes like

Precondition,

Description

Postcondition

Artifact details

Test Specification

14

Test Specification

Test Explorer

Guides through Process

Specialized Table Editors

Start on

Test Project

Product Line

and several other artifacts

The Explorer always starts with the corresponding Product Line as context …

but opens the right category and editor for the start artifact

15

Test Specification Editor

Test Specification

16

TE/TM


Test Specification

Implementation

Planning

Execution

Analysis

Additional material

Agenda

17

Tasks and Artifacts

Implementation

Create Test Implementations manually or via synthesis which then can be brought to execution.

Detail the Test Ideas in Manual Test Cases

Add Test Steps or Test Procedures, witch again may contain Test Steps

Add Measurement Value definitions to Test Cases or Test Steps to ask for detailed feedback for values of interest

Connect to the toolchain for Automated Test definition

Export Test Items as Traces

Import Test Scripts with references to prepare for test planning and later import from test execution tools

Test Implementation


Test script

TestEngineer

Synthesis

Create

Test Specification

Test Design & Implementation Tool

Vector vTESTstudio3rd party

SynchronizeTest Scripts

18

Test Script

Container for Manual Tests

Automatic Test Case

Connectors for Tracing of Automated tests

File Hierarchy

Container for test execution files

Manual Test Sequences

Container for Manual Tests

Manual Test Case

Executable Test Item

Connectors for Tracing

Test Procedure / Test Step

Detailing of Tests

Connectors for Tracing

Measurement Value Definition

Definition for additional values to be gathered during manual test execution

Artifact Details

Implementation

DHA

19

The Synthesis offers the option to synthesize Manual Test Sequences based on Test Specification artifacts.

ensure the same granularity

ease the work, in cases when specifications are already quite detailed

Starts on owner of Test Item (Test Specification or Test Heading)

Partial synthesis possible (only specific Test Headings)

Synthesis of Test Item Steps into

Procedures or

Test Steps

Update option if Test Items have changed

Functionality also available as Metric Block for Customizing

Synthesis

Implementation

Test Item Step

Test Heading

Test Item

Test Step or Test Procedure

Manual Test Sequence

Test CaseRelation

Synthesis

20

Platform Derivation

Implementation





Planningof Tests

Requirements


Test Specification

TestEngineer

Library

Reuse


ItemItem Step

Derive

Test Implementation


Test script

TestEngineer

Synthesis

Create


TestManager

Plan

Execute

Test Execution

Verdict

ar

Verdict

ar

TestEngineer

Test

Test Analysis

ReportReport

Report

TestManager

Analyze

SIL SIL

REQ

ECU SW-CCUF

Cluster-HIL

HIL

Cluster-HIL

HIL

Vehicle

FMU

Vehicle

FMU

21

Test Implementation Layer Editors

Implementation

Manual Test Sequence Editor

Test Implementation Editor

22

TE/TM


Test Specification

Implementation

Planning

Execution

Analysis

Additional material

Agenda

23

Tasks and Artifacts

Planning

During the planning the focus is on particular Test Items

Plan the Test Project

Define test phases in Test Execution Increments

Define the test scope via concrete Test Tasks and define in which increment they should be tested

Distribute Test Tasks to various testers or declare a Test Team for testing via the Picking Method.


TestManager

24

Test Tasks can be assigned to

Human Resources

Test Task Scope (partial test possible)

Test-Tasks have

Subject Under Test

Test Environment

Device under Test

Test content (Manual Test Sequences or Test Scripts)

Test Project for execution of tests has

A team (Resources)

A scope of Test Specifications & implementations (option: View Package)

Start and end date

Test Increments subdivide the Test Project into test phases

Test Increments form a straight sequence (no parallel increments)

Result of a recent Increment is superior to result of an older Increment

Used for planning and reporting

Within Test Increments a tree structure of Test Task Package and Test Tasks can be build

Test-increments of a project have i.e.

One or more Subjects Under Test

Scheduled Increment scope

Artifact details

Planning

25

Test Increment Editor can be opened on Test Project for Planning of

Scope to be tested

Requirement Base

Test Specifications

Test Implementations

Team

Responsibilities for Packages and Tasks

Schedule

Test Increment Editor

Planning

More options in Property Editor, i.e.

Technical Resources

Devices

Environments

Required Properties

26

Used for the test execution planning

Create a Test Task for every tester

Define the subject under test (SUT) and the test platform with the exact hardware or software version number

Shows Execution State and the results from the Test Execution Editor including the verdicts for controlling

Test Task Editor

Planning

27

TE/TM


Test Specification

Implementation

Planning

Execution

Analysis

Additional material

Agenda

28

Tasks and Artifacts

Execution

During the test execution phase, the planned tests are executed and the results received (verdicts) are incorporated.

Run the tests and add verdicts for the Test Steps of manual tests

Import test protocols for automated tests

Execute

Test Execution

Verdict

ar

Verdict

ar

TestEngineers

Test Execution Tool

Vector CANoe3rd party

Parse for Verdicts andRequirements / Test Specification Associations

29

Test Execution Editor

Execution

30

Highest verdict is Passed, lowest is Error

The lowest verdict is aggregated up to the artifact on top (up to Manual Test Sequence).

Verdict Logic

Execution

hig

hlo

w

31

Process for documented correction of measurement values

No delete / change of verdicts

Additional Verdict and Acceptance as separate steps

Verdict Override Aggregation

Execution

32

TE/TM


Test Specification

Implementation

Planning

Execution

Analysis

Additional material

Agenda

33

Overview

Analysis

During the test analysis phase, you keep track of the current test progress in the different test phases.

Test Project

Test Specification Analyzer: analyze the coverage of Requirements by Test Items.

Test Implementation Analyzer: analyze the coverage of Test Items by Test Cases.

Test Project Execution Analyzer: shows test results over the entire Test Project.

Device under Test History Analyzer: provides information about the test history of the Devices Under Test.

Test Increment

Planning Analyzer: provides an overview about the execution scope for a test increment. It lists all Test Items that are planned for the selected Test Increment

Execution Analyzer: shows all test results of the selected and executed Test Increment.

Trend Analyzers monitor different information over time of a Test Project.

Test Analysis

ReportReport

Report

TestManager

35

Analysis of

Coverage

Requirement covered by Test Specifications

Test Specifications covered by Implementations

Implementations covered by Test results

Current Result analysis

Results, based on Requirements / Test Specifications / Implementations

Progress Analysis – Analysis across the Test Project Phases

Results based on Requirements / Test Specifications / Implementations

Current State

Analysis

36

Cockpit Test Engineering and Test Management

Analysis

38

Cockpit Test Engineering and Test Management – Test Project Navigation

Analysis

Open the Details of Test project in TE/TM - Working

perspective

40

Reports as i.e. Test Project Summary Report

Define, save and apply settings to standard reports

Customized reports can be configured also

Various options to Report: Out of Box Reports

Analysis

41

Questions and Feedback

Analysis

42

TE/TM


Test Specification

Implementation

Planning

Execution

Analysis

Additional material

Agenda

43 © 2019. Vector Informatik GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V0.1 | 2019-03-11

Authors:Dannenhauer, MichaRotter, RobertVector Germany

For more information about Vectorand our products please visit

www.vector.com

http://www.vector.com/

44

„Big Picture“ Vector Toolchain i.e. for SiL

Additional material

CANalyzer Analysis Tool for single ECU’s, Networks and distributed systems

CANape Measurement, calibration, diagnosis as well as Flashing of ECU’s

CANdelaStudio Specification of Vehicle diagnose

CANoe Multi bus – Tool for Test, Simulation, Diagnose und Analysis of single ECU’s, Networks as well as distributed systems

CANoe.DiVa Automated test of software in ECU’s

DaVinci Configurator Pro

Configuration of AUTOSAR Basis software and RTE

DaVinci Developer Design of AUTOSAR-SWCs of ECU’s

Indigo Diagnosis tester

ODXStudio View and Edit ODX-Diagnosis data

PREEvision E/E-Engineering-Solution

vFlash Quick and Easy Flashing of ECU’S via CAN, CAN FD, LIN, FlexRay or Ethernet

vTESTstudio Comfortable Editing of automated Tests for embedded systems

vVIRTUALtarget Virtual test of AUTOSAR ECU’s

https://www.vector.com/de/de/produkte/produkte-a-z/software/canalyzer/https://www.vector.com/de/de/produkte/produkte-a-z/software/canape/https://www.vector.com/de/de/produkte/produkte-a-z/software/candelastudio/https://www.vector.com/de/de/produkte/produkte-a-z/software/canoe/https://www.vector.com/de/de/produkte/produkte-a-z/software/canoediva/https://www.vector.com/de/de/produkte/produkte-a-z/software/davinci-configurator-pro/https://www.vector.com/de/de/produkte/produkte-a-z/software/davinci-developer/https://www.vector.com/de/de/produkte/produkte-a-z/software/indigo/https://www.vector.com/de/de/produkte/produkte-a-z/software/odxstudio/https://www.vector.com/de/de/produkte/produkte-a-z/software/preevision/https://www.vector.com/de/de/produkte/produkte-a-z/software/vflash/https://www.vector.com/de/de/produkte/produkte-a-z/software/vteststudio/https://www.vector.com/de/de/produkte/produkte-a-z/software/vvirtualtarget/

45

Traceability of Artifacts

Additional material

Test Item

Requirement

Test Spec.

Product Goal LayerRequirements

Test Specification Layer

n

m

Subject to verify

n

Planning-Layer

n m findings

Ticket

Man. Test Seq.Test Implementation Layer

Test Case

n

m

n

m

n

external Tools

Spec. Artif. under Test

m n

Test Case Run

Test Run

Test Task

Test Execution Layer

m

Test Script

Test N

avig

atio

n

workshop test engineering and test management€¦ · robert rotter, product manager ... test...

Documents