HL7 V2 Test Case Authoring and Management Tool

Robert Snelick

National Institute of Standards and TechnologyMay 14th 2012

Contact: rsnelick@nist.gov

2

Specification to Testing: An End-to-End Approach• Proposed methodology and tools to support the development

HL7 V2 implementation guides and testing of systems that implement the implementation guides

• End-to-End Scope– Implementation Guide Authoring and Management– Test Data Authoring and Management– Message Generation and Inspection Document Generation– Test Case Authoring and Management– Testing Infrastructure

• Test Engine– Choreography

• Validation Engine– Messaging Validation– Functional Validation

• Communication Platforms

• Leverage and Extend Existing Concepts and Tools– VA Messaging Workbench (MWB), NIST Message Maker and Testing

Infrastructure

3

What will the tools address?

StandardsStandards Test CasesTest Cases

Test ExecutionTest ExecutionTest ScriptsTest Scripts ResultsResults AnalysisAnalysisTest MaterialTest Material

specification of test cases

specification of test cases

implementation guide development

implementation guide development

processable test cases

processable test cases

automated or human analysis

automated or human analysis

juror document or test report

juror document or test report

execution of the test scripts

execution of the test scripts

healthcare use case development

healthcare use case development create requirements to guide the standards development

create requirements to guide the standards development

create inspection document

provide validation test report

create inspection document

provide validation test report

XML message profiles

vocabulary validation

context files

XML message profiles

vocabulary validation

context files

test or certification managers determine pass/fail based on results

test or certification managers determine pass/fail based on results

ensure use case requirements are satisfied

create testable assertions criteria

ensure use case requirements are satisfied

create testable assertions criteria

tests cases should test all requirements specified in the implementation guide

tests cases should test all requirements specified in the implementation guide

methods and tools to transform test cases into machines process-able instructions

methods and tools to transform test cases into machines process-able instructions

Use CasesUse Cases

artifacts to support testing

artifacts to support testing

test engine interprets test scripts

validate message against test material

test engine interprets test scripts

validate message against test material

ImplementationImplementation

implementation

implementation

application implements the standard

application implements the standard

Test DataTest Data

development of test messages

development of test messages

messages support test cases

valid and invalid

messages support test cases

valid and invalid

Implementation Guide Authoring and Management Tool (IGAMT)

IGAMT and TCAMT

Test Case Authoring and Management Tool (TCAMT)

Message Maker Tool

Testing Infrastructure (Test Engine)

Test Case Authoring and Management Tool (TCAMT)

Vendors

Testing Infrastructure (Validation) and TCAMT

Testing Organizations

4

How can we Address the Increased Demand for Tools?

Test System (Allow NIST/others to build tools quickly)

Implementation Guide Authoring Tools (Create more “testable” specifications!)

Test Cases and Testing Artifacts

Test Case Management Tool (Let domain experts help write the test cases!)

ImplementationGuides

(Number is increasing rapidly—no testing tools

available)

HL7 v2 MUSyndromic

SurveillanceImplementation

Guide

ELINCSHL7 v2

Lab Orders and Results

ImplementationGuide

HL7 v2 MU incorporate Lab

Results & Reportable Lab

ResultsImplementation

Guides

HL7 v2MU Immunization

ReportingImplementation

Guides

IHE HL7 v2PCD

Technical Framework

IHE HL7 v2PIX/PDQTechnical

Framework

HL7 v2 MULab Results

Interface (LRI) Implementation

Guide

IHE HL7 v2Technical

FrameworkIn General

HL7 v2 Implementation

Guides in General(HL7, CDC, PHIN,ISDS, ONC, etc.)

20+ IHE integration profiles using V2!

5

Proposed Use and Goals Integrated Tooling• Develop tooling to automate process as much as possible

– Component integration—one tool output flows into the next• E.g., Conformance profile and conformance statements created during implementation guide development can

directly be used by validation engine

– Machine processable message profile specification• Used and built upon throughout and the development and testing cycle• Constraints as specified in Implementation Guide Authoring Tool are used directly by validation engine

• Goal is to enable domain experts to develop artifacts– Tooling will better enable domain experts to produce quality implementation guide

requirements and test criteria• HL7 Balloted Implementation Guides should use the tool to create (template)

– Ensure consistent definition and use of HL7 V2 Conformance Concepts– Ensure conformance constructs specified correctly and completely– Ensure compatibility with the standard or document variances

• Implementation Guides should have companion test plans complete with example/test messages– A quality level for implementation guides can be establish– Test plans can be balloted by HL7– Developing test plans in coordination with the implementation guide development promotes

better understanding of the requirements and force clear conformance requirements– Establish Implementation Guide Best Practices—Enforce with tooling– With HL7 CGIT WG participation the ONC S&I Framework Lab Results Interface (LRI)

Implementation Guide and NIST’s development of the LRI Test Plans and Test Tool is an example that is heading in the right direction

6

Tool Component Overview

• Implementation Guide Authoring and Management– Messaging Profiling and IG Document Generation

• Test Data Authoring and Management– Test Data Organization, Library Creation, and Management

• Message Generation and Inspection Document Generation– Assembly of Test Data, Message Generation, and Juror Inspection Documents

• Test Case Authoring and Management– Support complete Test Plan Life Cycle– Incorporates Conformance Profiles, Test Messages, Inspection Documents– Support Scenario and Application Functional Behavior Testing (Actor based)

• Testing Infrastructure– Test Engine

• Choreograph a sequence of transactions

– Validation Engine• Messaging Validation• Functional Validation

– Communication

7

ImplementationGuide

Authoring &Management

Tool

The Big Picture: HL7 V2 End-to-end Testing Support

HL7 v2StandardMessage

Definition

GeneralTransactionConstraints(e.g., IG , TF,

or user input)¹

HL7 v2Standard

Value Sets

Test CaseRequirements

VocabularyLibrary(XML)

ConformanceProfile(XML)

Test CaseManagement

and Authoring Tool

MessageProfiling

Test PlanExecution

Script(XML)

Test PlanTemplate

HL7 V2Impl.Guide

Template

Test PlanEnglish

Document

Data Sheets

Impl.Guide

EnglishDocument

SpecificationValidation

ContextFile

(XML)

¹ Support existing specifications

UseCases

VocabularyProfiling

Use CaseDevelopment

DispositionTraceability

Utilities

IG TemplateWizard

ConstraintGenerator

Test CaseDevelopment

Test ScriptGenerator

Data Sheet and Juror Document

Generator

Test PlanDevelopment

DocumentGenerator

Message Maker

DataManagement

MessageGeneration

TestingInfrastructure

DocumentGenerator

Execution Engine

ValidationEngine

Test Agents

Report Generation

Communication

TestingEnvironmentsCapabilities

ArtifactRepository

TestingTools

WebApplications

TestingInfrastructureComponents

Test PlanIntegration

Validation

ResourceBundle

ManagementMessageGeneration

Engine

VocabularyLibraries

(XML)

ConformanceProfiles(XML)

Test CaseSpecificContext

Files(XML)

Domain Specific

Tools

JurorDocuments

Utilities

Documentation

Communication

ResourceBundle

Test PlanExecution

Script(XML)

R. Snelick May 8th, 2012

DomainExperts

Domain Experts and Test Case Developers

User Input Tool Generated

Software Engineers

Tool Developers

ImportValue Sets

DITA XML

8

Test Case Management and Authoring Tool• Creates, Maintains, and Manages Test Plans• Embedded Standard Constraint Model or Imports Test Artifacts• Supports Multistep Test Cases (Test Scenarios)

– Defines Each Test Case (Test Steps)– Builds Test Environment– Defines transactions, actors, and sequence to execute tests– Defines generation context files and validation context files– Defines configuration parameters

• Generate Data Sheets• Generate Juror (Inspection) Documents• Incorporate Message Maker 2• Creates executable test script in XML format

– Independent of any execution engine

• Generates English Document Test Plan– Describes test cases and expected results– Provides traceability to specification reference

• Overall has to incorporate/use results of the other tools

9

TCAMT Design Overview

Test CaseRequirements

VocabularyLibrary(XML)

ConformanceProfile(XML)

Test CaseManagement

and Authoring Tool

Test PlanExecution

Script(XML)

Test PlanTemplate

Test PlanEnglish

Document

Data Sheets

Impl.Guide

EnglishDocument

SpecificationValidation

ContextFile

(XML)

ConstraintGenerator

Test CaseDevelopment

Test ScriptGenerator

Data Sheet and Juror Document

Generator

Test PlanDevelopment

Message Maker

DataManagement

MessageGeneration

DocumentGenerator

VocabularyLibraries

(XML)

ConformanceProfiles(XML)

Test CaseSpecificContext

Files(XML)

JurorDocuments

R. Snelick May 8th, 2012

Domain Experts and Test Case Developers

User Input Tool Generated

10

Conceptual View of System Components

Registry/ RepositoryRegistry/

Repository

TestDataTestData

AggregatedAggregated

ValidationValidation

ProxyProxy

SecuritySecurity

GenerationGeneration

LoggingLoggingLog

AnalyzerLog

Analyzer

Services

TimeTime

SpecializedSpecialized OtherServices

OtherServices

Test AgentTest Agent Evaluation AgentEvaluation Agent

NIST TEST INFRASTRUCTURE SYSTEM

Test Harness

Workflow Management /Execution Engine

Workflow Management /Execution Engine

Resource Repository

Templates, Common File Format Specifications

Templates, Common File Format Specifications

Test Cases, Machine-Readable Test ScriptsTest Cases, Machine-Readable Test Scripts

Test System Development Components

Test Case CreationTest Case Creation

Test Initiation and Test Identification Management

Test Initiation and Test Identification Management

Configuration Settings Management

Configuration Settings Management

User Repository ManagementUser Repository Management

Certification Body

NHIN

Other IndustryConsortia

Implementers

HITSP

STAKEHOLDERS

Vendors

Te

st

Ma

na

ge

me

nt

To

ols

an

d T

es

t S

ys

tem

s

ReportReport

Test Environments SupportedTest Environments Supported

Peer-to-Peer System TestingPeer-to-Peer System Testing Test AnalyzerTest Analyzer

System(s) Under Test

SUT

Real-Time MonitorReal-Time Monitor

Test Framework Test Framework Test Framework

Test System Instance

Instance TestingInstance Testing

Isolated System TestingIsolated System Testing

User InterfaceUser Interface

Test System ControlTest System Control

RemoteServicesRemoteServices

ExistingTool

ExternalService

11

Isolated System Test Environment E.g., IHE PIX Testing using a Web Application Client

ReportReport

IHE PIXConsumerTest Agent

IHE PIXConsumerTest Agent

HL7 V2Message

Generation

HL7 V2Message

Generation

IHE PIXSource

Test Agent

IHE PIXSource

Test Agent

HL7 V2MessageValidation

HL7 V2MessageValidation

Services Test Management

Router/Logger/ProxyRouter/Logger/Proxy

Vendor

System Under Test

Test Artifacts• Conformance Profiles• HL7 Tables• Validation Context Files• Generation Context Files

Test Artifacts• Conformance Profiles• HL7 Tables• Validation Context Files• Generation Context Files

IHE PIX ClientTest ScenarioIHE PIX ClientTest Scenario

ResultsHL7 V2 Message Validation Reports

ResultsHL7 V2 Message Validation Reports

Test Harness(Java Code)

Test Harness(Java Code)

Test Execution

Web Application

Client

12

Test Plan Structure and Format

Introduction and Overview

Test Plan Actors Transactions Test Cases

Test Cases Definition of Test Cases Reference to Conformance Requirements Test Setup Actor Description Configuration Generation Context File Test Data Test Steps Assertions Expected Results

List of Extracted Conformance Requirements and Test Case Mappings

Example Test MessagesDriver MessagesExpected Results Messages

HL7 V2Test Plan Title

Associated Files XML Conformance Profile XML Table Library XML Conditional Constraint Definitions XML Static Constraint Definitions (formats, etc) Context Value Specifications

13

Original of HL7 V2 Assertions

• Standard Defined Conformance Requirements– Ingrain to standard; often special case

– Built-in checks; design to turn on/off in authoring and validation tools

• Message Structure and Element Constraint Requirements– Conformance Message Profile or Standard Message Definition

• Implementation Guide Defined Conformance Requirements– Defined in constraint language; currently validation context file

• Dynamic Interchange Requirements– Acknowledgement Requirements

– Transport Protocol Requirements

• Application Functional Behavior Requirements– Sequence of Test Cases

– Data Content Requirements

– Defined in validation context file

14

Test Case Management Tool

• Instant Validation– Multiple Data Sets

– How do we handle the complex data sets?

– Right now we store in complex Excel Spreadsheets?

– E.g., MU testing today

• Isolated System Validation– Scenario Testing

– E.g., IHE Pre-connectathon testing today

• Peer-to-Peer System Testing– Interoperability Test Bed

– Future

15

TCAMT Detailed Design

ConformanceProfile(XML)

HL7 Table

Library(XML)

ImplementationGuide

EnglishDocument

Actor Builder

MessageBuilder

InteractionBuilder

List of Actors

List of Messages

List of Interactions

User DrivenTable Library

Message LevelAssertions

User DrivenTable Library

Message LevelAssertions

Interaction LevelAssertions

TransactionBuilder

List of Transactions

User DrivenTable Library

Message LevelAssertions

Interaction LevelAssertions

Transaction LevelAssertions

AssertionGenerator

AssertionGenerator

AssertionGenerator

Test Case Library Builder

Test Case Builder

Test PlanBuilder

Test CaseBuilder

Test StepBuilder

Test plan

List of Test Cases

List of Test Steps

User Driven Table Library

Message Level Assertions

Interaction Level Assertions

Transaction Level Assertions

Test Step Level Assertions

Test CaseExecution

Script(XML)

Table Library(XML)

ValidationContexts

(XML)

AssertionGenerator

MessageGenerator Message Generation Context

MessageContexts

(XML)

Test Suite(Test Cases)

English DocumentTest Plan

Table Library (HL7 + User’s)

Table Library (HL7 + User’s)

TC Library(Actor, MSG,

Inter, Transac)

TC Library(Actor, MSG,

Inter, Transac)

Test Assertion(TC, Transc, Inter,

MSG)

Test Assertion(TC, Transc, Inter,

MSG)

Message Generation(Message Context,Message example)

Message Generation(Message Context,Message example)

Test Suite(Test Plan Doc, Test Case Doc,

Test Case Scripts)

Test Suite(Test Plan Doc, Test Case Doc,

Test Case Scripts)

TCAMT Knowledge Base

Library Authoring

Library Reusing

Te

st C

ase

Au

tho

rin

g

Lib

rary

& T

C R

eu

sin

g

Test Case Level Assertions

16

Level of Assertion Assertion Type Target

•Data Value (Present, Value, RegExp)

•Single Message Correlation•Conditional

•Data Value (Present, Value, RegExp)

•Single Message Correlation•Conditional

•Multi Correlation AssertionSingle Value CheckCopy

•Data Value (Present, Value, RegExp)

•Single Message Correlation•Conditional

•Multi Correlation AssertionSingle Value CheckCopy

Assertions for TCAMT

Message Level Assertion

Interaction Level Assertion

Test Step Level Assertion

Transaction Level Assertion

Test Case Level Assertion

PCD01

DOR DOCPCD01

DOR DOC

PCD01

ACK

DOR(TA)

DOC(SUT)

DOR(TA)

DOC(SUT)

PCD01-001

ACK-001

PCD01-002

ACK-002

PCD01-001

ACK-001

17

TCAMT Process

TCAMT Process

Message Level Assertion

Message Level AssertionInteraction

Level Assertion

Message Level AssertionInteraction

Level Assertion

Test Step Level Assertion

Transaction Level Assertion

Transaction Level Assertion

Test Case Level Assertion

Message Generation Context

Message Generation Context

SUT Information

SUT Information

Test Plan Information

Test Plan Information

Library Test Case

Use Case Scenario

Use Case Scenario

Use Case Definition

Use Case Definition

Conformance Profile

Conformance Profile

Table LibraryTable Library

Supplement Validation Context

Supplement Validation Context

18

Test Case Schema

19

Test Case Archive

DOR(TA)

DOC(SUT)

PCD01-001

ACK-001

PCD01-002

ACK-002

TCAMT Test Plan

Description

TCAMT Test Case Script

XML

TCAMT Message Generation Context

PCD01-001

PCD01-002

TCAMT + imported Validation Context (Assertions)

ACK-001

ACK-002

HL7 Conformance Profile

PCD01

ACK

HL7 Table Library

PCD01

ACK

Test Case Authoring and Management Tool (TCAMT) - Screen shots

National Institute of Standards and Technology

April, 2012

21

CREATE LIBRARY

22

Home

TCAMT Library Menu

Welcome Message

23

List of Actors

Button to add a new Actor

Table List of existing Actors

Button to save/close

Select Command to delete or duplicate

24

Create Actor

Button to create Actor

Form to create Actor

25

List of Messages

Button to add a new Message

Select Command to delete or duplicate

Table List of existing Messages

Button to save/close

26

Create Message

Form to create Message

Button to create Actor

27

List of Interactions

Button to add a new Interaction

Select Command to delete or duplicate

Table List of existing Interaction

Button to save/close

28

Create Interaction

Form to create Interaction

Button to create Interaction

29

List of Transactions

Button to add a new Transaction

Select Command to delete or duplicate

Table List of existing Interaction

Button to save/close

30

Create Transaction

Form to create Transaction

Button to create Interaction

Select Interaction for each line

31

CREATE TEST PLAN & CASES

32

Home

Link for Test Plan & Cases

Welcome Message

33

List of Plans

Button to add a new Test Plan

Table List of existing Test Plans

34

Create Test Plan

Button to create Test Plan

Form to create Test Plan

35

List of Test Case

Button to add a new Test Case

Table List of existing Test Cases

Details for Test Plan

36

Create Test Case

Button to create Test Case

Form to create Test Case

37

View Meta Data for Test Case

Metadata for Test Case

Link to Test Steps

38

Define Test Steps

Button to add a new Test Step

Table List of existing Test Steps

Select Actor Tested

39

Export Test Case Script

Select to export Test Case

40

CREATE ASSERTIONSMessage Level Assertion

41

Link to Assertion Generator

Link to Message Level Assertion Generator

42

Message Level Assertion Generator

Table Tree of Message

Selected Tree Node (Yellow Highlighted)

Button to manage Assertions

Button to import assertions from existing Validation Context File

Button to export assertions to a new Validation Context File

43

Manage Assertion

Draggable selected Nodes

Dropable Form to create assertions

List of existing assertions

44

Create DataValue Assertion

Meta Information Form for Assertion

DataValue Form (PlainText/ RegExp/Presence) for Assertion

Location of Target Node (Dropped Node)

Add/remove check value

45

Create Single Message Correlation

Location of Target Node (First Dropped Node)

Location of Comparison Node (Second Dropped Node)

Meta Information Form for Assertion

46

Create Conditional Assertion

Meta Information Form for Assertion

If Part Form (It is same as DataValue Interface.)

Then Part Form (It is same as DataValue Interface.)

Else Part Form (It is same as DataValue Interface.)

47

CREATE ASSERTIONSInteraction Level Assertion

48

Link to Assertion Generator

Link to Interaction Level Assertion Generator

49

Interaction Level Assertion Generator

Table Tree of Interaction

Button to manage Assertions

Button to import assertions from existing Validation Context File

Button to export assertions to a new Validation Context File

Interaction Level Assertion Interface as same as Message Level interface

50

CREATE ASSERTIONSTransaction Level Assertion

51

Link to Assertion Generator

Link to Transaction Level Assertion Generator

52

Transaction Level Assertion Generator

Table Tree of Transaction

Selected Tree Node (Yellow Highlighted)

Button to manage Assertions

Button to import assertions from existing Validation Context File

Button to export assertions to a new Validation Context File

53

Create Correlation Assertion

Draggable selected Nodes

Comparison Node A (First Dropped Node)

List of existing assertions

Comparison Node B (Second Dropped Node)

Select correlation mode

54

CREATE ASSERTIONSTest Step Level Assertion

55

Link to Assertion Generator

Link to Test Step Level Assertion Generator

56

Test Step Level Assertion Generator

Table Tree of Test Step

Button to manage Assertions

Button to import assertions from existing Validation Context File

Button to export assertions to a new Validation Context File

Test Step Level Assertion Interface as same as Message & Interaction Level interface

57

CREATE ASSERTIONSTest Case Level Assertion

58

Link to Assertion Generator

Link to Test Step Level Assertion Generator

59

Test Case Level Assertion Generator

Table Tree of whole Test Case

Selected Tree Node (Yellow Highlighted)

Button to manage Assertions

Button to import assertions from existing Validation Context File

Button to export assertions to a new Validation Context File

Test Case Level Assertion Interface as same as Transaction Level interface