qrm teststand result processor - addq4.seaddq4.se/download/files/qrm teststand result processor...

Document Name Document Class Date Version

QRM Result Processor Documentation May 2014 1.4

Author Document Nr Page

Mattias Ericsson 4-100-200 1 (25)

1 (25)

TestStand QRM Result Processor

Documentation

The Compatible with LabVIEW logo is a trademark of National Instruments Corporation and is used

under a license from National Instruments Corporation.





2 (25)

Contents

1 Document Change History ..................................................................................... 3 2 Documents and Abbreviations ............................................................................... 3

2.1 Reference Documents ..................................................................................... 3 2.2 Abbreviations .................................................................................................. 3

2.3 Readers Guide ................................................................................................. 3 3 Overview ................................................................................................................ 4

3.1 QRM Client ..................................................................................................... 5

3.2 Evaluation........................................................................................................ 5 4 QRM Web Service ................................................................................................. 6

4.1 Installing .NET Framework 3.5SP1 (Windows 8 only) .................................. 6 4.2 Installation QRM Web Service ....................................................................... 7

4.3 Configuring Forwarding to Test Server .......................................................... 9 5 View Results in QRM Web Application ............................................................. 10 6 TestStand Result Processor .................................................................................. 13

6.1 Requirements ................................................................................................. 13

6.2 Installation ..................................................................................................... 13 7 Deployment of TestStand project with QRM Result Processor .......................... 16

8 QRM Result Processor Example ......................................................................... 16

8.1 Advanced QRM Sequences ........................................................................... 19

8.1.1 ModelPluginOptions callback ................................................................ 19 8.1.2 PreUUT callback .................................................................................... 19

9 Configuration of the QRM Result Processor ....................................................... 20

9.1 Result Set Execution Counter ....................................................................... 24 10 Support ................................................................................................................. 24

11 Troubleshooting ................................................................................................... 25





3 (25)

1 Document Change History Version Comment Author

1.0 First version. Mattias Ericsson

1.1 Updated after review. Mattias Ericsson

1.2 Added demo test server information Mattias Ericsson

1.3 Added information of enabling .NET 3.5

on Windows 8.

Mattias Ericsson

1.4 Updated Troubleshooting and

requirements for TestStand Result

Processor

Mattias Ericsson

2 Documents and Abbreviations

2.1 Reference Documents

Ref Doc Nr Document Name Content

[1] 4-100-88 QRM Windows Installation Guide Install guide

[2] 4-100-201 QRM Web Application Manual Manual

2.2 Abbreviations

QRM Quality Result Management

UUT Unit Under Test

2.3 Readers Guide

This document describes how to install and use the AddQ QRM TestStand Result

Processor Plugin. The Result Processor Plugin is a new feature of TestStand 2012.





4 (25)

3 Overview The QRM result management system has been created in order to store and maintain

test results. It is a Windows based (Windows 7/8/2008R2 Server/2012 Server)

distributed system based on web services, databases and ATML XML data. QRM

consists of two main parts:

QRM Web Service – This part is responsible for storing and synchronizing test

data in the system.

QRM Web Application Server – This part are responsible for presenting and

statistical analysis of test data through a web page based system.

QRM Database

QRM Web Service

Laptop

Test

Program

IIS Web Server

Test Station Server Client

QRM Web Service

Figure 1. QRM overview. This is typical setup. A test program on a test station post ATML

results to the QRM Web Service that forwards the ATML to another QRM Web Service that

store into the QRM database. A client makes a request to the IIS web server to get the results.

The only interaction between the two major parts is through the data storage, in all

other senses they are totally separated.

The QRM is a client-server based distributed system using web services. All

communication between the web services and clients are using HTTP protocol, which

means that it is very “firewall friendly”. All test data is sent as XML data, using the

ATML standard with some QRM extensions. This is a standard developed especially

for test results in automated systems. It is always the client who initiates the

communication to the web service. Notice that the web service itself could act as a

client. The database has been designed to meet all the data required by the ATML

standard.





5 (25)

3.1 QRM Client

The QRM client (TestStand QRM Result Processor) is quite simple and only formats

test results as ATML XML report and posts them by HTTP to a QRM Web Service.

The typical setup is a local QRM Web Service on the same machine as the QRM

client. This local QRM web service is always available even if the network is

temporary down. The client is not robust against network problems, but the QRM

Web Service is and will continue trying to forwarding the ATML reports until it

succeeds even if the test program running the client has stopped executing, since it is

a Windows background process.

3.2 Evaluation

In order to test the TestStand QRM Result Processor client without a QRM license,

there exists a test server on addq4.se that can be used to test the test station client

without using a license. The steps below needs to be performed:

1. Install the QRM Web Service and configure it to forward the addq4.se.

2. Install the TestStand QRM Result Processor.

3. Run a QRM example or run your own sequence.

4. View the results in the QRM Web Application at addq4.se.

The database at addq4.se will be recycled every night and all data will be lost. There

are some default example test results already stored in the database that also can be

viewed.

If you want to setup your own server with databases and QRM Web Application on

an IIS server, please contact AddQ for an evaluation license and instructions. Please

contact [email protected] or [email protected].

mailto:[email protected]






6 (25)

4 QRM Web Service

4.1 Installing .NET Framework 3.5SP1 (Windows 8 only)

By default, the .NET framework 3.5SP1 is not installed on Windows 8 (already

preinstalled on Windows 7).

In Control Panel, choose Programs and Features, choose Turn Windows features on or

off, and then select the .NET Framework 3.5 (includes .NET 2.0 and 3.0) check box.

This option requires an Internet connection. You do not need to select the child items.

For more information follow the link below:

http://msdn.microsoft.com/en-us/library/hh506443.aspx

http://msdn.microsoft.com/en-us/library/hh506443.aspx





7 (25)

4.2 Installation QRM Web Service

The QRM Web Service is a background Windows Service that is installed by a

standard installer. This is a 32-bit windows service that is used on both 32-bit and 64-

bit operating systems. The purpose of the QRM Web Service is to receive XML

ATML test data and store the data in the database and/or forward XML ATML test

data to another QRM Web Service. It is recommended to use a multicore computer,

since performance is increase by parallel processing of the XML results.

1. Browse to <Install Path>\QRM\QRM Web Service\<version>. Right-

click on the setup.exe and select Run as administrator.

2. Click Next.





8 (25)

3. Click Next.

4. Click Next. Installation is complete. If there is any error messages regarding

timeout, just ignore these. Reboot if prompt.

5. Test the QRM Web Service by opening the following URL in your web

browser: http://localhost:8080/QRMResult/Info

It should look like the picture below. No license is need to test the client

against the addq4.se test server.

6. Done!

http://localhost:8080/QRMResult/Info





9 (25)

4.3 Configuring Forwarding to Test Server

In order to make the QRM Web Service forward the generated ATML reports to the

addq4.se test server where it will be stored in a database and available to view in the

QRM Web Application.

1. Open the QRM Web Service configuration file:

C:\Users\Public\Documents\QRM\QRMWebServiceConfig.ini

2. Change the following key as shown below:

ForwardingActive=TRUE

3. Restart the QRM Web Service by entering the following URL in your web

browser:

http://localhost:8080/QRMResult/shutdown

Enter username qrmuser and password qrm123! when prompt.

4. Press the Start button and then the Info button.

http://localhost:8080/QRMResult/shutdown




Mattias Ericsson 4-100-200 10

(25)

10 (25)

5. Scroll down on the info start page to the field Forwarding. It should state

TRUE http://addq4.se:8080/QRMResult as shown below.

6. Done. Your QRM Web Service is now forwarding ATML reports to addq4.se

as shown in Figure 1.

Your TestStand QRM Result Processor client will post ATML results to

localhost:8080/qrmresult and the QRM Web Service will forward them to

addq4.se.

Please note that all test results that are posted to addq4.se test server is public

to everyone and all stored data will be destroyed each night and the database

recreated.

5 View Results in QRM Web Application Test results in QRM are available from a web browser. You can view your test results

for evaluation purpose at:

http://addq4.se/qrm

Login with the following user:

Username: TestUser

Password: qrm123!

http://addq4.se/qrm





(25)

11 (25)

Please notice that the QRM database at addq4.se will be recycled every night and all

data will be lost. There is some example data always available in database as well.

To just view your test results do the following simple steps:

1. Click Test Results tab and then the Execute button.

2. Expand any of the dates by clicking on them. The available result sets will be

shown.

3. Click any of the result sets to view the individual UUT test results.





(25)

12 (25)

Figure 2. The QRM Web Application at the demo test server at addq4.se/qrm

Please refer to reference [2] QRM Web Application Manual how to use the QRM

Web Application in full.

Figure 3. Example data in the QRM Web Application at addq4.se/qrm.





(25)

13 (25)

6 TestStand Result Processor The TestStand QRM Result Processor acts as a QRM client and posts ATML test

reports to a QRM Web Service. The TestStand QRM Result Processor is actually built

using the QRM LabVIEW client. The QRM Result Processor will interact with

TestStand in the following callbacks:

//Main

Initialize Entry Point -> Model Plugin - Initialize ## QRM Clear

Model Plugins - Begin -> Model Plugin - Begin, ModelPluginOptions

Callback,ModelPluginConfiguration Callback

Process Setup Callback

Pre UUTLoop Callback

// UUT loop

PreUUT Callback

Model Plugins - Pre UUT -> Model Plugin - Pre UUT

Model Plugins - UUT Start -> ## Initialization of QRM.

MainSequence Callback

Model Plugins - UUT Done -> ## Processing of QRM results, post to QRM

Model Plugins -> Model Plugin - Post UUT

PostUUT Callback

Post UUTLoop Callback

//Cleanup

Process Cleanup Callback

Model Plugins - End -> Model Plugin - End

6.1 Requirements

TestStand 2012 or later. Sequences must use the native TestStand callbacks to

implement UUT looping.

LabVIEW 2012 runtime engine

QRM Web Service installed as described in section 4.2.

.NET Framework 3.5

No license is required for the TestStand Result Processor, but QRM Web Service

requires an activated license in order to enable database storage and forwarding.

6.2 Installation

1. Run the setup.exe in the <QRM Install>\TestStand Result Processor\<TestStand version>\Volume1





(25)

14 (25)

2. Accept license agreement and press Next in the following Wizard steps.

Figure 4. Installation completed. No reboot is required unless the LabVIEW 2012 runtime was

installed.

3. When done, press Finish. If LabVIEW 2012 Runtime engine also was

installed a reboot is required and will be prompt.





(25)

15 (25)

The installation will install the following files in TestStand:

Figure 5. Files installed into TestStand by the QRM Result Plugin installer.

There are two main parts:

1. Result Processor Plugin – The actual result processor plugin. Consists of a

TestStand sequence, a LabVIEW packed library and .NET dll.

2. Examples – Examples that shows how to use and configure QRM.





(25)

16 (25)

7 Deployment of TestStand project with QRM Result Processor

There are two ways of deploying the TestStand QRM Result Plugin. Choose any of

them. In both cases make sure LabVIEW 2012 runtime engine is available and

included in your build if necessary.

1. Run the QRM Result Plugin Installer on each deployment station as described

in section 6.

2. In your deployment project include in the TestStand Directory:

Components\Language\English\AddQ_QRMResultProcessorStrings.ini

Components\Models\ModelPlugins\AddQ_QRMResultProcessor (both

sequence and folder)

Also see the structure shown in Figure 5. In both cases you might want to

include the QRM Result Plugin configuration file:

<TestStand>\Cfg\ModelPlugins\ResultProcessing.cfg

8 QRM Result Processor Example The easiest way to explain the QRM TestStand Result Processor Plugin is by an

example sequence. The examples are located in:

<TestStand>\Examples\AddQ

Figure 6. The QRM_DemoSeq.seq.

The demo sequence test steps are just dummy steps that output results of different step

types. Figure 7 shows what it looks like in the QRM Web Application. Open the QRM

Web Application in your browser and click on tab Test Results. Then click Execute





(25)

17 (25)

button. See QRM Web Application manual, ref [2] for more information. You can of

course test your own TestStand sequence without any additional QRM adaptation.

Figure 7. The QRM Web Application test results of one sequence execution of the

QRM_DemoSeq.seq

There are two example sequences:

1. QRM_DemoSeq.seq – Advanced additional information to QRM. This show

all you can do from TestStand in order to add additional information to QRM.

2. QRM_DemoSeq_Simple.seq – Simplest possible use of QRM with no

additional information at all in the sequence. The UUT Type will be set to

“Unknown”. TestStation will be default set as computer name and operation

the same as the TestStand user.

Figure 8. The result of the QRM_DemoSeq_Simple.seq.

Figure 9 and Figure 10 show the example TestStand demo sequences that is included

in the TestStand examples: <TestStand>/Examples/Demo/LabVIEW





(25)

18 (25)

Figure 9. The result of the TestStand demo sequence Computer Motherboard Test Sequence.seq.

The sequence has been executed without any changes.

Figure 10. The result of the TestStand demo sequence auto.seq. The sequence has been executed

without any changes.





(25)

19 (25)

8.1 Advanced QRM Sequences

In order to add more information to QRM, special TestStand steps and callbacks need

to be implemented. The ModelPluginOptions callback is called once at the start of the

sequence, even before Process Setup callback. In order to add UUT specific info,

there is a need of implementing the PreUUT callback as shown in the example below.

Please look at and run the QRM_DemoSeq.seq to see how this is achieved. It is

recommended to set the UUT Type (Part Number in TestStand) as shown in the

PreUUT callback. If there are results from different UUT types, the presentation and

statistics in QRM will be much more informative, if the UUT type is specified.

8.1.1 ModelPluginOptions callback

Location

Test Program and Test Program Version

Test Description and Test Description Version

Figure 11. The ModelPluginOption callback.

8.1.2 PreUUT callback

UUT Type (Part Number in TestStand) – Recommended to use!

UUT Variant and UUT Product Family

UUT Settings (array)

Equipment (array)

Figure 12. A subsequence of the PreUUT callback. Note the looping of equipment and UUT

settings.





(25)

20 (25)

9 Configuration of the QRM Result Processor

1. Launch QRM Result Processor Plugin option from Configure->Result

Processing… as shown in Figure 13.

Figure 13. Configuration of QRM Result Processor Plugin.

2. A dialog similar to the picture below will appear, showing all installed

TestStand ResultProcessors. Make sure AddQ_QRMResultProcessor is

enabled and press the option button. If the AddQ_QRMResultProcessor

doesn’t appear, press the Insert New button (+ and arrow button) and select it

from the list. You can also press Reset to Defaults… and the list will be rebuilt

by TestStand.

Figure 14. Installed TestStand Result Processors. If the AddQ_QRMResultProcessor doesn’t

appear, press the Insert New button (+ and arrow button) and select it from the list.

3. The dialog as shown in Figure 15 appears. The most important configuration

is the connection URL, user name and password. If QRMWebService is

installed locally (recommended) with default parameters, the connection URL

will not need to be changed. See reference [1].





(25)

21 (25)

Figure 15. Configuration of QRM Result Processor Plugin.

Option Description Exclude Setup and Cleanup Excludes results that are generated in setup and

cleanup section.

Exclude Post Actions Excludes results that are generated in post action

steps.

Exclude Action Steps Excludes results that are generated in action steps.

Exclude Statements Excludes results that are generated in statements.

Exclude Comments Excludes all comments.

Exclude Terminated Excludes UUTs that are terminated.

Exclude Errors Exclude steps that generated an error.

Multiple Numeric Tests as Arrays Selects if multiple numeric tests should be

considered as array of not by QRM. See Figure 16

and Figure 17.

Use Step Name as Test Name Instead of using TestStand Test names as “Main”,

“Sequence Call”, the step name will be used

instead. See Figure 18 and Figure 19.

Report Text as Attachment Outputs the test step report text as a text file

attachment.

Report Text as Comment Outputs the test step report text as a comment to the

result. Be care ful if report text is large. Table 1. QRM Configuration Options.

All settings changed in Table 1 will be stored in: <TestStand>\Cfg\ModelPlugins\ResultProcessing.cfg





(25)

22 (25)

Figure 16. Multiple Numeric Tests as Arrays = FALSE

Figure 17. Multiple Numeric Tests as Arrays = TRUE

Figure 18. Use Step Name as Test Name = TRUE.





(25)

23 (25)

Figure 19. Use Step Name as Test Name = FALSE.

Figure 20 shows an example where all exclude options have been set to TRUE.

Compare with Figure 18 to see the difference.

Figure 20. An example where all exclude options has been set to TRUE.





(25)

24 (25)

9.1 Result Set Execution Counter

Each time a test sequence execution has been performed, the sequence execution, the

result set name, will be named after the sequence file with a concatenated index

number. The counter is stored in an ini-file:

C:\Users\Public\Documents\QRM\QRM_ResultProcessorSequenceCounter.ini

It is the same location as the QRM license files and QRM web service configuration.

[Computer Motherboard Test Sequence.seq]

ExecutionCount = 7

[QRM_DemoSeq.seq]

ExecutionCount = 25

[QRM_DemoSeq_Simple.seq]

ExecutionCount = 1

Figure 21. An example of the content of the result set sequence counter file. One section for each

sequence is generated.

10 Support If you have questions, please post them at the QRM Community:

https://decibel.ni.com/content/groups/qrm

If you don’t have access to the community, please click Ask to join this group. For

any question, feel free to contact AddQ support at: [email protected]

https://decibel.ni.com/content/groups/qrm






(25)

25 (25)

11 Troubleshooting There are few things to check if you have problems with your TestStand Result

Processor.

Problem Solution

QRM Result Processor fails to execute. Make sure LabVIEW 2012 Runtime engine

is installed. The result processor always uses

this to execute.

QRM Result Processor is not visible in the

result processor list after installation.

If the AddQ_QRMResultProcessor doesn’t

appear, check Show More Option, press the

Insert New button (+ and arrow button) and

select it from the list. You can also press

Reset to Defaults… and the list will be

rebuilt by TestStand.

QRM Result Processor is not posting results. Make sure QRM Result Processor is enabled

in the active result processor list as shown in

Figure 13.

QRM Result Processor is not included when

deploy my TestStand system.

Please see section 7 of what to include when

deploying your system.

QRM Result Processor gets a HTTP error. Make sure you QRM Web Service is

available and that your URL setting is

correct. It is recommended to install a QRM

Web Service locally.

Make sure that there is no firewall problem

blocking port. QRM uses default port 8080.

From the test computer, in your browser, try

to open http://<url>:8080/qrmresult. If this is

not available, there is a firewall problem.

Error 1556 when posting results. Make sure you are not using the call

RunState.Engine.UnloadAllModules() in you

sequence. This will actually also unload the

QRM Result Processor.

For other problems, please refer to the troubleshooting section in reference [1], QRM

Installation Guide.