Monthly Program UpdateJanuary 12, 2012

Andrew J. Buckler, MSPrincipal Investigator

WITH FUNDING SUPPORT

PROVIDED BY NATIONAL

INSTITUTE OF STANDARDS AND

TECHNOLOGY

Agenda• Monthly snapshot in Jira

– (including status of installation at NIST)• QIBA 3A project snapshot• Theoretical development• Architecture and SW stack

22

3

BSD-2 licenseDomain is www.qi-bench.org.

Landing page provides • Access to

prototypes, • Repositories for

download and development,

• Acknowledgements,

• Jira issue tracking, and

• Documentation

3

Go to the site and workWith the apps and

Jira

QIBA 3A PROJECT SNAPSHOT(recalling that this is a testbed for us)

444

Basic structure of the challenges

5

Pilot

Pivotal

Investigation 1

Train Test Pilot

Pivotal

Investigation

Train Test Pilot

Pivotal

Investigation

Train Test Pilot

Pivotal

Investigation n

Train Test

P r i m a r y

S e c o n d a r y• Defined set of data• Defined challenge• Defined test set policy

555

First one: • Presently in pilot phase,

• Using StudyDescription method• Used batch scripting with reference method to

aid data curation• 10-12 participants (about 20 QI-Bench users)• First participant data received• Analysis plan using N-way ANOVA in R started

• Pivotal phase starting with batch assisted curation• Will be transitioning to database schema for

metadata (gradually away from spreadsheet)

Pooled Bias

Pooled Variability

Variability across shapesVariability across densities

Variability across slice thicknesses

0

5

10

Method AMethod BMethod CMethod DMethod EMethod FGroup

666

1. Relative performance is computed according to descriptive statistics

2. We determine a group value for each of the descriptive statistics, e.g., as the mean plus 1 stdev (or as wide as we think wise).

3. Results presented using radar plots

Bias Variability Repeatabilitycross-x

reproducibilitycross-y

reproducibilityNew Method 7.00 3.00 10.00 11.00 12.00Group 7.16 6.64 6.96 7.81 9.00

Bias

Variability

Repeatabilitycross-x reproducibility

cross-y reproducibility

0.00

10.00

20.00

New MethodGroup

In this example, the new proposed method does not perform well enough to be considered a valid method since it falls outside the group values.

777

In this example, the new proposed method is seen to perform within group values and may even help pave the way for an improved claim.

Bias VariabilityRepeatability

cross-x reproducibility

cross-y reproducibility

New Method 3.00 3.00 4.00 6.00 8.00Group 7.16 6.64 6.96 7.81 9.00

Bias

Variability

Repeatabilitycross-x reproducibility

cross-y reproducibility

0.00

5.00

10.00

New MethodGroup

888

THEORETICAL DEVELOPMENTprogress re: utilization of logical and statistical inference at each of two levels, technical performance of assay methods, and qualification of biomarker in specific clinical context

999

Another way to look at what needs to happen

10101010

Formulate

Statistical Analysis Results (Relation

strength)

Annotation and Image Markup,

Non-imaging Clinical Data

Primary Data: Images and other

Raw Data

Reference Data SetsQIBO

Specify

RDF Triple Store

CT Volumetry CT

obtained_by

Tumor growth

measure_of

TherapeuticEfficacy

used_for

Analyze

Y=β0..n+β1(QIB)+β2T+ eij

Execute

Feedbac k

Feed

bac

k

Specify: Establish a logical specification and setup terms for mathematical analysis

1111

• Functionality:• Establish means to semantically labeling imaging

biomarker data with emphasis on representing both the clinical context in which an imaging biomarker is used as well as the specifics of the imaging protocol used to acquire the images.

• Set up the logistic regression model:• Precisely specify dependant variable• Account for covariates• Enumerate independent variables and

error terms (sources of variability)• Establish database for collection of terms.

• Method:• Provide GUI to traverse the QIBO concepts

according to their relationships and create statements represented as RDF triples and stored in an RDF store.

• Each set of RDF triples will be stored as a “profile.”

• Relationship strength initialized based on prior estimates (if available) QIBO

Specify

RDF Triple Store

CT Volu

metryCT

obtained_by

Tumor growth

measure_of

Therapeutic

Efficacy

used_for

Ontologies supporting Specify

1212

• Extend the QIBO to link to existing established ontologies

1. leverage BFO upper ontology to align different ontologies

2. convert portions of BRIDG and LSDAM to ontology models in OWL

• Automated conversion would done in two steps:

1. convert current Sparx Enterprise Architect XMI EMF UML format

2. export resulting EMF UML into a RDF/OWL representation using TopBraid Composer

Formulate: advanced query framework made possible by Specify

1313

• allow users to select the profiles (or set of RDF triplets) created in Specify, execute a query and retrieve the results in various forms.

• assemble/transform the set of RDF triples to SPARQL queries:

1. form an uninterrupted chain linking the instance of the input class from the ontology to the desired output class

2. formulate/invoke necessary SPARQL queries against the web services deployed in SADI framework.

• interface with the query engine and will have offline (asynchronous) query execution capability.

• results to be exportable as serialized objects (RDF/XML and CSV)

Formulate

Statistical

Analysis

Results (Relati

on strengt

h)Annotation and

Image Markup, Non-imagin

g Clinical

DataPrimary Data: Images

and other Raw Data

Reference Data Sets

RDF Triple Store

CT Volu

metryCT

obtained_by

Tumor growth

measure_of

Therapeutic

Efficacy

used_for

Data Services supporting Formulate

1414

• wrap existing data services such as NBIA, caArray, caTissue, AIM and PODS using Semantic Automated Discovery and Integration (SADI)• this is enabled by metadata available

through the UML representations of the models exposed by these services and CDE annotations available for them through caDSR.

• describe service I/O semantically using the extended version of QIBO

• service registry of SADI will help the automated composition of computer-interpretable queries by the query engine. • example: “there is a service that

returns Biological Subjects that has undergone certain Biological Interventions”

Analyze: Use annotation and image markup to support statistical inference

1515

• Support Clinical Performance assessment (i.e., in addition to current Technical Performance) • Outcome studies• Integrated genomic/proteomic

correlation studies• Group studies for biomarker qualification

• (set up a basic multiple regression analysis, e.g.) Intent to treat analysis of the primary outcome via covariance model of the general form (QIBt)=β0..n+β1(QIB0)+β2T+ eij where QIBt and QIB0 are the QIB at a time after treatment and at randomization respectively, T is a treatment group indicator, and β0..n, β1, and β2 are model parameters. β2 represents the effect of treatment and its estimate is the difference between group means on the log scale, after adjustment for any imbalance between the groups in log QIB. The error terms in the model, eij, are assumed mutually independent and normally distributed. Depending on the nature of the QIB, the log transformation may be used instead of the direct value. Likewise calculations may be performed using z scores with corresponding conversion with raw values.

Quanti tative Ima ging Specification Lan guage

Batch Ana lysis Service

Reference Data Set Man ag er

UPICT Protocols, QIBA Profiles, literature papers and other sources

QIBO-

BatchMakeScripts

Reference Data Sets, Annotations, and Analysis Results

(red edgesrepresent

biostatisticalgeneralizability)

Source of clinical study results

Clinical Body of Evidence (formatted to enable SDTM and/or other standardized registrations

4. Output

3. Batch analysis scripts

UPICT Protocols, QIBA Profiles, entered with

Ruby on Rails web service

QIBO

Examples of output at biomarker (above the assay level)

1616

From Jack 1999. Note: W-score is the relative score of the measured HC volume corrected for intracranial volume and compared to age and sex adjusted normals.

Hypothetical model of dynamic biomarkers of the Alzheimer's pathological cascadeJack et al., 2010

161616

To inform thresholding

To substantiate surrogacy (or its weaker form of “activity”)

ARCHITECTURE AND SW STACKSo what is a cohesive architecture that maximizes leverage of best thinking, existing touchpoints, and stays current over time?

171717

STDM standard of CDISC into repositories like FDA’s

Janus.

MVT portion of AVT, re-useable

library of R scripts.

MIDAS, BatchMake, Condor Grid;

built using Zend on

PHP.

caB2B, NBIA,

PODS data elements, DICOM

query tools.

QIBO, AIM,RadLex/ Snomed/ NCIt; built

using Ruby on Rails.

•Specify context for use and assay methods.

•Use consensus terms in doing so.

Specify

•Assemble applicable reference data sets.

•Include both imaging and non-imaging clinical data.

Formulate •Compose and iterate

batch analyses on reference data.

•Accumulate quantitative read-outs for analysis.

Execute

•Characterize the method relative to intended use.

•Apply the existing tools and/or extend them.

Analyze •Compile evidence for regulatory filings.

•Use standards in transfer to regulatory agencies.

Package

1818

MVT: Reasonable framework, but many gaps

1919

There are multiple possibilities to deploy it as a web application, some of which we’ve considered:

1. Re-implement the existing implementation to use GWT in place of Swing, inclusive of both the XIPHost as well as MVT components, retaining the WG23 concept.

2. Re-implement only those parts necessary to perform the needed MVT functions using GWT with enough data handling to do so but without doing everything necessary to retain the full XIPHost capability.

3. Leverage the GUI design concept but otherwise implement without starting from the Swing code.

In all cases, there is the secondary design alternative of introducing a RESTful web service layer explicitly or not. (By the way, just for fun, I performed a conversion of the current Swing code to Ajax using AjaxSwing. I got most of AVT working over the web with minimal effort, but this isn’t a serious alternative because AjaxSwing has a license fee. I did it because I wanted to see how easy such a path would be. It’s an interesting capability! But irrelevant in the end.)

Pros: optimized for DICOM, works with workstationsCons: hard to create web apps, not optimized for semantic web

2020

HW

XIPApplication

InventorApplication Modules

WG 23 System Services PLUG

WG 23 System Services SOCKET

GRIDCLIENT

SERVICES

DICOMSERVICES(DCMTK)

OTHERSERVICES

VTK ITK AIMTK other

OS

NCIA

XIPIDE

RadLex

AIM

NCI

ProtégéEVS

XIP

MIDDLEWARE

DICOM

DICOM Services

IVI MiddlewarecaGrid

CaBIG

caDSR, EVS, RadLex, AIM ontology, etc

Client accessService access

Grid Data Service

Grid Analytical Service AIM Data

Service

XIP App

ServiceHost

WG23

DICOM Image

Sources

2020

Alternative architectural form…

21

SW Stack• J2SE (J2EE compliant)• MySQL• caGrid• Globus• Application:

• JBoss• caCore 212121

With pros and cons “opposite” that of the XIP based architecture

Functionality view annotated with architecture

2222

HW

XIPApplication

InventorApplication Modules

WG 23 System Services PLUG

WG 23 System Services SOCKET

GRIDCLIENT

SERVICES

DICOMSERVICES(DCMTK)

OTHERSERVICES

VTK ITK AIMTK other

OS

NCIA

XIPIDE

RadLex

AIM

NCI

ProtégéEVS

XIP

MIDDLEWARE

DICOM

DICOM Services

IVI MiddlewarecaGrid

CaBIG

caDSR, EVS, RadLex, AIM ontology, etc

Client accessService access

Grid Data Servi

ce

Grid Analytical Service

AIM

Data

Service

XIP App

Service

Host

WG23

DICOM

Image

Sources

2222

MIDAS Core

Apache File SystemPostGreSQL

Publication DB

MIDAS Data Server

MIDAS e-journal

MIDAS Compute Server

MIDAS Visualization

MIDAS Client

MIDAS C++ API

MIDAS Web API

When annotation and markup has already been done

Reference data

setsAnnotati

on

and marku

p

AIM-enabled (e.g., ClearCanvas) workstation

RIS worklist items

DICOM Q/R

First step to rationalizing architecture: mash them together and see what falls out

23232323

NCIA

RadLex

AIM

NCI

XIP

MIDDLEWARE

DICOM

DICOM Services

IVI MiddlewareSADI framework (e.g., wrapped caGrid)

CaBIG

caDSR, EVS, RadLex, AIM ontology, etc

Client accessService access

Grid Data Service

Grid Analytical Service AIM Data

Service

HW

XIPApplication

InventorApplication Modules

WG 23 System Services PLUG

WG 23 System Services SOCKET

GRIDCLIENT

SERVICES

DICOMSERVICES(DCMTK)

OTHERSERVICES

VTK ITK AIMTK other

OS

XIPIDE

ProtégéEVS

XIP App

ServiceHost

WG23

DICOM Image

Sources

This is an ongoing discussion. More to come!

2424

Value proposition of QI-Bench• Efficiently collect and exploit evidence establishing

standards for optimized quantitative imaging:– Users want confidence in the read-outs– Pharma wants to use them as endpoints– Device/SW companies want to market products that produce them

without huge costs– Public wants to trust the decisions that they contribute to

• By providing a verification framework to develop precompetitive specifications and support test harnesses to curate and utilize reference data

• Doing so as an accessible and open resource facilitates collaboration among diverse stakeholders

2525

Summary:QI-Bench Contributions• We make it practical to increase the magnitude of data for increased

statistical significance. • We provide practical means to grapple with massive data sets.• We address the problem of efficient use of resources to assess limits of

generalizability. • We make formal specification accessible to diverse groups of experts that are

not skilled or interested in knowledge engineering. • We map both medical as well as technical domain expertise into

representations well suited to emerging capabilities of the semantic web. • We enable a mechanism to assess compliance with standards or

requirements within specific contexts for use.• We take a “toolbox” approach to statistical analysis. • We provide the capability in a manner which is accessible to varying levels of

collaborative models, from individual companies or institutions to larger consortia or public-private partnerships to fully open public access.

2626

QI-BenchStructure / Acknowledgements• Prime: BBMSC (Andrew Buckler, Gary Wernsing, Mike Sperling, Matt Ouellette)

• Co-Investigators– Kitware (Rick Avila, Patrick Reynolds, Julien Jomier, Mike Grauer)– Stanford (David Paik, Tiffany Ting Liu)

• Financial support as well as technical content: NIST (Mary Brady, Alden Dima, Guillaume Radde)

• Collaborators / Colleagues / Idea Contributors– FDA (Nick Petrick, Marios Gavrielides)– UCLA (Grace Kim)– UMD (Eliot Siegel, Joe Chen, Ganesh Saiprasad)– VUmc (Otto Hoekstra)– Northwestern (Pat Mongkolwat)– Georgetown (Baris Suzek)

• Industry– Pharma: Novartis (Stefan Baumann), Merck (Richard Baumgartner)– Device/Software: Definiens (Maria Athelogou), Claron Technologies (Ingmar Bitter)

• Coordinating Programs– RSNA QIBA (e.g., Dan Sullivan, Binsheng Zhao)– Under consideration: CTMM TraIT (Andre Dekker, Jeroen Belien)

2727