john troxell, accenture - cdiscportal.cdisc.org/cdisc user networks/north america/atlantic... ·...
TRANSCRIPT
The CDISC Analysis Data Model (ADaM) Team has specified computer checks for ADaM compliance
Software tools implementing the checks produce error messages that conflict with ADaM Implementation Guide (ADaMIG)
Why does this happen, and how can we fix it?
2
Historically, SDTM validation checks were specified by a software tool developer and the FDA
In my opinion, CDISC should define validation checks for CDISC standards compliance
ADaM Validation Checks v1.0 published 2010◦ updated since then
3
1. The problem◦ Incorrect validation tool error messages
2. Understanding the problem◦ The space and vocabulary about it
◦ Six things that have to go right
◦ Why they don’t go right
3. Fixing the problem◦ Ideas
◦ Status
4
Check specifier:◦ The author of the specifications for a computer
compliance check of CDISC compliance
◦ Ideally, CDISC Team that developed the standard, e.g., the CDISC ADaM Team for the CDISC ADaMIG
Programmer or software developer:◦ The developer of a computer validation tool that
implements a computer compliance check
5
A computer validation tool is like a teacher
Correct error messages are instructive
Incorrect error messages are destructive:◦ teach the wrong thing
◦ waste time
to review and address
to explain in Analysis Data Reviewer’s Guide
◦ in the worst case, may cause someone to take the wrong “corrective” actions
7
“conformance”, “compliance”, and “validation” ◦ Words with many meanings
◦ Experience has shown that it is difficult to discuss the problem because of the vagueness
◦ Need better vocabulary
But before vocabulary, we need an understanding of what it is we are talking about◦ For discussion, an unofficial, suggested map of this
space is presented next
9
10
Regulatory Agency Technical Conformance
Aspects NOT Checkable by a
computer
Data Quality Checks
Accompanying Documentation SDRG, ADRG, ...
Version Acceptability
Tool-Readiness Checks
Data Standards Plan
Other Checks?
Agency-DefinedBusiness Rules
Non-CDISC Rules about CDISC
Agency Technical Conformance Computer Checks
CDISC-DefinedStandards Compliance
CDISC-Published Computer Checks
12
Regulatory Agency Technical Conformance
Aspects NOT Checkable by a
computer
Data Quality Checks
Accompanying Documentation SDRG, ADRG, ...
Version Acceptability
Tool-Readiness Checks
Data Standards Plan
Other Checks?
Agency-DefinedBusiness Rules
Non-CDISC Rules about CDISC
Agency Technical Conformance Computer Checks
CDISC-DefinedStandards Compliance
CDISC-Published Computer Checks
Compliance with CDISC standards as defined by CDISC in published CDISC standards documents
Some aspects are checkable by computer
Some aspects ARE NOT checkable by computer◦ maybe someday, with perfect metadata?
13
14
Regulatory Agency Technical Conformance
Aspects NOT Checkable by a
computer
Data Quality Checks
Accompanying Documentation SDRG, ADRG, ...
Version Acceptability
Tool-Readiness Checks
Data Standards Plan
Other Checks?
Agency-DefinedBusiness Rules
Non-CDISC Rules about CDISC
Agency Technical Conformance Computer Checks
CDISC-DefinedStandards Compliance
CDISC-Published Computer Checks
Computer checks of regulatory agency technical conformance defined by the agency
Does not include CDISC-published computer checks
15
16
Regulatory Agency Technical Conformance
Aspects NOT Checkable by a
computer
Data Quality Checks
Accompanying Documentation SDRG, ADRG, ...
Version Acceptability
Tool-Readiness Checks
Data Standards Plan
Other Checks?
Agency-DefinedBusiness Rules
Non-CDISC Rules about CDISC
Agency Technical Conformance Computer Checks
CDISC-DefinedStandards Compliance
CDISC-Published Computer Checks
Data Quality Checks◦ Checks having to do with the quality of collected
data
◦ Not about CDISC standards
Tool-Readiness Checks◦ Checks having to do with whether the data support
review tools
17
Non-CDISC Rules about CDISC◦ Checks about compliance to CDISC standards
invented by others than CDISC and that are used by the agency
◦ Example: The traditional SDTM checks, that were defined by a tool vendor and FDA
◦ May or may not be consistent with the CDISC standards
18
19
Regulatory Agency Technical Conformance
Aspects NOT Checkable by a
computer
Data Quality Checks
Accompanying Documentation SDRG, ADRG, ...
Version Acceptability
Tool-Readiness Checks
Data Standards Plan
Other Checks?
Agency-DefinedBusiness Rules
Non-CDISC Rules about CDISC
Agency Technical Conformance Computer Checks
CDISC-DefinedStandards Compliance
CDISC-Published Computer Checks
In regard to what a regulatory agency needs, “Conformance” seems like a good word◦ Technical Conformance Guide
In regard to adherence to a CDISC standard◦ “Conformance” is confusing because it is already
used in the broader context of agency needs
◦ “Compliance” seems like a good word, but it is a loaded word in some regulatory contexts
◦ “Adherence”?
◦ For now, I prefer “Compliance”
20
Some aspects of ADaM (or SDTM) compliance are checkable by computer
Some are NOT checkable by computer
If there are no error messages, it does not prove that the datasets are valid
“validation checks” could really be called “invalidation checks”◦ they can demonstrate non-compliance
◦ they cannot prove compliance
21
There is no consensus yet on better vocabulary for this space
We may never get agreement on one-word terms that eliminate confusion
Until we do, it is best to use as many adjectives as necessary so that both the speaker and the listener understand
22
A chain of six steps that all have to go right for correct validation tool error messages about ADaM compliance
Causes of breakdowns at each of the steps and discussion
Improvement ideas, activities, and status
24
There is a chain of steps that have to go right in order to prevent incorrect validation tool error messages about CDISC compliance
Failure at any step may cause an incorrect error message
25
1. CDISC IG is clear and consistent
2. Computer check specification does not conflict with IG
3. Computer check specification is clear and complete enough for a pure programmer
4. Computer check specification is understood by programmer
5. Check is implemented correctly by programmer
6. Content of error message is correct
26
The following slides identify some of the causes of breakdowns at each of the steps in the chain
Note: ADaM may be more challenging from a computer checks perspective than SDTM
But the chain and the causes apply in principle to computer checks for any CDISC data standard
27
1. CDISC IG is clear and consistent
2. Computer check specification does not conflict with IG
3. Computer check specification is clear and complete enough for a pure programmer
4. Computer check specification is understood by programmer
5. Check is implemented correctly by programmer
6. Content of error message is correct
28
We try hard to be clear and consistent but are not always 100% successful◦ it is a constant challenge and hard work
A major objective of ADaMIG v1.1 was to clarify v1.0◦ E.g., added scope to many statements (within a
study? dataset? parameter? ...?)
Clarification can mean:◦ expressing something clearly
◦ debate and resolution within the ADaM Team about the concept itself that is being clarified
29
1. CDISC IG is clear and consistent
2. Computer check specification does not conflict with IG
3. Computer check specification is clear and complete enough for a pure programmer
4. Computer check specification is understood by programmer
5. Check is implemented correctly by programmer
6. Content of error message is correct
30
Misunderstanding of the IG
Not reading other pertinent sections of the IG
Overly-aggressive hunts for rules, e.g.,◦ Misinterpreting an example as a general rule
◦ Misinterpreting statements made in the context of a particular variable to apply to all variables with certain variable name fragments
Likelier when computer check specifier is not the CDISC Team that developed the standard
31
1. CDISC IG is clear and consistent
2. Computer check specification does not conflict with IG
3. Computer check specification is clear and complete enough for a pure programmer
4. Computer check specification is understood by programmer
5. Check is implemented correctly by programmer
6. Content of error message is correct
32
Pure programmers were not the main audience implicitly assumed by authors of IG
Necessary information from other areas in IG not explicit in a particular check specification◦ For example, in ADaM, even for required variables,
nulls are allowed, unless otherwise specified
Computer check specification may ignore practical data scenarios, such as missing values
No examples provided of data passing and failing check
33
1. CDISC IG is clear and consistent
2. Computer check specification does not conflict with IG
3. Computer check specification is clear and complete enough for a pure programmer
4. Computer check specification is understood by programmer
5. Check is implemented correctly by programmer
6. Content of error message is correct
34
Likelier when the check is specified incompletely and when the programmer does not have an assumed background
Programmer does not read the whole IG◦ E.g., information about permissibility of missing
values might be found somewhere else in the documents than in a compliance check specification, or in the section of the document to which the check specification points as its source
35
1. CDISC IG is clear and consistent
2. Computer check specification does not conflict with IG
3. Computer check specification is clear and complete enough for a pure programmer
4. Computer check specification is understood by programmer
5. Check is implemented correctly by programmer
6. Content of error message is correct
36
1. CDISC IG is clear and consistent
2. Computer check specification does not conflict with IG
3. Computer check specification is clear and complete enough for a pure programmer
4. Computer check specification is understood by programmer
5. Check is implemented correctly by programmer
6. Content of error message is correct
38
Content of the message not specified by the check specifier
Specified content not provided in message
Message does not identify the check specifier
Opinion: some room for value-added additional explanation on the part of the software developer, but only if clearly identified as not originating from the specifier of the check
39
Some ideas for improvement contributed by many sources
CDISC Winter Intrachange 2016 and elsewhere
Not all of them are agreed upon
Some are currently underway
There is consensus that we need to improve
40
Clarify CDISC Standards; Ongoing
Define CDISC compliance computer checks; ADaM published; SDTM drafted
Do not be overly aggressive in extracting rules from standards
Specify compliance checks completely enough for a pure, CDISC-naïve programmer
Specify a standard error message
41
ADaM Leadership Team review ADaM checks for consistency with IG; Started
Improve ADaM Team process for tracking and managing issues and suggestions about checks; Started
Respond expeditiously and reliably to issues and suggestions about ADaM checks provided by software developers; Agreed
Consider NOTE/WARNING/ERROR structure instead of only ERROR for ADaM checks
42
Consider adding a Define-XML dataset metadata SUBCLASS attribute
Consider creating test data for computer checks (both pass and fail examples)
Consider a common format across standards for CDISC-specified computer checks
43
Maybe someday every computer check specified by CDISC will be complete & clear
Until then◦ Learn the standard, not just the check
specifications
◦ Read the whole IG, because important information is scattered
Submit ideas about computer checks of CDISC compliance to CDISC Team; Started
Do not release invented checks about ADaM compliance unilaterally; Started
44
Identify the specifier of the computer check that resulted in the error message ◦ E.g., CDISC check specification document name,
agency name, software company
When the check specification defines a standard error message, clearly differentiate the standard error message from additional developer explanation or advice
45
Clarify which agency business rule checks are about CDISC compliance, and which are not
Submit ideas about computer checks of CDISC compliance to CDISC Team
Do not release invented checks about ADaM compliance unilaterally
When requesting that software developers implement checks about CDISC compliance, involve the relevant CDISC team◦ This may prevent agency-originated checks that
conflict with the CDISC standards
46
Refer to specific software and version in issue reports and discussions◦ Some issues are fixed and then become obsolete
Do not rely exclusively on software tools◦ Passing the computer checks does not prove
compliance
◦ Manual review is essential
If in any doubt, seek expert advice
47
Understand the space
Use clear descriptions and enough adjectives to define what you are talking about
We will all benefit by better communication
48
John Troxell
Senior ADaM Consultant
Accenture Accelerated R&D Services
+1 908-612-1991
49