bonnie: tips for debugging clinical quality language ecqms€¦ · bonnie overview what is bonnie?...

Approved for Public Release; Distribution Unlimited. Case Number 18-0576. © 2018, The MITRE Corporation. All Rights Reserved.

Bonnie:

Tips for Debugging Clinical Quality Language

Electronic Clinical Quality Measures

James Bradley

MITRE, Senior Software Engineer

March 2018

The MITRE Corporation operates the Centers for Medicare & Medicaid Services (CMS) Alliance to

Modernize Healthcare (CAMH), a federally funded research and development center (FFRDC)

dedicated to strengthening the nation’s health care system. MITRE operates CAMH in partnership

with CMS and the Department of Health and Human Services.

| 2 |

Agenda

▪ Assumptions

▪ Bonnie Overview

▪ Debugging CQL eCQMs in Bonnie

▪ Common Points of Confusion in CQL Logic

▪ Resources

▪ Other Related Presentations at HIMSS 2018

▪ Questions


| 3 |

Assumptions

▪ Audience is familiar with Clinical Quality Language (CQL) and electronic clinical quality measures (eCQM)

▪ CQL packages being debugged are produced and exported using the Measure Authoring Tool (MAT)

▪ Audience is using Bonnie to debug CQL eCQMs


| 4 |

Bonnie Overview

What is Bonnie?

▪ Bonnie is an electronic Clinical Quality Measure (eCQM) testing tool that allows users to:

▪ Load eCQMs from the Health Quality Measure Format (HQMF) exported from

the Measure Authoring Tool (MAT)

▪ Build synthetic patients using Quality Data Model (QDM) data elements

defined as part of the measure definition

▪ Test new and updated eCQMs using synthetic patients

▪ Explore the behavior and complexity characteristics of eCQMs


| 5 |

Bonnie Overview

Bonnie Screenshot


| 6 |

Bonnie Overview

Bonnie Screenshot (continued)


| 7 |

Debugging CQL eCQMs in Bonnie

Common Debugging Issues

▪ Overlapping Value Sets

▪ Out of Date Codes

▪ Alias Not Referencing Expected Value

▪ Timing Issues:

– “starts after”

– “day of”

▪ Points of Confusion:

– “exists”

– Multi-source queries

– Comparing Direct Reference Codes


| 8 |


Problem: Overlapping Value Sets

▪ Problem: A “retrieve” is unexpectedly calculating as True.

▪ Debug Step: Check to see if the associated value set has overlaps that may be affecting calculations.


| 9 |


Debugging: Overlapping Value Sets

▪ Hover over the “retrieve” and see which data element highlights.


| 10 |


Debugging: Overlapping Value Sets (continued)

▪ Go to the Measure Details page, scroll to the bottom, and confirm there are overlapping codes.


| 11 |


Problem: Out of Date Codes

▪ Problem: A “retrieve” is unexpectedly calculating as False.

▪ Debug Step: Check to see if there is a warning on the top of the Patient Builder page.


| 12 |


Debugging: Out of Date Codes

▪ To address an outdated code, either:

– Select a new code from the code selection list and delete the old code.

– Drag a new data element onto the patient.


| 13 |


Problem: Alias Not Referencing Expected Value

▪ Problem: An alias is referencing a value different than the one the measure developer expects. This would manifest in the calculation not behaving as expected.


| 14 |


Debugging: Alias Not Referencing Expected Value

▪ Debug Step: Hover over the alias and see which element in the patient history highlights

▪ “Inpatient encounter” lit up, not “Emergency Department Visit”

▪ Review the logic of “Encounter with ED”


| 15 |


Problem: Timing Issues

▪ Pinpoint where the issue is occurring using Bonnie’s logic highlighting.

▪ Double-check the documentation for expected behavior.

An example: “starts after”:

−The intent was to check if PCI started after the start of the Hospitalization.

− “starts after” checks if something starts after the end of a given interval.


| 16 |


Problem: Timing Issues (continued)

▪ “day of”:

– “day of” causes the logic to pay attention to the date, rather than the time

– ‘ICM’ can start anywhere between 12am 1/15/12 and 11:59pm 1/16/12


| 17 |

Common Points of Confusion in CQL Logic

▪ “exists”

• Not intuitive for how it evaluates based on certain parameters

▪ “exists({})” -> false

▪ “exists({item})” -> true

▪ “exists(null)” -> false

▪ “exists({null}) “ -> true

• As long as the parameter is not an empty list or ‘null’, “exists” will evaluate to

‘true’.


| 18 |

Common Points of Confusion in CQL Logic (continued)

▪ Multi-source queries

– For example:

– If any of E, P, M, or D returns an empty set, this definition will not be evaluated.

– This is due to the Cartesian product of the sources. A Cartesian product with an empty set will always be the empty set.


| 19 |

Common Points of Confusion in CQL Logic (continued)

▪ Comparing Direct Reference Codes

▪ Use the Equivalent (“~”) operator instead of the Equal (“=”) operator

• For example:

performed.result ~ “Positive measurement finding (finding)”

▪ Equivalence (“~”) of codes is defined based on the code and system elements only

▪ “=” is too strict and will compare all elements of a direct reference code


| 20 |

Resources

▪ Main Bonnie site

– https://bonnie.healthit.gov/

▪ JIRA for tracking Bonnie-related issues

– https://oncprojectracking.healthit.gov/support/projects/BONNIE

▪ JIRA for tracking CQL-related issues

– https://oncprojectracking.healthit.gov/support/projects/CQLIT

▪ CQL Formatting and Usage Wiki

– https://github.com/esacinc/CQL-Formatting-and-Usage-Wiki/wiki/Questions-and-Answers

▪ Development team contact

– [email protected]

▪ Open Source on GitHub

– https://github.com/projecttacoma/bonnie


https://bonnie.healthit.gov/

https://oncprojectracking.healthit.gov/support/projects/BONNIE

https://oncprojectracking.healthit.gov/support/projects/CQLIT

https://github.com/esacinc/CQL-Formatting-and-Usage-Wiki/wiki/Questions-and-Answers

mailto:[email protected]

https://github.com/projecttacoma/bonnie

| 21 |

Other Related Presentations at HIMSS 2018

March 7, 2018

10:00 AM – 10:30 PM PT

Cypress: New Features, Clinical

Quality Language Support, and

Burden Reduction

March 7, 2018

2:00 PM – 3:30 PM PT

eCQM Strategy: The Future of

eCQMs

March 8, 2018

10:30 AM – 12:00 PM PT

eCQM Strategy: The Future of

eCQMs

Sessions will be held in the MITRE Government Relations Room (Zeno 4608)


| 22 |

Follow up


To download a copy of this presentation, visit:

https://health.mitre.org/himss18

Data Interoperability to Reduce Clinician Burden

Follow us on social media: @MITREhealth

https://health.mitre.org/himss18

https://www.facebook.com/search/top/?q=mitrehealth

https://twitter.com/MITREhealth

https://www.linkedin.com/company/mitre/

| 23 |

Questions?


| 24 |

Backup Slides


| 25 |

Common Points of Confusion in the Common CQL Library

▪ Hospitalization Function

– If an ED Encounter occurs within an hour of an Inpatient Encounter, “Hospitalization” returns the start of the ED Encounter and the end of the Inpatient Encounter.

– Events that occur within the gap between the ED Encounter and the Inpatient Encounter are within the Hospitalization period.


| 26 |

Common Points of Confusion in the Common CQL Library (continued)

▪ CalendarAgeInYearsAt vs. CalendarAgeInYears

– ‘CalendarAgeInYearsAt(BirthDateTime DateTime, AsOf DateTime)’

▪ Returns the number of years passed between the first and second DateTime.

– ‘CalendarAgeInYears(BirthDateTime DateTime)’

▪ Returns the number of years passed between the passed in DateTime and Now.

▪ Function returns “CalendarAgeInYearsAt(BirthDateTime, Today())”


bonnie: tips for debugging clinical quality language ecqms€¦ · bonnie overview what is bonnie?...

Documents