communicating risk reactively

Communicating Risk

Reactively

Mike Pearson

@grumplet

github.com/gmp26

Good Morning! It's great to be here this morning and I'm loving the conference so far - so many cool people doing great stuff!

Can you all hear me? ...

Great... My name is Mike Pearson.

I am gmp26 on github, and grumplet on Twitter.

http://github.com/gmp26

I’m fortunate enough to work in Cambridge at the Centre for Mathematical Sciences where I’ve been supporting a number of projects concerned with school mathematics education and the public understanding of risk and uncertainty.

My job now is to help people communicate risk accurately, fairly, and understandably by providing any web technology, graphics, and data visualisation needed.

Recently I’ve been using Clojure and Clojurescript to do this - and it feels like a really firm foundation to build on after trying to navigate the shifting sands of javascript for a few years.

Communicating Risk

But first a story

At the end of September I was planning a road trip to Leipzig and back to help my daughter move back to London

—

This is the vehicle we used.

A few days before leaving my conscience clicked in and nagged me to get a small chip in its windscreen seen to.

I called in for a repair.

Why am I telling you this?

Well, When the windscreen repair man arrived, he wanted me to sign a form saying I’d been warned that the repair could cause the whole windscreen to crack.

So I asked what were the chances of that happening. I reckoned he’d have done enough to give me a pretty good idea.

'50-50' he said.

—

'50-50? – What!!!? Really?? -- Do you mean half of all the repairs you make crack?

'Well it can happen' he said

'That seems a rather high risk! I don’t think I want to do this if the risk is that high.’

'Well maybe 70-30 then'

'So there's still a 30% risk of crack? Sorry to call you out, but I'm leaving in 2 days and I don't have time to get a whole new windscreen fitted.’

I was on my way back into the house when he said 'OK OK - I'll put £50 pounds on it not cracking'.

Now that gave me a lot more confidence, and I let him go ahead with the repair, which went just fine.

…pause…

COMMUNICATING RISK IS HARD.

Child Heart Surgery

But today I want to tell you about a project to communicate the risks of Children's Heart Surgery clearly and simply and truthfully to Doctors, to Consultants, Health Professionals, Journalists, and to Parents.

I’m going to talk about some of the history that created the need for this project, and which led to the philosophy and approach we adopted.

I’m going to talk about Data Visualisations and video animations - how these evolved into their present state.

and I’m going to talk a little about the web technology - Clojurescript with React, Rum, Figwheel and Devcards

Congenital Heart Disease

Congenital Heart Disease is the condition responsible for blue babies.

They were commonly referred to as blue babies because they had a bluish tinge when seen out in their prams and push chairs.

Sometimes they were called ‘hole in the heart babies’.

But for whatever reason, the heart was failing to pump blood around the lungs and body efficiently, and deoxygenated blood appears blue.

If the babies survived to be young children they would be small, easily tired and out of breath, and would retain that strange bluish colour that made them look so very different.

These days advances in diagnosis and more effective heart surgery make it quite unlikely you will encounter a blue baby on the street.

CHD affects about 6 babies in every 1000 live births

Congenital heart disease affects about 6 babies in every 1000 live births.

Not far from here, in Central Bohemia in Communist era Csechoslovakia, an autopsy was always performed on whenever a child died.

Nothing could be done for these children at the time, but the data collection was meticulous.

This data was collected for 27 years, between 1952 and 1979.

So we have a unique and unrepeatable picture of natural survival rates - what happens if there is no surgical intervention.

Natural survival over time

This is what happens to these children if nothing is done.

Of those that survive birth, 14% have such serious conditions they will die in their first week.

After 5 months a quarter will have died.

Other conditions like Tetralogy of Fallot, the classic hole in the heart condition, allowed survival into early adulthood,

Source: Šamánek, M. Pediatr Cardiol (1992) 13: 152. doi:10.1007/BF00793947

Making a blue child pink is one of the great stunts of modern medicine.

Open Hearts, Kate Bull

Kate Bull says in her recent book Open Hearts that “Making a blue child pink is one of the great stunts of modern medicine.”

Kate’s book talks about the early operations carried out in the States and subsequently in the UK, and about the extraordinary risks the families took to give their child a chance.

In the UK, Russel Brock, later Lord Brock, was one of the pioneers. When he was interviewed years later he stated that his first 6 operations went ‘Died Died Died Survived Died Died’

So that was about a 17% survival rate – in 1952. I was born in 1952, but fortunately without a heart defect.

CLICK

Post war progress

In the post war years, enormous strides were made.

Heart and Lung machines were developed, making open heart surgery much more possible

Post operative care improved hugely with oxygen tents and incubation units

In diagnosis, dangerous X-ray fluoroscopy gave way to ultrasound imaging

Good diagnosis allows surgeons to anticipate and prepare for what they will encounter when they go in. Fewer surprises means fewer deaths.

By 1989

The UK 30 day survival rate after congenital heart surgery had risen to

81%

So by 1989, survival rates were way better than 17%.

In fact, in the UK, 81% were surviving heart surgery.

But in Bristol, 30 day survival was about

63%.

Was this low figure due to chance variation?

The press didn’t think so, and there was one anaesthetist in Bristol who had serious concerns.

Eventually, a public inquiry was commissioned that ran through the early 90s.

The Bristol Inquiry into excess mortality

A data visualisation might help.

This is a simple scatter plot of Percentage Mortality in operations on children under 1 against the number of cases treated at each hospital.

We would predict the plot to show binomial variability with case volume, and we can draw in thresholds at 1 and 2 standard deviations from the mean.

Roughly 95% of all hospitals should lie inside the dark blue funnel, 99.8% inside the light blue funnel.

In fact there are a few more escapees than we’d predict - and this is because there is no risk adjustment for the different case profiles the hospitals have taken on. Some hospitals specialise in more risky cases.

CLICK

The Bristol Royal Infirmary shows up as the dot at the top - way outside the 99.8% funnel, giving a good indication that an investigation was needed.

After Bristol, UK hospitals standardised data collection

By 2010 there was enough good data to create a statistical model

After Bristol, UK hospitals standardised data collection.

Data was now systematically collected on age and weight, the conditions and complications that were presented, and the outcome 30 days after the operation.

By 2010 there was enough good data to create a statistical model

The Clinical Operational Research Unit at UCL obliged.

Here it is.

It’s a logistic model giving the probability of death by 30 days after surgery

Now hospitals could be compared fairly

Using the risk model, hospitals could now be compared fairly, but not understandably.

Maybe health professionals could understand how to interpret this graphic, but the general public found it to be rather opaque.

When another furore broke in 2013 over the performance of the Leeds General Infirmary - the dot in the grey zone here, the heart unit was closed for a week or so.

So there was a communications problem.

.Christina Pagel

Tim Rakow Emily Blackshaw

David Spiegelhalter Mike Pearson

Emily Jesper Joanne Thomas

When Christina Pagel of UCL put in for funding to update the statistical model, she added in a development budget for a web tool, and a focus group to improve on public communication.

The National Institute of Health Research thought this was an excellent idea, and suggested that focus groups should be involved right from the start and throughout the project.

Christina Pagel of UCL pulled together a team and asked for and got funding for focus groups to help sort out the communication.

This was quite a ground-breaking realisation – that these days, the funders would be prepared to support not just the academic work, but also public communications.

Christina and David Spiegelhalter from Cambridge were the statisticians on the team.

Emily Jesper and Joanne Thomas of Sense about Science had excellent contacts with health professionals, journalists, charities, and parents and would organise and run the focus groups.

Tim Rakow and Emily Blackshaw were the psychologists. They were to run experiments testing the success of the web tool in communicating the risks, comparing the understanding gained through use of the tool against simply reading the paper report. .. And I would develop the web tool.

..

46s of video

The plan

The plan was to drive the whole thing from the focus groups.

The first cycle was paper based - text and tables and a reworked NICOR graphic.

Sense about Science (Emily and Joanne) posed a number of questions, and we listened. They wrote up the results and proposed actions.

Each time round the cycle generated at least one new version of the web tool so we went 4 times round in all and had 8 small focus group.

I decided to use Clojurescript on this project because it had been working well for me in creating educational maths applications.

The end result was this site called childrensheartsurgery.info

Let’s try to improve on the funnel plot

You’ll notice there is no longer a horizontal line dividing better than average hospitals from worse than average.

Michael Gove, while Secretary of State for Education, famously asserted under questioning that he wanted all schools to be above average.

While this doesn’t make much sense, psychologists have shown that we are all over sensitive to that average line. We’d all want to go to a hospital with below average mortality rather than one with above average mortality – even if the difference between the two was very likely due to chance.

We can use colouring that begins to blur the boundaries – this was a suggestion from one of groups. Originally, we had started with a darker outside funnel.

While there is still no risk adjustment in this plot. The prediction intervals – the vertical coloured stripes – are calculated purely from the number of cases treated at each hospital.

This plot, by the way, was created in Clojurescript, and I’m in the process of creating an R htmlwidget from it so it can be used directly from an R package or a Shiny R application.

There’s still a problem labelling the hospitals here, and no easy way to do show risk adjustment.

So, onwards and upwards…

Another funnel, but turned through 90 degrees and sliced so we can name the hospitals properly.

The hospitals are sorted by the number of cases they treat so precision increases as you go down – but no longer in a smooth way.

This one was plotted in R

Now we can add risk adjustment to the plot…

The centering and spread are now adjusted according to the case mix of each hospital.

We no longer have the over dispersion we saw in the smooth funnel – the number of dots in each colour zone are roughly as predicted.

After going round the cycle a couple of times we arrived at this tabular display to display the results in any one year.

Project Cycles 2 & 3: The Map

One focus group parent noticed that we were trying to be really careful to avoid meaningless comparisons between hospitals, and suggested that instead of using a table and a funnel graph, we should display one hospital at a time. So that’s what we did – providing a map to help with navigation.

Again, quite a bit of Rum mixin hacking was necessary to get this going, coding directly in Clojurescript over the OpenLayers map interface.

Project Cycles 3 and 4

Levels Of Explanation

Project Cycles 3 and 4 were really all about introducing new levels of explanation.

We already had the deeper technical explanations – both in text and in graphics – but it was clear we needed more accessible levels.

We needed Headline, Summary, Detail everywhere.

So we extracted headline questions and summary answers from the texts

Notice that we needed to explain some terms to avoid misinterpretations

‘Chance’ or ‘Chance Factors’ was another problematic term. If your child is going under the surgeon’s knife you don’t want to hear there may be chance factors involved in the outcome.

But there is residual uncertainty, and for this we used the term ’unforeseeable factors’.

We needed key points. These were eagerly taken up by the press when we launched.

This is part of the more in-depth animation, which explained how the predicted ranges are calculated.

CLICK AND PLAY

The focus groups taught us to use the word ‘formula’ instead of ‘statistical model’

Typical misunderstandings

Average = Poor Chance Factor

= On the toss of a coin = 50/50 = Avoidable

Prediction Range ≠ Bar Chart Ratios and percentages

Individual (Child) Risk ≠ Aggregate (Hospital) Risk

Something I noticed was that you have to listen quite hard to notice the misunderstandings in the focus groups.

People don’t always say ‘I don’t understand that bit’.

Instead you have to realise that what they are saying isn’t quite right.

And scientific meanings are not the same as colloquial understandings.

If you are a (data) scientist

•Ask for funding to communicate your work • It may well be available

• Involve the public as well as your peers

The Techy BitClojure(script) Rum React The REPL Figwheel Devcards Intellij/Cursive

This technology stack works well.

Rum is sufficient to make React really easy to use. Just in time, Rum got support for server side rendering, which we used to make the web tool load up fast.

Iterative development with the REPL really helps.

Rum improved as we used it, enabling us to make the transition from animation to web app to web app plus server-side rendering seamlessly.

Figwheel is a joy to use - but can be delicate

Intellij is good, but can lose track of its own state. I learnt to ‘Reset Caches and Restart’ when things stopped making sense.

CLICK

These tools make development fast, but the real win is that we can code a complex component like this as a clojure expression - and that is it. No separate HTML file with some escaped templating language like moustache or angular.

No DataGrid library, no D3 or jQuery binding data to HTML.

It’s just a clojure data structure. Nothing more. This is the big win with a homoiconic language.

——

The Reactive Data Flow

It seems I like to think of reactive programming in terms of cycles. It’s a very simple view – probably an over-simplistic view of application development, but in some sense, the cycle with its one way flow is at the heart of what we mean by ‘Reactive’.

You may have seen it before somewhere.

On the left is the Browser View, and on the right is our Application State.

—- cut

There’s a rather beautiful symmetry to the diagram – it would not change that much if we turned it through 180 degrees. Changes to the View cause changes to the State. Changes to the State cause changes to the view.


Some of the browser user interface components are input controls that the user can manipulate.

Those inputs lead to actions that update the single global state in the application.

There’s a level change here – input events carry a lot of information – where the mouse is on the screen, what gesture triggered the event over what component and so on and so forth.

Usually we’re not interested in all that detail – we are more interested in the users intent – the intended action that will likely change the state of our system - so there is some event manipulation and filtering to be done.

Libraries such as RxJS are making a mark in this area. While we can use those libraries, in Clojure we can also build pipelines from core-async and use mechanisms such as publish/subscribe or the new mutator pattern in Om/Next.

——- cut

So we use the detailed input to calculate actions to dispatch. David Nolen calls these mutations.

Much of the early literature on Functional Reactive Programming was concerned with the bottom half of this cycle – does it make more sense to think of inputs as a sampled analog stream, or as a stream of discrete events? Well, it depends, but in the applications I have done so far, streams of discrete events have been sufficient. But then I am not usually processing audio or video.

Should you allow cycles within the input flow? That’s a trickier one to answer. Elm does not allow cycle because lazy evaluation can cause time or memory leaks

Sometimes it is absolutely fine to update state directly in the event handler. I don’t think we need to make this any more complex than the application demands – and the simpler approaches are often entirely adequate. It’s an area where the application programmer should be given choices.


On the right hand side, those actions update state. In Haskell or Elm this would be called a fold – in Clojure it’s a reduce operation.

And if I want to persist old state for logging or undo/redo purposes, clever immutable data structures allow me to do that with minimal memory overhead.

Those logs can be serialised state, transmitted over the wires and loaded back into another copy of the app at another time and place.

We used this idea to allow the researchers as Kings to record user sessions in a Google spreadsheet to play back and analyse later.


When the state updates, we have another piece of logic that I believe needs to be in the hands of the application programmer.

One of the reasons that I like Rum is that it gives me many options here.

Components can be configured to update when their parameters change, or when a state atom changes, or some value within an atom or some value derived from many atoms. Maybe in the future components we will set these things in motion with queries.


Before we leave this diagram, I want to mention the small loop labelled ‘local transient state’. This could be the transient state of a roll-over, or a drag gesture. Such state could be collected together with the app state on the right of the diagram, but with a little care it can kept locally in the component to avoid clutter. Once the gesture completes, the local state is forgotten.

Perhaps a more important use of local state would be to remember state associated with the life cycles of javascript libraries that React does not manage. In Rum we can attach event handlers in a :did-mount mixing, detach them in :will-unmount, remembering them in local state.

Anyway, I use the idea, and Rum supports it by providing a mixin to support local state variables.

OK, so that is my broad-brushstroke picture of Reactivity. Time to design a Rum component for the Child Heart Surgery Graphic.

Use Devcards

•They’re brilliant

•Thanks Bruce!

•e.g. github/ gmp26/ rcljs-widgets

Devcards are great.

Do you remember at school, the most difficult thing in plotting a graph was laying out the axes and scales?

Well these are a couple of devcard tests of the margin convention example from D3 - but written more understandably using Rum.

D3 is one of the best - if not - the best javascript library for scientific charts and plots, and I refer to it often. However, at the heart of D3 is the jQuery-like way it binds data to HTML elements.

We have better and more general ways to bind data to visuals using React, but that means writing axis and scale setup functions.

It’s good to test these out in devcards along the way.

Patch missing SVG attributes http://bit.ly/2dofCIe

One of the issues has been that React misses a few SVG attributes, but these can be patched in fairly easily.

For example, given a map of attributes and values missing in React, patch-svg-attrs returns a function that patches them into a Rum component through its state.

Rum state is accessible in a mixin, and in a defcs defined Rum component.

Using VideoJS Gist: http://bit.ly/2diAfFG

Another bit of javascript interop wizardry was necessary to support closed caption videos across all browsers using the VideoJS library.

Normally, you would use hiccup to embed the video element inside the React component,

But videoJS has a trick up it’s sleeve however which took me a while to fathom out.

When initialised, it wraps the video element you give it in a parent element of its own devising.

So to avoid a clash of responsibilities with React, we had to embed it using ‘dangerouslySetInnerHTML’.

You can also see here how a use-video-js lifecycle mixin is attached to the Rum component.

And here we’re using devcards as unit tests to check that the clojurescript calculations agree with R values.

It’s really nice to be able to handle both User Interface elements and pure function tests in one tool.

Questions for the future - 2

Do we have a good CSS-Modules story?

Probably - consider building on https://github.com/matthieu-beteille/cljs-css-modules

e.g. https://github.com/gmp26/css-modules-tester

https://github.com/gmp26/css-modules-tester

Using R from Clojure(script)

Try www.opencpu.org

It is very straightforward to set up an OpenCPU server able to run R and load R packages.

OpenCPU then provides a safe and secure way to call R functions over a JSON API.

Each call returns an API key which you can then use to retrieve the cached results of the call.

This approach allows you to use R packages from Clojure or Clojurescript.

Using in

Look up the R htmlwidgets package

The second approach allows us to generate Clojurescript visuals in R.

Since Hadley Wickham released his htmlwidgets packages, the R community has been very active in the javascript world, developing more interactive visualisations written in HTML5 and javascript.

RStudio provides a browser environment which allows these visualisations naturally within R.

There’s absolutely no reason why we can’t code up htmlwidget visualisations in Clojurescript using React, and I would far rather be developing data visualisations in Clojure than configuring JSON for the likes of Plotly and C3.

There’s a real opportunity for someone in this community to get their teeth into this.

So what are the chances of Clojurescript still being

in use in 5 years time?

Well I have no idea, but I’d put 50 pounds on it!

A final note

Those pioneering parents that Kate Bull talks about in her book still exist.

Children are still being born with congenital heart disease, and their parents are still doing everything possible to help.

We made some classic errors along the way and they were very good at pulling us up.

This rather ghostly icon for a dead child was one of them, but those parents very gently put us right. It was a privilege to work with them.

THANK YOU!

Mike Pearson @grumplet

http://childrensheartsurgery.info github.com/gmp26

Good Morning! It's great to be here this morning and I'm loving the conference so far - so many cool people doing great stuff!

Can you all hear me? ...

Great... My name is Mike Pearson.

I am gmp26 on github, and grumplet on Twitter.

Funding acknowledgement This project was funded by the National Institute for Health Research Health Services and Delivery Research Programme (project number 14/19/13)

Department of Health disclaimer The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the Health Services and Delivery Research Programme, NIHR, NHS or the Department of Health.

communicating risk reactively

Healthcare