introduction - whatsgoingoninhishead.files.wordpress.com€¦ · web viewthe methodology was...

What’s Going On in His Head? Evaluation report January 2015

Report of an External Evaluation of What’s Going On in His Head?

Author:

Sarah Jenkins

15 January 2015

www.jenesysassociates.co.uk


Contents1 Introduction..................................................................................................................... 2

2 Evaluation methodology and metrics...............................................................................2

2.1 Evaluation methodology............................................................................................2

2.2 Audience metrics.......................................................................................................2

2.3 Evaluation metrics.....................................................................................................3

3 Audience demographics and backgrounds.......................................................................4

4 Audience motivations.......................................................................................................7

5 Audience opinions............................................................................................................8

6 Immediate impacts.........................................................................................................10

7 Audience intentions....................................................................................................... 12

8 Longer-term impacts......................................................................................................13

9 Conclusions.................................................................................................................... 15

1


1 IntroductionThis report contains the findings of an external evaluation of What’s Going On In His Head?, which is a science communication show created and presented by science communicator James Piercy. Sponsored by the Wellcome Trust, the show uses the story of James’s own head injury, diagnosis, treatment and recovery to engage non-specialist audiences with various aspects of biomedical science and computer modelling.

The external evaluation was conducted in 2014 by Sarah Jenkins of Jenesys Associates Ltd. It aimed to address the following questions:

1. Who attends What’s Going On In his Head? and what are their science backgrounds, aims and motivations?

2. How do audiences react to the show? How challenging or emotive do they find it?3. What immediate and longer-term impacts does the show have on audience

members?4. How, if at all, does the show affect audience understanding of head injuries and

recovery?5. How, if at all, does the show affect audience attitudes towards head injuries?6. What factors contribute to the show’s most significant impacts? What factors limit

those impacts and what could be learned from these?

2 Evaluation methodology and metrics

2.1 Evaluation methodologyThe methodology was qualitative and quantitative. It comprised:

Observations and short interviews conducted by the external evaluator at 2 performances of What’s Going On In His Head?

An on-line audience questionnaire, which was publicised at all performances and via James Piercy’s blog and twitter feed. A link to the questionnaire was also emailed directly to audience members who provided their email addresses to the evaluator at the performances she observed.

Follow-up telephone interviews with audience members 3 to 6 months after they had seen the show to seek their longer term views and reflections.

2.2 Audience metricsThe evaluation covered 5 performances of What’s Going On in His Head? Although the show was performed in other locations, including outside the UK. The estimated audience numbers for the 5 evaluated performances are shown in bold in the table overleaf, alongside the total UK audience figure.

2


Fig. 1 Audience numbers

Location Estimated Audience Size

Addenbrookes medical school, UK 40

WI East Tuddenham, Norfolk, UK 48

Thinktank Birmingham 65

BSA North Wales, Wrexham, UK 80

Firstsite, Colchester, UK 69

King’s College London, UK 55

BIG Event, Oxford, UK 25

Exeter University,UK 85

Belfast University,UK 60

Total 527

2.3 Evaluation metricsA total of 116 audience questionnaires were completed. As the following graph shows, 100 responses were received for the 5 locations listed above, with 16 other respondents either not reporting where they had seen What’s Going On in His Head? Or having seen it in other locations. 66% (77) of respondents had seen the show at locations attended by the evaluator.

Fig. 2 Questionnaire responses by location (n=116)

East Tud-denham

Wrexham Colchester King's College London

Oxford Other/unknown

0

10

20

30

40

50

9

44

33

10

4

16

In addition to the questionnaire, qualitative feedback was captured through:

Observations and 10 short interviews were conducted during May 2014 at Wrexham (5 interviews) and Colchester (5 interviews).

24 telephone interviews were conducted between August and October 2014 with audience members who agreed to provide longer term feedback.

3


The remaining sections of this report document the evaluation findings and conclusions. Audience quotes are shown in blue italic.

4


3 Audience demographics and backgroundsThe majority (74%, 86) of questionnaire respondents were female. This figure is consistent with audiences observed in Wrexham and Colchester, where the evaluator estimated that two-thirds and three-quarters of the audiences respectively were female.

Fig. 3 Audience gender (n=116)

Female; 86; 74%

Male; 26; 22%

Unknown; 4; 3%

A majority of respondents (67%, 78) were age 45 or more. No respondents were age under 25. Again this figure is consistent with the observed audiences, where across 2 locations fewer than 10 individuals were estimated to be under 25.

Fig. 4 Audience age (n=116)

25 to 34; 10; 9%

35 to 44; 24; 21%

45 to 54; 20; 17%55 to 64; 32; 28%

65+; 26; 22%

Unknown; 4; 3%

5


In order to provide some indication of audience members’ science backgrounds, questionnaire respondents were asked to select the highest level at which they had studied science. Over half (52%, 60) reported they studied science after leaving school, and 24% (28) reported having studied a postgraduate degree or professional qualification. This suggests that the majority of audience members had some background in science, although they were not asked which field of science they had studied. 9% (10) reported that they had never studied science: all of these were retired and had seen the show in Colchester.

Fig. 5 Science background of audience (n=116)

Never studied

School: up to 16

School: 16 +

Undergraduate degree

Postgraduate degree

Professional qualification

Other

0 10 20 30

10

24

22

28

16

12

4

97 questionnaire respondents reported their occupations, which are categorised in the following table and suggest that the majority of audience members were not specialists or experts in the subjects of brain injury or healthcare. All occupations that were categorised as ‘other’ were not scientific or healthcare-related.

Fig. 6 Audience occupations (n=98)

Occupation n=

Retired 28

Healthcare professional/worker/ therapist 21

Teacher 10

Student 10

Academic/post doc 8

Science communicator/presenter 3

Other 18

6


As an indicator of their background knowledge of head injury, respondents were asked to select the extent to which they agreed or disagreed with the statement I knew a lot about head injury before this show. Over half (55%, 60) strongly disagreed or disagreed which suggests that they were not experts or did not have personal experience of head injury. In contrast 25% (28) agreed or strongly agreed that they knew a lot about head injury. Of the 4 respondents who ‘strongly agreed’ 2 had a head injury and 2 already knew James.

Fig. 7 Audience prior knowledge of head injury (n=110)

Strongly agree Agree Neither agree nor disagree

Disagree Strongly disagree

0

10

20

30

40

50

60

4

24 22

48

12

7


4 Audience motivationsQuestionnaire respondents were asked to select the main reason they came to What’s Going On In his Head? The most common answer was ‘It sounded interesting’ (35%, 40), indicating that the show content is inherently engaging. 24% (28) selected ‘I know James Piercy’ (24%, 28) and this answer was the most common choice of respondents who had seen the show at East Tuddenham and Colchester, which are relatively near to James’s home. 9 respondents who selected ‘Other’ described their reasons for attending. A majority of these were recommendation from a friend, colleague or another speaker at a multi-event location, such as a WI science day or science conference.

Fig. 8 Audience reasons for coming to the show (n=116)

I know James Piercy

I have had a head injury

I have a friend/family member who has a head injury

I work in healthcare

It sounded interesting

Relevant to my job/study

Other

0 10 20 30 40 50

28

4

4

12

40

8

20

Interviews with audience members at observed shows confirmed the above and explored reasons for finding the show interesting. The personal nature of the content and the fascinating nature of the brain were highlighted here.

For me the fact it is a personal story was most interesting. I wanted to hear about his experiences from the horse’s mouth, if you like.

We’ve heard from a lot of experts about these things like for Michael Schumacher. But I was interested to hear personally from someone directly affected by such an injury. You cannot get a better expert than someone who has had an injury.

Anything about the brain is interesting. It is amazing to think how complicated it all is. We are all still learning about it, so of course it is interesting.

Interviewees who knew James also reported that they had come to support him and see for themselves how his recovery was progressing.

8


5 Audience opinionsRatings

Questionnaire respondents rated What’s Going On In His Head? highly, with 93% (108) giving it the highest rating of ‘Excellent’ and no one rating it less than ‘Good’.

Fig. 9 Audience ratings of What’s Going On In His Head? (n=116)

Excellent Good Average Poor Very poor0

20

40

60

80

100

120108

80 0 0

Respondents’ explanations of their ratings fell into three main categories: Engaging and professional nature of the speaker coupled with his bravery Personal perspective applied to biomedical science The balance achieved between humour and serious scientific content

The following comments illustrate respondents’ opinions:

A good speaker who added just the right amount of humour to a very serious and potentially depressing subject.

An engaging speaker. Fascinating subject delivered with a unique personal perspective on science and healthcare.

Apart from the element of personal experience it was well researched and very well delivered. The science was easy to understand.

Expertly woven combination of a human journey, fascinating science and the strength of character that overcame astonishing odds.

Personal trauma and loss explained scientifically and made amusing! How amazing is that?

With one exception, whose comments indicated that this individual was expecting to attend a serious science lecture and that they found the humour inappropriate, interviewees also rated What’s Going On In His Head? highly. For example:

It was a delicate and accomplished blend of facts, some of which were horrific, and benefited from humour and a light hearted take on something that could have been quite dark.

9


It was so well presented by the speaker who was able to use humour at appropriate times. It was a real life experience which gave details which would probably not have come from an academic presentation. He had obviously done a lot of research.

Successes

Questionnaire respondents and interviewees at the observed performances were asked to describe the most successful aspects of the show. These reflect the rating and the most common success factors were:

The speaker’s ability to engage and connect with the audience. The use of a personal, first-hand experience. The quality and accessibility of the scientific facts that were presented

The following responses illustrate these success factors:

The warm human connection between the speaker and the audience allowed deeply personal experiences to be shared openly.

The strength is in the presentation and communication skills of the speaker – the way he is able to engage the audience in something quite difficult and communicate some great facts.

The personal account of the lecture and the laid back way in which he told it were never boring – he kept you interested all the way through. And because it’s a personal story it humanises the science.

The excellent presentation skills - he was engaging from the moment he started, informative and interesting. The emotional aspects and examples from personal experience served to help understanding and to make the information 'stick'.

It was very easy for me to follow and understand. A very informative insight into both the science and how a person has to readjust and learn to cope

He set out to tell us what was going on in his head without depressing us and he certainly managed to explain complex medical details clearly, but yet making it personal to him.

The pitching of the information. He managed to get complicated things explained so well. Punchy, interesting facts and presented at an accessible level for everyone!

Other reported successes were the interactive format of the show, which audiences enjoyed, and the emotional, sometimes challenging, nature of some of the content, which was deemed to have made the content memorable.

10


Suggestions

Respondents and interviewees were also asked what, if anything, could be improved about What's Going On In His Head? A majority said nothing. The most common suggestions were:

More promotion and marketing to increase awareness of the show. Post-event informal networking (particularly for people with direct experience of

head injuries). Content input from healthcare professionals to expand on some of the techniques

used in diagnosis and treatment of brain injury.

6 Immediate impacts General impacts

Questionnaire respondents and interviewees at observed performances were asked to describe the show’s main impacts on them. The most common descriptions fall into these categories:

Greater understanding of brain injury and its impacts on individuals. More positive feelings about the outcomes for individuals who suffer traumatic brain

injuries. A better appreciation of the technologies and experts involved in the treatment of

brain injuries.

Descriptions of these impacts included:

I now have a small understanding of how the very complex brain operates, by seeing what happens when things don't work properly.

To be mindful of the impact of brain trauma with regard to such things as lethargy. Need to eat regularly and realising that the brain needs a lot of energy to aid recovery.

I am much, much more aware of the complexities of a brain injury, and the lasting effects it has

A very positive story and I am so glad to have heard about the positive outcomes so far and hope that they continue for him.

That every brain injury is different and it is impossible from clinician's perspective to be definitive in prognosis.

Brain injuries are all about learning to live with them and trying to reconnect pathways rather than there being a magic wand 'cure'.

It was an emotional journey and I learned a lot about science too. I now have better awareness of the emergency services - including the Air Ambulance - and the closeness between scientists and medical professionals.

11


Other described impacts were: a better understanding of their own situation for individuals with head injuries; interest in further study or careers in healthcare; and admiration for human capacity to recover.

Specific impacts

Questionnaire respondents reported the extent to which they agreed or disagreed with five statements about the impacts What’s Going On In His Head? has had on them. The statements were developed from responses to an open question about impacts used in the 2013 evaluation of the show.

As shown in the graph below, by selecting ‘strongly agree’ or ‘agree’ 98% (108) of respondents indicated that they had ‘learned something new from the show’. Using the same response criteria, 98% (112) reported that they ‘now know more about diagnosis and treatment of head injury’; 84% (94) ‘now feel more optimistic about the outcomes from head injuries’; 89% (98) ‘now have more respect for those who suffer head injuries’; and 91% (110) ‘now have increased admiration for those who treat head injuries’.

Fig. 10 Immediate impacts on audiences (n=110 to 114)

I learned something new from this show

I now know more about diagnosis and treatment of head injuries

I now feel more optimistic about the outcomes from head injuries

I now have more respect for those who suffer head injuries

I now have increased admiration for those who treat head injuries

0 20 40 60 80 100 120

78

60

34

74

72

30

52

60

24

28

2

2

2

4

6

16

8

4

Strongly agree Agree Neither agree nor disagreeDisagree Strongly disagree

12


7 Audience intentionsQuestionnaire respondents reported their intentions to discuss or further explore the topic of head injury. 70% (66) indicated that they were ‘Extremely likely’ to discuss the content of this show with someone else (e.g. friend or family member). 19% (18) were ‘Extremely likely’ to research and/or read more about the topic of head injury.

Fig. 11 Audience intentions (n=94)

…discuss the content of this show with someone else (e.g. friend/family member)

…research and/or read more about the topic of head injury

0 10 20 30 40 50 60 70 80 90100

2

14

10

24

16

36

66

182

1-Not at all likely 2 34 5 - Extremely likely

Other comments made during interviews and questionnaire responses confirmed confirm the intentions reported above, as well as other intentions as these comments show.

My mind has been opened more to this sort of injury and with particular regard to the injured person and their recovery.

It was all so easy to understand, I have already told a friend all about it. I will probably not seek to read more about head injuries, but should I come across an article or something I will more than likely read it.

I’ll share information with family members - including a son who works in operating theatres

It is a very interesting talk about something that is a little out of the ordinary for a mid-week night out. I would definitely go to similar talks if available.

As an educator I found watching his presentation style really useful and I am taking away the point that it is both the content and ability to express it accessibly and to engage the audience that is the real skill

13


8 Longer-term impactsTelephone interviews with 24 audience members took place between 3 and 6 months after individuals had seen What’s Going On In His Head? Interview subjects were asked to describe any follow-up they had done and to reflect upon the impacts that the show had on them.

All interviewees said they would recommend or had recommended the show to others if it was being repeated. They said it would appeal to anyone, regardless of background, with some suggesting that the content may have to be adapted for young people to remove some of the more shocking elements.

I really think it is for everyone as you never know if you could suffer the same sort of injury or meet someone with such an injury.

I suppose children may need support to understand some parts of the show, perhaps he may have to take some more emotional things out for them, but leave in all the facts.

I suppose it felt slightly invasive for me, as a stranger, to know so much about this part of his life, but his delivery put the audience at ease and I overcame this to the point where I learned a great deal about the effects of a brain injury. I am able to talk to others about what I learned and I would now recommend the event to anyone over 14.

19 interviewees had discussed the content of the show with friends, colleagues or family members. Most of these indicated that they had shared their learning about the effects of brain trauma and told others about how impressed they were with James Piercy’s ability to communicate his experiences of injury and recovery.

I have discussed with my husband and children the need for us to be much more understanding of those who have brain injuries. I think it is important for everyone to understand the variability in how people struggle, on a day to day or hour to hour basis.

I raved about it to people I know via Facebook. The fact that James has turned a tragic accident around, done what he does and used it to explain about brain injury to others, to engage other people and explain to them about the impact. It was a very positive story.

As a nurse this talk has made me think very hard about how I care for badly injured people, which I have discussed with colleagues. This man is inspirational – he has been through such a lot both through his injuries and the loss of his wife.

Talking to my family about it has helped me realise that my mind has been opened more to this sort of injury and it impacts, with particular regard to the injured person and their recovery.

7 interview subjects reported that they were now more aware of head injury coverage and reporting in the media.

14


I would definitely say that I’ve noticed it more in the news and I don’t just mean the cases involving F1 drivers, but if I read about a traumatic accident I am now able to think about how victims may be diagnosed and treated.

Other reported follow-up activities were attending another event (organised by a brain injury charity) about head injury; researching more about the topic; mentioning James’s story in a work newsletter; being inspired to talk publicly about their own head injury; and being more aware of the situation of a friend who has a head injury.

When asked to reflect on the impacts of What’s Going On In His Head? all interviewees described how their understanding of head injury and its treatment had been improved.

Looking at it now, I understand the importance of rapid response to this kind of accident. I’ve also got increased empathy towards head injury sufferers about the effort that everyday living makes on a person.

I now know that determination and the right support mean you can achieve your goals in the most trying circumstances.

Other than much greater awareness of the impact of head injuries, I learned a lot about the range of skills, professionals and scientific disciplines that come together to treat patients.

I was amazed to learn the range of effects head injuries can have, but also how time can restore functions temporarily lost. I was left amazed at what our healthy brains do automatically every day, and we only ever notice when it stops working.

I suppose it felt slightly invasive for me, as a stranger, to know so much about this part of his life, but his delivery put the audience at ease and I overcame this to the point where I learned a great deal about the effects of a brain injury. I am able to talk to others about what I learned and I would now recommend the event to anyone over 14.

Some interview subjects also made suggestions to increase the impact of What’s Going On In His Head? All expressed the view that there is the potential for the topic of brain injury to be explored more widely in public forums. Specific suggestions were:

To explore more aspects of brain injury and recovery by including examples based on the experiences of others who have injuries.

To add content from healthcare professionals, thus expanding on the biomedical content.

To make the format of the show more immersive or theatrical and therefore even more memorable.

To adapt the content for TV or a book in order to increase the potential size of the audience.

15


9 ConclusionsWhat’s Going On In his Head? attracted an adult audience of which most were age 45 or more and over half had studied science after leaving school. A quarter had studied science at post-graduate level or professional qualification level. Audience members’ occupations suggest that most were not experts in brain injury and this suggestion is supported by the fact that over half indicated that they did not know a lot about head injury before the show. Audience members who strongly agreed that they knew a lot about head injury comprised 2 individuals with head injuries and 2 who already knew James Piercy.

The most common reason for attending What’s Going On In His Head? was ‘It sounded interesting’ with the personal nature of the content deemed to be a factor in stimulating interest. A quarter said they came because they knew know James Piercy, which is unsurprising as 2 of the show locations were close to James’s home and 1 took place at the BIG event, and James is a prominent member of the BIG network.

Audiences reacted positively to What’s Going On In His Head? and rated it highly. They attributed these ratings to the quality of the speaker, personal nature of the content; and the blend of humour and scientific facts.

What’s Going On In His Head? has a number of short and longer-term impacts on audience members of all backgrounds. The most significant of these were: greater understanding of brain injury and its impacts on individuals and learning about how brain injuries are diagnosed and treated, including some physiological facts and how some specific technologies are applied. The show also stimulated audiences to discuss the content of the show with others with a majority of individuals indicating that they would discuss the content and half being likely to research or read more about head injury. Longer-term feedback indicates that the content had been discussed with family members, friends and healthcare colleagues and that individuals were now more attuned to head injuries, including media reporting.

The evaluation findings suggest that What’s Going On In His Head? successfully enhances audiences’ knowledge about head injuries, including the individual, specific nature of such injuries and improves audience understanding of how those who are affected by these injuries may have to adapt.

The personal nature of the content coupled with humour and excellent communication were the main success factors. Audiences believe that these successes can be built upon to maximise impacts through increased marketing and promotion; adoption of higher production levels; exploiting other formats such as TV or books; and adding content from relevant healthcare professionals and other individuals who have head injuries.

Overall, What’s Going On In His Head? has been a success and addresses an appealing topic. There is some scope to develop other activities around the same topic. However, much of the success is due to James Piercy’s skill as a science communicator and basing the content around his personal experiences. Therefore, it will be important to maintain excellent communication standards and include first-hand experiences in any extended activities.

16

introduction - whatsgoingoninhishead.files.wordpress.com€¦ · web viewthe methodology was...

Documents