On 31st October 2020, the UK government announced a second national lockdown to mitigate the spread of COVID-19. See Figure 1 [1].
Figure 1: A timeline of UK government responses to the Covid-19 pandemic between March 2020 and December 2021 (Courtesy of the Institute for Government).
The briefing papers supplied by various expert groups to inform that policy decision, along with the slides used to explain the reasoning to the public, are available, and show that amongst the key pieces of information informing the decision were medium- and longer-term projections of numbers of deaths and hospitalisations due to COVID-19, produced by a variety of modelling groups in the UK [2].
These projections are communicated in a number of different ways, and contain two different kinds of uncertainty. One is the (‘direct’) modelled uncertainty provided by each individual group’s model, usually communicated in the form of a range (credible interval or interquartile range) around it. The other is the uncertainty about the quality of the evidence underlying the modelling (‘indirect uncertainty’) [3], which can be communicated as verbal descriptions (e.g. “There is wide uncertainty … both for individual models and between models”[4]), or by graphically representing the individual modelled projections in the form of an ensemble plot, which gives the audience an impression of the degree of consensus (or lack of it) amongst experts. Both forms of uncertainty can be combined and represented as a single range around a median estimate, as was also done in some projections of COVID-19 deaths at this time [5].
We decided to investigate the effects of the communication of uncertainty around projections in a pair of experiments within a single online survey: the first experiment presenting people with a one of a range of graphical formats illustrating the projected numbers (as also used by the scientific advisory groups), the second experiment presenting people with a text form of the information (using terms used by the scientific advisory groups to the UK government and investigated in a previous experiment). The measures in each experiment were planned to be the same, and were based on a previous experiment.
Conditions
Experiment 1
Participants will be randomly assigned to one of five conditions to see graphs depicting the projection of the number of deaths, from 27th October to 8th December, based on the actual data and charts provided by SPI-M-O on 28th October 2020 [5,6]. They will be told that the graph is produced by the experts advising the government and that it is based on the results of modelling done by several independent research groups.
The control condition will show a single line illustrating the ‘most likely’ scenario, with no direct or indirect uncertainty illustrated, and it will be stated as being based on several models.
The fan chart condition will show a ‘most likely’ line with two shaded areas representing the interquartile range and 90% credible interval around it (i.e. a combined direct & indirect uncertainty in one representation), and it will be stated as being based on several models
The best/worst/most likely case line condition will show three lines marked as ‘most likely’, ‘reasonable worst case scenario’, and ‘reasonable best case scenario’, and it will be stated as being based on several models. This is a variation on the fan chart, in that it illustrate the combined direct and indirect uncertainties but in a different form.
The ensemble fan condition will show a series of ‘most likely’ lines, each with a shaded area around it, representing the 90% credible interval, and each labelled as coming from a different model. This is therefore illustrating both the direct and indirect uncertainties separately.
The ensemble line condition will show a series of ‘most likely’ lines, each labelled as coming from a different model but not showing the direct uncertainty around each. This is therefore illustrating the indirect uncertainty only.
See Figure 2 for the stimuli.
Figure 2: stimuli for Experiment 1. a) control condition – consensus ‘most likely’ (no uncertainty), b) fan chart of the consensus (combined direct and indirect uncertainties), c) ensemble fan (separate direct and indirect uncertainties), d) best/worst/most likely case taken from the consensus (combined direct and indirect uncertainties), e) ensemble line (indirect uncertainties only). When first shown to participants, each will have text explaining what is represented in the chart.
Participants will answer questions about the perceived usefulness and ease of comprehension of the graph, how risky it made the future feel, and the trustworthiness of the experts who produced it. They will also be asked, if they had to make a decision based on that projection, how likely they would be to decide to impose tighter restrictions to prevent the spread of Covid-19. They will then be asked ‘If we now told you that the number of deaths on November 24th actually turned out to be [x], how surprised would you be?’, and how trustworthy they think the experts are who produced the projection(s) are, in retrospect. They will each be given two outcomes for the number of deaths, one of 150 deaths - low compared to the projection and one of 900 deaths – high compared to the projection. The stimulus graph will always remain available for them to look at when answering questions.
Experiment 2
Participants will be randomly assigned to one of five conditions to read statements about the projected number of deaths on November 24th.
Control (no uncertainty): “The SPI-M group state that according to the current trend, their consensus projection for the number of deaths in the UK due to COVID-19 on November 24th is 500.” [The number matches the value shown for deaths on this date in the control graph]
Direct uncertainty only: “The SPI-M group state that according to the current trend, their consensus projection for the number of deaths in the UK due to COVID-19 on November 24th is between 380-610.” [The range matches the range shown for deaths on this date in the best/worst/most likely graph]
Indirect uncertainty (verbal) only: “The SPI-M group state that according to the current trend, their projection for the number of deaths in the UK due to COVID-19 on November 24th is 500 but that the consensus on this number is low.”
Indirect uncertainty (numerical) only: “The SPI-M group state that according to the current trend, their projections for the number of deaths in the UK due to COVID-19 on November 24th range from the worst case scenario of 500-800 to a best case scenario of 200-500.” [The ranges match those around the best and worst case lines in the ensemble fan chart]
Indirect and direct uncertainty: “The SPI-M group state that according to the current trend, their projections for the number of deaths in the UK due to COVID-19 on November 24th range from the worst case scenario of 610 to a best case scenario of 380.” [The range matches that implied by the ensemble line chart]
Participants will answer the same set of questions as around the graphs. The stimulus text will always remain available for them to look at when answering questions.
Dependent variables
Trustworthiness (Initial)
How trustworthy do you think the experts making this projection are?
[Slider 0-100; not at all trustworthy – very trustworthy]
Subjective comprehension
How much effort did you have to put in to understand the projection?
[Slider 0-100; no effort at all – a lot of effort]
Informedness
How well informed did the projection make you feel about the COVID-19 situation?
[Slider 0-100; not very informed – very informed]
If you had to make a decision about whether to impose national restrictions, how helpful would you find this projection?
[Slider 0-100; not at all helpful – very helpful]
Decision
If you had to make a decision on October 27th about whether or not to impose national restrictions to reduce the spread of COVID-19 how likely would you be to decide to move the country to tighter restrictions?
[Slider 0-100; not at all likely – very likely]
Risk perception
How risky does the Covid-19 situation feel to you in the period covered by the projection? [Slider 0-100; very low risk – very high risk]
Uncertainty perception
How certain or uncertain do you think the experts’ projection is?
[Slider 0-100; not at all certain – very certain]
Perception of the uncertainty/trustworthiness (post)
(these two scenarios will be presented in a randomised order)
Now please imagine that the actual number of deaths on November 24th was confirmed to be 150:
Given the projection, how surprised would you be that the number of deaths on November 24th was confirmed as 150?
[slider 0-100; not at all surprised – very surprised]
Given that the number of deaths on November 24th was 150, how trustworthy do you think the experts making the projection were?
[slider 0-100; not at all trustworthy – very trustworthy]
Now please imagine that the actual number of deaths on November 24th was confirmed to be 900:
Given the projection, how surprised would you be that the number of deaths on November 24th was confirmed as 900?
[slider 0-100; not at all surprised – very surprised]
Given that the number of deaths on November 24th was 900, how trustworthy do you think the experts making the projection were?
[slider 0-100; not at all trustworthy – very trustworthy]
Participants will also complete a numeracy test and demographic information.
Analysis
Analyses will be the same for Experiments 1 & 2.
All analyses will be exploratory. Our main outcomes of interest are the perceived uncertainty of the projections and perceived trustworthiness (measured in response to a) the initial plot and following updates stating the number of deaths was b) above or c) below range described.
For the following outcomes, differences between condition will be analysed using a one-way ANOVA: Subjective comprehension, informedness, usefulness, likelihood of imposing tighter restrictions, risk perception, perceived trustworthiness (measured at initial presentation), surprise at deaths being high, surprise at deaths being low. Significant effects will be followed up with pairwise comparisons using Tukey’s post hoc tests.
Differences in perceived trustworthiness measured in response to different information (‘information scenario'; either the first time the information is seen – with no feedback as to actual result, or after being informed that the number of deaths was high compared to the projection, or after being informed that the number of deaths was low compared to the projection), will be analysed by a two-way mixed 5(condition, between)x3(information scenario, within) ANOVA. Significant interactions will be followed up with repeated measures ANOVAs examining the effect of information scenario for each condition. Pairwise differences between scenarios will be analysed using paired t-tests with Bonferroni correction.
Participants who fail a simple attention check (“please select ‘quite worried’”) will be excluded from analyses.
Participants
A sample size of 1500 will be recruited, To calculate the sample size we conducted power calculations in GPower [7] based on a linear model with one, between subjects categorical predictor with 5 levels/arms specifying 80% power and an expected difference of d = 0.3. We have used an overly-conservative Bonferroni correction to account for all 10 multiple comparisons that stem from each combination of 5 arms in the between subjects design, correcting alpha to 0.005 for the purposes of the input parameters in our power calculation (i.e. 0.05 divided by the 10 comparisons). This gives n = 1500 in total. In analyses, we will use Tukey’s post hoc procedure to account for multiple testing unless otherwise stated.