Assessing and Communicating Uncertainty

With government issued statistics and charts becoming a semi-regular feature of prime-time television viewing, the need to effectively communicate risk levels and uncertainty is more important than ever.

In 2020 a brand-new toolkit was developed by a cross-departmental team for Assessing and Communicating Uncertainty. This is targeted at all government analysts and is available at Uncertainty Toolkit for Analysts in Government.

The aim is to assist analysts in providing decision makers with data in such a way that the inherent uncertainty of the given situation is understood. This should help to ensure that fully informed decisions are made.

However, the toolkit could also be applied to the work of government analysts who are at the forefront of providing data for the general public. The desired result will be for the public to gain a better understanding of the risks faced with regard to the COVID-19 pandemic.

The Nature of Risk

When quantifying risk it is important to consider 2 parameters. One to measure the magnitude of the risk, and another to measure its likelihood.

It is uncontroversial to assert that in order to understand risk, one must understand both the level of uncertainty involved and the potential magnitude of that risk.

The chart below, ‘New infections levelling off’ was originally from the Office for National Statistics (ONS) and was on the BBC in November 2020. It shows the magnitude of the risk faced and includes a measure of uncertainty.

This is but one example of where a simple technique, such as including a margin of error, can greatly improve the information being conveyed by a chart. In doing so, it can improve the audience’s understanding of the underlying risk.

 Source 

Although it may be natural to assume that projections into the future are inherently uncertain, it may be less obvious that recently gathered data will also be laden with uncertainty. The chart below (Wales, Percentage testing positive for COVID-19) employs error bars in a way that shows the uncertainty of the data itself.

Source

A further example of recent reporting where careful consideration has been given to the communication of uncertainty is contained in the Coronavirus (COVID-19) Infection Survey antibody data for the UK: 2 March 2021.

Here, error bars are included on every chart, and a clear explanation of the meaning of these bars is offered: “Black lines represent 95% credible intervals around the estimates, which reflect a 95% probability that the true value lies within the ranges shown.”

Source 

What is the toolkit?

The Assessing and Communicating Uncertainty Toolkit is a new guide for analysts working in government to use when confronted by a problem that includes elements of uncertainty. 

The toolkit breaks down the process of conducting analysis involving uncertainty into a 4-step process:

  1. agreeing how uncertainty should be used to help answer the question
  2. defining and identifying uncertainty
  3. measuring and understanding uncertainty
  4. presenting and communicating uncertainty

For each step the toolkit guides the analyst with examples and explanations, providing key points to consider and suggestions of actions to take. It also includes several case studies so the analyst can see the whole cycle in context from start to finish.

Why does this matter?

As we saw during 2020, the government’s actions to mitigate the risks due to the COVID-19 pandemic have had a profound effect on our daily lives.

It’s important that data prepared for decision makers in government is presented in a way as to ensure that the nature of the uncertainty and subsequent results are fully understood.

It’s also vitally important that the data being presented to people allows them to understand the level of uncertainty inherent in the results. The Assessing and Communicating Uncertainty Toolkit helps analysts to do this, and in doing so is designed to ensure that properly informed decisions can be made by all.

The Communicating Uncertainty Team

The toolkit has been a collaborative effort by representatives from a wide range of government departments. It represents evidence and best practice from across the analytical community within and outside of government.

The current iteration of the toolkit is the result of extensive research and discussions surrounding good practice for dealing with and communicating uncertainty. This work is carried out by analysts working on the front line in the field of presenting technical analysis to decision-makers.

Phase 2 of the project, which is well underway, is to engage more widely with academics and to learn from even more published papers in this field. This will help the team ensure the advice being offered reflects the latest findings of academic researchers and make sure all relevant considerations have been taken into account, with the goal of making the toolkit more robust. 

Actuaries are experts at risk and uncertainty, so the Government Actuary’s Department has naturally been a key contributor to the toolkit. It’s been a privilege to work with colleagues from a range of government departments and organisations that have contributed to the production of the toolkit.

Where can I find it?

To see (and use) the toolkit for yourself, please follow the link below:

Uncertainty Toolkit for Analysts in Government.

  • Scott Waygood

    Senior Trainee Actuary

    Government Actuary's Department