The danger of building strong narratives on weak data

As per today's quote of the day, John Burn-Murdoch has a superb, must-read article in the FT this weekend about

The danger of building strong narratives on weak data | Financial Times (ft.com).

He makes a really important point about how compiling data about what's happening to a country's economy, or to national health, is difficult, and some of the key data does not come in until months or even years after the first estimates have been published.

Furthermore, some countries' statistics are much more accurate than others, and some are more pessimistic.

Based on the absolute size of subsequent revisions to economic data among 24 countries who bother to go back and check their data and publish the revised ones, correcting the initial figures if they were wrong, Britain's ONS is one of the five most accurate among equivalent agencies - behind France, Canada, Italy and Spain but ahead of the US, Germany, Australia or Belgium and way ahead of most other democracies (all of which in turn will be producing far more accurate figures than most authoritarian or totalitarian regimes.)

However, when you look at the direction of the revisions, Britain stands revealed as publishing the most pessimistic initial statistics - ONS figures are the most likely to be revised up rather than down, where for the US it is the reverse.

The combination sounds counter-intuitive, but the figures back John Burn-Murdoch up on this: Britain manages to combine some of the most accurate initial economic stats in absolute terms with the most pessimistic ones in terms of the direction of errors.

This tends to mean that false narratives all too often get established based on initial data which turns out to be wrong.

For example, how many people remember the "double dip" recession of 2011/2012? It never actually happened. And this week we discovered that another commonly believed and shared narrative was based on wrong data: the allegation that Britain was the only country in the G7 whose economy had yet to recover past pre-COVID levels. Turns out that it wasn't quite like that.

The ONS now say that the data on which this statement had been based was inaccurate and has been revised.  The UK economy shrank less than was previously thought during the pandemic, and returned to pre-Covid levels almost two years ago: the ONS now think Britain's GDP was 0.6% higher than before Covid in the last quarter of 2021, not 1.2% smaller. The revision also means that UK growth over the pandemic is now estimated to have been the third fastest in the G7 over the pandemic, not the slowest.

Oops.

As I wrote last week, "I suppose the lesson to learn here is not to get too invested in the initial version of any economic statistic."

And the same applies to health. All governments around the world got some things right and other things wrong during the pandemic. At the time a whole raft of people were suggesting that Britain had a far worse rate of excess deaths than most comparable countries.

We now know that this claim arose because Britain was publishing far more honest statistics about deaths from COVID-19 than any authoritarian state and far more accurate statistics about COVID death rates than most other democracies. Far from having a death rate which was an outlier, we were right in the middle of comparable nations but being given far more realistic information about the atrocious number of excess deaths which we and almost all other nations suffered.

Compare the scatter of excess death rates (left to right axis, right is worse) and economic impact (vertical axis) on these two charts of respectively the 2021 early view (on the left) and the 2023 current view (on the right.)

Bit different isn't it?

Here's the key learning point: whether you are on the left or the right, whether you were on the safety first side of the COVID debate who thought the government was not locking down hard enough or the lockdown-sceptic side who thought government lockdowns were doing more harm than COVID, be careful before buying into a narrative based on early data which may turn out to tell you what you want to hear - and to be completely wrong.