Explainer

Understanding Accuracy

How exactly should we interpret our measures of accuracy? The one-sentence description of the measure is as follows: “accuracy” captures the degree to which media coverage of changes in spending correspond with actual changes in spending by the federal government in the previous, current and upcoming fiscal year. This is, we believe, a relatively straightforward working definition of accuracy.

There are however a good number of measurement-related decisions required to get to this kind of accuracy measure. And there are real advantages of — but also limitations to — this approach to measuring the accuracy of media coverage. We consider some of these below.

Federal Policy. Our policy measure focuses on federal policy change. This means that the version of “policy” to which we compare our media signal does not account for spending at the state or local level. If local- or state-level budgetary policy differs from federal policy, and if media coverage focuses on local- or state-level policy, we may observe inaccuracy that is more about the level of government that media focus on than it is about inaccurate reporting of federal policy change. We expect this to be a minor issue for national television broadcasters, and for newspapers with large national audiences as well, but it may be more impactful for more regionally-focused newspapers.

Spending Change. Our measures of policy and media coverage focus on changes in spending. This means that our account of accuracy does not consider policy changes that aren’t reflected in spending (although we do attempt to account for environmental regulation in our measures in Information and Democracy). Put differently, our measures allow us to look at the accuracy of coverage about budgetary policy change. In the large domains we consider here, this is much – but not all – policy.

Accuracy Over Time. Our calculation of accuracy captures average levels of accuracy over decades, not the accuracy of media sources on any specific single piece of legislation. This approach to accuracy does not consider, for instance, whether media accurately reported the changes implemented in Obamacare. It does however capture the degree to which, over several decades, media coverage reflected annual changes in health care spending by the federal.

Sources of (In)accuracy. Accuracy (and inaccuracy) in media coverage may be the consequence of independent reporting by individual media outlets, but there are other sources. A news outlet’s use of syndicated work by affiliated outlets, or the use of news agencies/wire services, may determine reporting accuracy, as might direct reporting of information provided by government agencies or elected officials . We cannot easily distinguish the initial sources of accurate or inaccurate information in media outlets. We can however capture the accuracy of information that finds its way into news coverage, regardless of its source. Insofar as one outlet is more accurate than another one, then, we can likely attribute that difference to some combination of journalistic and editorial decisions.

Social Media. We do not examine the accuracy of content on social media here. We offer a preliminary analysis of Facebook content in Information and Democracy. And a good amount of news coverage in social media comes from the “legacy” outlets we consider here.

Congruence vs Correspondence. Our approach to measuring accuracy focuses on the degree to which spending and media coverage of spending are correlated. This is what might be called correspondence between policy and coverage. This is not the same thing as what is referred to as congruence, the degree to which policy and coverage reflect the same level of spending. Put differently: our measure captures over-time parallelism between change in budgetary policy and the media policy signal, but it ignores whether actual levels of spending are accurately captured in media content. We do this for a number of substantive and methodological reasons, outlined in Information and Democracy.

Standardized Variables. Our measure of accuracy relies on standardized variables for spending and the media policy signal. This means that our measures of accuracy “factor out” differences in the volume of spending across domains, as well as differences in the volume of content across both policy domains and media outlets. This has some consequences for media outlets’ accuracy scores, of course. A media outlet scores better on our accuracy measure with a moderate amount of highly accurate coverage than with a large amount of only moderately accurate coverage. Our graphics show accuracy alongside volume, of course — so we can consider a combination of accuracy and volume there. But it is worth noting that our approach to accuracy scores means that they will on their own privilege accuracy over volume.

In sum, there are limitations to what we can show with a single measure of media accuracy. In this case, we have focused on all media content about federal budgetary policy in five major spending domains, over time. We regard this as a valuable measure; and as a useful starting point for further considerations of media accuracy as well — in other policy domains, focused on spending or otherwise, zeroing in on individual policies, considering social media and other news outlets, and in other countries as well.