About these endnotes

This is where we provide references and in-depth information about everything in the Judgment playbook.

Acknowledgements

We honor and thank the scientists whose research inspired this Playbook. Any errors or omissions are ours.

Bias and noise in judgments

We define “judgments” here as questions on which the correct answer is not known (a matter of judgment is not a matter of fact), but where there is a reasonable expectation that a correct answer exists and that the extent of disagreement between qualified people will be limited (a matter of judgment is not a matter of opinion).  

In statistical terms, judgment error includes two components: bias and noise. Bias is a shared, directional error, while noise seems random. Noise can be measured between different people or within the same person making the same judgment at different times (Kahneman, Sibony, & Sunstein, 2021).

The judgment process

There are many mental mechanisms that lead to judgment errors. The mental shortcuts we take (called heuristics) often lead us to answer a different question from the one that is posed. We can also be influenced by our preconceived opinions or neglect information that does not “fit” neatly into the coherent picture we need to reach a conclusion. We are also influenced by the opinions of others in a group, especially people we trust or who are in a position of authority. All these mechanisms can lead to errors that are shared (producing bias) or that are specific to an individual or a situation (producing noise) (Kahneman, 2011).

The magnitude of noise

There is evidence of noise in important judgments in all domains, including medicine, law, public health, economic forecasting, hiring decisions, performance reviews, food safety, forensic science, child protection, patent applications, and many more. A simple summary: “Wherever there is judgment, there is
noise—and more of it than you think” (Kahneman, Rosenfield, Gandhi, & Blaser, 2016).

Reducing judgment error

There are multiple ways to reduce errors in judgment. “Debiasing” targets specific biases in order to reduce shared errors (Soll, Milkman, & Payne, 2015). Other approaches aim to change the context of judgments, or “choice architecture,” to “nudge” people towards better decisions (Thaler & Sunstein, 2008). Organizations, too, can review their decision-making processes to achieve a sound “decision architecture” (Sibony, 2020). And an integrated approach to “decision hygiene” can reduce both biases and noise (Kahneman et al., 2021).