About these endnotes
This is where we provide references and in-depth information about everything in the Judgment playbook.
We honor and thank the scientists whose research inspired this Playbook. Any errors or omissions are ours.
We define “judgments” here as questions on which the correct answer is not known (a matter of judgment is not a matter of fact), but where there is a reasonable expectation that a correct answer exists and that the extent of disagreement between qualified people will be limited (a matter of judgment is not a matter of opinion).
In statistical terms, judgment error includes two components: bias and noise. Bias is a shared, directional error, while noise seems random. Noise can be measured between different people or within the same person making the same judgment at different times (Kahneman, Sibony, & Sunstein, 2021).
There are many mental mechanisms that lead to judgment errors. The mental shortcuts we take (called heuristics) often lead us to answer a different question from the one that is posed. We can also be influenced by our preconceived opinions or neglect information that does not “fit” neatly into the coherent picture we need to reach a conclusion. We are also influenced by the opinions of others in a group, especially people we trust or who are in a position of authority. All these mechanisms can lead to errors that are shared (producing bias) or that are specific to an individual or a situation (producing noise) (Kahneman, 2011).
There is evidence of noise in important judgments in all domains, including medicine, law, public health, economic forecasting, hiring decisions, performance reviews, food safety, forensic science, child protection, patent applications, and many more. A simple summary: “Wherever there is judgment, there is
noise—and more of it than you think” (Kahneman, Rosenfield, Gandhi, & Blaser, 2016).
There are multiple ways to reduce errors in judgment. “Debiasing” targets specific biases in order to reduce shared errors (Soll, Milkman, & Payne, 2015). Other approaches aim to change the context of judgments, or “choice architecture,” to “nudge” people towards better decisions (Thaler & Sunstein, 2008). Organizations, too, can review their decision-making processes to achieve a sound “decision architecture” (Sibony, 2020). And an integrated approach to “decision hygiene” can reduce both biases and noise (Kahneman et al., 2021).
Kahneman, D. (2011). Thinking, fast and slow (1st ed.). Farrar, Straus and Giroux.
Kahneman, D., Rosenfield, A. M., Gandhi, L., & Blaser, T. (2016, October). Reducing noise in decision making. Harvard Business Review, 36–43.
Kahneman, D., Sibony, O., & Sunstein, C. R. (2021). Noise: A flaw in human judgment. Little Brown Spark.
Sibony, O. (2020). You’re about to make a terrible mistake! How biases distort decision-making and what you can do to fight them. Little Brown Spark.
Soll, J. B., Milkman, K. L., & Payne, J. W. (2015). A user’s guide to debiasing. In G. Keren & G. Wu (Eds.), The Wiley Blackwell handbook of judgment and decision making (p. 684). John Wiley & Sons. https://doi.org/10.1002/9781118468333.ch33
Thaler, R. H., & Sunstein, C. R. (2008). Nudge : Improving decisions about health, wealth, and happiness. Yale University Press.