Noise – A Flaw in Human Judgement is a book by Daniel Kahneman, Oliver Sibony, and Cass Sunstein. The noise that is their subject is not that of honking geese or backfiring motorcycles; it’s the unwanted variability in judgement or decision making when the facts behind the decision or judgement are the same.
Noise and bias are the two main topics of the book. Basis is a consistent deviation from a correct judgement. For example, a judge may always give harsher sentences to obese defendants because he has a bias against obesity. Noise is when a judge gives a harsh sentence when his favorite baseball teams loses, but gives a lenient one when the team wins.
Bias has received enormous attention and continues to be the subject of intense interest. Noise, which the three authors consider to be equally or even more problematic, has received little attention – hence this volume. Kahneman, et al discuss noise in a variety of environments. I’ll focus on two – the law and medicine.
First a few general remarks. The tome is very detailed and its style varies a lot. I suspect each author wrote different parts on his own with a subsequent general review by his two colleagues. This separation of assignments, if I’m correct, would explain the numerous dry passages and sections that go into detail greater than that which holds the attention of a nonspecialist reader.
Two principles are the foundation of the book. First, standardized tests are the best single predicter of subsequent performance. It thus follows that multiple choice tests are much less noisy than essays which require individual grading and necessarily will contain both noise and bias. Second, rules or algorithms yield better results – ie, fewer errors and less noise – than decisions made on the basis of variable individual judgements. The authors also state that long-term predictions are virtually always wrong.
A common example of noisy system is the tax code. One IRS agent will allow a deduction whereas another will disallow the same deduction according to a different interpretation of the same regulation. The authors stay away from the tax code as its orders of magnitude worse than Uncle Remus’s tarpit, medicine offers an easier target.
Consider a patient who sees two expert specialists for the same complaint and who receives two different diagnoses and treatment plans. Both doctors can’t be right, though they can both be wrong. This outcome is an example of noise which the authors rightly consider a major problem in medical practice. None of the authors is a physician which I think, at least partly, explains their underestimation of the problem of medical noise and error. They recommend the use of algorithms (rules) rather than guidelines or standards which are so flexible that noise typically overwhelms them.
A rule based process will eliminate noise, but not necessarily error. There are several reasons for this failure. Some of the rules are implemented by bureaucrats to save money, but are unrelated to improving patient care. Consider the patient whose doctor prescribes a nonstandard does of a drug – for a very good medical reason. The insurance company denies coverage because its rule doesn’t allow for this dose. Noise has been eliminated at the cost of error. Also, the rules may change every year. The recommendations for PSA screening for prostate cancer change more often than a newborn’s diapers. The rule has to be right to eliminate both noise and error. Medicine changes, usually for the better, sometimes so quickly that error is hard to discard as rules may lag well behind the latest data. Getting rid of medical noise is important but complicated. I think Kahneman and colleagues underestimate the difficulty of riding the profession of noise despite recognizing how much noise it contains.
The authors are on surer ground with respect to the law. Sunstein is a distinguished legal scholar who fully understands the vagaries of our legal system in all its complexity, though he chooses to ignore a substantial part of it – the Supreme Court (see below). The law is manmade and thus not subject to the persistent lack of knowledge that surrounds medical science. The law should be applied equally; a statement which is indisputable. Yet it’s not.
Almost everyone is aware of bias in the legal system and proclaims his intent to eliminate it. Noise according to Kahneman, et al is at least as big a problem, yet it is mostly ignored. The way to rid the system of it, say the authors, is to use algorithms; an example is sentencing rules. Most judges, or decisionmakers of importance, think they are free of both noise and bias and accordingly rebel against rules that limit their agency. They make passionate pleas that different cases need different interpretations of the same laws to allow for differences in those on whom sentence is to be passed.
Kahneman and associates mimic Plastic Man in attempting to resolve these objections though they convincingly argue that following rules yields better outcomes than letting judges follow their instincts. Judges seem to prefer flexibility to noise reduction despite evidence that that this preference leads to poorer outcomes. In the name of fairness Kahneman, et al discuss so many permutations of judgement and decision making that all but the most dogged reader’s attention may drift.
The book is notable for what it avoids. The authors are each politically correct and hence avoid dangerous areas. Start with the Supreme Court which isn’t even mentioned despite the legal system being one of the main subjects of the book. If there’s a noisier, more biased, error prone institution in American life, I can’t think what it might be. One of its justices summed up their situation thus: “We are not final because we’re infallible. We’re infallible because we’re final.” The Court is capable of provoking mayhem and horror. Think of Dred Scott or Plessy v Ferguson. Five justices can completely upend American society and life by finding hitherto unseen penumbras and rights lurking in the constitution – rights that somehow have escaped detection for centuries. Noise, bias, and error are inextricably attached to the court. Kahneman. et al don’t examine the problem because, I think, they’re afraid of where such an examination might take them.
Having proclaimed the superiority of standardized tests of general mental ability as the single best predicter of future accomplishment, they ignore the abandonment of these tests by a succession of teaching institutions. They are felt to interfere with the diverse outcomes desired our schools and enlightened politicians. Indeed the whole issue of diversity and its attendant biases are avoided by the Kahneman team. The reason is obvious. The issue is too hot to even approach.
Similarly, climate change is mentioned once, but only in passing. If long term predictions are almost always wrong, what might one conclude about assessments of the climate decades hence? This is another important issue germane to Noise, but which was likely felt to be another minefield best not trodden.
These caveats aside, the book is an excellent follow-up to Kahneman and Tversky’s Thinking, Fast and Slow. Fast thinking is the immediate response to a problem that requires deliberate thought – ie, slow thinking. Basically, noise is what results when complicated issues are incompletely and inconsistently analyzed. Noise is a good guide to placing judgements of all kinds on as rational and effective foundation as human activities will allow. There is much more to the volume than I have touched. Decision making of a host of types are dissected. Though sometimes a slog, Noise is worth the effort required to place judgements on as sound a foundation as possible.
