Overconfidence in a World Governed by Randomness

This blog post is inspired by a New York Times article titled The Hazards of Confidence about Daniel Kahneman's new book titled Thinking, Fast and Slow. Daniel Kahneman is emeritus professor of psychology and of public affairs at Princeton University and a winner of the 2002 Noble Prize in Economics.

Below are some of the fascinating quotes from the article followed by my analysis which was inspired by multiple email conversations I had with friends regarding this article:

We are prone to think that the world is more regular and predictable than it really is, because our memory automatically and continuously maintains a story about what is going on, and because the rules of memory tend to make that story as coherent as possible and to suppress alternatives. Fast thinking is not prone to doubt.

The confidence we experience as we make a judgment is not a reasoned evaluation of the probability that it is right. Confidence is a feeling, one determined mostly by the coherence of the story and by the ease with which it comes to mind, even when the evidence for the story is sparse and unreliable. The bias toward coherence favors overconfidence. An individual who expresses high confidence probably has a good story, which may or may not be true. 

Nevertheless, the evidence from more than 50 years of research is conclusive: for a large majority of fund managers, the selection of stocks is more like rolling dice than like playing poker. At least two out of every three mutual funds underperform the overall market in any given year.

More important, the year-to-year correlation among the outcomes of mutual funds is very small, barely different from zero. The funds that were successful in any given year were mostly lucky; they had a good roll of the dice. There is general agreement among researchers that this is true for nearly all stock pickers, whether they know it or not — and most do not. The subjective experience of traders is that they are making sensible, educated guesses in a situation of great uncertainty. In highly efficient markets, however, educated guesses are not more accurate than blind guesses.

The illusion of skill is not only an individual aberration; it is deeply ingrained in the culture of the industry. Facts that challenge such basic assumptions — and thereby threaten people’s livelihood and self-esteem — are simply not absorbed. The mind does not digest them. This is particularly true of statistical studies of performance, which provide general facts that people will ignore if they conflict with their personal experience.

Overconfidence arises because people are often blind to their own blindness. True intuitive expertise is learned from prolonged experience with good feedback on mistakes.

I completely agree with the article and Mr. Kahenman's views and observation. I feel like people construct opinions and make decisions emotionally and use statistics and information that supports their opinion and decision to justify their ideologies. 

The truth is the universe is governed by physical and biological laws but the interactions of particles, organisms and people is mostly governed by randomness. And randomness is something our brain has a hard time processing that's why we always reduce things down to "Good vs Evil", "Black and White" and "Us vs Them". It makes life easier to digest for most people but does not solve problems, it's too simplistic of a view. Cancer, terrorism economic/financial disasters, poverty, racism are all like you said products of very complex and often unconscious underlying processes and mechanisms in society and simplifying them will not help us solve these problems (i.e. making NFL players wear pink for a month will no cure breast cancer). 

Politicians know this, that's why they always offer simplistic, easy to digest causes and solutions for our country's complex problems, which I think is dishonest and counterproductive but again the honest ones would never be understood and elected by the majority of the population. 

In my opinion, if you live your life as if everything happens for a reason then you'll find and/or create reasons for everything (our brains are obsessed with pattern recognition), but accepting randomness as life's driving force allows for humility and appreciation for life as it has come to be from the random interactions of trillions of starts and galaxies and the 15 billion.

The universe is arbitrary. Things don't happen for a reason, they just happen. All the attempts to understand the "why" behind everything are to some extent manifestations of the biggest question of all: Why are we here? Must be a reason.

-A friend's response