I’m currently reading Noise: A Flaw in Human Judgment, the latest book by Daniel Kahneman wo also wrote Thinking, Fast and Slow.
I’m only halfway in the book but in one chapter the authors talk about an experiment done by 2 researchers, Edward Vul and Harold Pashler where they gave a person a specific question not once but twice. The hypothesis was that the average of 2 answers was always closer to the truth than each answer independently.
And indeed they were right.
One knows more than one
Turns out that this is related to the wisdom-of-crowds-effect; if you take the average of a number of independent answers from different people it typically leads to a more correct answer.
I never heard about this effect before, but it turns out that I’m applying this principle for a long time based on something I discovered in the Righting Software book by Juval Löwy, the broadband estimation technique.
This technique allows you to estimate the implementation effort for a complex system in a fast and efficient way. The idea is that you bring a large group of people(diversity of the group is key!) together and ask them to estimate the effort to build this system. By doing this with a large enough(think somewhere between 12 and 30 people) you are applying the wisdom-of-the-crowds effect to produce more accurate results.
UPDATE: I forgot to emphasize something when applying the wisdom-of-crowds-effect. It is really important that the estimations are done independently(!) of each other. If you allow people to align first, you end up with group-thinking and skewed estimations.
Important when applying this technique is that the outliers, values that were at least one standard deviation removed from the average, are not ignored but serve as a way to identify uncertainties. After the reasoning behind these outliers is explained a new estimation round is conducted. This process is repeated until all estimations fall within one standard deviation.
I had some very good results applying this technique but it was only when reading about the wisdom-of-crowds-effect that I started to understand the science behind it.
And now it is time to continue reading as I want to find out how I can further reduce the noise in our estimations…
More information
Noise: A Flaw in Human Judgment – Wikipedia
Wisdom of the crowd – Wikipedia
Righting Software | A Method for System and Project Design by Juval Löwy
Measuring the Crowd Within: Probabilistic Representations Within Individuals (ucsd.edu)