Revisiting the "belief in the law of small numbers" : replication and extensions of problems reviewed in Tversky and Kahneman (1971)

Abstract

The belief in the law of small numbers is the phenomenon that people tend to think small random samples are highly representative of the population they are drawn from. In a Registered Report with an American online Amazon Mechanical Turk sample using CloudResearch (N = 1089), we conducted a conceptual replication and extension of seven problems reported by Tversky and Kahneman (1971) examining laypersons’ intuitions. In the control condition where participants were presented with scenarios that have the same sample size as the original study, we found support for people’s tendency to (1) overestimate the likelihood of successful replications, (2) assume the replication sample to carry the same characteristics of the population, (3) conclude the replication and reject the null hypothesis without sufficient evidence, and (4) overestimate the replicability of population characteristics. We failed to find support for people’s tendency to underestimate the required sample size in a replication. Extending the replication, we multiplied the sample sizes in the scenarios by 10 and 100, and found that generally as the sample size increased, the participants’ estimates also increased. We concluded that in the control condition where the sample sizes were small, the participants produced answers that support the aforementioned tendencies, and as the sample size increased, participants’ answers were also increased.