Big Data and Winning the Lottery

Sarah Jons Author

There is no rhyme or reason. If there is no rhyme or reason why something happens, there is no obvious explanation for it.

There is no rhyme or reason. If there is no rhyme or reason why something happens, there is no obvious explanation for it. Imagine you had to randomly select one person, from a giant database of everyone whom has ever lived on earth. I know this is unlikely but bear with me. What are the chances that person is Steve Jobs?

The answer is a very big number: 1 in 107,600,000,000 .

So, now imagine being able to pick the 6 numbers of tomorrow’s lottery: the chances are one in 13,983,816. Not too bad?

So consider this real event and think about the chances of it really happening. On September 6th, 2009, the Bulgarian lottery randomly selected as the winning numbers 4, 15, 23, 24, 35, 42.

On September 10th, the Bulgarian lottery randomly selected as the winning numbers 4, 15, 23, 24, 35, 42—exactly the same numbers as the previous week.

• Fix?
• What are the odds?
• Could a massive fraud have been perpetrated?
• Had the previous numbers somehow been copied?
• More on this later.

For a long time, economists, scientists and science-fiction writers alike have pursued the question whether you can accurately predict the future from the past given sufficiently large groups, big data, historical information and computational power. In one of science-fiction’s classics books the Foundation Series by Isaac Asimov, introduced the fictional scientific concept of Psychohistory. The essential idea in psychohistory, is that while one cannot foresee the actions of a particular individual, the laws of statistics as applied to large groups of people could predict the general flow of future events. Asimov used the analogy of a gas: an observer has great difficulty in predicting the motion of a single molecule in a gas, but can predict the mass action of the gas to a high level of accuracy. Maybe that is the future.

You may have read the book by Nassim Taleb on the black swan theory. It is an excellent and still relevant example of understanding data. Worth watching him explain it here. It is a metaphor that describes an event that comes as a surprise, has a major effect, and is often inappropriately rationalized after the fact with the benefit of hindsight. The theory was developed to explain:

• The disproportionate role of high-profile, hard-to-predict, and rare events that are beyond the realm of normal expectations in history, science, finance, and technology.
• The non-computability of the probability of the consequential rare events using scientific methods (owing to the very nature of small probabilities).
• The psychological biases that make people individually and collectively blind to uncertainty and unaware of the massive role of the rare event in historical affairs.

Black Swans are events that can totally change the course of history, sort of like the unexpected appearance of a mutant with psychic powers in Asimov’s Foundation, except that they occur far, far more often. As examples of such Black Swan events he cites the rise of the Internet, the personal computer, World War I, the 9/11 attacks, and our ongoing financial meltdown.

Models based on analyzing historical data are very good at accurately measuring the risk in a portfolio under normal market conditions, the kinds of markets that explain 99 percent of events and follow the familiar bell curve or normal distribution. But, every so often, say one percent of the time, improbable events happen that are way outside a normal distribution. Such market events are totally unpredictable, that is, the future could not have been predicted based on past behavior, because the improbable event is something that has rarely, if ever, happened before.

But there is a massive neuroscience and psychology problem called cognitive biases. These are tendencies to think in certain ways. Cognitive biases can lead to systematic deviations from a standard of rationality or good judgment, and are often studied in psychology and behavioural economics. To give you my two of my current favourites (I have already dealt with Gamblers Fallacy and the God of Gaps).

Firstly the rhyme-as-reason effect is a cognitive bias whereupon a saying or aphorism is judged as more accurate or truthful when it is rewritten to rhyme. Yes truly. Researchers looked at people who judged variations of sayings which did and did not rhyme, and tended to evaluate those that rhymed as more truthful (controlled for meaning). For example, the statement ‘What sobriety conceals, alcohol reveals’ was judged to be more accurate than by different participants who saw ‘What sobriety conceals, alcohol unmasks’.

One of the most famous examples of this persuasive quality of the rhyme-as-reason effect, see ‘If it doesn’t fit, you must acquit’ the signature phrase of Johnnie L Cochran, Jr. in the O.J.Simpson trial.

The second joyful example of a cognitive bias is the so-called IKEA effect. It is a cognitive bias that occurs when consumers place a disproportionately high value on products they partially created. The IKEA effect is thought to contribute to the sunk costs effect. It occurs when managers continue to devote resources to sometimes failing projects they have invested their labour in. Have a watch of this brilliant Dan Ariely talk on What makes us feel good about our work? –

Back to the Bulgarian lottery result. It was unusual in that the duplicate sets of numbers occurred in consecutive draws. But the law of truly large numbers, combined with the fact that there are many lotteries around the world regularly rolling out their numbers, means we shouldn’t be too surprised—and so we shouldn’t be taken aback to hear that it had happened before. For example, the North Carolina Cash 5 lottery produced the same winning numbers on July 9 and 11, 2007. The lottery is a six-out-of-49 lottery, so the chance of any particular set of six numbers coming up is one in 13,983,816. That means that the chance that any particular two draws will match is one in 13,983,816. But what about the chance thatsome two draws among three draws will match? Or the chance that some two draws among 50 draws will match? There are three possible pairs among three draws but 1,225 among 50 draws. The law of combinations is coming into play. If we take it further, among 1,000 draws there are 499,500 possible pairs. In other words, if we multiply the number of draws by 20, increasing it from 50 to 1,000, the impact on the number of pairs is much greater, multiplying it by almost 408 and increasing it from 1,225 to 499,500. We are entering the realm of truly large numbers. How many draws would be needed so that the probability of drawing the same six numbers twice was greater than one half—so that this event was more likely than not? Using the same method we used in the birthday problem results in an answer of 4,404.If two draws occur each week, making 104 in a year, this number of draws will take less than 43 years. That means that after 43 years, it is more likely than not that some two of the sets of six numbers drawn by the lottery machine will have matched exactly.

As our world becomes increasingly integrated, fast changing and unpredictable, we expect large improbably disturbances or black swans, to occur more frequently, not only in finance but across business, government and society in general. Mathematical models, information analysis and fast computers will continue to be extremely valuable tools, critical to the smooth functioning of our complex systems. But, when the going gets really rough, no machine or model can ever make up for the wisdom that only comes from human judgment and experience. As long as you understand your bias!

Categories: : blog