Dark Data: Why What You Don't Know Matters

Chapter 7: Dark Data and the Big Picture

By David Hand

With data doubling every two years, it’s easy to assume that any observation, decision, or conclusion one makes can be an informed one supported by data. That’s not always the case says author David Hand. Professor emeritus of mathematics at Imperial College London, former president of the Royal Statistical Society and a fellow of the British Academy, Hand suggests that our seemingly data-saturated world is full of Dark Data, the expanse of data that we don’t see.

While it’s impossible to know everything, missing data could be crucial to our understanding of a situation or problem. In his latest book, Dark Data, Dr. Hand explores the many ways we can be blind to missing or unseen data and how, in our rush to be a data-driven society, we might be missing things that matter, leading to dangerous decisions that can sometimes have disastrous consequences.

In this excerpt from Chapter Seven, Dark Data and the Big Picture, Hand discusses the effect of dark data in the scientific and medical communities. Building on professor of medicine and statistics at Stanford John Ioannidis', claim that “it can be proven that most claimed research findings are false,” Dr. Hand argues that the very nature of scientific discovery lends itself to dark data errors. Along the way, Dr. Hand introduces data phenomena like p-hacking, spurious measurement errors, and false discovery rates, and “HARKing.”

Excerpted from DARK DATA: Why What You Don’t Know Matters by David J. Hand. Copyright © 2020 by David J. Hand. Published by Princeton University Press. Reprinted by permission.”