What P-Values Actually Mean (And The Lie You Were Taught)

Open almost any published scientific paper, scroll to the results section, and you will find it: $p < .05$.

It is the holy grail of research. It is the threshold that determines if a drug gets approved, if a paper gets published, or if a researcher gets funding. But if you stop a student in the hallway and ask them to define what a p-value actually is, 90% of them will confidently give you the exact wrong answer.

The Big Lie

Here is the most common—and entirely false—definition of a p-value:

“A p-value of 0.04 means there is a 4% chance that my results were a fluke, and a 96% chance that my hypothesis is correct.”

This sounds incredibly logical. It is also mathematically impossible. A p-value cannot tell you the probability that your hypothesis is right, because the entire formula is built on the assumption that you are wrong.

The Truth: The “Innocent Until Proven Guilty” Rule

To understand the p-value, you have to understand the Null Hypothesis ($H_0$).

The Null Hypothesis is the ultimate skeptic. It is the baseline assumption that your fancy new drug doesn’t work, your new teaching method is useless, and there is absolutely no difference between Group A and Group B.

In statistics, you are never trying to prove that you are right. You are trying to put the Null Hypothesis on trial and prove that it is ridiculously unlikely.

The real definition of a p-value is this:If we assume the Null Hypothesis is 100% true, what is the probability of getting data this extreme?

Let’s step away from data and look at a coin.

You hand me a rupee. My Null Hypothesis is that this is a perfectly fair, normal coin. It has a 50% chance of landing on Heads, and a 50% chance of landing on Tails.

• Flip 1: Heads. (No surprise. Probability is 50%.)
• Flip 2: Heads. (A little lucky, but normal. Probability of two in a row is 25%.)
• Flip 3: Heads. (Probability is 12.5%.)
• Flip 4: Heads. (Probability is 6.25%.)
• Flip 5: Heads.

The probability of flipping 5 Heads in a row with a fair coin is roughly 3% (0.03).

That 0.03 is your p-value!

I am now faced with a choice. Which of these two realities makes more logical sense?

1. Reality A: The coin is perfectly fair, and I just witnessed a very rare, 3% fluke.
2. Reality B: My Null Hypothesis is garbage. The coin is rigged.

Because a 3% chance is so low, most rational people will reject Reality A and accept Reality B. You have just “rejected the null hypothesis” because $p < .05$.

Use this interactive explorer to see how shifting your data changes the p-value under the curve of the Null Hypothesis:

Why 0.05?

Why is $p < .05$ the magic number?

There is no complex mathematical law behind it. In the 1920s, a brilliant statistician named Ronald Fisher essentially said, “Hey, a 1-in-20 chance (5%) feels like a pretty good cut-off for when we should stop blaming luck and start looking for a real effect.”

And the scientific world just ran with it.

The danger is treating 0.05 like a magical cliff. A p-value of 0.049 is not a “massive success,” and a p-value of 0.051 is not an “absolute failure.” They represent virtually the exact same amount of evidence against the null hypothesis.

The Takeaway: A p-value is simply a measure of surprise. If the Null Hypothesis is true, how surprised should you be by your data? If the p-value is tiny, you should be very surprised—so surprised, in fact, that you stop believing the Null Hypothesis.