P-Value

What Is a P-Value?

A p-value is a number produced by a hypothesis test that measures how surprising a dataset would be if a starting assumption were correct. That starting assumption is called the null hypothesis, and it usually says there is no effect, no difference, or no relationship. The p-value gives the probability of obtaining results at least as extreme as the ones observed, assuming the null hypothesis holds. A small p-value suggests the observed data would be unlikely under that assumption, while a large one suggests the data are consistent with it.

How It Works

Researchers first choose a test statistic, such as the difference in average returns between two groups, and then work out how that statistic would be distributed if the null hypothesis were true. The p-value is the share of that hypothetical distribution that falls at or beyond the observed value. Results are often compared with a cutoff chosen in advance, commonly 0.05, and a result below that cutoff is described as statistical significance at that level. The cutoff is a convention, not a law of nature.

An Example

Suppose an analyst tests whether a trading strategy beat the market over ten years. If the strategy's excess return is so large that a result this strong would appear only 2% of the time under a no-skill assumption, the p-value is 0.02. This does not mean there is a 98% chance the strategy works, and it does not mean there is a 2% chance it is luck. It only describes how the data relate to the null assumption.

Limitations

P-values are easy to misread. They do not measure the size or practical importance of an effect, and a tiny effect can produce a small p-value in a very large sample. Testing many strategies and reporting only those that pass raises the risk of a Type I error, meaning false positives appear more often than the chosen threshold suggests. For this reason, many researchers pair p-values with a confidence interval and with testing on data not used to build the strategy.