Hypothesis Testing

What Is Hypothesis Testing?

Hypothesis testing is a structured way to check a claim about a population using sample data. It starts with two competing statements. The null hypothesis is the default position, usually that there is no effect or no difference. The alternative hypothesis is the claim the researcher wants to test, such as the idea that a stock's average return differs from zero. The test then asks how surprising the observed data would be if the null hypothesis were true.

How It Works

The process generally follows a few steps. First, the analyst states both hypotheses and sets a significance level, the maximum risk of a false alarm that is considered acceptable. Second, the analyst calculates a test statistic from the sample, which measures how far the observed result sits from what the null hypothesis predicts, taking standard deviation and sample size into account. Third, that statistic is converted into a p-value, the probability of seeing a result at least as extreme as the one observed if the null hypothesis were true. If the p-value falls below the chosen significance level, the null hypothesis is rejected.

Errors and Limitations

Two kinds of mistakes are possible. A Type I error occurs when a true null hypothesis is rejected, a false positive. A Type II error occurs when a false null hypothesis is not rejected, a false negative. Rejecting the null does not prove the alternative is true, nor does it show the effect is large enough to matter. Failing to reject does not prove the null is true; it only means the data were not strong enough. Testing many ideas on the same data also raises the odds that some will look significant purely by chance, a common issue in backtesting trading rules.

Why It Matters to Investors

Hypothesis testing underlies much of the academic research behind market findings and is used to judge whether a strategy's past performance is likely to reflect more than luck. When an investment claim is presented, it helps to ask what the null hypothesis was, what significance threshold was used, how many variations were examined, and whether a confidence interval around the estimated effect is reported alongside the p-value. These questions help readers judge the strength of evidence without implying that any particular investment is good or bad.