What is statistical significance?
A result is statistically significant when its p-value is at or below a prespecified threshold, such as 0.05, under the test's null hypothesis and model assumptions. It does not give the probability that an effect is real, or establish that the effect is large or valuable.
Last reviewed .
What significance tells you
A p-value measures how incompatible the observed data are with the null hypothesis and the test's assumptions. It is the probability, under that model, of a test statistic at least as extreme as the observed one. It is not the probability that the null hypothesis is true.
A common prespecified threshold, alpha, is 0.05. Under a valid testing procedure with a true null hypothesis, this controls the long-run false-positive rate at 5%. It does not mean a particular significant finding has a 5% probability of being false.
What significance doesn't tell you
A significant result can still be tiny. With a very large sample, small differences can reach significance and have no business value. Look at the size of the effect and the confidence interval too.
A result that isn't significant isn't proof of no effect. It may mean the test was too small to detect one.
Common questions
What is 95% statistical significance?
The phrase usually refers to testing at alpha = 0.05, often paired with a corresponding 95% confidence interval. Under the null hypothesis and a valid procedure, 5% of repeated tests would reject a true null. This is not a 95% probability that this result is real.
How do you reach statistical significance?
Plan a big enough sample before the test and run it for the planned length. You can't force significance, and stopping early when it appears can produce false positives.
Is statistical significance the same as practical significance?
No. Statistical significance compares a p-value with a prespecified threshold under the test's assumptions. Practical significance asks whether the estimated effect is large enough to matter for the business; consider its size and uncertainty.
What if my test never reaches significance?
Report the estimated difference and its confidence interval, including what effects the data can and cannot rule out. Follow the planned stopping rule. A new, larger test may be useful, but repeatedly extending the same test until it becomes significant can inflate false positives.
Want help putting this to work?
Tell us where your marketing stands today, and we'll show you what to do first.