Significance
Significance
Definition
Significance is often referred to as a p-value below 0.05. However, p<0.05 is merely a threshold value and does not explain the meaning of significance.
Significance is the probability (p) that the result of the sample occurs when the null-hypothesis in the population is true (this is related to Type I and type II errors). In other words, the probability that the respective result occurs in the sample due to chance alone.
Example
It is assessed whether boys are faster compared to girls on the 60m sprint. The difference in mean sprint time is 2s and the result of the statistical testis a p-value of 0.03.
This indicates that there is a probability of 3% that the mean difference between boys and girls in our sample is 2s, while in the population, there is no difference between boys and girls.
In other words, there is a probability of 3% that the difference between boys and girls of 2s is observed due to (bad) chance alone.
Interpretation
Commonly, when the p-value is below 0.05, we reject the null-hypothesis and accept the alternative hypothesis. However, this does not mean that the null-hypothesis is not true. It merely shows that it is very unlikely that the null-hypothesis is true.
Vice versa, when the p-value is above 0.05, this does not indicate that the null-hypothesis is true. It merely indicates, that the respective effect is not big enough to assume that it is due to something else than chance alone. In other words, the probability of the result occurring due to chance alone is deemed to high, that it can't be assumed that it did not occure due to chance alone.
Additionally, a statistically significant effect does not indicate that the effect is relevant.