The following is an attempt to clarify my own understanding of p values and effect size measurements and is in no way a comprehensive survey or formal definition of these concepts.
Introduction
The use of p values are typically used to demonstrate the effectiveness of an intervention, that is to determine the relationship between the variable that is being studied and the inputs that are being applied to the studied variable. With regards to technical terminology, the variable being studied is the dependent variable, e.g. the effect and the independent variables are the inputs, e.g. the causes.The null hypothesis is the pivotal concept in significance testing as it is the null hypothesis that all scientific studies are attempting to prove or disprove. The null hypothesis is the assertion that there is no effect due to the causes being applied, that is, there is no difference in the independent variable due to the actions of the dependent variables on the independent variable.
Rejection of the null hypothesis basically suggests that there is a change in the independent variable due to the actions of the independent variable. The problem for me regarding this concept is its double negative construction. I constantly have to stretch my brain to do the mental gymnastics to first understand the concept and since I use the concept so infrequently, I have to relearn it each time I return to it. Frustrating!
On to p values
To determine if the change in the independent variable was greater than a random chance, the statistical tool called p value which calcualted from the data and then compared to the significance level (or alpha). Alpha is a predetermined level that is chosen prior to the initiation of the experiment that denotes the level of Type I error that will be allowed. Type I error is essentially a false positive error akin to a fire alarm going off when there isn't a fire. Formally, it is the "an incorrect rejection of a true null hypothesis" More negatives! So essentially I can understand this by saying that I have made a type I error if I determine that there is an effect when there actually was no effect.
So to recap, the p value is calculated from the data and compared to the predetermined level (alpha) that will signify whether I should reject or accept the null hypothesis. A p value less than or equal to that predetermined level (alpha) suggests that the null hypothesis is not true. More negatives! In my own words, if p </= alpha then there was a significant effect, that is the cause did have an effect, the dependent variable did influence the independent variable.
P values can be affected by directionality e.g. whether the effect is negative and/or positive. This is determined by using statistical measures called one or two tailed tests. These terms are used because they measure the area in the "tails" of the bell curve (standard distribution or Gaussian distribution are other terms to describe this graphical representation of the way random processes are distributed).
Now for effect sizes
So if p values tell us if the effects we see during an experiment are significant or relevant, then the effect size can tell us the strength of the treatment effect. One example to understand these types of statistics that I have recently discovered is to consider the difference in magnitude between two explosives, a firecracker and a stick of dynamite. The stick of dynamite has a much larger effect than the firecracker in a similar way that a treatment with a large effect size will have a greater effect on a problem compared to a treatment with a small effect size.
There are 3 groups of effect size statistical measures
1) standardized difference measures
Cohen's d (2 group)
Hedge's d (2 group)
Eta squared and partial Eta squared (> 2 groups)
Omega squared (>2 groups)
Intraclass correlation coefficients
2) correlational measures - used to measure the standardized covariance among two variables
3) odds ratios - quantifies how strongly the absence or presence of a variable is associated to the absence or presence of another variable in a given population. These are reported with confidence intervals that signify the relative precision of the reported odds ratio.
Relevance
Currently, there is a push for researchers to go beyond p values. This is because very large studies can reach statistical significance due the effect that the large number of samples has on the resulting p value, since p value is the probability that observed differences are due to chance. As outlined above, a p value that is greater than the most commonly chosen alpha level of .05, then the effect of the independent variable on the dependent variable is assumed to be due to chance.Effect size statistics are independent of the sample size and are therefore more meaningful when attempting to determine the relevance of a studies outcome. In the case of a large number of measurements, even a small effect in the dependent variable can be calculated to be very significant due to the effect of the large sample size. Effect size rationalizes this across sample sizes and provides an accurate reading of the change seen in the dependent variable.
A note about power
So the above is a discussion of significance vs effect which basically assess avoidance of making a type I error i.e. assuming that the outcome is due to an actual effect and not due to chance. These types of errors are also known as alpha errors. Power statistics assess the opposite error, a false negative, no fire alarm when there actually is one. These are known as beta errors. Statistical power depends on the combination of significance level (alpha), effect size and sample size. That is, if one of these factors is not ideal, there is the danger of making the assumption that there is an effect on the dependent variable when there actually wasn't. Power calculations are also useful in that these can be used to determine the size of a future study
Fun with the null hypothesis
| Null hypothesis (H0) is | |||
|---|---|---|---|
| Valid/True | Invalid/False | ||
| Judgement of Null Hypothesis (H0) | Reject | Type I error False Positive | Correct inference True Positive |
| Fail to reject (accept) | Correct inference True Negative | Type II error False negative | |
No comments:
Post a Comment