Rules of thumb on magnitudes of effect sizes

The scales of magnitude are taken from Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates (see also here). The scales of magnitude for partial $$\omega^text{2}$$ are taken from Table 2.2 of Murphy and Myors (2004).

There is also a table of effect size magnitudes at the back of Kotrlik JW and Williams HA (2003) here. An overview of commonly used effect sizes in psychology is given by Vacha-Haase and Thompson (2004).

Kraemer and Thiemann (1987, p.54 and 55) use the same effect size values (which they call delta) for both intra-class correlations and Pearson correlations. This implies the below rules of thumb from Cohen (1988) for magnitudes of effect sizes for Pearson correlations could also be used for intra-class correlations. It should be noted, however, that the intra-class correlation is computed from a repeated measures ANOVA whose usual effect size (given below) is partial eta-squared. In addition, Shrout and Fleiss (1979) discuss different types of intra-class correlation coefficient and how their magnitudes can differ.

The general rules of thumb given by Cohen are for eta-squared, which uses the total sum of squares in the denominator, but these would arguably apply more to partial eta-squared than to eta-squared. This is because partial eta-squared in factorial ANOVA arguably more closely approximates what eta-squared would have been for the factor had it been a one-way ANOVA and it is presumably a one-way ANOVA which gave rise to Cohen's rules of thumb.

Also:Haddock et al (1998) state that $$\sqrt{3/\pi}$$ multiplied by the log of the odds ratio is a standardised difference equivalent to Cohen's d.

• Using partial eta-squared in an ANCOVA in SPSS

• Effect size for multilevel models

• Further details on the derivation of the Odds Ratio effect sizes

• Cohen's d adjusted for base rates

• A quick guide to choice of sample sizes for Cohen's effect sizes

• A nonparametric analogue of Cohen's d and applicability to three or more groups

Definitions

For two-sample t-tests Cohen's d = (difference between a pair of group means) / (averaged group standard deviation) = t / Sqrt [(1/n1) + (1/n2)] (Pustejovsky (2014), p.95 and Borenstein (2009), Table 12.1)

For a one-sample t-test Cohen's d = difference between the mean and its expected value / standard deviation = t / Sqrt(N). Cohen's d also equals t / Sqrt(N) in a paired t-test since t / Sqrt(N) = difference between two means / standard deviation of the difference

Other effect sizes using t-ratios

$$\eta² $$ = SS(effect) / [ Sum of SS(effects having the same error term as effect of interest) + SS(the error associated with these effects) ]

Cohen's f = Square Root of eta-squared / (1-eta-squared)

There is also a $$\mbox{Partial } \eta² $$ = SS(effect) / [ SS(effect) + SS(error for that effect) ]

Multivariate $$\eta^text{2}$$ = 1 - $$\Lambda^1/s $$ where $$\Lambda$$ is Wilk's lambda and s is equal to the number of levels of the factor minus 1 or the number of dependent variables, whichever is the smaller (See Green et al (1997)). It may be interpreted as a partial eta-squared.

$$\kappa²$$ = ab / (Maximum value of ab) where a and b are the regression coefficients representing the independent variable to mediator effect and the mediator to outcome respectively to estimate the indirect effect of IV on outcome. See Preacher and Kelley (2011) for further details including MBESS procedure software for fitting this in R. For further details on mediation analysis see also here. Field (2013) also refers to this measure.

Suggestion : Use the square of a Pearson correlation for effect sizes for partial $$\eta^text{2}$$ giving 0.01 (small), 0.09 (medium) and 0.25 (large) giving intuitively larger values than eta-squared.

Cohen's $$\omega²$$ = Sum over all the groups $$ (\mbox{(observed proportion - expected proportion)}²) $$ / (expected proportion)

* Odds Ratio definition

References

Borenstein, M. (2009) Effect sizes for continuous data. In H. Cooper, L. V.Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 221–235). Sage Foundation:New York, NY.

Cohen, J. (1988) Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.

Field, A. (2013) Discovering statistics using IBM SPSS Statistics. Fourth Edition. Sage:London.

Green, SB, Salkind, NJ & Akey, TM (1997). Using SPSS for Windows:Analyzing and understanding data. Upper Saddle River, NJ:

Haddock, CK, Rinkdskopf, D. & Shadish, C. (1998) Using odds ratios as effect sizes for meta-analysis of dichotomous data: A primer on methods and issues. Psychological Methods 3 339-353.

Kotrlik, JW and Williams, HA (2003) The incorporation of effect size in information technology, learning, and performance research. Information Techology, Learning, and Performance Journal 21(1) 1-7.

Kraemer HC and Thiemann S (1987) How many subjects? Statistical power analysis in research. Sage:London. In CBSU library.

Murphy KR and Myors B (2004) Statistical power analysis: A Simple and General Model for Traditional and Modern Hypothesis Tests (2nd ed.). Lawrence Erlbaum, Mahwah NJ. (Alternative rule s of thumb for effect sizes to those from Cohen are given here in Table 2.2).

Preacher, KJ and Kelley, K (2011) Effect size measures for mediation models: quantitative strategies for communicating indirect effects. Psychological Methods 16(2) 93-115.

Pustejovsky JE (2014) Converting From d to r to z When the Design Uses Extreme Groups, Dichotomization, or Experimental Control. Psychological Methods 19(1) 92-112. This reference also gives several useful formulae for variances of effect sizes such as d and also on how to convert d to a Pearson r.

Shrout, PE and Fleiss, JL (1979) Intraclass Correlations: Uses in Assessing Rater Reliability, Psychological Bulletin, 86 (2) 420-428. (A good primer showing how anova output can be used to compute ICCs).

Tabachnick, BG and Fidell, LS (2007) Using multivariate statistics. Fifth Edition. Pearson Education:London.

Vacha-Haase, T and Thompson, B (2004) How to estimate and interpret various effect ssizes. Journal of Counseling Psychology 51(4) 473-481.