Exact test vs chi-square?

Howell (2002) on p.158 explains a difficulty using the Pearson chi-square test for testing the independence of frequencies in two-way tables. Namely that when the expected cell frequencies are less than five the Pearson statistics does not follow a chi-square distribution.

Instead it is recommended by Howell that Fisher's exact test is used when any expected frequency is less than five although Pett (1997) suggests problems using alternatives to chi-square tests if fewer than a fifth of expected values are less than five. Ian Campbell (2007) mentions here that the exact test is too conservative for 2x2 tables and suggests, instead, using an alternative chi-square, The N-1 chi-square, which performs well provided all expected counts are 1 or greater. Bruce Weaver shows here that the N-1 chi-square is equivalent to the linear-by-linear chi-square test outputted by SPSS CROSSTABS and it may also be computed using the on-line calculator on Ian Campbell's website.

For a 2x2 table only Bruce shows here that the N-1 chi-square can easily be obtained from the Pearson chi-square since

'N -1' chi-square = Linear-by-Linear chi-square = Pearson chi-square x (N -1) / N

CROSSTABS in SPSS, consequently, outputs how many expected cell frequencies are less than five whenever a chi-square test is performed. Fisher's exact test may be requested by pressing on the exact button in the crosstabs dialogue box and requesting 'exact'. Alternatively the locally available software program fishrc computes this test on a UNIX machine.

The p-value can then be quoted. This p-value is always two-sided except in the case of a 2 by 2 table where it can also be one-sided.

For EXCEL users the BINOMDIST function can be used for exact probabilities for the special case of a single proportion (equivalently a 2 by 1 table) using its cumulative distribution function.

For example BINOMDIST(11,23,0.5,TRUE) = the probability of observing at most 11 'correct' out of a possible 23 assuming the chance of observing a correct is equal to a half = 0.5 (since 11 is the midpoint of the distribution and the binomial distribution is symmetric).

References

Campbell I (2007) Chi-squared and Fisher-Irwin tests of two-by-two tables with small sample recommendations. Statistics in Medicine, 26, 3661 - 3675. A pre-print of this paper is available in pdf format from here.

Howell DC (2002) Statistical methods for psychology fifth edition: Duxbury Press:Pacific Grove, CA.

Pett MA (1997) Nonparametric statistics for Health Care Research. Sage publications:London.