Generalizability and Sample Size in EFA and PCA
In addition to its role in determining statistical power, sample size also influences the generalizability of results by the ratio of observations to predictor variables.
Two different approaches have been taken: suggesting a minimum total sample size, or examining the ratio of subjects to variables, as in multiple regression. There is no concensus on which is the more effective criterion but in practice both a large sample size (for generalizability) and a high subject to variable ratio (to avoid overfitting) should be considered important in determining sample sizes in multivariate analyses.
Comfrey and Lee (1992) suggest that “the adequacy of sample size might be evaluated very roughly on the following scale: 50 – very poor; 100 – poor; 200 – fair; 300 – good; 500 – very good; 1000 or more – excellent” (p. 217). Guadagnoli and Velicer (1988) review several studies that conclude that absolute minimum sample sizes, rather than subject to item ratios, are more relevant. These studies range in their recommendations from an N of 50 (Barrett & Kline, 1981) to 400 (Aleamoni, 1976).
Hair et al (1998) suggest A general rule is that there should be at least five observations for each independent variable. This ratio is also advocated by Bryant & Yarnold (1995) and Costello & Osborne (2005). Any ratio below this runs the risk of overfitting making the results too sample specific. Gorsuch (1983, p.332) and Hatcher (1994, p. 73) also recommend a minimum subject to item ratio of at least 5:1 in EFA, but they also have stringent guidelines for when this ratio is acceptable, and they both note that higher ratios are generally better. In particular Gorsuch (1974, p.296) suggests the 5:1 ratio applies only when the expected variable communalities (the amount each variable has in common with all other variables) are high and there are many variables loading highly on each factor. Tabachnick and Fidell suggest smaller sample sizes are allowable if there are strong inter-item correlations (and communalities - the proportion of variance an item has in common with all the other items) and a few distinct factors. This is also supported by de Winter et al (2009). Here is a pdf copy of their paper.
There is a widely-cited rule of thumb from Nunnally (1978, p. 421) that the subject to item ratio for exploratory factor analysis should be at least 10:1, but that recommendation was not supported by published research. Moons et al (2009) also, however, suggest at least 10 times as many subjects (outcomes) as predictors are used, in general, in multivariate analyses.
The following is taken from Osborne, Jason W. & Anna B. Costello (2004) who among others find absolute sample sizes simplistic given the variance in the types of scales researchers examine. Each scale differs in the number of factors or components, the number of items on each factor, the magnitude of the item-factor correlations, and the correlation between factors, for example. This discomfort has led some authors to focus on the ratio of subjects to items, or more recently, the ratio of subjects to parameters (as each item will have a loading for each factor or component extracted), as authors do with regression, rather than absolute sample size when discussing guidelines concerning EFA and PCA.
Although PCA and EFA are two distinct procedures, the mathematics and processes behind each are related and similar, in practice the outcome of a PCA and EFA is often identical, and these results relating to PCA should generalize to EFA handily and vice-versa.
There is no one ratio that will work in all cases; the number of items per factor and communalities and item loading magnitudes can make any particular ratio overkill or hopelessly insufficient (MacCallum, Widaman, Preacher, & Hong, 2001).
Thus researchers seeking guidance concerning sufficient sample size in EFA or PCA are left between two entrenched camps-- those arguing for looking at total sample size (e.g. Arrindell & van der Ende, 1985) and those looking at ratios. This is unfortunate, because both probably matter in some sense, and ignoring either one can have the same result: errors of inference. Failure to have a representative sample of sufficient size results in unstable loadings (Cliff, 1970), random, non-replicable factors (Aleamoni, 1976; Humphreys, Ilgen, McGrath, & Montanelli, 1969), and lack of generalizability to the population (MacCallum, Widaman, Zhang, & Hong, 1999).
The ratio of 5 to 1 applies to multiple regression, factor analysis and discriminant analyses including logistic regression. Ideally there should be 15 to 20 observations per independent variable. If a stepwise procedure is used it is recommended that there are 50 times more observations than variables (Wilkinson, L. (1975)).
The 5 to 1 ratio is also recommended, additionally for discriminant analyses, that the smallest group size should exceed the number of predictor variables. Ideally each group should have at least 20 observations in it.
If the discriminant group sizes vary markedly then there can be a tendency to classify a disproportionately large number of observations into the large sized groups.
In factor analysis sample size can also influence the thresholds for determining what is a high factor loading.
In cluster analysis Mooi and Sarstedt (2011) (page 263 in Table 9.11) suggest sample sizes of at least 2^m (for m clustering variables) but smaller sample sizes can be justified provided a few distinct homogeneous clusters with well separated cluster means are produced.
References
Aleamoni, L. M. (1976). The relation of sample size to the number of variables in using factor analysis techniques. Educational and Psychological Measurement, 36, 879-883.
Arrindell, W.A. & van der Ende, J. (1985). An empirical test of the utility of the observer-to-variables ratio in factor and components analysis. Applied Psychological Measurement 9 165-178. This paper does test cases-to-variables ratios and concluded for factor analysis that it made little difference to the stability of the solutions.
Baggaley, A. R. (1983). Deciding on the ratio of number of subjects to number of variables in factor analysis. Multivariate Experimental Clinical Research, 6(2), 81-85.
Barrett, P. T., & Kline, P. (1981). The observation to variable ratio in factor analysis. Personality study and group behavior, 1, 23-33.
Bryant, F. B., & Yarnold, P. R. (1995). Principal-components analysis and exploratory and confirmatory factor analysis. In L.G. Grimm, P.R. Yarnold (Eds.), Reading and understanding multivariate statistics, American Psychological Association, Washington, DC pages 99-136.
Cliff, N. (1970). The relation between sample and population characteristic vectors. Psychometrika, 35, 163-178.
Cohen, J., & Cohen, P. (1983). Applied multiple regression/correlation analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
Comfrey, A. L., & Lee, H. B. (1992). A First Course in Factor Analysis. Hillsdale, NJ: Lawrence Erlbaum Associates.
Costello, A. B., & Osborne, J. W. (2003). Exploring best practices in Factor Analysis: Four mistakes applied researchers make. Paper presented at the Paper presented at the annual meeting of the American Educational Research Association, Chicago, Ill, April.
Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor analysis:four recommendations for getting the most from your analysis. Practical Assessment, Research and Evaluation 10 1-9.
de Winter, JCF, Dodou, D and PA Wieringa (2009) Exploratory factor analysis with small sample sizes. Multivariate Behavioural Research, 44, 147-181.
Ford, J. K., MacCallum, R. C., & Tait, M. (1986). The application of exploratory factor analysis in applied psychology: A critical review and analysis. Personnel Psychology, 39, 291-314.
Gorusch, R. L. (1974). Factor Analysis (1st ed.). Philadelpha, W.B. Saunders.
Gorusch, R. L. (1983). Factor Analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
Guadagnoli, E., & Velicer, W. F. (1988). relation of sample size to the stability of component patterns. Psychological Bulletin, 103, 265-275.
Hair Jr, JF, Anderson, RE, Tatham, RL and Black WC (1998) Multivariate data analysis fourth edition. Prentice-Hall:London.
Hatcher, L. (1994). A Step-by-Step Approach to Using the SAS® System for Factor Analysis and Structural Equation Modeling. Cary, N.C.: SAS Institutte, Inc.
Humphreys, L. G., Ilgen, D., McGrath, D., & Montanelli, R. (1969). Capitalization on chance in rotation of factors. Educational and Psychological Measurement, 29(2), 259-271.
MacCallum, R. C., Widaman, K. F., Preacher, K. J., & Hong, S. (2001). Sample size in factor analysis: The role of model error. Multivariate Behavioral Research, 36, 611-637.
MacCallum, R. C., Widaman, K. F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4, 84-99.
Mooi, E. and Sarstedt, M. (2011). A Concise Guide to Market Research. Springer-Verlag Berlin Heidelberg. DOI 10.1007/978-3-642-12541-6_9.
Moons, KGM, Royston, P, Vergouwe, Y, Grobbee, DE and Altman, DG (2009) Prognosis and prognostic research: what, why, and how? BMJ 2009;338:b375
Nunnally, J. C. (1978). Psychometric Theory (2nd ed.). New York: McGraw Hill.
Osborne, Jason W. & Anna B. Costello (2004). Sample size and subject to item ratio in principal components analysis. Practical Assessment, Research & Evaluation, 9(11).
Pedhazur, E. J. (1997). Multiple Regression in Behavioral Research: Explanation and Prediction. Fort Worth, TX: Harcourt Brace College Publishers.
Tabachnick, B. G., & Fidell, L. S. (2001). Using Multivariate Statistics (4th ed.). New York:: Harper Collins.
Wilkinson L. (1975) Tests of significance in stepwise regression Psychological Bulletin 86 168-74.