Running head: NORMALITY AND OUTLIERS

Checking for Non-normality and Outliers in ANOVA and MANOVA Rhonda D. Williams University of Calgary

NORMALITY AND OUTLIERS Checking for Non-normality and Outliers in ANOVA and MANOVA In statistics, statistical assumptions underlie most univariate and multivariate statistical analysis (Meyers, Gamst & Guarino, 2006). Some of the main assumptions include normality, linearity and homoscedasticity. It is vital to ensure these basic statistical assumptions are met before using the data set in an analysis. If one or more of these basic assumptions is violated,

then the statistical results may become biased or distorted, resulting in false statements about the phenomenon being studied (Meyers, Gamst & Guarino, 2006). For the purpose of this paper, the focus will be on the assumption of normality and detection of outliers. Most phenomenons investigated by researchers are based on the assumption of normal distribution in the sample population. Meeting the normality assumption allows researchers to draw certain conclusions and implications from the data. In addition, many observed variables actually are normally distributed, which is another reason why the normal distribution represents a "general feature" of empirical reality (Statsoft Inc., 2012). When beginning an analysis of variance (ANOVA) or multivariate analysis of variance (MANOVA), researchers should check for non-normality and outliers as it provides valuable information regarding their data set prior to continuing on with the analysis. This paper focuses on defining normal distribution and outliers as well as the graphical and statistical ways the assumption of normality and outliers can be identified for both ANOVA and MANOVA. Normal Distribution Many variables naturally fall within a normal distribution and can be represented empirically by this fundamental law of natural science (e.g. biological variables such as blood pressure, height and weight; Gordon, 2007; Statsoft Inc., 2012). One of the common assumptions

NORMALITY AND OUTLIERS

of many statistical procedures assumes the sample population used for the analysis is normally distributed (assumption of normality). This is particularly true when the size of the samples is large. It is also assumed that when dependent variables are measured, they should be normally distributed within the groups (French, Macedo, Poulsen, Waterson & Yu, n.d.; Statsoft Inc., 2012). The symmetrical bell-shaped curve of the normal distribution has only two parameters: mean and standard deviation. The normal distribution has 68% of all observations fall within a range of 1 standard deviation from the mean, and a range of 2 standard deviations includes 95% of the scores (Statsoft Inc., 2012). The shape of distribution of continuous variables in both univariate and multivariate analysis should correspond to the bell-shaped curve of the normal distribution (Meyers, Gamst & Guarino, 2006). When the distributional properties of the data set depart widely from the assumption of normality, there may be an elevated risk of committing erroneous statistical conclusions such as Type I and Type II errors (Theodore & Gatchel, 2008). Therefore, it is very important for researchers to make sure every effort is taken to detect nonnormality within their study.

Figure 1. The bell curve of the normal distribution. = mean; = standard deviation. Adapted from www.allpsych.com. by Heffner, C.L., 2012. Copyright 1999-2003 by AllPsych and Heffner Media Group Inc.

NORMALITY AND OUTLIERS Checking for Normality in ANOVA and MANOVA There are both graphical and statistical methods for evaluating the assumption of

normality within univariate and multivariate analysis. In an ANOVA, graphic approaches such as histograms, stem-and-leaf plots and normal probability plots can be used to visually assess if the assumption of normality has been met for the data set being analyzed. Statistically the assumption can also be evaluated by looking at the skewness and kurtosis within the distribution of the data set as well as implementing the Kolmogorov-Smirnov test and the Sharpiro-Wilk test in SPSS. In a MANOVA, although there is no specific statistical analysis to run in SPSS, there are a few suggestions put forth by researchers to test for the assumption of normality in a multivariate analysis. The following sections will provide more detail in each of these graphical and statistical approaches in assessing the assumption of normality in ANOVA and MANOVA. ANOVA Graphical Approaches for Normality Histograms graphically provide an approximation of the visual distribution of the data shape in an ANOVA and helps determine if the assumption of normality has been met. In SPSS, the program can superimpose a drawing of the normal curve onto the distribution chart to determine how close the data scores come to being normally distributed (Gordon, 2007; Meyers, Gamst & Guarino, 2006).

Figure 2. A histogram representing the distribution of a data set with the normal curve superimposed. Adapted from http://psychstat.com. by Gordon, O., 2007. Copyright 2012 by Tangient LLC.

NORMALITY AND OUTLIERS

Stem-and-leaf plots also graphically represent the assumption of normality and are used to display the frequency of a quantitative value by using the actual values within the plot, demonstrating the distribution within an ANOVA. The plots are comprised of the frequency with which the value occurs, the base value (stem) and the leaf portion which expresses the full value. For example, in the stem-and-leaf plot below (Figure 3), the lowest score has a frequency of 1.00 with a stem of 1 and a leaf of 5, representing the one lowest score of 15. In the next row down, the plot represents three scores within the twenties of 20, 20 and 21. By looking at the leaves a researcher can get a quick snapshot of the general shape of the distribution and check for skewness and kurtosis (discussed next in the statistical approaches section; Meyers, Gamst & Guarino, 2006).

Figure 3. A hypothetical Stem-and-Leaf Plot. Adapted from Data Screening by Meyers, L.S., Gamst, G., Guarino, A.J., 2006, Applied Multivariate Research: Design and Interpretation. Copyright 2000-2012 by Sage Publications.

NORMALITY AND OUTLIERS A more precise graphical method for representing the assumption of normality for an ANOVA is a normal probability plot. In a normal probability plot, the variable values are rank ordered and plotted against the expected values of the normal distribution, creating a straight

diagonal line for the normal distribution. Values are then plotted in comparison with the diagonal to determine if the assumption of normality has been met (Meyers, Gamst & Guarino, 2006; Stevens, 2009). Both the quantile-quantile (Q-Q) plot and probability-probability (P-P) plot are an excellent way to see whether the data deviate from the normal distribution. In SPSS a scatter plot for either/both the Q-Q plot and P-P plot can be created shows the relationship between the actual 'observed' values and what those values would be 'expected' to be if the data were normally distributed. Although either plot can be utilized, the normal quantile plot is more sensitive to deviances from normality in the tails of the distribution, whereas the normal probability plot is more sensitive to deviances near the mean of the distribution. The following examples demonstrate Q-Q plots with normal, negatively skewed and positively skewed presentations (Gordon, 2007).

Figure 4.The three plots depict a) a normal Quantile-Quantlie (Q-Q) plot with a normal distribution, b) Q-Q plot with a negative skew to the distribution of the data points, and c) Q-Q plot with a positive skew to the distribution of the data. Adapted from http://psychstat.com. by Gordon, O., 2007. Copyright 2012 by Tangient LLC.

NORMALITY AND OUTLIERS

Despite the quick assessment graphical representations provide regarding the assumption of normality for researchers, analysis based on a small or moderate sample size can make it difficult to determine whether non-normality is real or apparent due to the potential for sampling error (Stevens, 2009). Therefore, statistical approaches should also be considered as another measure to check for non-normality. ANOVA Statistical Approaches for Normality Statistical approaches for investigating the assumption of normality in an ANOVA often start with measuring the skewness and kurtosis of the distribution in comparison to the normal distribution. A researcher can use the SPSS Frequencies, Descriptives or Explore functions to calculate skewness and kurtosis for the distribution of a data set for each variable (Gordon, 2007; Meyers, Gamst & Guarino, 2006). Skewness is a measure of the symmetry of a distribution and can be identified as normal, positive or negative. Normal skewness occurs when the sample is drawn from a normal population with a symmetrical distribution. Positive skewness occurs when the scores are more concentrated at the lower end of the distribution, while negative skewness occurs when the scores are more grouped at the higher end of the distribution (Gordon, 2007; Meyers, Gamst & Guarino, 2006). For example, in clinical data, skewness may be a product of floor or ceiling effects associated with self-report measures or out-of-range values because of data entry errors (Theodore & Gatchel, 2008). A value of skewness (estimate of skewness in the population) and the standard error can be calculated in SPSS. The mean of the population of skewness scores will be zero if the data set is symmetrical and normally distributed. If the mean value is greater than 2.00, the assumption of normality is not met and caution should be used due to potential bias and distortion (Gordon, 2007; Meyers, Gamst & Guarino, 2006).

NORMALITY AND OUTLIERS

Figure 5. The three plots depict a) a normal symmetrical distribution, b) a distribution with a positive skew and c) a distribution with a negative skew. Adapted from http://psychstat.com. by Gordon, O., 2007. Copyright 2012 by Tangient LLC.

In addition to skewness, kurtosis also provides information about the distribution of a data set that tests for the assumption of normality in an ANOVA. Kurtosis measures the general peakedness of the clustering of the scores toward the center of the distribution (Meyers, Gamst & Guarino, 2006). Normal kurtosis, or mesokurtosis, follows the bell curve in the normal distribution. Leptokurtosis (positive kurtosis) indicates an extreme peak appearance in the center of the distribution and occurs when values are clustered more around the middle values of the distribution. Platykurtosis (negative kurtosis) occurs when the values have a relatively flat distribution in appearance (Meyers, Gamst & Guarino, 2006). SPSS can calculate the value for kurtosis and its standard error. The value of kurtosis is zero when the data set is normally distributed. If the mean kurtosis value is greater than 2.00, the assumption of normality is not met and the data may become biased and distorted just as it is with skewness (Gordon, 2007; Meyers, Gamst & Guarino, 2006).

NORMALITY AND OUTLIERS

Figure 6. Three distributions demonstrating the types of kurtosis (peakedness of the clustering of scores). a) mesokurtic curve represents the normal bell-curved distribution b) leptokurtic curve represents a positive kurtosis where the values cluster around the middle values of the distribution c) platykurtic curve represents a negative kurtosis with a relatively flat distribution. Adapted from www.allpsych.com. by Heffner, C.L., 2012. Copyright 1999-2003 by AllPsych and Heffner Media Group Inc.

Another statistical approach that tests the assumption of normality for ANOVA is the Kolmogorov-Smirnov test. This test is based on the largest vertical distance between the normal cumulative distribution function (CDF) and the sample cumulative frequency distribution. Although it has poorer power to detect non-normality than the Shapiro-Wilk test, the Kolmogorov-Smirnov test could be used when the data sample size is greater than 50 participants. The Kolmogorov-Smirnov test can be obtained through the Explore function of SPSS menu as Normality plots with tests available when you click on the Plots button (Gordon, 2007; Meyers, Gamst & Guarino, 2006, Stevens, 2009). Finally, the Shapiro-Wilk test is a regression-type test that uses the correlations of sample order statistics (the sample values arranged in ascending order) with those of a normal distribution. Generally this test is recommended over others because it appears to be the most powerful in detecting even small non-normality within distributions. (Meyers, Gamst & Guarino, 2006) However, the Shapiro-Wilk test should only be used for sample sizes of less than 50

NORMALITY AND OUTLIERS

10

participants as it is not suitable for very large sample sizes (Gordon, 2007). Again, the ShapiroWilk test can be obtained through the Explore function of SPSS (the same as stated above). While normality tests are useful, they are not infallible. There are some limitations to using both the Kolomogorov-Smirnov test and the Shapiro-Wilk test. The null hypothesis for both tests is that the population is normally distributed, and the alternative hypothesis is that it is not normally distributed. Therefore, researchers can prove the population is not normally distributed but can not prove it is normally distributed. In addition, if the population is found to not be normally distributed, the analysis does not tell the researcher the reason for non-normality. Finally, both tests are influenced by the power of the analysis. If the sample size is small, the test may not have enough power to detect non-normality within the population. If the sample size is very large, it may become too sensitive and detect even trivial deviations from normality. To buffer some of these limitations, researchers should always look at the graphical approaches in addition to the statistical findings (Gordon, 2007). MANOVA Assumption of Normality Since there is not a direct test for multivariate normality, researchers generally test each variable individually for normality. After checking for normality for each variable, more analysis is required to determine the assumption of normality for multivariate analysis. Linear combinations and pairwise combinations of the variables should also be normally distributed within a multivariate analysis. Although SPSS does not offer a statistical test for multivariate normality, some researchers suggest a thorough univariate normality analysis along with a bivariate scatterplot of key pairs of variables be used to check for normality (Meyers, Gamst & Guarino, 2006, Stevens, 2009).

NORMALITY AND OUTLIERS Outliers

11

An outlier is a value or score that lies an abnormal distance for other values in a random sample of the population. These extreme cases or unusual values can appear in both a single variable (univariate) or in a combination of variables (multivariate). Before abnormal observations can be singled out, it is necessary to characterize normal observations (Meyers, Gamst & Guarino, 2006; NIST/SEMATECH, 2012). Outliers can signify a variety of things when researching a phenomenon. Outliers could represent the presence of extraordinary events or unusual circumstances within a data set, but could also identify anomalies within the data that may need to be attended to before further statistical analysis is completed. For example, outliers may be caused by data entry errors or improper coding. In multivariate analysis, outliers could show the uniqueness within a combination of values for several variables (Meyers, Gamst & Guarino, 2006). Tests for outliers should be run before performing an ANOVA or MANOVA. The presence of outliers may also increase the chances of committing a Type I or Type II error and give no indication as to which type of error is occurring in the analysis (French et al., n.d.). Checking for Outliers in ANOVA and MANOVA When checking for outliers in both ANOVA and MANOVA, researchers examine the overall shape of the graphed data for important features that are far removed from the mass of the data. Outliers should be investigated carefully and are of particular interest when they exercise a disproportionate influence on the outcome of the statistical analysis of the data set (Zijlstra, Andries van der Ark & Sijtsma, 2007). When an outlier is detected, the researcher needs to cross-reference the value to make sure the corresponding value is not out of the range possible for that particular variable (Meyers, Gamst & Guarino, 2006; Theodore & Gatchel,

NORMALITY AND OUTLIERS

12

2008). Such mistakes can stem from data entry or transcribing errors from one record source to another. If the value is cross-referenced with the original hard copy source and the value is correct and remains out of range, that particular case should be deleted in order to exclude it from any further analysis (Theodore & Gatchel, 2008). For outliers that are within the accurate range of values but widely differ from the bulk of the data, several strategies could be used to deal with the outlier. For a small number of outliers, the data points could simply be deleted from the data set and excluded from further analysis. However, this approach runs the risk of biasing the results in the opposite direction. In addition, the outliers may also provide interesting and valuable data that could demonstrate new information about the phenomenon under examination. Each outlier should be evaluated individually to determine the potential reason for the extreme score and the impact of retaining or deleting the data value from the analysis (Gordon, 2007; Theodore & Gatchel, 2008). Before considering the possible elimination of these points from the data, one should try to understand why they appeared and whether it is likely similar values will continue to appear (NIST/SEMATECH, 2012). A researcher may also choose to investigate the influence of the outliers by analysing the data with and without them to get a full understanding of the impact on the data set. Outliers exercising a disproportionate influence on the statistical results may be deleted from the data set but could be representative of a group that is misrepresented in the sample, and a new sample may need to be collected to adhere to the assumption of normality. Outliers may also be studied as separate interesting cases to gain further understanding about the area of focus for the study (Zijlstra, Andries van der Ark & Sijtsma, 2007). There are several approaches such as frequency distributions, z scores, histograms, normal probability plots and box plots that can be used to detect outliers within an ANOVA. In

NORMALITY AND OUTLIERS

13

addition to these approaches, bivariate scatterplots and the Mahalanobis distance can be used to identify outliers within a MANOVA. Each of these detection methods will be further explored in the next sections (Gordon, 2007; Meyers, Gamst & Guarino, 2006; Stevens, 2009). Graphical Approaches to Outliers in ANOVA When processing a data set for the detection of univariate outliers, a frequency distribution can be created for each variable. This allows a researcher to quickly scan the data for any extreme values and determine the possible causes for the outlier. Outliers can also be easily detected through the use of graphical techniques such as histograms, normal probability plots and box plots because univariate outliers distinguish themselves by the physical separation of the values from the bulk of the cases on a particular variable (Meyers, Gamst & Guarino, 2006). As already discussed previously regarding the assumption of normality, both histograms and normal probability plots can be used to detect outliers within the data. Histograms provide a visual approximation of the distribution of the data set and when the normal curve is superimposed, any extreme data points can be seen. Normal probability plots represent the normal distribution as a horizontal line and each value is represented on the plot so outliers can be easily visually detected (Gordon, 2007; Meyers, Gamst & Guarino, 2006). Box plots are a useful graphical display for describing the behavior of the data in the middle as well as at the ends of the distributions. Also known as box and whiskers plots, box plots are used to identify extreme scores and provide a considerable amount of information about the data in a succinct way. Extreme scores (outliers) can have an adverse effect on statistical analysis by not showing a true representation of the majority of the scores and distorting results. Box plots are based on the median rather than the mean of the data set so it is not impacted by

NORMALITY AND OUTLIERS extreme scores in the distribution (Meyers, Gamst & Guarino, 2006). As displayed in Figure 7

14

below, the borders of the box are set at the 25th percentile (first quartile) for the lower border and the 75th percentile (third quartile) for the upper border of the box with a solid line drawn across the box to locate the median (NIST/SEMATECH, 2012). This range captures the majority of the scores and the distance between the first and third quartile is called the interquartile range (IQR). Above and below the IQR are two boundary lines called inner fences which indicate the acceptable limits of scores (within 1.5 IQRs of the box). Any values found outside this range are then considered to be outliers (Gordon, 2007; Meyers, Gamst & Guarino, 2006). Outliers are indicated by SPSS with a O while extreme scores are represented with a *. The whiskers of the plot that extend vertically out of the box from above and below only extend as far as the smallest and largest values that fall within the lower and upper inner fences. The distribution of the outliers can also provide some information about the potential skewness of the data (positive or negative) if it is present (Meyers, Gamst & Guarino, 2006).

Figure 7. The general form of a box and whiskers plot based on Cohens (1996) description. Adapted from Data Screening by Meyers, L.S., Gamst, G., Guarino, A.J., 2006, Applied Multivariate Research: Design and Interpretation. Copyright 20002012 by Sage Publications.

NORMALITY AND OUTLIERS

15

Figure 8. These three box plots provide a graphical representation of three different data sets. a) box plot with a symmetrical normal distribution and no outliers b) box plot with outliers that are positively skewed c) box plot with outliers that are negatively skewed. Adapted from http://psychstat.com. by Gordon, O., 2007. Copyright 2012 by Tangient LLC.

Statistical Approaches to Outliers in ANOVA In addition to a frequency distribution, each value for continuous variables can be converted to a standard score (e.g. z scores) with a mean of 0 and a standard deviation of 1. This conversion can be done using the Explore or Descriptives function on SPSS to get the z scores. Any scores exceeding 2.5 can be considered an outlier and should be carefully considered for possible deletion (Gordon, 2007; Meyers, Gamst & Guarino, 2006). Approaches to Outliers in MANOVA After inspecting the data set for univariate outliers, an assessment for multivariate outliers can also be implemented. The first step recommended by Meyers, Gamst and Guarino (2006) is to run bivariate scatterplots for combinations of key variables. Most cases fall within the elliptical (oval-shaped) pattern mass. Outliers are visually represented outside this pattern. The explore function within SPSS can also be used to create an extreme values output that lists the five highest and five lowest values within the data set so researchers can quickly check the values collected. In addition to the bivariate scatterplots, the Mahalanobis distance can also be

NORMALITY AND OUTLIERS calculated to measure the multivariate distance between each case and the group multivariate

16

mean (centroid). The chi-square distribution (with an alpha level of .001) is calculated for each case to identify outliers and possible candidates for elimination (Meyers, Gamst & Guarino, 2006). Summary In statistics, it is important to have statistical assumptions that guide analysis and reduce the potential for biases and distortions within the data analysis (Meyers, Gamst & Guarino, 2006). The assumption of normality is based on the normal distribution that is a natural phenomenon that assumes variables can be distributed along a symmetrical bell-shaped distribution curve (French et al., n.d.; Meyers, Gamst & Guarino, 2006; Statsoft Inc., 2012). Outliers are extreme values that can occur during an analysis for a variety of reasons. Each case needs to be individually investigated to determine the potential reason for the outlier that could range from a data entry error to discovering new and exciting data about the phenomenon under investigation (Meyers, Gamst & Guarino, 2006; Theodore & Gatchel, 2008; Zijlstra, Andries van der Ark & Sijtsma, 2007). In the beginning stages of implementing an ANOVA and MANOVA, researchers need to check for non-normality and outliers in the data set by using both graphical and statistical approaches. When testing for non-normality within an ANOVA, histograms, stemand-leaf plots and normal probability plots (P-P plots and Q-Q plots) can be used graphically to provide researchers with a quick way to visually check for non-normality. Statistical approaches include checking for skewness and kurtosis and using either the Kolmogorov-Smirnov test or the Shapiro-Wilk test. There are currently no statistical processes specifically for MANOVA regarding non-normality. When researchers are detecting outliers within an ANOVA, frequency distributions, box plots, z scores, histograms and normal probability plots can be used. For a

NORMALITY AND OUTLIERS MANOVA, researchers must rely on univariate techniques, bivariate scatterplots and the

17

Mahalanobis distance to detect outliers. Given the importance of identifying non-normality and outliers before continuing with an analysis, researchers need to consider all of the statistical and graphical tools to increase the reliability and validity of their research.

NORMALITY AND OUTLIERS References French, A., Macedo, M., Poulsen, J., Waterson, T., & Yu, A. (n.d.). Multivariate analysis of variance. Retrieved from http://userwww.sfsu.edu/~efc/classes/biol710/manova/ MANOVAnewest.pdf Gordon, O. (2007). Assumptions of normality. Retrieved from http://psystats.wikispaces.com/ Assumption+of+Normality Meyers. L. S., Gamst, G., & Guarnio, A. J. (2006). Data screening. In Applied multivariate research: Design and interpretation. Retrieved from http://www.sagepub.com/upmdata/6418_Chapter_3A-Meyers_I_Proof_2.pdf NIST/SEMATECH (2012) NIST/SEMATECH e-handbook of statistical methods. Retrieved from http://www.itl.nist.gov/div898/handbook/ StatSoft, Inc. (2012). Electronic Statistics Textbook. Tulsa, OK: StatSoft. Retrieved from http://www.statsoft.com/textbook/.

18

Stevens, J. P. (2009). Applied multivariate statistics for the social sciences (5th Ed.). New York, NY: Routledge. Theodore, B. R., & Gatchel, R. J. (2008). Management an interpretation of data obtained from clinical trials in pain management. Pain Practice, 8(6), 461-472. doi: 10.1111/j.15332500.2008.00240.x Zijlstra, W. P., Andries van der Ark, L. A., & Sijtsma, K. (2007). Outlier detection in test and questionnaire data. Multivariate Behavioural Research, 42(3), 531-555. doi: 10.1080/00273170701384340

NORMALITY AND OUTLIERS Paper Evaluation Rubric Item Evaluation

19

Good/ Barely Satisfactory Adequate (0-1) (2) Knowledge and Based on Based on Based on Based on Comprehension information from information from information from information from different sources, sources that are a few sources limited sources 4 presented in a well described with some with only basic highly coherent and explained relationships and explanation and and systematic explanation few details form Integration Strong base of Points are well Points are well Points are clearly evidence and developed and explained and presents but there 4 sound based on some some evidence is is a lack of interpretation of evidence with included with evidence and links links between links between one or two links between theory, theory, research research and between theory, research and and practice practice research, practice practice Critical Strong well Well developed A few critical Little or no Evaluation developed arguments with arguments are attention given to 3 arguments with careful appraisal presents with critical evaluation Could have used critical appraisal of of some key some attempt at of problems and more critical problems, issues problems, issues appraisal/ issues analysis and your evaluation own reflections

Outstanding (4)

Very Good (3)

Organization and Coherent, Integrated and flow of the paper integrated and well systematic organised presentation 4 presentation although transitions are abrupt Presentation Presents the Presents Style information in a information clear format using clearly but it is 4 multiple methods not appealing to (i.e., graphs, the reader (all in diagrams, one format) descriptions) and it is appealing to the reader

Generally coherent but some breaks or lack of transition evident

Lack of coherence and attention to relationships between topics and points

Ideas are not Disorganized style presented clearly that is difficult to read and understand and it is not appealing to the reader

NORMALITY AND OUTLIERS Language 4 Use language that a day to day researcher would understand Some ideas are Majority of presented in day language is too to day language complex but some are also presented in a complex manner Correct use of APA A few APA errors 3-5 APA errors format (2-3) occur Language is inappropriate

20

APA Format 3

Significant number of APA errors are made or serious APA violations occur (for example, not referencing)

26/28 (93%)