Graziano & Raulin
Research Methods (9th edition)
Planned Comparisons
and Post Hoc Tests
Conducting an ANOVA and finding a significant F-ratio only means that at least one mean is statistically different from at least one other mean. To find out which means are different from which other means requires that you compute either planned comparisons or post hoc tests.
Planned comparisons are, as the name implies, comparisons of specific means or combinations of means that are planned in advance of the data collection. These comparisons are usually based on strong theoretical reasons. Planned comparisons are sometimes referred to as contrasts.
If you have not made specific predictions about the pattern of scores to justify a planned comparison, you can still look to see which group means are different from which other group means using post hoc tests. They are called post hoc because they are done after the fact, that is, without prior planning.
Planned Comparisons for a One-way ANOVA
Planned comparisons are based on specifying a contrast with integer coefficients. This process is spelled out in the section on planned comparisons in the statistical concepts section of the Student Resource Website.
To illustrate the process, we are going to use the data from Table 10.1. To refresh your memory, these are typing speed data collected under six temperature conditions (55, 60, 65, 70, 75, and 80 degrees, respectively). It was an independent groups design, with 8 different typists in each of the six conditions.
We are going to test the hypothesis that typing speed is better when the temperature is moderate than when it is excessively hot. We will define moderate temperature as 65, 70, or 75 degrees, and excessive temperature as 55, 60, or 80 degrees. You learned in the section on setting up planned comparisons that you need to identify weightings that will identify the contrast you want to make AND that sum to zero. In this case, since we have 3 groups in one cluster of the contrast and three groups in the other cluster, we can use weightings of +1 and -1 to identify the clusters, as shown in the table below.
|
Temperature Conditions |
|||||
| 55 | 60 | 65 | 70 | 75 | 80 |
| +1 | +1 | -1 | -1 | -1 | +1 |
To conduct the planned comparison as part of our one-way ANOVA, we start by opening the file that contains the data in Table 10.1 using the SPSS for Windows program, which gives us this screen. [Note that we have already created these files for you and posted them for you to download on elsewhere on this website.]
Then we select the Analyze menu, the Compare Means submenu, and the One-Way ANOVA choice, which gives you this screen. To set up the contrast, we click on the Contrasts button, which gives us this screen. To insert the coefficients for the contrast, we need to put them in one at a time in the order of the conditions. So we put the first coefficient (+1) into the coefficients box and click Add, as shown in this screen. We continue until the values of +1, +1, -1, -1, -1, and +1 have been added in that order. Then you click the Continue button. Clicking the OK button will run the ANOVA and the specified planned comparison, which produces this output file.
The output of shows the coefficients in one box by condition. Be sure to check them to make sure that they were applied as you intended. The next box shows the results of the contrast. SPSS for Windows actually does two statistical tests, one that assumes equal variances and one that does not make that assumption. In this case, both are clearly significant, so it does not matter which one you use. If you are concerned about whether the variances are equal, you can select the Homogeneity of Variance test on the options menu when you run the ANOVA. We will illustrate that step in the animation below.
Animations
We have prepared an animation that will walk you through this procedure. To run the animation, simply click on the title of the animation in the table below.
Note that we do not recommend that you try to run the animations if you have a slow connection, such as a dial-up connection. You will find that the animations take forever to load with a slow connection.
Post Hoc Tests for a One-way ANOVA
The logic of post hoc tests was covered in the section on statistical concepts on this Student Resource Website. In this section, we will show you how to perform the various post hoc tests in SPSS for Windows. For our example, we will use the same data on temperature and typing speed used to illustrate planned comparisons above (from Table 10.1).
We start by opening the SPSS program and opening the data file, which give us this screen. Just like in the planned comparison, we set up that one-way ANOVA by selecting the Analyze menu, the Compare Means submenu, and the one-way ANOVA choice, which gives us this screen. We then move the variables to set up the ANOVA (typing speed to the dependent list and conditions to the factor box). we then click on the post hoc button, which gives us this screen.
As you can see, SPSS for Windows gives you a large choice of post hoc tests, some of which assume equal variances in the groups and others that do not. For the purposes of illustration, we are going to select three post hoc procedures: the Least Squared Difference test (LSD), Tukey-B test, and Dunnett's C test. This screen shows those selections. We press the continue button to return to the one-way ANOVA screen.
Which test we want to use will depend on whether we have homogeneity of variance. SPSS will test for that if we request it by clicking on the options button, which gives us this screen. While we are at it, we will select descriptive statistics and a plot of the means in addition to a test for the homogeneity of variance. Clicking continue returns us to the one-way ANOVA page, and clicking OK runs the analyses.
Because we have asked the program to do several procedures, this is a large output file, with several screens worth of information. This first screen shows the summary statistics, the test for homogeneity of variance, and the ANOVA. Note that the ANOVA is statistically significant and that the test for homogeneity of variance is not significant. This means that we do have homogeneity of variance and therefore can use the LSD and Tukey-B tests as our post hoc tests. This also means that we can look at the post hoc tests because post hoc tests make sense ONLY if we have a significant F-ratio. Remember that the F-ratio tells us if we have at least one mean that is statistically significant from at least one other mean, so if the F is not significant, we clearly have no significant group differences to look for with post hoc test.
When statistical procedures were all computed by hand, One did the ANOVA first to see whether there were any differences to probe. You then did the test for homogeneity of variance to see which post hoc test you should use. Then, and only then, did you do the post hoc tests. With modern computers, it takes just a few second to set up all of the procedures in a single run and you just ignore the unnecessary tests. So in this case, we will ignore the Dunnett's C test, because we have homogeneity of variance, and we will look at the results of the LSD tests, which are shown in this screen.
Note that SPSS for Windows gives you 30 separate comparisons, although in reality there are only 15 comparisons. Comparing the 55 degree with the 80 degree condition is that same as comparing the 80 degree with the 55 degree condition. The only difference is that the sign of the difference is reversed, but the statistical significance is identical. SPSS for Windows does this kind of thing with a lot of their analyses. With fast computers, it is easier to just compute everything, even though some computations are redundant.
We also asked the SPSS program to include a graph of the distribution of means, which is at the bottom of the output file, as shown here. It took one extra mouse click to produce this graph, and so we recommend that you include such graphs routinely in your analyses. There is nothing like a picture to help you organize in your mind complex results.
Animations
We have prepared an animation that will walk you through this procedure. To run the animation, simply click on the title of the animation in the table below.
Note that we do not recommend that you try to run the animations if you have a slow connection, such as a dial-up connection. You will find that the animations take forever to load with a slow connection.