Ninth Edition CoverGraziano & Raulin
Research Methods (9th edition)

Chapter 8 Summary
Hypothesis Testing, Validity,
and Threats to Validity

Hypothesis Testing

Hypothesis testing is crucial in experimental research, and developing the research hypothesis is a major task. The researcher begins with initial ideas (theoretical concepts) and refines them into a statement of the problem, which is converted into the research hypothesis when operational definitions of variables are added to the problem statement. By combining the statement of the problem and operational definitions of variables, the researcher makes a prediction about the effects of the specific, operationally defined, independent variable on the specific, operationally defined, dependent variable. That prediction is the research hypothesis.

Starting the Research with an Initial Idea

The research process begins with one or more initial ideas, which are phrased as questions. The researcher then searches the literature and refines, modifies, retains, or discards the initial ideas.

Statement of the Problem

Problem statements for experimental research are in the form of questions that concern causality. They contain a good deal of information and we find in each: (1) a statement about an expected causal effect, (2) the identification of at least two major variables, and (3) an indication of the direction of the expected causal effects.

There are several criteria for a good problem statement: (1) It should state clearly the expected relationships between variables; (2) it must be stated in the form of a question; and (3) it must at least imply the possibility of an empirical test of the problem.

Operational Definitions

Constructs in research must be developed to the level of an operational definition in order to measure and manipulate a variable and to communicate to other researchers exactly what was done in the research. The operational definition specifies exactly how variables are to be measured or manipulated. Operational definitions make it possible to precede a step closer to formulating the research hypothesis.

Research Hypothesis

A good research hypothesis: (1) makes a statement about the relationship among variables and (2) implies clearly that the relationship can be empirically tested. The research hypothesis is a declarative statement.

The Contribution of Theory to the Research Hypothesis

Theory is critical in developing the research hypothesis. Detailed theories guide the researcher's thinking, suggesting which research studies will uncover the most interesting and useful information.

Testing the Research Hypothesis

The experimental research hypothesis is not a single statement, but actually encompasses three hypotheses, each of which must be carefully checked: the null or statistical hypothesis, the confounding variable hypothesis, and the causal hypothesis.

  • Null Hypothesis. "Null" means no or none. The null hypothesis states that there are no differences between conditions or groups beyond chance differences. If we find a statistically significant difference between the conditions or groups, then we reject the null hypothesis. If the differences are within chance limits, we conclude that there is not sufficient evidence to reject the null hypothesis.
  • Confounding Variable Hypothesis. Testing and rejecting the null hypothesis, while necessary, is not sufficient to draw a causal inference. We must also rule out the possibility that factors other than the independent variable may have had an effect on the dependent variable (that is, we must rule out the possibility that differences were due to confounding variables). The confounding variable hypothesis holds that any observed statistically significant difference may be due to extraneous factors that had systematic effects on the dependent measure(s), rather than being due to manipulation of the independent variable. 
  • Causal hypothesis. After testing and rejecting the null hypothesis and carefully ruling out confounding variable hypotheses, then we are ready to turn to the original question and draw causal inferences. Causal conclusions are always tentative in science, no matter how carefully the experiment was carried out. We can have high confidence in the conclusions, but not certainty.

Most problem statements can be developed into several different research hypotheses, each of which can be tested. Thus, several different studies can be generated from the same problem statement. This can be done because the variables indicated in the problem statement can be operationally defined in several different ways. By doing that, we can develop more than one research hypothesis and, essentially, study many facets of the original problem statement. This gives us a degree of replication and produces a more thorough study of the problem.

Validity and Threats to Validity

A major concern in scientific research is the validity of the procedures and conclusions. Validity refers to the methodological soundness of the research, which is a concern at all levels of constraint. Many factors may affect the outcome of an experiment; some of these factors are likely to be threats to the validity of the experiment. Therefore, two of the major tasks of the experimenter are to anticipate potential threats to validity and create procedures to eliminate or reduce them. Threats to validity and their control are two sides of the same coin, and control over threats to validity is a major concern for the researcher.

Statistical Validity

Two threats to statistical validity are (1) unreliable dependent variable and (2) violations of statistical assumptions.

Construct Validity

Construct validity refers to how well the study's results support the theory or constructs behind the research and asks if the theory that is supported by the findings provides the best available explanation of the results. Threats to construct validity can be reduced by using clearly stated operational definitions and by building the hypotheses on solid, well-validated constructs.

External Validity

External validity is the degree to which we are able to generalize the results of an experiment to other participants, conditions, times, and places. Threats to external validity are best controlled by having an adequate, representative sample of participants.

Internal Validity

Internal validity is of great concern for the experimenter because it involves the very heart of the experimental goals: the demonstration of causality. In an experiment, internal validity concerns the question, "Was the independent variable, and not an extraneous variable, responsible for the observed changes in the dependent variable?" An experiment is internally valid when we can conclude with confidence that it was the independent variable that brought about the observed changes in the dependent variable. Any factor that reduces that confidence is a threat to the internal validity of the study. 

Major Confounding Variables

There are a number of confounding variables that researchers must avoid.

  • Maturation. Observed changes in the dependent variable may be due to natural changes in the participants over time.
  • History. Observed changes may be due to events that occur over time but are not controlled in the experiment. This is a particular problem in long-term research.
  • Testing. The effects of repeated testing of participants may threaten internal validity as participants gain proficiency on the testing measure.
  • Instrumentation. Apparent pre-post changes in the dependent measure may be due to changes in the measuring instrument, rather than to the experimental manipulation.
  • Regression to the Mean. Whenever participants are selected because their scores on a measure are extreme (either very high or very low), they will tend to be less extreme on a second testing.
  • Selection. Confounding due to selection can occur when care is not taken to ensure that two or more groups being compared are equivalent before the manipulations begin.
  • Attrition. Confounding due to attrition can occur when participants drop out of the study; this is especially problematic if more participants drop out of one group than another or if participants with certain characteristics drop out while others remain. We must be careful not to use procedures or situations that will bias some participants against completing the study, thus differentially affecting the outcome of the study.
  • Diffusion of Treatment. When participants in one condition in an experiment communicate something about the procedures to participants in another condition, the differences between conditions may be affected. Such information exchanges may erode the planned experimental differences between the groups.
  • Sequence Effects. Sequence effects are a threat to validity in within-subjects designs, in which the same participants are measured under different conditions. Participants' experience with earlier conditions may affect their response to later conditions.

Subject and Experimenter Effects

Many threats to the validity of a study are due to the expectations and biases of the participants and the experimenters.

Subject Effects

Participants enter experiments with a variety of motivations, biases, and expectations, all of which may affect the outcome. Demand characteristics are characteristics of the setting and procedures that unintentionally suggest the behavior expected of the participants. A related phenomenon, the placebo effect, can occur when participants expect some fairly specific effect of an experimental manipulation.

Experimenter Effects

Experimenters may carry their own biases and expectations into a study, which may affect the outcome. Experimenter expectancies are potential threats to validity and must be controlled.

Many of the most important threats to validity will be controlled if the researcher designs into each experiment the random selection of participants, the random assignment of participants to conditions, and a proper control group.

Ethical Principles

The researcher has an ethical obligation to conduct carefully planned research so that the research will be useful. Failing to control confounding makes the research of little value.

Some confounding variables can only be controlled by violating ethical principles. For example, to avoid selection problems, we would have to force all selected individuals to participate, which violates the principle of informed consent. We never violate ethics to enhance validity.